How to run an etcd cluster among pod replicas?

Question

I have a pod/service running an application that consumes etcd as a synchronization system and datastore. I want to run etcd within the pod, such that all of the replicas form a coherent cluster. In other words, so the application in replica #1 can write "foo" to localhost:4001/v2/keys/my_key and then replica #2 can then read localhost:4001/v2/keys/my_key and get "foo" as a result.

It's not clear how this can be done, since pod replicas are not individually addressable. I could in theory create an "etcd" service exposing the cluster ports, but any requests would round-robin to all the replicas so the individual etcd nodes would not be able to find each other.

Am I approaching this problem the correct way?

score 3 · Answer 1 · answered Jun 08 '17 at 09:28

You can deploy etcd on kubernetes using an Operator (from the extensions/v1beta1) and the quay.io/coreos/etcd-operator image.

An example deployment with a cluster size of 3 looks like this:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: etcd-operator
spec:
  replicas: 1
  template:
    metadata:
      name: etcd-operator
      labels:
        app: etcd
        component: operator
    spec:
      containers:
      - name: etcd-operator
        image: quay.io/coreos/etcd-operator:v0.3.0
        env:
        - name: MY_POD_NAMESPACE
          valueFrom: { fieldRef: { fieldPath: metadata.namespace } }
        - name: MY_POD_NAME
          valueFrom: { fieldRef: { fieldPath: metadata.name } }

---

apiVersion: etcd.coreos.com/v1beta1
kind: Cluster
metadata:
  name: etcd-cluster
  labels:
    app: etcd
    component: cluster
spec:
  size: 3
  version: "3.1.8"

Please be aware of the beta status of this project. However according to the maintainers the operator is now stable. I have deployed the configuration above successfully but I didn't run any of this in production.

The operator code is available on github. You can find additional documentation there.

You are using `apiVersion: etcd.coreos.com/v1beta1` and `kind: Cluster`, whereas etcd-operator example uses `apiVersion: etcd.database.coreos.com/v1beta2` and `kind: EtcdCluster`. What are the differences? https://github.com/coreos/etcd-operator/blob/master/example/example-etcd-cluster.yaml — akauppi, Nov 03 '17 at 13:51

score 1 · Answer 2 · answered Feb 18 '16 at 01:58

1

There's a pretty good example of a three node etcd cluster here: https://github.com/coreos/etcd/tree/master/hack/kubernetes-deploy

They make use of separate rc's and services for each replica as a workaround until nominal services are added.

answered Feb 18 '16 at 01:58

JKnight

1,831
1
14
22

CAUTION: This hack is completely unsuitable for production since you CANNOT remove a node/replica and then add one again. There is no service discovery in this setup! – peedee Jul 08 '19 at 14:43

score 0 · Answer 3 · edited Oct 11 '16 at 07:20

0

I added your question to kubernetes/kubernetes#5017

If someone knows the answer, they will hopefully post it there.

I think it may require the "nominal services" feature (kubernetes/kubernetes#260) which is not implemented yet, but I'm not sure.

edited Oct 11 '16 at 07:20

Steel Brain

3,774
25
38

answered Oct 04 '15 at 00:27

DavidO

1,473
9
7

How to run an etcd cluster among pod replicas?

3 Answers3