etcdctl throws Error: context deadline exceeded error

Question

I'm trying to create a one node etcd cluster on AWS using coreos cloud-config. I have created a Route53 recordset with value etcd.uday.com which has a alias to the ELB which points to the ec2 instance. Etcd is running successfully but when I run the etcd member list command I get below error

ETCDCTL_API=3 etcdctl member list \
--endpoints=https://etcd.udayvishwakarma.com:2379 \
--cacert=./ca.pem \
--cert=etcd-client.pem \
--key=etcd-client-key.pem

Error: context deadline exceeded

However, it lists members when --insecure-skip-tls-verify flag is added to the etcdctl member list command. I have generated certificated using cfssl using below configs

ca.json

{
"CN": "Root CA",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "UK",
      "L": "London",
      "O": "Kubernetes",
      "OU": "CA"
    }
  ],
  "ca": {
    "expiry": "87658h"
  }
}

ca.config

 {
  "signing": {
    "default": {
      "expiry": "2190h"
    },
    "profiles": {
      "client": {
        "expiry": "8760h",
        "usages": [
          "signing",
          "key encipherment",
          "client auth"
        ]
      },
      "server": {
        "expiry": "8760h",
        "usages": [
          "signing",
          "key encipherment",
          "server auth"
        ]
      },
      "peer": {
        "expiry": "8760h",
        "usages": [
          "signing",
          "key encipherment",
          "server auth",
          "client auth"
        ]
      },
      "ca": {
        "usages": [
          "signing",
          "digital signature",
          "cert sign",
          "crl sign"
        ],
        "expiry": "26280h",
        "is_ca": true
      }
    }
  }
}

etcd-member.json

 {
  "CN": "etcd",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "hosts":[
    "etcd.uday.com"
  ],
  "names": [
    {
      "O": "Kubernetes"
    }
  ]
}

etcd-client.json

 {
  "CN": "etcd",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "hosts":[
    "etcd.uday.com"
  ],
  "names": [
    {
      "O": "Kubernetes"
    }
  ]
}



     cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -hostname="etcd.uday.com" \
          -config=ca-config.json -profile=peer \
etcd-member.json | cfssljson -bare etcd-member

    cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -hostname="etcd.uday.com" \
    -config=ca-config.json -profile=client\
 etcd-client.json | cfssljson -bare etcd-client

My etcd-member.service systemd unit cloudconfig is as below

    units:
  - name: etcd-member.service
    drop-ins:
    - name: aws-etcd-cluster.conf
      content: |
        [Service]
        Environment=ETCD_USER=etcd
        Environment=ETCD_NAME=%H
        Environment=ETCD_IMAGE_TAG=v3.1.12
        Environment=ETCD_SSL_DIR=/etc/etcd/ssl

        Environment=ETCD_CA_FILE=/etc/ssl/certs/ca.pem
        Environment=ETCD_CERT_FILE=/etc/ssl/certs/etcd-client.pem
        Environment=ETCD_KEY_FILE=/etc/ssl/certs/etcd-client-key.pem
        Environment=ETCD_CLIENT_CERT_AUTH=true
        Environment=ETCD_TRUSTED_CA_FILE=/etc/ssl/certs/ca.pem
        Environment=ETCD_PEER_CA_FILE=/etc/ssl/certs/ca.pem
        Environment=ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd-member.pem
        Environment=ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd-member-key.pem
        Environment=ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/certs/ca.pem


        Environment=ETCD_INITIAL_CLUSTER_STATE=new
        Environment=ETCD_INITIAL_CLUSTER=%H=https://%H:2380
        Environment=ETCD_DATA_DIR=/var/lib/etcd3
        Environment=ETCD_LISTEN_CLIENT_URLS=https://%H:2379,https://127.0.0.1:2379
        Environment=ETCD_ADVERTISE_CLIENT_URLS=https://%H:2379
        Environment=ETCD_LISTEN_PEER_URLS=https://%H:2380
        Environment=ETCD_INITIAL_ADVERTISE_PEER_URLS=https://%H:2380
        PermissionsStartOnly=true
        Environment="RKT_RUN_ARGS=--uuid-file-save=/var/lib/coreos/etcd-member-wrapper.uuid"
        ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/lib/coreos/etcd-member-wrapper.uuid
        ExecStartPre=/usr/bin/sed -i 's/^ETCDCTL_ENDPOINT.*$/ETCDCTL_ENDPOINT=https:\/\/%H:2379/' /etc/environment
        ExecStartPre=/usr/bin/mkdir -p /var/lib/etcd3
        ExecStartPre=/usr/bin/chown -R etcd:etcd /var/lib/etcd3
        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/lib/coreos/etcd-member-wrapper.uuid
    enable: true
    command: start

Is cert generation wrong or something I have missed?

score 1 · Answer 1 · answered Jun 26 '19 at 07:45

1

The certificates are generated for etcd.uday.com. You are trying to connect using etcd.udayvishwakarma.com while certificate is valid for etcd.uday.com. Change endpoint on etcdctl from etcd.udayvishwakarma.com to etcd.uday.com.

answered Jun 26 '19 at 07:45

Junaid

728
6
12

score 0 · Answer 2 · answered Dec 23 '19 at 06:29

I ran into the same problem today, probably this is not going to be useful for you now, but it will for anybody who runs into the same problem in the future. I think you might be missing

etcd.udayvishwakarma.com

from your cert in

--cert=etcd-client.pem

To verify that etcd.udayvishwakarma.com exists in your cert, you can run:

openssl x509 -in etcd-client.pem -text

and you should be able to see it under X509v3 Subject Alternative Name. If you don't, you will probably need to recreate the certificate adding that DNS name.

etcdctl throws Error: context deadline exceeded error

2 Answers2