8

I'm trying to set up a kubernetes cluster on 2 nodes , centos 7.1 using this guide. However when I attempt to start the services on the minion like so:

for SERVICES in kube-proxy kubelet docker flanneld; do
    systemctl restart $SERVICES
    systemctl enable $SERVICES
    systemctl status $SERVICES 
done

I get the following error:

-- Logs begin at Wed 2015-12-23 13:00:41 UTC, end at Wed 2015-12-23 16:03:54 UTC. --
Dec 23 16:03:47 sc-test2 systemd[1]: docker-storage-setup.service: main process exited, code=exited, status=1/FAILURE
Dec 23 16:03:47 sc-test2 systemd[1]: Failed to start Docker Storage Setup.
-- Subject: Unit docker-storage-setup.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit docker-storage-setup.service has failed.
-- 
-- The result is failed.
Dec 23 16:03:47 sc-test2 systemd[1]: Unit docker-storage-setup.service entered failed state.
Dec 23 16:03:48 sc-test2 flanneld[36477]: E1223 16:03:48.187350 36477 network.go:53] Failed to retrieve network config: 100: Key not found (/atomic.io)
Dec 23 16:03:49 sc-test2 flanneld[36477]: E1223 16:03:49.189860 36477 network.go:53] Failed to retrieve network config: 100: Key not found (/atomic.io)
Dec 23 16:03:50 sc-test2 flanneld[36477]: E1223 16:03:50.192894 36477 network.go:53] Failed to retrieve network config: 100: Key not found (/atomic.io)
Dec 23 16:03:51 sc-test2 flanneld[36477]: E1223 16:03:51.194940 36477 network.go:53] Failed to retrieve network config: 100: Key not found (/atomic.io)
Dec 23 16:03:52 sc-test2 flanneld[36477]: E1223 16:03:52.197222 36477 network.go:53] Failed to retrieve network config: 100: Key not found (/atomic.io)
Dec 23 16:03:53 sc-test2 flanneld[36477]: E1223 16:03:53.199248 36477 network.go:53] Failed to retrieve network config: 100: Key not found (/atomic.io)
Dec 23 16:03:54 sc-test2 flanneld[36477]: E1223 16:03:54.201160 36477 network.go:53] Failed to retrieve network config: 100: Key not found (/atomic.io)

I'm sure I set the key on the master with : etcdctl mk /coreos.com/network/config '{"Network":"172.17.0.0/16"}'

By far installation seems to be the hardest bit on using kubernetes :(

Jonathan
  • 8,558
  • 3
  • 46
  • 70
  • 1
    If you look at the error message, it's not looking under `/coreos.com`; it appears to be looking under `/atomic.io`: `Key not found (/atomic.io)` – larsks Dec 23 '15 at 16:30
  • True..here's an excerpt from the config file on the minion `# etcd config key. This is the configuration key that flannel queries # For address range assignment FLANNEL_ETCD_KEY="/atomic.io/network" ` – Jonathan Dec 23 '15 at 16:39
  • Do I change this here? I'm using centos.. – Jonathan Dec 23 '15 at 16:45
  • 1
    You can change the value of `FLANNEL_ETCD_KEY` in `/etc/sysconfig/flanneld`, or you can just create the appropriate path in `etcd`. Are you sure you're using a current CentOS atomic image? In my environment (CentOS 7 atomic; `rpm-ostree status` says `7.20151118`), `FLANNEL_ETCD_KEY` is already set to `"/coreos.com/network"`. – larsks Dec 23 '15 at 16:49
  • I don't have the rpm-ostree status command..my file now looks like `# Flanneld configuration options # etcd url location. Point this to the server where etcd runs FLANNEL_ETCD="http://x.x.x.x:2379" # etcd config key. This is the configuration key that flannel queries # For address range assignment FLANNEL_ETCD_KEY="/coreos.com/network" # Any additional options that you want to pass #FLANNEL_OPTIONS="" ` However flannel still fails but with a different error : ` Failed to retrieve network config: invalid character 'e' after top-level value` – Jonathan Dec 23 '15 at 17:41
  • Changing the key to `/atomic.io/network` worked for me. – Jonathan Dec 30 '15 at 06:21

1 Answers1

7

Today's christmas but I spent the whole day trying to get this to work :) This is what I did:

#1 FLANNEL

As mentioned I'd set the flannel etcd key on the master with:

etcdctl mk /coreos.com/network/config '{"Network":"172.17.0.0/16"}'

but I got this error when trying to start flannel on the minion:

Failed to retrieve network config: 100: Key not found (/atomic.io)

So I edited the /etc/sysconfig/flanneld file on the minion from:

# Flanneld configuration options  

# etcd url location.  Point this to the server where etcd runs
FLANNEL_ETCD="http://master:2379"

# etcd config key.  This is the configuration key that flannel queries
# For address range assignment
FLANNEL_ETCD_KEY="/coreos.com/network"

# Any additional options that you want to pass
#FLANNEL_OPTIONS=""

to:

# Flanneld configuration options  

# etcd url location.  Point this to the server where etcd runs
FLANNEL_ETCD="http://master:2379"

# etcd config key.  This is the configuration key that flannel queries
# For address range assignment
FLANNEL_ETCD_KEY="/atomic.io/network"

# Any additional options that you want to pass
#FLANNEL_OPTIONS=""

i.e. changed the FLANNEL_ETCD key.

After this systemctl start flanneld worked.

#2 DOCKER

I didn't find a way to make the version installed as a dependency by kubernetes work so I uninstalled it and following the docker docs for Centos installed docker-engine and manually created a docker.service file for systemctl.

cd /usr/lib/systemd/system

and the contents of the docker.service:

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target docker.socket
Requires=docker.socket
Requires=flanneld.service
After=flanneld.service

[Service]
EnvironmentFile=/etc/sysconfig/flanneld
ExecStart=/usr/bin/docker daemon -H fd:// --bip=${FLANNEL_SUBNET}
Restart=on-failure
RestartSec=5


[Install]
WantedBy=multi-user.target

then start and enable the daemon with systemctl as well as query the status.

systemctl restart docker
systemctl enable docker
systemctl status docker
Jonathan
  • 8,558
  • 3
  • 46
  • 70
  • 1
    I've been working on the same thing. The instructions from severalnines.com seem to depend on more storage; I was trying this out in centos guests in virtual box, and the docker startup failed because docker-storage-setup failed because it was unable to create a logical volume to hold docker images. Since I was working with virtual box, I just created larger (12GB) centos guests, and that worked. Then I was stuck on the flannel problem. It looks like their setup for the flanneld configuration was wrong. Your change here fixed it for me. – Chris Westin Dec 29 '15 at 23:03
  • Nope, spoke too soon. 12GB VMs wasn't large enough. Any ideas how big I should make it? – Chris Westin Dec 29 '15 at 23:11
  • I don't think storage is an issue, I was able to get kubernetes working by uninstalling the existing docker package (installed via yum install docker) and installing docker-engine as per the docker docs. Then swapping out the service file values with the above. – Jonathan Dec 30 '15 at 05:17
  • I did find that adding the argument `--storage-opt="dm.blocksize=64k"` to the `docker daemon` command fixed it. From https://github.com/docker/docker/issues/6325 – Jonathan Dec 30 '15 at 06:13