8

I accidentally deleted the kubernetes svc:

service "kubernetes" deleted

using:

 kubectl delete svc --all

what should I do? I was just trying to remove services so I could launch new ones.

  • I think you'll have to do etcd surgery, since IIRC Kubernetes remembers which ClusterIPs it has given out and will not let you assign `${service_cidr}.1` to the Service named `kubernetes`. You can certainly try along with some `--validate=false` and take your chances, but I'm pretty sure it won't work – mdaniel Jun 13 '19 at 06:19
  • 3
    I did the same yesterday, but for me the service is created again. I am not sure, how though. – Malathi Jun 13 '19 at 10:47

1 Answers1

8

A bit theory first ;) Whenever you delete kubernetes svc, you also delete endpoint and this is where Reconciler comes in. It is actually a controller manager for the core bootstrap Kubernetes controller loops, which manage creating the "kubernetes" service, the "default", "kube-system" and "kube-public" namespaces, and provide the IP repair check on service IPs.

So, in healthy clusters default.kubernetes service should be automatically recreated by controller manager.

If it's not, I'd recommend to:

Check api-server logs

kubectl logs -f kube-apiserver-master -n kube-system

You should see something like:

Resetting endpoints for master service "kubernetes" to [10.156.0.3]

If you don't see it, try to manually remove etcd key for this service

Because the current state of the cluster is stored in etcd, it may happen that the key remain when you deleted a service:

a. exec to etcd-master pods

kubectl exec -it etcd-master -n kube-system sh

b. get the etcd key value

ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --key=/etc/kubernetes/pki/etcd/server.key --cert=/etc/kubernetes/pki/etcd/server.crt get /registry/services/endpoints/default/kubernetes

c. if you get any value like:

v1    Endpointst
O

kubernetesdefault"*$eafc04cf-90f3-11e9-a75e-42010a9c00032����z!


10.156.0.3
https�2TCP"

just remove it by 

ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --key=/etc/kubernetes/pki/etcd/server.key --cert=/etc/kubernetes/pki/etcd/server.crt rm /registry/services/endpoints/default/kubernetes

After you did it, check the api-server logs once again.

A_Suh
  • 2,923
  • 3
  • 18