K8s - DNS Resolution between pods in different namespaces

Question

I have a simple pod in a namespace called "a", and another pod in a namespace "b"...

I also have a test script that makes a grpc call from "a" to "b".

This script does not work in using the FQDN (e.g. "some-service.b.cluster.local")
This script does work if using the exact IP Address of the pod in "b" namespace

I suppose is some error in DNS Resolution, but I can't really move forward.

Any help?

kubectl exec -n a somepod-f647b7d95-mrvfr cat /etc/resolv.conf

nameserver 10.12.0.10
search chimera.svc.cluster.local svc.cluster.local cluster.local c.<company-name>-staging.internal <provider>.internal
options ndots:5

kubectl get pods -n kube-system

event-exporter-v0.2.4-6d4c69fbfb-f4xpf                           1/1     Running   0          24d
fluentd-gcp-scaler-6965bb45c9-mzvw6                              1/1     Running   0          7d6h
fluentd-gcp-v3.2.0-2m2bf                                         1/1     Running   0          7d6h
fluentd-gcp-v3.2.0-2v6bq                                         1/1     Running   0          7d6h
fluentd-gcp-v3.2.0-4xpbc                                         1/1     Running   0          7d6h
fluentd-gcp-v3.2.0-7g5hm                                         1/1     Running   0          7d6h
fluentd-gcp-v3.2.0-8mqvc                                         1/1     Running   0          7d6h
fluentd-gcp-v3.2.0-f9hrs                                         1/1     Running   0          7d6h
fluentd-gcp-v3.2.0-fr58c                                         1/1     Running   0          7d6h
fluentd-gcp-v3.2.0-hzrsb                                         1/1     Running   0          7d6h
fluentd-gcp-v3.2.0-kq8hc                                         1/1     Running   0          7d6h
fluentd-gcp-v3.2.0-kt6p5                                         1/1     Running   0          7d6h
fluentd-gcp-v3.2.0-nsztm                                         1/1     Running   0          7d6h
fluentd-gcp-v3.2.0-qcl4r                                         1/1     Running   0          7d6h
fluentd-gcp-v3.2.0-qggv9                                         1/1     Running   0          7d6h
fluentd-gcp-v3.2.0-qkkp5                                         1/1     Running   0          7d6h
fluentd-gcp-v3.2.0-rm9hn                                         1/1     Running   0          5d5h
fluentd-gcp-v3.2.0-sv52h                                         1/1     Running   0          7d6h
fluentd-gcp-v3.2.0-t75fp                                         1/1     Running   0          7d6h
fluentd-gcp-v3.2.0-v49fv                                         1/1     Running   0          7d6h
kube-dns-6cd7bbdf65-jnntn                                        4/4     Running   0          24d
kube-dns-6cd7bbdf65-txmlj                                        4/4     Running   0          24d
kube-dns-autoscaler-8687c64fc-29jgq                              1/1     Running   0          7d6h
kube-proxy-gke-iceberg-api-v2-201908101259587443-01f0b55b-q0k3   1/1     Running   0          217d
kube-proxy-gke-iceberg-api-v2-201908101259587443-0d661dfb-3zhx   1/1     Running   0          217d
kube-proxy-gke-iceberg-api-v2-201908101259587443-92bbd393-w96w   1/1     Running   1          115d
kube-proxy-gke-iceberg-es-single-202003021919386-1b520a2e-sn9m   1/1     Running   0          5d6h
kube-proxy-gke-iceberg-es-single-202003021919386-bf6046bf-7wsp   1/1     Running   0          5d5h
kube-proxy-gke-iceberg-es-single-202003021919386-d64daa4e-1jqz   1/1     Running   0          5d5h
kube-proxy-gke-iceberg-general-20190810125958886-21ed2623-4m0p   1/1     Running   0          217d
kube-proxy-gke-iceberg-general-20190810125958886-8b185cf9-x1j2   1/1     Running   0          217d
kube-proxy-gke-iceberg-general-20190810125958886-eaf63d3c-k338   1/1     Running   0          217d
kube-proxy-gke-iceberg-kafka-2019081012595876540-429586da-m2qf   1/1     Running   0          217d
kube-proxy-gke-iceberg-kafka-2019081012595876540-76ebb654-z7xx   1/1     Running   0          217d
kube-proxy-gke-iceberg-kafka-2019081012595876540-c3abee6e-4q76   1/1     Running   0          217d
kube-proxy-gke-iceberg-rabbitmq-2019081012595876-552d6676-8z2k   1/1     Running   0          217d
kube-proxy-gke-iceberg-rabbitmq-2019081012595876-662980f7-76jc   1/1     Running   0          217d
kube-proxy-gke-iceberg-rabbitmq-2019081012595876-b269df22-6zqj   1/1     Running   0          217d
kube-proxy-gke-iceberg-redis-2019081012595877180-38264a5e-c0ch   1/1     Running   0          217d
kube-proxy-gke-iceberg-redis-2019081012595877180-9412d5f5-pt3w   1/1     Running   0          217d
kube-proxy-gke-iceberg-redis-2019081012595877180-947dc20b-c002   1/1     Running   0          217d
kube-state-metrics-67b67d8fdd-nkpt4                              2/2     Running   0          24d
l7-default-backend-fd59995cd-cvqwb                               1/1     Running   0          24d
metrics-server-v0.3.1-5c8f664b95-sthjz

add content of /etc/resolv.conf from pod a and check if coredns pod are running in kube-system namespace — Arghya Sadhu, Mar 27 '20 at 14:54

score 3 · Accepted Answer · answered Mar 28 '20 at 00:31

From your description it appears you're running KubeDNS. My First piece of advice to you would be to migrate to CoreDNS as KubeDNS is on a deprecation path.

Secondly, two things leap out at me.

You're talking about making calls between pods instead of services. While Kubernetes does provide Service Discovery between your applications, it does this through DNS as you know. However, just because pods can resolve each other doesn't mean a container will have its ports exposed outside of its pod. To do this, even to an application within the cluster than can resolve it, you must declare a Service resource for each Pod or Controller.
When you talk about making a call that references a FQDN of your B pod/service, you do not specify the default FQDN schema and do not mention having customized this.

First, can you please kubectl get svc -n NAMESPACE for both namespaces in which your A and B pods are running and confirm that a Service of type ClusterIP has been created and an IP Address has been associated with the Service?

Secondly, can you try and perform a connection attempt from Application A to the Service of Application B by specifying the following FQDN format instead?

some-service.b.svc.cluster.local

Notice the svc portion. You mentioned some-service.b.cluster.local in your OP.

Lastly, if everything comes back normal we can start troubleshooting kube-dns. it does appear that all three pods are running. However, have you tried to describe them and/or grab their logs? Could you try the following and share a summary if anything looks interesting?

kubectl describe pod -n kube-system kube-dns-6cd7bbdf65-jnntn
kubectl describe pod -n kube-system kube-dns-6cd7bbdf65-txmlj
kubectl describe pod -n kube-system kube-dns-autoscaler-8687c64fc-29jgq

kubectl logs -n kube-system kube-dns-6cd7bbdf65-jnntn
kubectl logs -n kube-system kube-dns-6cd7bbdf65-txmlj
kubectl logs -n kube-system kube-dns-autoscaler-8687c64fc-29jgq

I imagine the logs commands will provide you with the answer you're looking for. Let me know if you need any further clarification or help on this issue. I'm happy to help.

K8s - DNS Resolution between pods in different namespaces

1 Answers1