Failing consul health check on local machine

Question

I have Consul running via docker using docker-compose

version: '3'

services:  
  consul:
    image: unifio/consul:latest
    ports:
      - "8500:8500"
      - "8300:8300"
    volumes: 
      - ./config:/config
      - ./.data/consul:/data
    command: agent -server -data-dir=/data -ui -bind 0.0.0.0 -client 0.0.0.0 -bootstrap-expect=1
  mongo:
    image: mongo
    ports:
      - "27017:27017"
      - "27018:27018"
    volumes:
      - ./.data/mongodb:/data/db
    command: mongod --bind_ip_all

and have a nodejs service running on port 6001 which exposes a /health endpoint for health checks. I am able to register the service via this consul package. However, visiting the consul UI I can see that the service has a status of failing because the health check is not working.

The UI show this message:

Get http://127.0.0.1:6001/health: dial tcp 127.0.0.1:6001: getsockopt: connection refused

Not sure why is not working exactly, but I kind of sense that i may have misconfigured consul. Any help would be great.

Have you been able to fix this issue @MrFoh – Jerry Okafor Feb 11 '20 at 18:45 — Jerry Okafor, Feb 11 '20 at 18:45

score 3 · Accepted Answer · answered Jun 01 '19 at 17:26

Consul is running in your docker container. When you use 127.0.0.1 in this container, it refers to itself, not to your host.

You need to use a host IP that is known to your container (and of course make sure your service is reachable and listening on this particular IP).

In most cases, you should be able to contact your host from a container through the default docker0 bridge ip that you can get with ip addr show dev docker0 from your host as outlined in this other answer.

The best solution IMO is to discover the gateway that your container is using which will point to the particular bridge IP on your host (i.e. the bridge created for your docker-compose project when starting it). There are several methods you can use to discover this ip from the container depending on the installed tooling and your linux flavor.

score 3 · Answer 2 · answered Jun 01 '19 at 20:10

While Zeitounator's answer is perfectly fine and answers your direct question, the "indirect" solution to your problem would be to manage the nodejs service through docker-compose.

IMHO it's a good idea to manage all services involved using the same tool, as then their lifecycles are aligned and also it's easy to configure them to talk to each other (at least that's the case for docker-compose).

Moreover, letting containers access services on the host is risky in terms of security. In production environments you usually want to shield host services from containers, as otherwise the containers lose their "containment" role.

So, in your case you would need to add the nodejs service to docker-compose.yml:

services:
  (...)
  nodejs-service:
    image: nodejs-service-image
    ports:
    - "6001:6001" [this is only required if you need to expose the port on the host]
    command: nodejs service.js

And then your Consul service would be able to access nodejs-service through http://nodejs-service:6001/health.

Failing consul health check on local machine

2 Answers2