I have a docker compose stack with a few containers. The two in question are an extended python:3-onbuild container (see base image here) with a falcon webserver running and a basic node:8.11-alpine container that is attempting to make post requests to the python webserver. This is a simplified version of my compose file:
version: '3.6'
services:
app: # python:3-onbuild
ports:
- 5000
build:
context: ../../
dockerfile: infra/docker/app.Dockerfile
lambda: # node:8.11-alpine
ports:
- 10000
build:
context: ../../
dockerfile: infra/docker/lambda.Dockerfile
depends_on:
- app
I know the networking is working because if I ssh into the lambda container
docker exec -it default_lambda_1 /bin/ash
and run ping app I get a response back.
$ ping app
PING app (172.18.0.4): 56 data bytes
64 bytes from 172.18.0.4: seq=0 ttl=64 time=0.184 ms
64 bytes from 172.18.0.4: seq=1 ttl=64 time=0.141 ms
I can even run ping app:5000
ping app:5000
PING app:5000 (172.18.0.3): 56 data bytes
64 bytes from 172.18.0.3: seq=0 ttl=64 time=0.105 ms
64 bytes from 172.18.0.3: seq=1 ttl=64 time=0.176 ms
I even created a /status endpoint on my api to attempt to work through this issue. It queries the names of all the tables in the database, so it should fail if the database isn't ready. But, if I run curl http://app:5000/status I get a response back with all my table names, no problem at all.
The problem only arises when my node process attempts to make a post request (using axios)
this.endpointUrl = 'http://app:5000';
const options: AxiosRequestConfig = {
method: 'POST',
url: `${this.endpointUrl}/session/get`,
data: {
'client': 'alexa'
},
responseType: 'json'
};
axios(options)
.then((response: AxiosResponse<any>) => {
console.log(`[SessionService][getSession]: "/session/get" responseData = ${JSON.stringify(response.data)}`);
})
.catch(err => {
console.error(`[SessionService][getSession]: error = ${err}`);
});
I get the following error:
console.error src/index.ts:111
VirtualConsole.on.e at project/node_modules/jsdom/lib/jsdom/virtual-console.js:29:45
Error: Error: getaddrinfo ENOTFOUND app app:5000
at Object.dispatchError (/project/node_modules/jsdom/lib/jsdom/living/xhr-utils.js:65:19)
at Request.client.on.err (/project/node_modules/jsdom/lib/jsdom/living/xmlhttprequest.js:676:20)
at emitOne (events.js:121:20)
at Request.emit (events.js:211:7)
at Request.onRequestError (/project/node_modules/request/request.js:878:8)
at emitOne (events.js:116:13)
at ClientRequest.emit (events.js:211:7)
at Socket.socketErrorListener (_http_client.js:387:9)
at emitOne (events.js:116:13)
at Socket.emit (events.js:211:7) undefined
console.error src/index.ts:111
axios_1.default.then.catch.err at services/Service.ts:94:25
[SessionService][getSession]: error = Error: Network Error
I learned more about this ENOTFOUND error message from here
getaddrinfo is by definition a DNS issue
So then, I looked into using the node dns packge to look up app
> dns.lookup('app', console.log)
GetAddrInfoReqWrap {
callback: [Function: bound consoleCall],
family: 0,
hostname: 'app',
oncomplete: [Function: onlookup],
domain:
Domain {
domain: null,
_events: { error: [Function: debugDomainError] },
_eventsCount: 1,
_maxListeners: undefined,
members: [] } }
> null '172.18.0.3' 4
> dns.lookup('app:5000', console.log)
GetAddrInfoReqWrap {
callback: [Function: bound consoleCall],
family: 0,
hostname: 'app:5000',
oncomplete: [Function: onlookup],
domain:
Domain {
domain: null,
_events: { error: [Function: debugDomainError] },
_eventsCount: 1,
_maxListeners: undefined,
members: [] } }
> { Error: getaddrinfo ENOTFOUND app:5000
at errnoException (dns.js:50:10)
at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:92:26)
code: 'ENOTFOUND',
errno: 'ENOTFOUND',
syscall: 'getaddrinfo',
hostname: 'app:5000' }
So it looks like it can find app but not app:5000? Weird. Since finding this out, I tried changing the python webserver port to 80 so I could make the axios requests to plain ol' http://app, but that resulted in the same ENOTFOUND error.
I cannot find much information on how to fix this. One idea is from here
the problem is that http.request uses dns.lookup instead of dns.resolve
The answer suggests using an overlay network. I don't really understand what that is, but if it is required, I'll go ahead and do that. But, I'm wondering if there is a more simple solution now that I'm using compose yml version 3.6. In any case, dns.resolve gives me similar output to dns.lookup
> dns.resolve('app', console.log)
QueryReqWrap {
bindingName: 'queryA',
callback: [Function: bound consoleCall],
hostname: 'app',
oncomplete: [Function: onresolve],
ttl: false,
domain:
Domain {
domain: null,
_events: { error: [Function: debugDomainError] },
_eventsCount: 1,
_maxListeners: undefined,
members: [] },
channel: ChannelWrap {} }
> null [ '172.18.0.3' ]
> dns.resolve('app:5000', console.log)
QueryReqWrap {
bindingName: 'queryA',
callback: [Function: bound consoleCall],
hostname: 'app:5000',
oncomplete: [Function: onresolve],
ttl: false,
domain:
Domain {
domain: null,
_events: { error: [Function: debugDomainError] },
_eventsCount: 1,
_maxListeners: undefined,
members: [] },
channel: ChannelWrap {} }
> { Error: queryA ENOTFOUND app:5000
at errnoException (dns.js:50:10)
at QueryReqWrap.onresolve [as oncomplete] (dns.js:238:19)
code: 'ENOTFOUND',
errno: 'ENOTFOUND',
syscall: 'queryA',
hostname: 'app:5000' }
EDIT: By the way, I can make the same POST requests from the lambda container across https to the outside world (a production server running the same code as the app service)
EDIT: If I remove http:// from the endpointUrl as suggested here, I get Error: Invalid protocol: app
EDIT: I thought it might be related to this issue with the node-alpine base image, but changing to node:8.11 or node:carbon resulted in the same DNS issue ENOTFOUND
EDIT: I'm sure it is not a timing issue because I'm waiting to run my tests for 100 seconds, making a test request every 10 seconds. If I skip the test step and let the container spin up like normal, I'm able to curl app from inside the lambda container far before a 100-second mark...
