13

The problem

Inside nginx-proxy Docker container (more info below), I always see the same IP adress for every connection: 172.18.0.1 (which is the nginx-proxy network gateway). For example:

nginx.1    | www.my-site.tld 172.18.0.1 - - [28/Nov/2017:17:22:21 +0000] "GET /some/path HTTP/2.0" 200 46576 "https://www.my-site.tld/some/path" "Mozilla/5.0 (Linux; Android 4.4.2; PSP5507DUO Build/KVT49L) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 YaBrowser/17.10.0.446.00 Mobile Safari/537.36"

No matter if I make requests from the external network, of from the local machine, or even from the server (host) itself, the IP address is always 172.18.0.1.

What I need is to see the real client IP.

Attempts to fix

I have googled a lot, and tried different solutions, but none worked for me. I tried the following:

  • disabling firewalld (nothing works at all in that case, of course, iptables are empty);
  • using --userland-proxy=false (no luck);
  • using --net=host (not a solution at all, and does not work, too).

Related links:

So, any ideas? Seems to be either iptables routing issues or internal Docker bug. Anyway, it is for sure related to Docker, because the following displays correct remote user IP:

[root@server]# nc -lv 12345

[user@remote-client]$ nc -vz MY.IP.ADDRESS.HERE 12345

Very weird!

Now I'm going to provide full information about the system, so please be patient. :)

Basic

I have Centos 7.4 and Docker installed:

# cat /etc/centos-release
CentOS Linux release 7.4.1708 (Core)

# uname -r
3.10.0-693.5.2.el7.x86_64

# docker version
Client:
 Version:      17.09.0-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:41:23 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.09.0-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:42:49 2017
 OS/Arch:      linux/amd64
 Experimental: false

firewalld is used as a firewall.

Network configuration

Interfaces

There are three physical interfaces:

# ifconfig | grep -A 7 enp
enp1s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::1ad6:c7ff:fe00:fc98  prefixlen 64  scopeid 0x20<link>
        ether 18:d6:c7:00:fc:98  txqueuelen 1000  (Ethernet)
        RX packets 11530516  bytes 2220554890 (2.0 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 24237462  bytes 31702254967 (29.5 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp2s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::1ad6:c7ff:fe06:ef36  prefixlen 64  scopeid 0x20<link>
        ether 18:d6:c7:06:ef:36  txqueuelen 1000  (Ethernet)
        RX packets 948513  bytes 143294797 (136.6 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2371584  bytes 3207775040 (2.9 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp3s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 31.172.141.111  netmask 255.255.255.128  broadcast MY.EXTERNAL.IP.HERE
        inet6 fe80::127b:44ff:fe46:c2d1  prefixlen 64  scopeid 0x20<link>
        ether 10:7b:44:46:c2:d1  txqueuelen 1000  (Ethernet)
        RX packets 46772225  bytes 57416420859 (53.4 GiB)
        RX errors 0  dropped 1  overruns 0  frame 0
        TX packets 28814037  bytes 10943786995 (10.1 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

And a virtual bridge (enp1s0 and enp2s0):

# ifconfig | grep -A 7 virbr
virbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.1  netmask 255.255.255.0  broadcast 192.168.0.255
        inet6 fe80::1ad6:c7ff:fe00:fc98  prefixlen 64  scopeid 0x20<link>
        ether 18:d6:c7:00:fc:98  txqueuelen 1000  (Ethernet)
        RX packets 11542053  bytes 2042945874 (1.9 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 26586017  bytes 34919742970 (32.5 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Firewall zones

The enp3s0 interface is connected to the Internet (--zone=public), and virbr0 is connected to the internal local network (--zone=internal).

Forwarding and masquerading

IPv4 forwarding is enabled:

# cat /proc/sys/net/ipv4/ip_forward
1

and masquerading on public zone is configured with firewalld:

# firewall-cmd --zone=public --list-all
success

Docker configuration

Docker has default configuration.

Networking

I only created a network:

# docker network create -d bridge nginx-proxy

Here is it's configuration:

# docker network inspect nginx-proxy
[
    {
        "Name": "nginx-proxy",
        "Id": "192821446c9a5891fd1a7e240533cefb81ba0548033acb605eea805abec83505",
        "Created": "2017-11-24T14:25:10.687199435+02:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "1c563c33a45c9feabee4cb5ec3194464c66bfad1e233dedb797e269649067159": {
                "Name": "nginxproxy_nginx-proxy_1",
                "EndpointID": "683ceaced8eb416b485775734ca6828e9eced5ea6e3c7a76960c4eb807b521d9",
                "MacAddress": "02:42:ac:12:00:02",
                "IPv4Address": "172.18.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}
    }
]

Images and containers

I'm using jwilder/nginx-proxy image. Here is my docker-compose.yml:

version: '3'
services:
  nginx-proxy:
    image: jwilder/nginx-proxy:alpine
    labels:
      - 'com.github.jrcs.letsencrypt_nginx_proxy_companion.nginx_proxy=true'
    volumes:
      - /var/run/docker.sock:/tmp/docker.sock:ro
    ports:
      - '80:80'
      - '443:443'
    networks:
      nginx-proxy:
networks:
  nginx-proxy:
depends_on:
  - nginx-proxy

networks:
  nginx-proxy:
    external:
      name: nginx-proxy

As you can see, nginx-proxy container is listening on ports 80 and 443 at all interfaces.

# docker ps | grep nginx-proxy
1c563c33a45c        jwilder/nginx-proxy:alpine                        "/app/docker-entry..."   8 hours ago         Up 8 hours          0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp   nginxproxy_nginx-proxy_1

iptables

Container-related rules

Here are iptables rules related to the container:

# iptables-save | grep -e br-192821446c9a -e 172.18.0 -e '\*'
*mangle
*security
*raw
*filter
-A FORWARD -o br-192821446c9a -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-192821446c9a -j DOCKER
-A FORWARD -i br-192821446c9a ! -o br-192821446c9a -j ACCEPT
-A FORWARD -i br-192821446c9a -o br-192821446c9a -j ACCEPT
-A DOCKER -d 172.18.0.2/32 ! -i br-192821446c9a -o br-192821446c9a -p tcp -m tcp --dport 443 -j ACCEPT
-A DOCKER -d 172.18.0.2/32 ! -i br-192821446c9a -o br-192821446c9a -p tcp -m tcp --dport 80 -j ACCEPT
*nat
-A POSTROUTING -s 172.18.0.0/16 ! -o br-192821446c9a -j MASQUERADE
-A POSTROUTING -s 172.18.0.2/32 -d 172.18.0.2/32 -p tcp -m tcp --dport 443 -j MASQUERADE
-A POSTROUTING -s 172.18.0.2/32 -d 172.18.0.2/32 -p tcp -m tcp --dport 80 -j MASQUERADE
-A DOCKER -i br-192821446c9a -j RETURN
-A DOCKER ! -i br-192821446c9a -p tcp -m tcp --dport 443 -j DNAT --to-destination 172.18.0.2:443
-A DOCKER ! -i br-192821446c9a -p tcp -m tcp --dport 80 -j DNAT --to-destination 172.18.0.2:80

Complete iptables rules list

And here is the complete output of iptables-save:

# iptables-save
# Generated by iptables-save v1.4.21 on Tue Nov 28 19:16:31 2017
*mangle
:PREROUTING ACCEPT [9063259:7480026035]
:INPUT ACCEPT [250719:30179859]
:FORWARD ACCEPT [8795875:7445096680]
:OUTPUT ACCEPT [1078071:1430559812]
:POSTROUTING ACCEPT [9842972:8867742437]
:FORWARD_direct - [0:0]
:INPUT_direct - [0:0]
:OUTPUT_direct - [0:0]
:POSTROUTING_direct - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_internal - [0:0]
:PRE_internal_allow - [0:0]
:PRE_internal_deny - [0:0]
:PRE_internal_log - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A INPUT -j INPUT_direct
-A FORWARD -j FORWARD_direct
-A OUTPUT -j OUTPUT_direct
-A POSTROUTING -j POSTROUTING_direct
-A PREROUTING_ZONES -i enp3s0 -g PRE_public
-A PREROUTING_ZONES -i virbr0 -g PRE_internal
-A PREROUTING_ZONES -g PRE_public
-A PRE_internal -j PRE_internal_log
-A PRE_internal -j PRE_internal_deny
-A PRE_internal -j PRE_internal_allow
-A PRE_public -j PRE_public_log
-A PRE_public -j PRE_public_deny
-A PRE_public -j PRE_public_allow
COMMIT
# Completed on Tue Nov 28 19:16:31 2017
# Generated by iptables-save v1.4.21 on Tue Nov 28 19:16:31 2017
*security
:INPUT ACCEPT [207361:25027156]
:FORWARD ACCEPT [8764800:7437165960]
:OUTPUT ACCEPT [1078077:1430561824]
:FORWARD_direct - [0:0]
:INPUT_direct - [0:0]
:OUTPUT_direct - [0:0]
-A INPUT -j INPUT_direct
-A FORWARD -j FORWARD_direct
-A OUTPUT -j OUTPUT_direct
COMMIT
# Completed on Tue Nov 28 19:16:31 2017
# Generated by iptables-save v1.4.21 on Tue Nov 28 19:16:31 2017
*raw
:PREROUTING ACCEPT [9063267:7480026451]
:OUTPUT ACCEPT [1078080:1430562828]
:OUTPUT_direct - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_internal - [0:0]
:PRE_internal_allow - [0:0]
:PRE_internal_deny - [0:0]
:PRE_internal_log - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A OUTPUT -j OUTPUT_direct
-A PREROUTING_ZONES -i enp3s0 -g PRE_public
-A PREROUTING_ZONES -i virbr0 -g PRE_internal
-A PREROUTING_ZONES -g PRE_public
-A PRE_internal -j PRE_internal_log
-A PRE_internal -j PRE_internal_deny
-A PRE_internal -j PRE_internal_allow
-A PRE_public -j PRE_public_log
-A PRE_public -j PRE_public_deny
-A PRE_public -j PRE_public_allow
COMMIT
# Completed on Tue Nov 28 19:16:31 2017
# Generated by iptables-save v1.4.21 on Tue Nov 28 19:16:31 2017
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1078082:1430564354]
:DOCKER - [0:0]
:DOCKER-ISOLATION - [0:0]
:FORWARD_IN_ZONES - [0:0]
:FORWARD_IN_ZONES_SOURCE - [0:0]
:FORWARD_OUT_ZONES - [0:0]
:FORWARD_OUT_ZONES_SOURCE - [0:0]
:FORWARD_direct - [0:0]
:FWDI_internal - [0:0]
:FWDI_internal_allow - [0:0]
:FWDI_internal_deny - [0:0]
:FWDI_internal_log - [0:0]
:FWDI_public - [0:0]
:FWDI_public_allow - [0:0]
:FWDI_public_deny - [0:0]
:FWDI_public_log - [0:0]
:FWDO_internal - [0:0]
:FWDO_internal_allow - [0:0]
:FWDO_internal_deny - [0:0]
:FWDO_internal_log - [0:0]
:FWDO_public - [0:0]
:FWDO_public_allow - [0:0]
:FWDO_public_deny - [0:0]
:FWDO_public_log - [0:0]
:INPUT_ZONES - [0:0]
:INPUT_ZONES_SOURCE - [0:0]
:INPUT_direct - [0:0]
:IN_internal - [0:0]
:IN_internal_allow - [0:0]
:IN_internal_deny - [0:0]
:IN_internal_log - [0:0]
:IN_public - [0:0]
:IN_public_allow - [0:0]
:IN_public_deny - [0:0]
:IN_public_log - [0:0]
:OUTPUT_direct - [0:0]
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -j INPUT_direct
-A INPUT -j INPUT_ZONES_SOURCE
-A INPUT -j INPUT_ZONES
-A INPUT -m conntrack --ctstate INVALID -j DROP
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o br-828c38e3582b -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-828c38e3582b -j DOCKER
-A FORWARD -i br-828c38e3582b ! -o br-828c38e3582b -j ACCEPT
-A FORWARD -i br-828c38e3582b -o br-828c38e3582b -j ACCEPT
-A FORWARD -o br-7cbbdaf3d8fe -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-7cbbdaf3d8fe -j DOCKER
-A FORWARD -i br-7cbbdaf3d8fe ! -o br-7cbbdaf3d8fe -j ACCEPT
-A FORWARD -i br-7cbbdaf3d8fe -o br-7cbbdaf3d8fe -j ACCEPT
-A FORWARD -o br-92f8d769de0b -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-92f8d769de0b -j DOCKER
-A FORWARD -i br-92f8d769de0b ! -o br-92f8d769de0b -j ACCEPT
-A FORWARD -i br-92f8d769de0b -o br-92f8d769de0b -j ACCEPT
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -o br-6a38b645b1c7 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-6a38b645b1c7 -j DOCKER
-A FORWARD -i br-6a38b645b1c7 ! -o br-6a38b645b1c7 -j ACCEPT
-A FORWARD -i br-6a38b645b1c7 -o br-6a38b645b1c7 -j ACCEPT
-A FORWARD -o br-2d90be830c58 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-2d90be830c58 -j DOCKER
-A FORWARD -i br-2d90be830c58 ! -o br-2d90be830c58 -j ACCEPT
-A FORWARD -i br-2d90be830c58 -o br-2d90be830c58 -j ACCEPT
-A FORWARD -o br-192821446c9a -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-192821446c9a -j DOCKER
-A FORWARD -i br-192821446c9a ! -o br-192821446c9a -j ACCEPT
-A FORWARD -i br-192821446c9a -o br-192821446c9a -j ACCEPT
-A FORWARD -o br-b61364bf1724 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-b61364bf1724 -j DOCKER
-A FORWARD -i br-b61364bf1724 ! -o br-b61364bf1724 -j ACCEPT
-A FORWARD -i br-b61364bf1724 -o br-b61364bf1724 -j ACCEPT
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i lo -j ACCEPT
-A FORWARD -j FORWARD_direct
-A FORWARD -j FORWARD_IN_ZONES_SOURCE
-A FORWARD -j FORWARD_IN_ZONES
-A FORWARD -j FORWARD_OUT_ZONES_SOURCE
-A FORWARD -j FORWARD_OUT_ZONES
-A FORWARD -m conntrack --ctstate INVALID -j DROP
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
-A OUTPUT -j OUTPUT_direct
-A DOCKER -d 172.18.0.2/32 ! -i br-192821446c9a -o br-192821446c9a -p tcp -m tcp --dport 443 -j ACCEPT
-A DOCKER -d 172.18.0.2/32 ! -i br-192821446c9a -o br-192821446c9a -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER -d 172.20.0.2/32 ! -i br-2d90be830c58 -o br-2d90be830c58 -p tcp -m tcp --dport 1194 -j ACCEPT
-A DOCKER-ISOLATION -j RETURN
-A FORWARD_IN_ZONES -i enp3s0 -g FWDI_public
-A FORWARD_IN_ZONES -i virbr0 -g FWDI_internal
-A FORWARD_IN_ZONES -g FWDI_public
-A FORWARD_OUT_ZONES -o enp3s0 -g FWDO_public
-A FORWARD_OUT_ZONES -o virbr0 -g FWDO_internal
-A FORWARD_OUT_ZONES -g FWDO_public
-A FWDI_internal -j FWDI_internal_log
-A FWDI_internal -j FWDI_internal_deny
-A FWDI_internal -j FWDI_internal_allow
-A FWDI_internal -p icmp -j ACCEPT
-A FWDI_public -j FWDI_public_log
-A FWDI_public -j FWDI_public_deny
-A FWDI_public -j FWDI_public_allow
-A FWDI_public -p icmp -j ACCEPT
-A FWDO_internal -j FWDO_internal_log
-A FWDO_internal -j FWDO_internal_deny
-A FWDO_internal -j FWDO_internal_allow
-A FWDO_public -j FWDO_public_log
-A FWDO_public -j FWDO_public_deny
-A FWDO_public -j FWDO_public_allow
-A FWDO_public_allow -m conntrack --ctstate NEW -j ACCEPT
-A INPUT_ZONES -i enp3s0 -g IN_public
-A INPUT_ZONES -i virbr0 -g IN_internal
-A INPUT_ZONES -g IN_public
-A IN_internal -j IN_internal_log
-A IN_internal -j IN_internal_deny
-A IN_internal -j IN_internal_allow
-A IN_internal -p icmp -j ACCEPT
-A IN_internal_allow -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW -j ACCEPT
-A IN_internal_allow -d 224.0.0.251/32 -p udp -m udp --dport 5353 -m conntrack --ctstate NEW -j ACCEPT
-A IN_internal_allow -p udp -m udp --dport 137 -m conntrack --ctstate NEW -j ACCEPT
-A IN_internal_allow -p udp -m udp --dport 138 -m conntrack --ctstate NEW -j ACCEPT
-A IN_internal_allow -p tcp -m tcp --dport 2049 -m conntrack --ctstate NEW -j ACCEPT
-A IN_internal_allow -p tcp -m tcp --dport 20048 -m conntrack --ctstate NEW -j ACCEPT
-A IN_internal_allow -p udp -m udp --dport 20048 -m conntrack --ctstate NEW -j ACCEPT
-A IN_internal_allow -p tcp -m tcp --dport 111 -m conntrack --ctstate NEW -j ACCEPT
-A IN_internal_allow -p udp -m udp --dport 111 -m conntrack --ctstate NEW -j ACCEPT
-A IN_internal_allow -p tcp -m tcp --dport 139 -m conntrack --ctstate NEW -j ACCEPT
-A IN_internal_allow -p tcp -m tcp --dport 445 -m conntrack --ctstate NEW -j ACCEPT
-A IN_internal_allow -p tcp -m tcp --dport 58846 -m conntrack --ctstate NEW -j ACCEPT
-A IN_internal_allow -p tcp -m tcp --dport 8112 -m conntrack --ctstate NEW -j ACCEPT
-A IN_internal_allow -p tcp -m tcp --dport 53 -m conntrack --ctstate NEW -j ACCEPT
-A IN_internal_allow -p udp -m udp --dport 53 -m conntrack --ctstate NEW -j ACCEPT
-A IN_public -j IN_public_log
-A IN_public -j IN_public_deny
-A IN_public -j IN_public_allow
-A IN_public -p icmp -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW -j ACCEPT
-A IN_public_allow -p udp -m udp --dport 67 -m conntrack --ctstate NEW -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 80 -m conntrack --ctstate NEW -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 443 -m conntrack --ctstate NEW -j ACCEPT
-A IN_public_allow -p udp -m udp --dport 1194 -m conntrack --ctstate NEW -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 1195 -m conntrack --ctstate NEW -j ACCEPT
COMMIT
# Completed on Tue Nov 28 19:16:31 2017
# Generated by iptables-save v1.4.21 on Tue Nov 28 19:16:31 2017
*nat
:PREROUTING ACCEPT [118631:13941821]
:INPUT ACCEPT [6147:464301]
:OUTPUT ACCEPT [18836:1989867]
:POSTROUTING ACCEPT [2593:197858]
:DOCKER - [0:0]
:OUTPUT_direct - [0:0]
:POSTROUTING_ZONES - [0:0]
:POSTROUTING_ZONES_SOURCE - [0:0]
:POSTROUTING_direct - [0:0]
:POST_internal - [0:0]
:POST_internal_allow - [0:0]
:POST_internal_deny - [0:0]
:POST_internal_log - [0:0]
:POST_public - [0:0]
:POST_public_allow - [0:0]
:POST_public_deny - [0:0]
:POST_public_log - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_internal - [0:0]
:PRE_internal_allow - [0:0]
:PRE_internal_deny - [0:0]
:PRE_internal_log - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -j OUTPUT_direct
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.101.0.0/24 ! -o br-828c38e3582b -j MASQUERADE
-A POSTROUTING -s 172.102.0.0/24 ! -o br-7cbbdaf3d8fe -j MASQUERADE
-A POSTROUTING -s 172.101.0.0/24 ! -o br-92f8d769de0b -j MASQUERADE
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.102.0.0/24 ! -o br-6a38b645b1c7 -j MASQUERADE
-A POSTROUTING -s 172.20.0.0/16 ! -o br-2d90be830c58 -j MASQUERADE
-A POSTROUTING -s 172.18.0.0/16 ! -o br-192821446c9a -j MASQUERADE
-A POSTROUTING -s 172.19.0.0/16 ! -o br-b61364bf1724 -j MASQUERADE
-A POSTROUTING -j POSTROUTING_direct
-A POSTROUTING -j POSTROUTING_ZONES_SOURCE
-A POSTROUTING -j POSTROUTING_ZONES
-A POSTROUTING -s 172.18.0.2/32 -d 172.18.0.2/32 -p tcp -m tcp --dport 443 -j MASQUERADE
-A POSTROUTING -s 172.18.0.2/32 -d 172.18.0.2/32 -p tcp -m tcp --dport 80 -j MASQUERADE
-A POSTROUTING -s 172.20.0.2/32 -d 172.20.0.2/32 -p tcp -m tcp --dport 1194 -j MASQUERADE
-A DOCKER -i br-828c38e3582b -j RETURN
-A DOCKER -i br-7cbbdaf3d8fe -j RETURN
-A DOCKER -i br-92f8d769de0b -j RETURN
-A DOCKER -i docker0 -j RETURN
-A DOCKER -i br-6a38b645b1c7 -j RETURN
-A DOCKER -i br-2d90be830c58 -j RETURN
-A DOCKER -i br-192821446c9a -j RETURN
-A DOCKER -i br-b61364bf1724 -j RETURN
-A DOCKER ! -i br-192821446c9a -p tcp -m tcp --dport 443 -j DNAT --to-destination 172.18.0.2:443
-A DOCKER ! -i br-192821446c9a -p tcp -m tcp --dport 80 -j DNAT --to-destination 172.18.0.2:80
-A DOCKER ! -i br-2d90be830c58 -p tcp -m tcp --dport 1195 -j DNAT --to-destination 172.20.0.2:1194
-A POSTROUTING_ZONES -o enp3s0 -g POST_public
-A POSTROUTING_ZONES -o virbr0 -g POST_internal
-A POSTROUTING_ZONES -g POST_public
-A POST_internal -j POST_internal_log
-A POST_internal -j POST_internal_deny
-A POST_internal -j POST_internal_allow
-A POST_public -j POST_public_log
-A POST_public -j POST_public_deny
-A POST_public -j POST_public_allow
-A POST_public_allow ! -o lo -j MASQUERADE
-A PREROUTING_ZONES -i enp3s0 -g PRE_public
-A PREROUTING_ZONES -i virbr0 -g PRE_internal
-A PREROUTING_ZONES -g PRE_public
-A PRE_internal -j PRE_internal_log
-A PRE_internal -j PRE_internal_deny
-A PRE_internal -j PRE_internal_allow
-A PRE_public -j PRE_public_log
-A PRE_public -j PRE_public_deny
-A PRE_public -j PRE_public_allow
COMMIT
# Completed on Tue Nov 28 19:16:31 2017
ololoepepe
  • 540
  • 5
  • 14
  • `172.18.0.1` _is_ a real IP address. What is a fake IP address, and why would that be a fake IP address? – Ron Maupin Dec 20 '17 at 20:37
  • 3
    @RonMaupin `172.18.0.1` is IP address of the virtual Docker network gateway. When a user with an IP address, for example, `111.222.111.222` connects, his/her address should be recognized as `111.222.111.222`, and surely not `172.18.0.1` (which actually used to happen). As I answered below, this was caused by masquerading the real IP. In simple words, iptables thought that a packet coming from my external interface (say, `1.2.3.4`) was going to another external interface (actually local Docker virtual interface) and IP in it should have been replaced by the IP of the interface (`172.18.0.1`). – ololoepepe Dec 21 '17 at 09:33
  • My point is that `172.18.0.1` is a real IP address. The dotted-decimal representation can be correctly converted to a 32-bit IPv4 address. It can appear as a source or destination address in an IP packet, and it is a forwardable address. People toss around phrases like "real IP address," but I don't understand. An IP address that isn't real would have an octet with a number larger than `255` (`300.298.407.33`) that could not be correctly converted to a 32-bit IPv4 address. – Ron Maupin Dec 21 '17 at 17:14
  • 3
    @RonMaupin I understand your point, but that's more a matter of terminology rather than the actual sense. If you look at https://github.com/jwilder/nginx-proxy/issues/130 and https://github.com/jwilder/nginx-proxy/issues/133 you would see that people use to call *that thing we are talking about* the "real IP address". Anyway, that's not related to the topic and neither helps to cllarify anything, nor brings the solution. Edit: there are sites like this: http://www.whatsmyrealip.com/ on the Internet, so I think my understanding of what a "real IP" means is quite common. – ololoepepe Dec 21 '17 at 19:35
  • I wasn't answering the question; I was clarifying the terminology, which is why I put it as a comment rather than an answer. Inaccurate terminology leads to errors. The fact is that `172.18.0.1` is a completely real IPv4 address, and implying that is not is inaccurate and misleading. – Ron Maupin Dec 21 '17 at 19:43

1 Answers1

8

Finally, answering my own question.

After some research I have found that it was IP masquerading that caused Docker to recognize client addresses incorrectly.

The cause

In my case, firewalld is configured to share internet connection with other machines in local network. This is accomplished by setting zone to public for the external interface (enp3s0) and to internal for the local interfaces (enp1s0, enp2s0), and by enabling IP masquerading for outgoing packages.

And that's it. Masquerading is also applied to packages coming from public network to docker networks, as if the package was going to (another) external interface.

Solution

As far as I understand all that iptables and Docker stuff, there is one acceptable workaround: add all virtual network interfaces created by Docker to the internal zone, so masquerading rule is not applied to packages going to the corresponding networks. For example:

# firewall-cmd --permanent --zone=internal --change-interface=br-192821446c9a

To make the solution persistent, it is required to create the corresponding script in the /etc/sysconfig/network-scripts directory. For example, for the br-192821446c9a interface, a file /etc/sysconfig/network-scripts/ifcfg-br-192821446c9a should be created, with the following content:

DEVICE=br-192821446c9a
TYPE=Bridge
BOOTPROTO=none
IPADDR=172.18.0.1
PREFIX=16
DEFROUTE=yes
IPV4_DNS_PRIORITY=100
NAME=br-192821446c9a
ONBOOT=no
ZONE=internal

Hope this answer will save somebody's time.

ololoepepe
  • 540
  • 5
  • 14