Last night I was trying to figure out why one of my swarm services (Drone, drone.cubieserver.de) could not access another swarm service (Gitea, git.cubieserver.de). Both are running in the same swarm stack and network and both are proxied by Traefik. All the services are running on the same host.
I could also let the Drone service directly access Gitea through Docker swarms internal mesh network, but then I’d loose the flexibility of moving the individual services to different hosts and stacks. Also, the connection would no longer be encrypted.
When I was trying to access one of the services (via the external hostname) I got a connection timeout:
root@drone $ wget https://git.cubieserver.de Connecting to git.cubieserver.de (126.96.36.199:443) wget: can't connect to remote host (188.8.131.52): Operation timed out
After looking extensively into my firewall rules and searching the web, I figured out that there must be something wrong with them.
Usually when running Docker services and
publishing the ports, Docker automatically creates the appropriate firewall rules.
-A DOCKER -i docker0 -j RETURN -A DOCKER -i docker_gwbridge -j RETURN -A DOCKER ! -i docker_gwbridge -p tcp -m tcp --dport 443 -j DNAT --to-destination 172.18.0.7:443 -A DOCKER ! -i docker_gwbridge -p tcp -m tcp --dport 80 -j DNAT --to-destination 172.18.0.7:80 -A DOCKER-INGRESS -p tcp -m tcp --dport 5443 -j DNAT --to-destination 172.18.0.2:5443
You can see that the two rules on line 3 and 4 are from ports published in
host networking mode.
This is required when you want the service to be able to see the real origin IP of a client (otherwise it will always just see the IP of the Docker gateway).
The last rule (line 5) is from a port that was published in “regular” mode (i.e. in the overlay network):
You can see that the first two rules do not apply to the input interace
docker_gwbridge, thus an internal (swarm) service trying to connect to ports 80 and 443 won’t be able to reach them.
The fix is to create an additional firewall rule (like you would do for your regular web server, too) that allows traffic from all interfaces to these ports.
-A INPUT -p tcp -m multiport --dports 80 -m comment --comment "100 allow HTTP traffic" -j ACCEPT -A INPUT -p tcp -m multiport --dports 443 -m comment --comment "100 allow HTTPS traffic" -j ACCEPT
Or in my case simply with Puppet:
So if you ever have the issue that you cannot connect to a port published in host networking mode (but can connect to one published in the overlay network) from inside the swarm, have a look at your firewall rules and try this fix.