Last night I was trying to figure out why one of my swarm services (Drone, drone.cubieserver.de) could not access another swarm service (Gitea, git.cubieserver.de). Both are running in the same swarm stack and network and both are proxied by Traefik. All the services are running on the same host.

I could also let the Drone service directly access Gitea through Docker swarms internal mesh network, but then I’d loose the flexibility of moving the individual services to different hosts and stacks. Also, the connection would no longer be encrypted.

When I was trying to access one of the services (via the external hostname) I got a connection timeout:

root@drone $ wget https://git.cubieserver.de
Connecting to git.cubieserver.de (144.76.91.184:443)
wget: can't connect to remote host (144.76.91.184): Operation timed out

After looking extensively into my firewall rules and searching the web, I figured out that there must be something wrong with them. Usually when running Docker services and publishing the ports, Docker automatically creates the appropriate firewall rules. Well, almost.

-A DOCKER -i docker0 -j RETURN
-A DOCKER -i docker_gwbridge -j RETURN
-A DOCKER ! -i docker_gwbridge -p tcp -m tcp --dport 443 -j DNAT --to-destination 172.18.0.7:443
-A DOCKER ! -i docker_gwbridge -p tcp -m tcp --dport 80 -j DNAT --to-destination 172.18.0.7:80
-A DOCKER-INGRESS -p tcp -m tcp --dport 5443 -j DNAT --to-destination 172.18.0.2:5443

You can see that the two rules on line 3 and 4 are from ports published in host networking mode. This is required when you want the service to be able to see the real origin IP of a client (otherwise it will always just see the IP of the Docker gateway).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

services:
  traefik:
    image: traefik
    ports:
    - target: 80
      published: 80
      protocol: tcp
      mode: host
    - target: 443
      published: 443
      protocol: tcp
      mode: host

The last rule (line 5) is from a port that was published in “regular” mode (i.e. in the overlay network):

1
2
3
4
5
6

services:
  ejabberd:
    image: ejabberd/ecs
    ports:
    - 5222:5222
    - 5443:5443

You can see that the first two rules do not apply to the input interace docker_gwbridge, thus an internal (swarm) service trying to connect to ports 80 and 443 won’t be able to reach them.

The fix is to create an additional firewall rule (like you would do for your regular web server, too) that allows traffic from all interfaces to these ports.

-A INPUT -p tcp -m multiport --dports 80 -m comment --comment "100 allow HTTP traffic" -j ACCEPT
-A INPUT -p tcp -m multiport --dports 443 -m comment --comment "100 allow HTTPS traffic" -j ACCEPT

Or in my case simply with Puppet:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

firewall { '100 allow HTTP traffic':
  dport  => 80,
  proto  => tcp,
  action => accept,
}
firewall { '100 allow HTTPS traffic':
  dport  => 443,
  proto  => tcp,
  action => accept,
}

So if you ever have the issue that you cannot connect to a port published in host networking mode (but can connect to one published in the overlay network) from inside the swarm, have a look at your firewall rules and try this fix.