Docker Address Pools

I'm working on consolidating my lab (5 servers -> 1), and that means that I currently have roughly 20 stacks/60 containers running on a single server. If your docker environment is smaller than this, you may not have ever had to think about docker network pools. I know I didn't.

At least for the default bridge interface/mode, containers and stacks automatically get a network created for them, unless you manually specify an existing network. When this happens, docker creates a new bridge on the host system and configures a subnet for the containers to use. As you can see, a docker network using the bridge driver predictably creates a bridge:

#> docker network ls
NETWORK ID     NAME                       DRIVER    SCOPE
1544f66cc446   ghost_default              bridge    local

#> brctl show br-1544f66cc446
bridge name        bridge id            STP enabled   interfaces
br-1544f66cc446    8000.02422bc97989    no            veth1c20ae6
                                                      veth2103a84
                                                      veth56d45eb

So we can see that we have a docker network with a matching bridge, and in order for that to be even remotely helpful, we also need a route:

#> ip route show | grep 1544f66cc446
172.30.0.0/16 dev br-1544f66cc446 proto kernel scope link src 172.30.0.1

This particular bridge is using the 172.30.0.0/16 subnet, with 172.30.0.1 as the gateway (which actually uses the docker0 gateway, usually 172.17.0.1). With a /16 subnet, this means we have 65,534 usable hosts in this docker network, which is currently being used by... 3 containers (the veth interfaces shown by brctl). That's kind of absurd for a homelab, but normally it would never cause any real issues. In some cases, though, once you create enough networks you may run into address space conflicts. This is the default network pool used by docker:

{
  "default-address-pools": [
    {"base":"172.17.0.0/16","size":16},
    {"base":"172.18.0.0/16","size":16},
    {"base":"172.19.0.0/16","size":16},
    {"base":"172.20.0.0/14","size":16},
    {"base":"172.24.0.0/14","size":16},
    {"base":"172.28.0.0/14","size":16},
    {"base":"192.168.0.0/16","size":20}
  ]
}

I'm not going to pretend to know subnets well enough to break down how many networks this makes, but it's more than enough for any use. The problem is that it includes {"base":"192.168.0.0/16","size":20} which means all 192.168.0.0 addresses may be used by docker. This is potentially not good, as most home networks default to 192.168.0.0/24 or 192.168.1.0/24.

The result, given enough networks you will eventually have a subnet created in that space, let's use 192.168.4.0/24 as an example. In a typical home network, you might have client devices on 192.168.0.0/24 (that is, 192.168.0.2-254). When docker assigns 192.168.4.2 to a container it won't conflict with any client devices, but that doesn't mean you're safe. The subnet size is still /20 which means that your home network address space of 192.168.0.* is included, and when the network and bridge are created, docker also creates a route for that subnet on the new bridge gateway.

If you are familiar with routing you can already see how this would be an issue, but let me cover the basics first. A route is kind of like the postal system. You have a local address like 192.168.1.18 and a gateway, or post office in this metaphor, of 192.168.1.1. When you want to send something to someone you take your mail to the post office and give them the destination address, and the post office knows where that address is. In the case of a local address, that post office can take it directly. But if the address is in the next town over, it first needs to send the message to that town's post office (router), and they can deliver it to the destination.

In the case of this specific issue, when that new route gets created with a new gateway all the traffic related to that subnet will now go to the new gateway. The problem is that gateway doesn't have any information on where any other addresses in town might be, it only knows the local neighborhood. So your traffic, destined for the other side of town, never makes it out of the neighborhood. The solution is simple, just remove the docker network and by extension the bridge and route. Then the traffic will once again correctly be sent to your router which does have all the address and route info for other clients.

My homelab is on the 10.8.8.0/24 subnet so the impact wasn't huge, I only lost connection to some cameras in 192.168.1.0/24, but in the worst case the docker host could have lost internet connectivity entirely. Luckily preventing this is simple, you just need to configure the default docker pools. Docker's own example is plenty for me, and it just involves setting:

{
  "default-address-pools": [
    { "base": "172.17.0.0/16", "size": 24 },
    { "base": "172.18.0.0/16", "size": 24 }
  ]
}

All this does is limit the available subnets to anything starting with 172.17 or 172.18, with each subnet being size /24 which allows for 254 usable hosts per subnet. That's a total of ~512 networks that can be created, which is certainly more than enough for any homelab while avoiding any potential IP collisions or routing issues.