Clustering

Why there is no controller

A central controller is one more thing that has to be up before you can manage your load balancers, which is exactly backwards. When something is broken at three in the morning you still need to get in and fix nginx.

So every node runs the same container, holds a full copy of the configuration in its own SQLite database, and can be signed into on its own. Losing a node loses that node.

Active and standby

At any moment one node is active and the rest are standby. Both hold identical configuration and both are serving traffic the whole time.

How the active node is chosen

Priority decides, and higher wins. Two things stop this going wrong:

• A node will not make itself active unless it can see more than half the fleet. A network split cannot leave you with two nodes both taking writes.
• Forced standby stops a node becoming active without removing it from the fleet. Use it on a machine you are working on, and remember to turn it off afterwards, because a fleet where one node can never take over is a fleet with no failover and nothing will remind you.

Staying in step

Nodes send each other a heartbeat every few seconds saying what they are and which configuration version they are running. If one stops hearing from another for long enough, it decides that node has gone.

Configuration cannot drift through normal use, because every apply goes to every node before any node uses it. A node that was switched off during a change gets it queued and replayed when it comes back.

Adding a node

On a node that is already running, go to Cluster and press Add a node. You get a command with a token in it. The token works once and expires in two hours.

sudo git clone https://git.failoverlb.net/failoverlb.git /data/docker/failoverlb \
  && sudo /data/docker/failoverlb/failoverlb_setup.sh \
       --join 10.0.0.10:7444 --token <token>

The new node makes its own key pair, sends a signing request with the token, gets back a certificate signed by the cluster authority, and then asks for the whole roster so it learns about every other member rather than only the one it joined through. It comes up as standby and pulls the current configuration, so you never set your sites up twice.

Why SQLite

Every node keeps its own full copy and the nodes sync with each other. Adding Postgres would mean one more service that has to be healthy before you can manage your load balancers, and it would become the thing that fails first.

Three nodes and beyond

Nothing about the design is limited to two. A third node joins exactly the same way. Quorum works better with an odd number, so three is a genuinely better shape than two if you have the machines.

Common questions