Fleet
Active and standby failover
What happens when a node fails, what it fixes, and, just as importantly, what it does not fix.
When the active node fails
The standby notices the heartbeats have stopped, promotes itself to active, and carries on. Traffic already going to the surviving node is unaffected throughout, because it was already serving.
What this does not solve
Traffic that was going to the dead node
Nothing on this page can help with that, because the decision about where to send a visitor was made before the connection was even attempted. Their computer looked up your address, was given the address of a node that has since died, and is now trying to reach it. Only the lookup itself can send them somewhere else, which is what DNS failover is for.
Failing over on purpose
There is a button to hand over to another node. Use it before planned work such as a reboot. Handing over deliberately is calmer than pulling the plug and letting the software work it out, and it lets you confirm the other node is healthy first.
Forced standby
Stops a node from becoming active without removing it from the fleet. Use it on a machine you are working on, so a reboot you did not expect to change anything does not hand it the fleet halfway through your work.
Remember to turn it off
A fleet where one node can never take over is a fleet with no failover, and nothing will remind you. It is the single most common thing left switched on after maintenance.
The order for maintenance
Coming out and going back in are mirror images, and doing it in this order is the whole point.
| Order | Coming out | Going back in |
|---|---|---|
| 1 | Check both nodes healthy | Check it is online again |
| 2 | Check nothing is saved and unapplied | Turn off forced standby |
| 3 | Turn off its DNS answer, then wait | Turn its DNS answer back on |
| 4 | Hand over if it is active | Watch it for a few minutes |
| 5 | Turn on forced standby | |
| 6 | Do the work |
Health first, then being allowed to be in charge, then traffic. Turning the DNS answer back on before the node is actually well sends people to something that is not ready.
Test it on purpose
In daylight, before you need it
A failover that has never been tried is a plan, not a capability, and the two look identical right up until the moment they do not. Stop the active node deliberately, watch what happens, and bring it back. An hour spent on that is worth more than any amount of documentation.
Common questions
How long does failover take?
A few seconds for the promotion itself. What takes longer is visitors whose computers already have the old address, which is a DNS problem rather than a cluster one.
Do existing connections to the failed node survive?
No. They were being served by a machine that has gone. The visitor retries and reaches the survivor.
Does the standby need warming up?
No. It was serving traffic the whole time.
Step by step instructions
The how to section has searchable, task shaped answers. Search it for failover.
Related features
Clustering
Every node holds the whole config. No central controller.
Read moreDNS failover and GSLB
Hand out the address of a node that is actually up.
Read moreTwo phase apply
Nothing goes live until every node says the config is valid.
Read moreUtilization charts
Requests, connections, processor and memory over time.
Read more