Build a backup routine
There are two different questions here and they have different answers. One is undoing a change you just made. The other is getting everything back after losing a machine.
Two questions, two answers
| Question | Answer | How long |
|---|---|---|
| Undo the change I just made | Go back to the previous version on the Changes page. | Seconds |
| Get everything back after losing a machine | Restore a backup onto a new one. | Minutes |
The first is the answer nearly every time. It is quick, it needs no files, and it works while you are still standing there wondering what you just did.
What a backup holds
Sites, backend pools, zones and access lists, TCP and UDP services, error pages, certificates, paid authorities, settings and accounts, plus the certificate files themselves.
That last part matters. It means a restored fleet can serve traffic immediately, rather than waiting to be issued new certificates it cannot get until it is already serving traffic.
The file is equivalent to your private keys
It contains every private key this fleet holds, which is enough on its own to serve every site you serve, to anybody who has the file and the password. Keep it where you would keep the keys themselves.
What is already kept for you
| What | Where | Covers |
|---|---|---|
| Previous nginx configs, before every apply | /var/backups/nginx-fleet-manager | Getting nginx started when the GUI is not running |
| The configuration database | Docker volume nfm-data | Everything, if you snapshot the machine |
| Certificates and keys | /etc/nginx/nginxmgr/certs | Serving traffic |
| Settings and the secret key | The .env file | Reading your own encrypted values |
Guard .env separately
Without it, a restored database cannot decrypt its own stored passwords and authenticator seeds. It is one small file per node and it is the one people forget.
A routine that actually gets done
- Before any change you are unsure about. A backup taken before a change is the cheapest way to undo it, and it costs ten seconds.
- On a schedule that matches how often your configuration changes. Weekly for most people. Monthly if it barely moves.
- After any big change. A new site, a certificate migration, a WAF rollout.
- Snapshot the virtual machines if they are virtual. That covers everything in the table above at once and is the simplest thing that works.
Where to keep them
- Not only on the load balancers.
- Somewhere access is controlled.
- Somewhere you will still have access if you are locked out of this system.
- Inside something with proper key derivation if it is going somewhere you do not control. The zip encryption derives its key with fewer rounds than a modern password vault, so the file is only as strong as the password against somebody with time to spend.
Delete old copies rather than letting them accumulate. Every one of them stays valid for as long as the keys inside it do.
Test it, once, properly
An untested backup is a hope. Test it on a throwaway machine, not on your live fleet.
- Build a spare server exactly as you built the first one.
- Take a backup from the live fleet.
- Restore it onto the spare, using See what is in it first.
- Check the Sites page and the Certificates page match what you expect.
- Read the report the restore printed for anything it could not do.
- Throw the machine away.
An hour, once, and you know. Do it again after any major upgrade.
The trap when restoring onto different servers
Almost everything in a backup is names and settings, which mean the same thing wherever they land. DNS failover answers are the exception: an answer is a literal address whose whole job is to be handed out to the internet.
Carried across unchanged, those answers would still name the old servers. The new load balancers would come up, report themselves healthy, and send every visitor to machines that are not theirs. Both ends would look correct.
So a restore repoints them: an answer tied to a load balancer by name is rewritten to that load balancer's address on the new fleet, matching the kind of address. Anything it cannot match is left alone and reported to you by name, because guessing which new machine replaces which old one is not something to do quietly with a DNS record.
After a restore
- The accounts in the backup replace the accounts here, so sign in again if your own account came from the backup.
- Set the management network allowlist afterwards. It is deliberately not restored, because restoring it is a good way to be locked out of the system you are recovering.
- Re-enroll any tunnel workloads, because each machine still points at the old load balancers.
- Check certificates. One that expired while everything was down needs renewing.