Disaster recovery
What happens to your fleet if the panel itself is lost — and the three ways back. The recovery kit you generate at setup, restoring the panel from a backup, claiming still-running agents over their own connection, and break-glass into a hardened node.
The panel is a control plane, not a runtime — your nodes, instances, and sites keep running even if the panel goes away (re-adopt covers reconciling workloads after the panel forgets them). But there is one thing only the panel holds that the nodes cannot rebuild on their own: the keys to get back to them.
This page is about that: how a panel loss can lock you out of your own fleet, and the layered ways to make sure it never does.
Why a panel loss can lock you out
When you enroll a node, the panel connects over SSH once to install a small agent, then hangs up. From then on the agent dials out to the panel over a secure WebSocket — that connection, not SSH, is the control channel. The agent's first job is to harden the box, which turns off password login and root SSH.
That hardening is what makes a panel loss dangerous. After a database wipe the panel no longer holds:
- the per-node token the agent authenticates with, so a still-running agent is no longer recognised;
- the SSH credential you first enrolled with; and
- even if you had it, sshd now refuses passwords — so your original password is rejected too.
The result: a hardened node, a forgotten panel, and no obvious way back in. Everything below exists so that situation is either prevented or quickly recoverable.
The two secrets to carry out of band
All recovery ultimately rests on two bootstrap secrets surviving a panel loss: MASTER_KEY (decrypts
every stored credential) and AGENT_GATEWAY_SECRET (lets agents re-authenticate). The recovery
kit below is the easiest way to keep both safe. Keep your backup's repository password separate from
the kit.
Layer 1 — Generate a recovery kit at setup
The chicken-and-egg problem with a panel backup is that it needs somewhere to go and a key to decrypt it — and at first boot you have neither configured. The recovery kit closes that gap. It is a small, passphrase-encrypted file you generate the moment you finish setup.
The kit contains the four boot secrets (including MASTER_KEY and AGENT_GATEWAY_SECRET) plus
non-secret restore hints (your panel hostname and where the backup target lives). It does not
contain your backup's repository password — keep that separate, so a leaked kit and a leaked backup
can't be combined.
Set a real MASTER_KEY first. The wizard refuses to make a kit under the development placeholder —
a kit made then would be useless.
Choose a passphrase (≥ 12 characters). It is never stored, logged, or emailed — it's the only thing that opens the kit, so keep it somewhere separate from the file (a password manager).
Generate. The kit downloads immediately, and — if you provide an email and SMTP is configured — a copy is sent as an attachment. The email body carries no secret; only the encrypted attachment.
Store it offline and confirm. Click I've saved it safely to clear the checklist step. Re-generate the kit whenever you rotate a boot secret.
Re-generate after a rotation
An old kit carries stale secrets. If you rotate MASTER_KEY, BETTER_AUTH_SECRET,
AGENT_GATEWAY_SECRET, or the database password, generate a fresh kit.
The recovery-at-risk warning
Because the kit is the safety net, the panel watches whether you actually have one. When you have nodes but no saved recovery kit and no successful panel backup, the dashboard shows a red banner — losing the panel right now would lock you out, and this is the one warning that says so.
It clears as soon as either leg is in place: a saved recovery kit or a successful panel self-backup.
Layer 2 — Restore the panel from a backup
The cleanest recovery is to bring the panel's database back. With a panel self-backup armed (see Backups), you restore it into a fresh VM at the same hostname and the still-running agents reconnect on their own — no SSH, no per-node steps. The full procedure, timed for under 30 minutes, is the migrate / restore the panel runbook.
The recovery kit is your out-of-band source for the boot secrets that runbook needs.
Layer 3 — Claim agents calling home (no SSH)
If you restore into a blank panel — or stand a new one up from just the recovery kit's secrets — the agents on your nodes keep dialing home. A panel that doesn't recognise them does not drop them: it quarantines them, holding the connection open with no control over the box, and lists them for you to review.
Review. Each quarantined agent shows the facts it reported — hostname, addresses, how many instances it's running — so you can confirm it's yours before acting.
Claim. Claiming back-fills the node's record and re-uses the token the agent is already presenting, so its very next connection authenticates — with no change on the box and no SSH.
Re-adopt its workloads. Once the node is managed again, run Re-adopt to reclaim the instances and sites still running inside it.
Why this is safe
A quarantined agent can do nothing — the panel never sends it a command and it never counts as online until you claim it. Quarantine is rate-limited per source address, and claiming is an explicit, audited action. A random host on the internet can neither flood the list nor gain any trust.
Layer 4 — Break-glass into a hardened node
If you have no backup, no kit, and the agent can't reach the panel, the only trust root left is your
hosting provider. Boot the server into the provider's rescue mode (or use IPMI/KVM), mount the root
disk, and either re-enable password SSH or add a fresh key — then re-enroll and
re-adopt. Step-by-step, per-provider instructions are in the
break-glass runbook shipped with the source
(runbooks/break-glass-node.md).
This is the slow, manual path — the layers above exist so you never need it.
In one table
| You have… | Recover by… | SSH needed? |
|---|---|---|
A panel backup + repo password + (kit or MASTER_KEY) | Restore the panel — agents reconnect unattended | No |
| No backup yet, but the recovery kit | Stand up a panel from the kit's secrets, then claim the calling-home agents | No |
| Neither backup nor kit | Provider rescue/KVM into each box, then re-enroll + re-adopt | Yes |
The takeaway: generate the recovery kit at setup and arm a panel backup. With either in place, a panel loss is an unattended reconnect — never a lock-out.
Backups
Back up your sites and the panel itself with restic — deduplicated, encrypted, append-only — on a schedule or on demand, with a policy that cascades from the fleet down to a single app and a safety floor that won't delete your last good backup.
Storage nodes
Provision a backup-only box you own — it runs restic's rest-server in append-only mode, so a compromised site can add backups but never delete its own history. One dependency, enrolled like any other node.