Disaster recovery
Bring a lost or rebuilt server back with the configuration backup, the recovery key, account backups on S3 and the zopanel ctl dr-restore command.
Account backups bring back websites, databases and mail. To rebuild a whole server you also need what account backups do not hold: the panel's settings, packages, administrator and reseller logins and its keys. ZoPanel keeps these in an encrypted configuration backup (.zpb). This page explains what is kept where, and the steps for each kind of failure.
Test the "server lost" procedure on a spare VPS at least once a quarter.
What is kept, and where
| What | Where | How long |
|---|---|---|
| Panel database (accounts, sites, DNS, mail, settings) | /var/lib/zopanel/db-backups/zopanel-YYYYMMDD.db |
7 days |
| Panel database before each update | /var/lib/zopanel/pre-update/ |
Last 3 updates |
Configuration backup (.zpb) |
S3, under <prefix>/<hostname>/_panel/ (daily, when remote backups are on), or downloaded by hand |
As many as account backups |
| Account backups (files, databases, mailboxes) | ~/backups on the server, and S3 when remote backups are on |
Settings → General and Settings → Remote backups |
| Incremental backups (restic) | Local repository or S3 | Incremental backup retention |
| Deleted accounts | /home/.zp-terminated/<user>-<time>/ and /var/vmail/.zp-terminated/<user>-<time>/ |
Keep deleted accounts (days), 7 by default |
The configuration backup
A configuration backup contains:
- the settings: S3, SMTP relay, notifications and so on,
- packages, administrator and reseller logins, and external database servers,
- a full copy of the panel database,
- all of
/etc/zopanel, including the restic repository password, DKIM and DNS keys, the ACME account and app environments.
Machine-specific settings, such as the panel domain and fleet tokens, are not carried over.
The file is encrypted with XChaCha20-Poly1305 using the recovery key. With remote backups on, a new copy is uploaded to S3 every day. You can also download one at any time from Backups → Disaster recovery → Download configuration (.zpb).
The recovery key
The recovery key opens the configuration backup and every encrypted copy on S3. To see it, click Backups → Disaster recovery → Show key. The same card shows when the configuration was last pushed to S3.
Keep the key outside the server, in a password manager or on paper in a safe. If the server is lost and you do not have the key, the backups on S3 cannot be decrypted, and nobody can recover them for you.
First checks: the panel does not start
Before you rebuild anything, check whether the server can be repaired:
zopanel ctl doctor # services, agent, database, disks, memory, certificate, clock
journalctl -u zopanel -u zopanel-agent -n 200
zopanel ctl support-bundle # logs and configuration for support, without passwords or keys
A bad update
Updates roll back by themselves, both the binary and the database, when the new version is not healthy within 3 minutes. To roll back by hand later:
zopanel rollback # previous version, current database
zopanel rollback --db # previous version and the database saved before the update
Damaged panel database
Restore one of the daily copies:
systemctl stop zopanel
cp /var/lib/zopanel/db-backups/zopanel-<day>.db /var/lib/zopanel/zopanel.db
rm -f /var/lib/zopanel/zopanel.db-wal /var/lib/zopanel/zopanel.db-shm
chown zopanel:zopanel /var/lib/zopanel/zopanel.db
systemctl start zopanel
Any changes made since that day, such as new accounts or settings, have to be made again.
Server lost: restore on a new server
You need the S3 settings, the recovery key and a new server running the same OS family.
- Install ZoPanel on the new server with
install.sh. Use the same hostname if you can. - Connect the same storage. In Settings → Remote backups, enter the same endpoint, bucket, folder prefix and keys as on the old server, then Save.
- Restore the configuration. Open Backups → Disaster recovery → Browse S3, open the old server's folder (its hostname), then
_panel, and click Restore configuration on the newest.zpbfile. Enter the recovery key. You can also upload a downloaded.zpbfile with Restore configuration.- Settings are overwritten. Packages, administrator and reseller logins, and external database servers are added unless one with the same name already exists.
- The old server's recovery key and incremental backup password come back, so its backups can be read on this server.
- Restore the accounts. In Browse S3, open the old server's folder, then each account, and click Restore on its newest backup. Missing accounts are created as they were, with their websites, databases, mailboxes and DNS.
- Re-issue the panel certificate. Set the panel domain under Settings → General → Panel domain & SSL.
- Switch DNS. Point the domains, or the old IP address, to the new server. Website certificates are issued again automatically once the domains resolve to it.
Same server rebuilt (OS reinstalled, /home kept)
When the operating system was reinstalled but the disk that holds /home survived, restore the whole panel state from the command line instead:
# after installing ZoPanel again
zopanel ctl dr-restore zopanel-config-<date>.zpb --key <RECOVERY-KEY> --dry-run # check the file
zopanel ctl dr-restore zopanel-config-<date>.zpb --key <RECOVERY-KEY>
zopanel ctl rebuild
--dry-runonly decrypts and checks the file. It shows the server it came from, the ZoPanel version and the date.- The key can also be passed in the
ZOPANEL_RECOVERY_KEYenvironment variable, which keeps it out of your shell history. dr-restorestops ZoPanel, replaces the whole panel database and/etc/zopanel, and starts ZoPanel again. The previous files are kept as*.before-restore-<time>.zopanel ctl rebuildhas the panel apply every account and website again (nginx, PHP, limits) from the restored database.
If some accounts' data was not on the surviving disk, restore those accounts from S3 as in step 4 above.
Account deleted by mistake
Deleted accounts are kept for the number of days set in Settings → General → Keep deleted accounts (days): 7 by default, at most 90, and 0 deletes them at once. During that time the data is still on the server:
ls /home/.zp-terminated/ # <user>-<time>/home, databases/*.sql.gz, info.json
ls /var/vmail/.zp-terminated/ # mailboxes
To recover:
- Create the account again with the same username and package.
- Copy the website files back, working as the account.
- Import the database dumps from Databases → Import / restore.
- Create the mailboxes again, copy the Maildir folders back and run
chown -R vmail:vmailon them.
After a security incident
- Check Security Center: activity log integrity, malware and 2FA.
- Send the activity log to a remote syslog server (Settings → General → Send the activity log to), so an intruder cannot erase the record.