Proxmox Disaster Recovery: How to Prepare for Server Failure

Disaster recovery for Proxmox is not only about having VM backups. It is about knowing what to do when a node dies, when shared storage fails, or when the entire site is unavailable, and having tested it before. This guide walks through the failure scenarios and the preparation each one requires.

Start with the failure scenarios

Different failures need different responses. Planning for "a disaster" in general usually means planning for none of them in particular.

High availability is not disaster recovery

Proxmox VE's HA manager restarts guests on surviving nodes when a node fails, as long as the guest disks are on shared or replicated storage. That reduces downtime for hardware failures, but it does nothing against deleted data, corruption replicated to every node, ransomware, or the loss of the site. HA and backups solve different problems; you need both.

Similarly, storage replication (ZFS replication between nodes or Ceph redundancy) copies mistakes as quickly as it copies good data. A DROP TABLE is replicated in seconds.

What must be backed up

Guests

Every VM and container you would need to rebuild the service, backed up with vzdump to Proxmox Backup Server or file storage. Include the guest configuration, which vzdump stores in the archive.

Application data

For databases, add application-level dumps or physical backups inside the guest. A VM backup taken while a database is writing is crash-consistent at best.

Node and cluster configuration

The cluster configuration lives in /etc/pve, a database-backed file system shared by all nodes. If you lose all nodes, you also lose it. Back up at least:

• /etc/pve (storage definitions, firewall rules, HA groups, user and permission configuration, guest configs);
• /etc/network/interfaces and any SDN configuration;
• /etc/hosts, /etc/hostname and custom files under /etc;
• ZFS pool layouts and Ceph configuration notes;
• certificates if you use custom ones, and the subscription keys you need to reinstall.

A simple scheduled archive of those paths, copied offsite, saves hours when rebuilding a node:

tar czf /root/pve-config-$(hostname)-$(date +%F).tar.gz \
  /etc/pve /etc/network/interfaces /etc/hosts /etc/hostname /etc/vzdump.conf

Proxmox Backup Server's client can also back up host directories directly to a datastore with proxmox-backup-client backup.

Encryption keys and credentials

If PBS client-side encryption is enabled, the key file is required to restore. Keep it in a password manager and a sealed offline copy. The same applies to credentials of the offsite storage.

Restoring guests on another node

Backups are not tied to the node that created them. Any node with access to the backup storage can restore them:

# On a surviving node, list backups available on the PBS storage
pvesm list pbs-main --vmid 100

# Restore VM 100 under the same ID on this node's storage
qmrestore pbs-main:backup/vm/100/2026-10-18T22:00:03Z 100 --storage local-zfs

If the original guest still exists in the cluster configuration (for example, the node is dead but the VM is still assigned to it), restore under a new VMID or remove the stale configuration first. Check bridge names and VLANs: a guest configured for vmbr1 will not start on a node where that bridge does not exist.

For large VMs, PBS live-restore starts the guest while data is still being copied, which shortens downtime. It needs good bandwidth between the node and PBS.

Recovering from shared storage failure

When Ceph or a SAN fails, most guests are affected at the same time. Restore capacity becomes the bottleneck: how many terabytes per hour can your backup server deliver, and to which storage? Prepare by:

• keeping enough local storage on nodes, or spare capacity elsewhere, to restore critical guests;
• ranking guests by priority (identity services, DNS and databases first);
• measuring restore throughput during tests and calculating how long the full restore would take.

Recovering from a site loss

This is the scenario where offsite backups matter. You need:

1. a copy of the backups outside the site (remote PBS, S3-backed datastore or cloud archives);
2. somewhere to run the guests: spare hardware, a secondary site, or rented servers where Proxmox VE can be installed;
3. the configuration backup and network documentation to recreate bridges, VLANs and IP addressing;
4. enough bandwidth or a plan to bring data physically if the dataset is large.

Restoring tens of terabytes over the internet can take days. For critical systems, consider keeping a warm copy in the secondary location (a remote PBS that receives sync jobs) so restores happen on a local network there. Our article on RPO and RTO explains how to set targets for this.

Protecting against ransomware

Attackers who gain administrator access to the cluster often try to delete backups. Reduce that risk:

• Use a dedicated PBS user or API token for Proxmox VE with the DatastoreBackup role, which allows creating backups without the Datastore.Prune privilege needed to delete snapshots. Run pruning on the PBS side instead, and confirm the privileges in the PBS documentation for your version.
• Let the offsite PBS pull from the primary, so primary credentials cannot reach the offsite copy.
• Consider immutable storage (S3 Object Lock) or offline media for at least one copy, the "1" in 3-2-1-1-0.
• Enable two-factor authentication for Proxmox VE and PBS web interfaces.

Testing the plan

A recovery plan that has never been executed is a hypothesis. Schedule tests with increasing scope:

1. Monthly: restore one guest to a new VMID on an isolated network and check the application.
2. Quarterly: restore a set of related guests (application, database, DNS) and test the full service.
3. Yearly: simulate the loss of the primary site and rebuild critical services from offsite backups on other hardware.

Document each test: what was restored, from which backup, how long each step took and what went wrong. Update the runbook after every test.

A recovery runbook template

• Contacts and escalation: who decides to declare a disaster.
• Location of backups, credentials and encryption keys.
• Restore order of guests, with dependencies.
• Network plan for the recovery site (bridges, VLANs, IPs, DNS changes).
• Steps to reinstall Proxmox VE and join or create a cluster.
• Verification checklist for each service.
• Communication plan for users and customers.

Next steps

1. Write down your failure scenarios and the maximum acceptable downtime for each service.
2. Add a configuration backup of /etc/pve and network settings.
3. Restrict PBS permissions used by Proxmox VE and set up a pull-based offsite copy.
4. Run your first isolated restore test this month.

For the offsite layer, see how to back up Proxmox virtual machines to the cloud and the Backnix Proxmox backup page.

References

• Proxmox VE: High Availability
• Proxmox VE: Proxmox Cluster File System (pmxcfs)
• Proxmox Backup Server: user management and privileges

Topics: Disaster recoveryHigh availabilityProxmox VE