R1Soft SBM disaster recovery: rebuild the server and reattach disk safes

Every hosting operator who runs R1Soft eventually asks the awkward second-order question: what happens when the Server Backup Manager itself dies. The agents are fine, the disk safes are sitting intact on the storage volume, but the Tomcat process that knew which safe belonged to which customer is gone — and with it the Derby database that held every policy, schedule, user, and recovery-point manifest reference.

This guide rebuilds an SBM from scratch and re-binds it to existing disk safes so you keep the recovery point history you have already paid to store. It assumes the disk safes survived (they live on a separate volume — see SBM sizing for why that matters) and that the original SBM host is gone or unrecoverable. End-to-end on a fleet of 30–50 agents: about three hours, most of it spent waiting for agent heartbeats.

What survives, what doesn't

The SBM stores two distinct kinds of state, in two distinct places, and the failure mode depends on which one you lost.

• Disk safes — the actual backup data — live under /var/lib/r1soft-volumes/<volume-id>/<disk-safe-id>/ on whatever storage volume you registered. Each safe is a self-describing directory with DiskSafe.cdp metadata, the BlockStore, the ChainStore, and the per-recovery-point manifests. As long as this directory tree survives byte-for-byte, every recovery point you ever took is still restorable.
• SBM configuration — policies, schedules, agents, users, retention rules, license activation, SSL keys — lives in the Apache Derby database under /usr/sbin/r1soft/data/ (path varies slightly by version; on 6.16+ it's typically /usr/sbin/r1soft/data/cdp.dbs/). This is what you lose when the SBM root disk dies.

If you have a recent dump of the Derby database, recovery is mostly mechanical: install SBM, restore the database, point it at the volume, done. If you don't, you can still rescue every disk safe — you just have to recreate the policies and re-register the agents by hand.

Step 1 — Salvage the storage volume

Before touching software, make the disk safes visible to the new host. The cleanest path is to detach the volume from the dead host and mount it on the rebuild host without modification.

If the safes lived on a SAN LUN, iSCSI target, or external JBOD, re-present the LUN to the new host. If they were on local disks, physically move the disks. In either case, mount the volume read-only first and verify the directory tree is intact:

mount -o ro /dev/sdX1 /mnt/r1soft-recover
ls /mnt/r1soft-recover/r1soft-volumes/

You should see one subdirectory per volume ID, and inside each, one subdirectory per disk safe ID. Spot-check a DiskSafe.cdp file with file — it should report as a binary data file, not zero bytes. Remount read-write only once you have confirmed the layout.

Step 2 — Install a fresh SBM at the same version

Match the SBM version exactly. A safe written by 6.18 will not always re-attach cleanly on a 6.16 rebuild — Derby schema migrations are one-way. Find the version on your monitoring or licensing portal records; if you don't know it, install the latest 6.x and accept that some older safes may need a schema fix-up on first attach.

Provision a host that meets the sizing baseline for the agent count you expect to re-register. Install SBM using the official repository:

wget http://repo.r1soft.com/r1soft.sh -O /tmp/r1soft.sh
sh /tmp/r1soft.sh
yum install -y serverbackup-enterprise
serverbackup-setup --user admin --pass <new-admin-password>
systemctl enable --now cdp-server

Confirm the web UI comes up on https://<host>:8443/ with the default empty database before you go any further.

Step 3 — Restore the Derby database (if you have a backup)

If you have a Derby snapshot from before the loss — and you should, see the next section on prevention — restore it now. Stop the SBM, replace the data directory, then start the service:

systemctl stop cdp-server
rm -rf /usr/sbin/r1soft/data/cdp.dbs
tar -xzf /backup/cdp.dbs.tar.gz -C /usr/sbin/r1soft/data/
chown -R cdp:cdp /usr/sbin/r1soft/data/cdp.dbs
systemctl start cdp-server

When the SBM comes back, log in with the credentials from the restored database — not the temporary admin you set in step 2, which is now overwritten. Policies, users, agents, and retention rules are all back. Skip to step 6 to reconnect the volume.

Step 4 — Register the storage volume (no Derby backup path)

If you did not have a Derby backup, mount the salvaged volume at its original path so existing volume-relative metadata still resolves. The original mount was almost certainly /var/lib/r1soft-volumes — keep it there:

umount /mnt/r1soft-recover
mkdir -p /var/lib/r1soft-volumes
mount /dev/sdX1 /var/lib/r1soft-volumes
chown -R cdp:cdp /var/lib/r1soft-volumes

Add the matching entry to /etc/fstab before you forget. In the SBM UI, go to Configuration → Volumes → New Volume, name it whatever you want, and point it at the existing /var/lib/r1soft-volumes/<volume-id> directory. SBM will accept the path and warn that disk safes exist; that is exactly what you want.

Step 5 — Re-attach orphaned disk safes

For each disk safe directory under the volume, run Disk Safes → Add Existing Disk Safe and point it at the directory. SBM reads DiskSafe.cdp, populates a new row in the Derby database, and rebuilds the recovery point list by walking the manifests on disk. On a 500 GB safe with two years of hourly recovery points this takes 2–10 minutes.

For more than a handful of safes, script it via the SBM REST API rather than clicking through the UI:

for safe in /var/lib/r1soft-volumes/<volume-id>/*/; do
  safe_id=$(basename "$safe")
  curl -k -u admin:<password> \
    -X POST "https://localhost:8443/api/disk-safes/attach" \
    -H "Content-Type: application/json" \
    -d "{\"volumeId\":\"<volume-id>\",\"path\":\"$safe_id\"}"
done

Each safe comes back without an assigned agent. You'll bind it in step 6.

Step 6 — Reconnect agents

The agents are still alive on the customer servers, still listening on TCP/1167, and still holding the same agent key in /etc/r1soft-server-backup-agent/conf/. From the SBM, Agents → Add Agent, supply the hostname or IP and the agent key, and the agent reconnects in seconds. Bind the previously-orphaned disk safe to that agent on the agent's Disk Safes tab.

For a fleet, pull the agent list from your CMDB or Blesta installation and loop with the REST API. Skipping the UI also avoids the 30-second per-add wait the JS frontend imposes.

Once the agent is bound and you click Force a Backup, the next recovery point appends to the existing chain — no full re-baseline, no extra storage cost. That is the whole reason you went to the trouble of preserving the safes.

Prevent the next outage

Two cron jobs would have saved most of step 4:

# /etc/cron.d/cdp-derby-backup
0 3 * * * cdp /usr/sbin/r1soft/scripts/cdp-dump-db.sh /backup/cdp-$(date +\%F).dbs.tar.gz
0 4 * * * cdp rsync -a /backup/cdp-*.dbs.tar.gz backup@dr-host:/srv/r1soft-derby-backups/

The first dumps Derby with the database online. The second ships the dump off-host. Combine with Disk Safe Replication for the data side and you have a complete DR posture: replicated safes for the bytes, exported Derby dumps for the metadata, and a runbook for stitching them back together.

Next steps

• Set up Disk Safe Replication so the safes survive a datacenter event, not just an SBM rebuild
• Migrate disk safes to a new volume to move recovered safes onto larger storage post-recovery
• Size your rebuilt SBM correctly before re-onboarding the full fleet