Skip to content

Runbook: disaster recovery

The droplet is a deliberate single point of failure (ADR 0003). This is the plan for losing it entirely, and for the smaller disasters on the way there.

What is backed up, where

Asset Mechanism Location Retention
Every app Postgres (9 databases incl. website, portal, all demo shops, GrowCentric) /usr/local/bin/pg-backup, nightly 03:30 UTC via /etc/cron.d/pg-backup DigitalOcean Spaces keferboeck/db-backups/<host>/ 14 days
Application code and config Git GitHub (keferboeck/* repos) forever
Deploy secrets (.kamal/ files, master keys, AR encryption keys) not in Git by design Georg's laptop + password manager manual
Uploaded files (Active Storage) inside app volumes on the droplet ⚠ droplet only none
DNS DigitalOcean DNS zone DO account live

Known gaps

Active Storage uploads (portal document attachments, logos, receipts) live only on the droplet volume today; move them to Spaces-backed storage or include volumes in the nightly job. The .kamal secret files exist only on the laptop: keep the password manager copies current. Verify a restore quarterly, not just the backup log.

Scenario: one app is broken after a deploy

kamal rollback <previous-sha> from that app's repo. Two minutes, no data touched.

Scenario: one database is corrupted

  1. ssh root@161.35.42.199, fetch the wanted dump from Spaces (or /var/backups/postgres for the same-day copy).
  2. Stop the app container: kamal app stop from the app repo.
  3. zcat dump.sql.gz | docker exec -i <app>-db psql -U <user> <database> (drop and recreate the schema first for a clean restore).
  4. kamal app boot, verify, note the data loss window (max 24 h) in the incident log.

Scenario: the droplet is gone

Recovery order: DNS first (stops the bleeding), then portal (operations), then website, then shops.

  1. Create a droplet (lon1, Docker preinstalled or apt install docker.io), attach the reserved IP if one exists; otherwise update the A records for all keferboeck.com hosts in DigitalOcean DNS to the new IP (doctl compute domain records list keferboeck.com).
  2. Bootstrap Kamal: from each app repo on the laptop, kamal setup (installs kamal-proxy, boots the app and its Postgres accessory). Secrets come from the local .kamal/ files — this is why they must survive independently of the droplet.
  3. Restore databases from Spaces into each freshly created accessory (scenario above).
  4. Re-create the uptime monitor: copy /opt/uptime-monitor from this repo's documentation (or the GrowCentric docs copy) and reinstate /etc/cron.d/uptime-monitor and /etc/cron.d/pg-backup.
  5. Verify TLS: kamal-proxy re-issues Let's Encrypt certificates on first request per host once DNS resolves to the new machine.
  6. Write the incident up in incidents.

Practical RPO / RTO: up to 24 hours of data (nightly dumps), roughly half a day of rebuild for all apps. If either stops being acceptable, the upgrade path is: hourly dumps, volume backups for uploads, and DigitalOcean droplet snapshots as a warm layer.