ops-docs/maintenance/backups.md

Backups

There are two independent backup systems, on two different machines, with two different targets and two different passphrases. Confusing them is the most likely mistake — they do not back each other up, and neither one covers the other's host.

	Agent-host backup	Cloudron app backup
What	The agent host's state (all agent homes, ~/.hermes, ~/.ssh, ~/repos, gateway unit files)	Every Cloudron app's /app/data + addon databases, plus platform config
Machine	Agent host ov-0f018e (179.237.103.165)	The Cloudron server 179.237.86.116
Tool	restic (client-side encrypted, run nightly by a systemd timer)	Cloudron's built-in backup (dashboard-configured)
Target	Infomaniak Swiss Backup (S3-compatible)	Infomaniak Swiss Backup (S3-compatible, s3-v4-compat)
Schedule	04:30 UTC + up to 5 min jitter (agent-backup.timer)	daily 23:00 UTC (00 00 23 * * *)
Retention	7 daily / 4 weekly / 3 monthly, no grouping	7 days (keepWithinSecs: 604800)
Config lives	~/.hermes/scripts/backup.sh, unit at /etc/systemd/system/agent-backup.*	Cloudron dashboard (no write API — see below)
Last verified	2026-11-04 (13 snapshots in the repo)	2026-11-02 (runs 2026-10-30 → 11-03 all normal, ~5.4 GB)

Nothing on the Cloudron host OS is backed up by either system (netplan, sshd, authorized_keys, hand-edited /etc) — build the box, do not expect to restore it. This was the migration's central caveat and it still holds.

---

1. Agent-host backup (restic)

agent-backup.timer → agent-backup.service → ~/.hermes/scripts/backup.sh.

• The unit runs as User=root on purpose. The agent homes under /home/* are mode 0700, so only root can enumerate them; ubuntu sees EACCES and a non-root [ -d .../.hermes ] test reports false for a home that exists — which would silently skip that agent. The script therefore discovers agents by enumerating /home/* as root and stages each one; a new agent is covered without editing any list. Coverage is still unproven until its files show up in a snapshot.
• It backs up: /home/ubuntu/.hermes (config, .env, skills, cron, memory, logs, state DBs), /home/ubuntu/.ssh, /home/ubuntu/repos, every other agent's /home/<agent>/.hermes, and /etc/systemd/system/hermes-gateway*.service. It excludes rebuildable bulk (tools/, hermes-agent/, installs/, cache/, lsp/, model caches) and the live SQLite files — consistent copies of those are staged first with sqlite3 .backup.
• Failure is visible, not silent. On failure the script appends a note to ~/.hermes/cron/updates_notes.jsonl, which the daily stewarding digest reads. Exit codes: 0 ok, 1 config, 2 backup, 3 prune.

Deliberate-but-surprising: the PER-USER restic cache split

RESTIC_CACHE_DIR defaults to /var/cache/restic (root's run), while manual runs by ubuntu use ~/.cache/restic-manual. This is intentional — do not "tidy" it back into one shared ~/.cache/restic.

A shared cache directory let each user create files the other could not read; restic then retried every snapshot with exponential backoff (30s, 1m25s, …) until the run crawled or timed out. The visible symptom (2026-11-03) was a prune that "failed" three nights running — which looked exactly like retention being broken, when it was a permissions/retry storm. Splitting the cache per user ended it. The lock and log also live under $HOME/.hermes/ (not /tmp): with fs.protected_regular, a root run is denied a sticky-dir file owned by another user (EACCES), and under set -e that killed the root run silently.

Deliberate-but-surprising: --group-by "" on retention

Retention is:

restic forget --tag nightly --group-by "" \
  --keep-daily 7 --keep-weekly 4 --keep-monthly 3 --prune

--group-by "" (no grouping) is required, not cosmetic. restic's default is --group-by host,paths, which splits the nightly snapshots into one group per distinct paths set. The backup glob /etc/systemd/system/hermes-gateway*.service grew from one unit to three when the infrabot and socialbot gateways were added (2026-10-30), so the paths set changed mid-history: snapshots before 2026-10-30 formed one group and snapshots after formed another. Each group then sat under the keep-7-daily limit on its own, so nothing was ever expired and the repo could never shrink. Empty grouping applies one policy across all nightly snapshots. --group-by host would also collapse today's groups, but the box has already migrated once and a hostname change would reintroduce the same split — no grouping is immune to both path-set drift and a hostname change.

Current state (honest): the repo still holds 13 snapshots — the pre-fix accumulation. The script on disk carries the fix, and a --dry-run (2026-11-04) confirms it keeps 8 and removes 5. The next timer run (2026-11-05 04:31 UTC) applies it. Until it runs, treat "13 snapshots" as expected, not as the fix having failed.

Other mechanics

• Weekly restic check on Sundays (repo metadata); full data check monthly.
• The passphrase (RESTIC_PASSWORD) lives in ~/.hermes/.env and the co-op Vault. A backup whose key exists only on the backed-up box is not a backup.
• S3 credentials come from SWB_S3_ACCESS_KEY / SWB_S3_SECRET_KEY (mapped to AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY in the script).

2. Cloudron app backup

• Configured in the dashboard (Settings → Backups). There is no write API: GET /api/v1/backup_sites works, but PUT/POST on it and POST /backups all 404. Schedule and retention are dashboard-only.
• Two sites exist: the live one (s3-v4-compat, daily 23:00, 7-day retention, id 03cfbdca-88fc-4ccc-97c3-7e1b58cf892d) and an inactive filesystem site with schedule never. Do not be alarmed by the second one.
• A run reads state=normal even when it failed. The failure's real error text is in GET /api/v1/tasks (task id backup_<siteId>, field error.message), not in the run list. /events needs a ?since= range and /system/logs does not exist — a 404 there is not "no failures".
• /backup_sites carries schedule as a 6-field cron and retention.keepWithinSecs. Check retention before claiming it needs changing.
• Quota math: a failure whose upload pushed the retained set over the provider ceiling is the signature. The old box's EntityTooLarge failures came from a 4.82 GB run twice daily against the ceiling; the ceiling was raised and the schedule moved to once daily. Per-run size still grows (mostly registry image blobs) — watch the number.
• Two boxes on one S3 bucket interleave and prune each other's backups. The old box's schedule must stay disabled while it is kept for rollback.
• Cross-check siteId per run: runs on an old site id are pre-migration leftovers, kept state=normal forever and never pruned by the current site's retention.

Restoring

• Cloudron restores apps per-app, from the newest box-backup entry: GET /apps/:id/backups → pick by creationTime → POST /apps/:id/restore with {"backupId": ...}. Restore apps sequentially (parallel restores exhaust disk). repair recreates a container but restores no data — a repaired app comes up empty.
• Full procedure, triage and the post-restore checker: the cloudron-ops skill and migration/cloudron-server-migration.md.
• Agent host: restic restore latest --target / (or a specific snapshot) with RESTIC_PASSWORD and the S3 keys in the environment, run as root.

The credential that carries the backups

The agent-host RESTIC_PASSWORD and the Cloudron backup passphrase are two different secrets, both held by the GM and recorded in the token runbook's inventory. The Cloudron backup passphrase + S3 secretAccessKey are the chicken-and-egg pair a bare-metal rebuild needs first — they must be held off-box.