# Backups There are **two independent backup systems**, on two different machines, with two different targets and two different passphrases. Confusing them is the most likely mistake — they do not back each other up, and neither one covers the other's host. | | Agent-host backup | Cloudron app backup | |---|---|---| | What | The agent host's state (all agent homes, `~/.hermes`, `~/.ssh`, `~/repos`, gateway unit files) | Every Cloudron app's `/app/data` + addon databases, plus platform config | | Machine | Agent host `ov-0f018e` (`179.237.103.165`) | The Cloudron server `179.237.86.116` | | Tool | `restic` (client-side encrypted, run nightly by a systemd timer) | Cloudron's built-in backup (dashboard-configured) | | Target | Infomaniak **Swiss Backup** (S3-compatible) | Infomaniak **Swiss Backup** (S3-compatible, `s3-v4-compat`) | | Schedule | 04:30 UTC + up to 5 min jitter (`agent-backup.timer`) | daily 23:00 UTC (`00 00 23 * * *`) | | Retention | 7 daily / 4 weekly / 3 monthly, no grouping | 7 days (`keepWithinSecs: 604800`) | | Config lives | `~/.hermes/scripts/backup.sh`, unit at `/etc/systemd/system/agent-backup.*` | Cloudron dashboard (no write API — see below) | | Last verified | 2026-11-04 (13 snapshots in the repo) | 2026-11-02 (runs 2026-10-30 → 11-03 all `normal`, ~5.4 GB) | Nothing on the Cloudron **host OS** is backed up by either system (netplan, sshd, `authorized_keys`, hand-edited `/etc`) — build the box, do not expect to restore it. This was the migration's central caveat and it still holds. --- ## 1. Agent-host backup (restic) `agent-backup.timer` → `agent-backup.service` → `~/.hermes/scripts/backup.sh`. - **The unit runs as `User=root` on purpose.** The agent homes under `/home/*` are mode `0700`, so only root can enumerate them; `ubuntu` sees `EACCES` and a non-root `[ -d .../.hermes ]` test reports **false for a home that exists** — which would silently skip that agent. The script therefore discovers agents by enumerating `/home/*` **as root** and stages each one; a new agent is covered without editing any list. Coverage is still unproven until its files show up in a snapshot. - **It backs up:** `/home/ubuntu/.hermes` (config, `.env`, skills, cron, memory, logs, state DBs), `/home/ubuntu/.ssh`, `/home/ubuntu/repos`, every other agent's `/home//.hermes`, and `/etc/systemd/system/hermes-gateway*.service`. It **excludes** rebuildable bulk (`tools/`, `hermes-agent/`, `installs/`, `cache/`, `lsp/`, model caches) and the live SQLite files — consistent copies of those are staged first with `sqlite3 .backup`. - **Failure is visible, not silent.** On failure the script appends a note to `~/.hermes/cron/updates_notes.jsonl`, which the daily stewarding digest reads. Exit codes: 0 ok, 1 config, 2 backup, 3 prune. ### Deliberate-but-surprising: the PER-USER restic cache split `RESTIC_CACHE_DIR` defaults to **`/var/cache/restic`** (root's run), while manual runs by `ubuntu` use **`~/.cache/restic-manual`**. This is intentional — **do not "tidy" it back into one shared `~/.cache/restic`.** A shared cache directory let each user create files the other could not read; restic then **retried every snapshot with exponential backoff** (30s, 1m25s, …) until the run crawled or timed out. The visible symptom (2026-11-03) was a prune that "failed" **three nights running** — which looked exactly like retention being broken, when it was a permissions/retry storm. Splitting the cache per user ended it. The lock and log also live **under `$HOME/.hermes/`** (not `/tmp`): with `fs.protected_regular`, a root run is denied a sticky-dir file owned by another user (`EACCES`), and under `set -e` that killed the root run *silently*. ### Deliberate-but-surprising: `--group-by ""` on retention Retention is: ``` restic forget --tag nightly --group-by "" \ --keep-daily 7 --keep-weekly 4 --keep-monthly 3 --prune ``` **`--group-by ""` (no grouping) is required, not cosmetic.** restic's default is `--group-by host,paths`, which splits the nightly snapshots into one group per distinct **paths set**. The backup glob `/etc/systemd/system/hermes-gateway*.service` grew from one unit to three when the infrabot and socialbot gateways were added (2026-10-30), so the paths set changed mid-history: snapshots before 2026-10-30 formed one group and snapshots after formed another. Each group then sat **under** the keep-7-daily limit *on its own*, so **nothing was ever expired and the repo could never shrink**. Empty grouping applies one policy across all nightly snapshots. `--group-by host` would also collapse today's groups, but the box has already migrated once and a hostname change would reintroduce the same split — no grouping is immune to both path-set drift and a hostname change. **Current state (honest):** the repo still holds **13 snapshots** — the pre-fix accumulation. The script on disk carries the fix, and a `--dry-run` (2026-11-04) confirms it **keeps 8 and removes 5**. The next timer run (2026-11-05 04:31 UTC) applies it. Until it runs, treat "13 snapshots" as expected, not as the fix having failed. ### Other mechanics - Weekly `restic check` on Sundays (repo metadata); full data check monthly. - The passphrase (`RESTIC_PASSWORD`) lives in `~/.hermes/.env` **and** the co-op Vault. A backup whose key exists only on the backed-up box is not a backup. - S3 credentials come from `SWB_S3_ACCESS_KEY` / `SWB_S3_SECRET_KEY` (mapped to `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` in the script). ## 2. Cloudron app backup - Configured in the **dashboard** (*Settings → Backups*). There is **no write API**: `GET /api/v1/backup_sites` works, but `PUT`/`POST` on it and `POST /backups` all 404. Schedule and retention are dashboard-only. - **Two sites exist:** the live one (`s3-v4-compat`, daily 23:00, 7-day retention, id `03cfbdca-88fc-4ccc-97c3-7e1b58cf892d`) and an inactive `filesystem` site with schedule `never`. Do not be alarmed by the second one. - **A run reads `state=normal` even when it failed.** The failure's real error text is in `GET /api/v1/tasks` (task id `backup_`, field `error.message`), not in the run list. `/events` needs a `?since=` range and `/system/logs` does not exist — a 404 there is not "no failures". - `/backup_sites` carries `schedule` as a 6-field cron and `retention.keepWithinSecs`. Check retention **before** claiming it needs changing. - **Quota math:** a failure whose upload pushed the retained set over the provider ceiling is the signature. The old box's `EntityTooLarge` failures came from a 4.82 GB run twice daily against the ceiling; the ceiling was raised and the schedule moved to once daily. Per-run size still grows (mostly registry image blobs) — watch the number. - **Two boxes on one S3 bucket interleave and prune each other's backups.** The old box's schedule must stay **disabled** while it is kept for rollback. - **Cross-check `siteId` per run:** runs on an old site id are pre-migration leftovers, kept `state=normal` forever and never pruned by the current site's retention. ## Restoring - **Cloudron** restores apps per-app, from the **newest** box-backup entry: `GET /apps/:id/backups` → pick by `creationTime` → `POST /apps/:id/restore` with `{"backupId": ...}`. Restore apps **sequentially** (parallel restores exhaust disk). `repair` recreates a container but restores **no data** — a repaired app comes up empty. - Full procedure, triage and the post-restore checker: the `cloudron-ops` skill and `migration/cloudron-server-migration.md`. - **Agent host:** `restic restore latest --target /` (or a specific snapshot) with `RESTIC_PASSWORD` and the S3 keys in the environment, run as root. ## The credential that carries the backups The agent-host `RESTIC_PASSWORD` and the Cloudron backup passphrase are **two different secrets**, both held by the GM and recorded in the token runbook's inventory. The Cloudron backup passphrase + S3 `secretAccessKey` are the chicken-and-egg pair a bare-metal rebuild needs first — they must be held off-box.