inference-bot
2eb88f55b4
Audit of the operator docs against live reality (2026-11-04): - app-inventory.md: all 19 Cloudron apps with id, fqdn and registry image pin. Corrects the portal path (code/member-portal, not co-op/member-portal) and the panel path (local/coop.inference.admin-panel, not code/admin-panel). - maintenance/backups.md: the two backup systems; the deliberate per-user restic cache split and the --group-by "" retention fix, with why. - maintenance/woodpecker-ci.md: CI server + agent, image build/pin, manual-only trigger, git-lfs requirement, and its current not-yet-green state (agent's job-collection long-poll gets HTTP 500 from the server). - provider-pricing/: index + GreenPT, Tinfoil (local EHBP proxy architecture) and the Metis /v1/systemone decision-model route; regolo.md updated to record the gateway prices were applied. - agent-fleet.md: steward, infrastructure-bot and socialbot (Unix users, gateway units, teams); member-bot tier 2 planned but not yet provisioned. - README: file list; operator-token-runbook: correct the membership_sweep cadence (3x daily, not nightly); migration doc: post-migration addendum (16->19 apps, DNS split resolved, old box still on with apps stopped).
138 lines
7.9 KiB
Markdown
138 lines
7.9 KiB
Markdown
# Backups
|
|
|
|
There are **two independent backup systems**, on two different machines, with two
|
|
different targets and two different passphrases. Confusing them is the most likely
|
|
mistake — they do not back each other up, and neither one covers the other's host.
|
|
|
|
| | Agent-host backup | Cloudron app backup |
|
|
|---|---|---|
|
|
| What | The agent host's state (all agent homes, `~/.hermes`, `~/.ssh`, `~/repos`, gateway unit files) | Every Cloudron app's `/app/data` + addon databases, plus platform config |
|
|
| Machine | Agent host `ov-0f018e` (`179.237.103.165`) | The Cloudron server `179.237.86.116` |
|
|
| Tool | `restic` (client-side encrypted, run nightly by a systemd timer) | Cloudron's built-in backup (dashboard-configured) |
|
|
| Target | Infomaniak **Swiss Backup** (S3-compatible) | Infomaniak **Swiss Backup** (S3-compatible, `s3-v4-compat`) |
|
|
| Schedule | 04:30 UTC + up to 5 min jitter (`agent-backup.timer`) | daily 23:00 UTC (`00 00 23 * * *`) |
|
|
| Retention | 7 daily / 4 weekly / 3 monthly, no grouping | 7 days (`keepWithinSecs: 604800`) |
|
|
| Config lives | `~/.hermes/scripts/backup.sh`, unit at `/etc/systemd/system/agent-backup.*` | Cloudron dashboard (no write API — see below) |
|
|
| Last verified | 2026-11-04 (13 snapshots in the repo) | 2026-11-02 (runs 2026-10-30 → 11-03 all `normal`, ~5.4 GB) |
|
|
|
|
Nothing on the Cloudron **host OS** is backed up by either system (netplan, sshd,
|
|
`authorized_keys`, hand-edited `/etc`) — build the box, do not expect to restore
|
|
it. This was the migration's central caveat and it still holds.
|
|
|
|
---
|
|
|
|
## 1. Agent-host backup (restic)
|
|
|
|
`agent-backup.timer` → `agent-backup.service` → `~/.hermes/scripts/backup.sh`.
|
|
|
|
- **The unit runs as `User=root` on purpose.** The agent homes under `/home/*`
|
|
are mode `0700`, so only root can enumerate them; `ubuntu` sees `EACCES` and a
|
|
non-root `[ -d .../.hermes ]` test reports **false for a home that exists** —
|
|
which would silently skip that agent. The script therefore discovers agents by
|
|
enumerating `/home/*` **as root** and stages each one; a new agent is covered
|
|
without editing any list. Coverage is still unproven until its files show up in
|
|
a snapshot.
|
|
- **It backs up:** `/home/ubuntu/.hermes` (config, `.env`, skills, cron, memory,
|
|
logs, state DBs), `/home/ubuntu/.ssh`, `/home/ubuntu/repos`, every other
|
|
agent's `/home/<agent>/.hermes`, and `/etc/systemd/system/hermes-gateway*.service`.
|
|
It **excludes** rebuildable bulk (`tools/`, `hermes-agent/`, `installs/`,
|
|
`cache/`, `lsp/`, model caches) and the live SQLite files — consistent copies of
|
|
those are staged first with `sqlite3 .backup`.
|
|
- **Failure is visible, not silent.** On failure the script appends a note to
|
|
`~/.hermes/cron/updates_notes.jsonl`, which the daily stewarding digest reads.
|
|
Exit codes: 0 ok, 1 config, 2 backup, 3 prune.
|
|
|
|
### Deliberate-but-surprising: the PER-USER restic cache split
|
|
|
|
`RESTIC_CACHE_DIR` defaults to **`/var/cache/restic`** (root's run), while manual
|
|
runs by `ubuntu` use **`~/.cache/restic-manual`**. This is intentional — **do not
|
|
"tidy" it back into one shared `~/.cache/restic`.**
|
|
|
|
A shared cache directory let each user create files the other could not read;
|
|
restic then **retried every snapshot with exponential backoff** (30s, 1m25s, …)
|
|
until the run crawled or timed out. The visible symptom (2026-11-03) was a prune
|
|
that "failed" **three nights running** — which looked exactly like retention being
|
|
broken, when it was a permissions/retry storm. Splitting the cache per user ended
|
|
it. The lock and log also live **under `$HOME/.hermes/`** (not `/tmp`): with
|
|
`fs.protected_regular`, a root run is denied a sticky-dir file owned by another
|
|
user (`EACCES`), and under `set -e` that killed the root run *silently*.
|
|
|
|
### Deliberate-but-surprising: `--group-by ""` on retention
|
|
|
|
Retention is:
|
|
|
|
```
|
|
restic forget --tag nightly --group-by "" \
|
|
--keep-daily 7 --keep-weekly 4 --keep-monthly 3 --prune
|
|
```
|
|
|
|
**`--group-by ""` (no grouping) is required, not cosmetic.** restic's default is
|
|
`--group-by host,paths`, which splits the nightly snapshots into one group per
|
|
distinct **paths set**. The backup glob `/etc/systemd/system/hermes-gateway*.service`
|
|
grew from one unit to three when the infrabot and socialbot gateways were added
|
|
(2026-10-30), so the paths set changed mid-history: snapshots before 2026-10-30
|
|
formed one group and snapshots after formed another. Each group then sat **under**
|
|
the keep-7-daily limit *on its own*, so **nothing was ever expired and the repo
|
|
could never shrink**. Empty grouping applies one policy across all nightly
|
|
snapshots. `--group-by host` would also collapse today's groups, but the box has
|
|
already migrated once and a hostname change would reintroduce the same split — no
|
|
grouping is immune to both path-set drift and a hostname change.
|
|
|
|
**Current state (honest):** the repo still holds **13 snapshots** — the pre-fix
|
|
accumulation. The script on disk carries the fix, and a `--dry-run` (2026-11-04)
|
|
confirms it **keeps 8 and removes 5**. The next timer run (2026-11-05 04:31 UTC)
|
|
applies it. Until it runs, treat "13 snapshots" as expected, not as the fix
|
|
having failed.
|
|
|
|
### Other mechanics
|
|
|
|
- Weekly `restic check` on Sundays (repo metadata); full data check monthly.
|
|
- The passphrase (`RESTIC_PASSWORD`) lives in `~/.hermes/.env` **and** the co-op
|
|
Vault. A backup whose key exists only on the backed-up box is not a backup.
|
|
- S3 credentials come from `SWB_S3_ACCESS_KEY` / `SWB_S3_SECRET_KEY` (mapped to
|
|
`AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` in the script).
|
|
|
|
## 2. Cloudron app backup
|
|
|
|
- Configured in the **dashboard** (*Settings → Backups*). There is **no write
|
|
API**: `GET /api/v1/backup_sites` works, but `PUT`/`POST` on it and
|
|
`POST /backups` all 404. Schedule and retention are dashboard-only.
|
|
- **Two sites exist:** the live one (`s3-v4-compat`, daily 23:00, 7-day
|
|
retention, id `03cfbdca-88fc-4ccc-97c3-7e1b58cf892d`) and an inactive
|
|
`filesystem` site with schedule `never`. Do not be alarmed by the second one.
|
|
- **A run reads `state=normal` even when it failed.** The failure's real error
|
|
text is in `GET /api/v1/tasks` (task id `backup_<siteId>`, field `error.message`),
|
|
not in the run list. `/events` needs a `?since=` range and `/system/logs` does
|
|
not exist — a 404 there is not "no failures".
|
|
- `/backup_sites` carries `schedule` as a 6-field cron and `retention.keepWithinSecs`.
|
|
Check retention **before** claiming it needs changing.
|
|
- **Quota math:** a failure whose upload pushed the retained set over the
|
|
provider ceiling is the signature. The old box's `EntityTooLarge` failures came
|
|
from a 4.82 GB run twice daily against the ceiling; the ceiling was raised and
|
|
the schedule moved to once daily. Per-run size still grows (mostly registry
|
|
image blobs) — watch the number.
|
|
- **Two boxes on one S3 bucket interleave and prune each other's backups.** The
|
|
old box's schedule must stay **disabled** while it is kept for rollback.
|
|
- **Cross-check `siteId` per run:** runs on an old site id are pre-migration
|
|
leftovers, kept `state=normal` forever and never pruned by the current site's
|
|
retention.
|
|
|
|
## Restoring
|
|
|
|
- **Cloudron** restores apps per-app, from the **newest** box-backup entry:
|
|
`GET /apps/:id/backups` → pick by `creationTime` → `POST /apps/:id/restore`
|
|
with `{"backupId": ...}`. Restore apps **sequentially** (parallel restores
|
|
exhaust disk). `repair` recreates a container but restores **no data** — a
|
|
repaired app comes up empty.
|
|
- Full procedure, triage and the post-restore checker: the `cloudron-ops` skill
|
|
and `migration/cloudron-server-migration.md`.
|
|
- **Agent host:** `restic restore latest --target /` (or a specific snapshot) with
|
|
`RESTIC_PASSWORD` and the S3 keys in the environment, run as root.
|
|
|
|
## The credential that carries the backups
|
|
|
|
The agent-host `RESTIC_PASSWORD` and the Cloudron backup passphrase are **two
|
|
different secrets**, both held by the GM and recorded in the token runbook's
|
|
inventory. The Cloudron backup passphrase + S3 `secretAccessKey` are the
|
|
chicken-and-egg pair a bare-metal rebuild needs first — they must be held off-box.
|