docs: infrastructure audit — app inventory, backups, CI, provider pricing, agent fleet
Audit of the operator docs against live reality (2026-11-04): - app-inventory.md: all 19 Cloudron apps with id, fqdn and registry image pin. Corrects the portal path (code/member-portal, not co-op/member-portal) and the panel path (local/coop.inference.admin-panel, not code/admin-panel). - maintenance/backups.md: the two backup systems; the deliberate per-user restic cache split and the --group-by "" retention fix, with why. - maintenance/woodpecker-ci.md: CI server + agent, image build/pin, manual-only trigger, git-lfs requirement, and its current not-yet-green state (agent's job-collection long-poll gets HTTP 500 from the server). - provider-pricing/: index + GreenPT, Tinfoil (local EHBP proxy architecture) and the Metis /v1/systemone decision-model route; regolo.md updated to record the gateway prices were applied. - agent-fleet.md: steward, infrastructure-bot and socialbot (Unix users, gateway units, teams); member-bot tier 2 planned but not yet provisioned. - README: file list; operator-token-runbook: correct the membership_sweep cadence (3x daily, not nightly); migration doc: post-migration addendum (16->19 apps, DNS split resolved, old box still on with apps stopped).
This commit is contained in:
inference-bot committed
1 parent
8b33d823c1
commit
2eb88f55b4
12 files changed
+688
-3
No files matched your search
@@ -0,0 +1,137 @@
|
||||
# Backups
|
||||
|
||||
There are **two independent backup systems**, on two different machines, with two
|
||||
different targets and two different passphrases. Confusing them is the most likely
|
||||
mistake — they do not back each other up, and neither one covers the other's host.
|
||||
|
||||
| | Agent-host backup | Cloudron app backup |
|
||||
|---|---|---|
|
||||
| What | The agent host's state (all agent homes, `~/.hermes`, `~/.ssh`, `~/repos`, gateway unit files) | Every Cloudron app's `/app/data` + addon databases, plus platform config |
|
||||
| Machine | Agent host `ov-0f018e` (`179.237.103.165`) | The Cloudron server `179.237.86.116` |
|
||||
| Tool | `restic` (client-side encrypted, run nightly by a systemd timer) | Cloudron's built-in backup (dashboard-configured) |
|
||||
| Target | Infomaniak **Swiss Backup** (S3-compatible) | Infomaniak **Swiss Backup** (S3-compatible, `s3-v4-compat`) |
|
||||
| Schedule | 04:30 UTC + up to 5 min jitter (`agent-backup.timer`) | daily 23:00 UTC (`00 00 23 * * *`) |
|
||||
| Retention | 7 daily / 4 weekly / 3 monthly, no grouping | 7 days (`keepWithinSecs: 604800`) |
|
||||
| Config lives | `~/.hermes/scripts/backup.sh`, unit at `/etc/systemd/system/agent-backup.*` | Cloudron dashboard (no write API — see below) |
|
||||
| Last verified | 2026-11-04 (13 snapshots in the repo) | 2026-11-02 (runs 2026-10-30 → 11-03 all `normal`, ~5.4 GB) |
|
||||
|
||||
Nothing on the Cloudron **host OS** is backed up by either system (netplan, sshd,
|
||||
`authorized_keys`, hand-edited `/etc`) — build the box, do not expect to restore
|
||||
it. This was the migration's central caveat and it still holds.
|
||||
|
||||
---
|
||||
|
||||
## 1. Agent-host backup (restic)
|
||||
|
||||
`agent-backup.timer` → `agent-backup.service` → `~/.hermes/scripts/backup.sh`.
|
||||
|
||||
- **The unit runs as `User=root` on purpose.** The agent homes under `/home/*`
|
||||
are mode `0700`, so only root can enumerate them; `ubuntu` sees `EACCES` and a
|
||||
non-root `[ -d .../.hermes ]` test reports **false for a home that exists** —
|
||||
which would silently skip that agent. The script therefore discovers agents by
|
||||
enumerating `/home/*` **as root** and stages each one; a new agent is covered
|
||||
without editing any list. Coverage is still unproven until its files show up in
|
||||
a snapshot.
|
||||
- **It backs up:** `/home/ubuntu/.hermes` (config, `.env`, skills, cron, memory,
|
||||
logs, state DBs), `/home/ubuntu/.ssh`, `/home/ubuntu/repos`, every other
|
||||
agent's `/home/<agent>/.hermes`, and `/etc/systemd/system/hermes-gateway*.service`.
|
||||
It **excludes** rebuildable bulk (`tools/`, `hermes-agent/`, `installs/`,
|
||||
`cache/`, `lsp/`, model caches) and the live SQLite files — consistent copies of
|
||||
those are staged first with `sqlite3 .backup`.
|
||||
- **Failure is visible, not silent.** On failure the script appends a note to
|
||||
`~/.hermes/cron/updates_notes.jsonl`, which the daily stewarding digest reads.
|
||||
Exit codes: 0 ok, 1 config, 2 backup, 3 prune.
|
||||
|
||||
### Deliberate-but-surprising: the PER-USER restic cache split
|
||||
|
||||
`RESTIC_CACHE_DIR` defaults to **`/var/cache/restic`** (root's run), while manual
|
||||
runs by `ubuntu` use **`~/.cache/restic-manual`**. This is intentional — **do not
|
||||
"tidy" it back into one shared `~/.cache/restic`.**
|
||||
|
||||
A shared cache directory let each user create files the other could not read;
|
||||
restic then **retried every snapshot with exponential backoff** (30s, 1m25s, …)
|
||||
until the run crawled or timed out. The visible symptom (2026-11-03) was a prune
|
||||
that "failed" **three nights running** — which looked exactly like retention being
|
||||
broken, when it was a permissions/retry storm. Splitting the cache per user ended
|
||||
it. The lock and log also live **under `$HOME/.hermes/`** (not `/tmp`): with
|
||||
`fs.protected_regular`, a root run is denied a sticky-dir file owned by another
|
||||
user (`EACCES`), and under `set -e` that killed the root run *silently*.
|
||||
|
||||
### Deliberate-but-surprising: `--group-by ""` on retention
|
||||
|
||||
Retention is:
|
||||
|
||||
```
|
||||
restic forget --tag nightly --group-by "" \
|
||||
--keep-daily 7 --keep-weekly 4 --keep-monthly 3 --prune
|
||||
```
|
||||
|
||||
**`--group-by ""` (no grouping) is required, not cosmetic.** restic's default is
|
||||
`--group-by host,paths`, which splits the nightly snapshots into one group per
|
||||
distinct **paths set**. The backup glob `/etc/systemd/system/hermes-gateway*.service`
|
||||
grew from one unit to three when the infrabot and socialbot gateways were added
|
||||
(2026-10-30), so the paths set changed mid-history: snapshots before 2026-10-30
|
||||
formed one group and snapshots after formed another. Each group then sat **under**
|
||||
the keep-7-daily limit *on its own*, so **nothing was ever expired and the repo
|
||||
could never shrink**. Empty grouping applies one policy across all nightly
|
||||
snapshots. `--group-by host` would also collapse today's groups, but the box has
|
||||
already migrated once and a hostname change would reintroduce the same split — no
|
||||
grouping is immune to both path-set drift and a hostname change.
|
||||
|
||||
**Current state (honest):** the repo still holds **13 snapshots** — the pre-fix
|
||||
accumulation. The script on disk carries the fix, and a `--dry-run` (2026-11-04)
|
||||
confirms it **keeps 8 and removes 5**. The next timer run (2026-11-05 04:31 UTC)
|
||||
applies it. Until it runs, treat "13 snapshots" as expected, not as the fix
|
||||
having failed.
|
||||
|
||||
### Other mechanics
|
||||
|
||||
- Weekly `restic check` on Sundays (repo metadata); full data check monthly.
|
||||
- The passphrase (`RESTIC_PASSWORD`) lives in `~/.hermes/.env` **and** the co-op
|
||||
Vault. A backup whose key exists only on the backed-up box is not a backup.
|
||||
- S3 credentials come from `SWB_S3_ACCESS_KEY` / `SWB_S3_SECRET_KEY` (mapped to
|
||||
`AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` in the script).
|
||||
|
||||
## 2. Cloudron app backup
|
||||
|
||||
- Configured in the **dashboard** (*Settings → Backups*). There is **no write
|
||||
API**: `GET /api/v1/backup_sites` works, but `PUT`/`POST` on it and
|
||||
`POST /backups` all 404. Schedule and retention are dashboard-only.
|
||||
- **Two sites exist:** the live one (`s3-v4-compat`, daily 23:00, 7-day
|
||||
retention, id `03cfbdca-88fc-4ccc-97c3-7e1b58cf892d`) and an inactive
|
||||
`filesystem` site with schedule `never`. Do not be alarmed by the second one.
|
||||
- **A run reads `state=normal` even when it failed.** The failure's real error
|
||||
text is in `GET /api/v1/tasks` (task id `backup_<siteId>`, field `error.message`),
|
||||
not in the run list. `/events` needs a `?since=` range and `/system/logs` does
|
||||
not exist — a 404 there is not "no failures".
|
||||
- `/backup_sites` carries `schedule` as a 6-field cron and `retention.keepWithinSecs`.
|
||||
Check retention **before** claiming it needs changing.
|
||||
- **Quota math:** a failure whose upload pushed the retained set over the
|
||||
provider ceiling is the signature. The old box's `EntityTooLarge` failures came
|
||||
from a 4.82 GB run twice daily against the ceiling; the ceiling was raised and
|
||||
the schedule moved to once daily. Per-run size still grows (mostly registry
|
||||
image blobs) — watch the number.
|
||||
- **Two boxes on one S3 bucket interleave and prune each other's backups.** The
|
||||
old box's schedule must stay **disabled** while it is kept for rollback.
|
||||
- **Cross-check `siteId` per run:** runs on an old site id are pre-migration
|
||||
leftovers, kept `state=normal` forever and never pruned by the current site's
|
||||
retention.
|
||||
|
||||
## Restoring
|
||||
|
||||
- **Cloudron** restores apps per-app, from the **newest** box-backup entry:
|
||||
`GET /apps/:id/backups` → pick by `creationTime` → `POST /apps/:id/restore`
|
||||
with `{"backupId": ...}`. Restore apps **sequentially** (parallel restores
|
||||
exhaust disk). `repair` recreates a container but restores **no data** — a
|
||||
repaired app comes up empty.
|
||||
- Full procedure, triage and the post-restore checker: the `cloudron-ops` skill
|
||||
and `migration/cloudron-server-migration.md`.
|
||||
- **Agent host:** `restic restore latest --target /` (or a specific snapshot) with
|
||||
`RESTIC_PASSWORD` and the S3 keys in the environment, run as root.
|
||||
|
||||
## The credential that carries the backups
|
||||
|
||||
The agent-host `RESTIC_PASSWORD` and the Cloudron backup passphrase are **two
|
||||
different secrets**, both held by the GM and recorded in the token runbook's
|
||||
inventory. The Cloudron backup passphrase + S3 `secretAccessKey` are the
|
||||
chicken-and-egg pair a bare-metal rebuild needs first — they must be held off-box.
|
||||
@@ -0,0 +1,94 @@
|
||||
# Woodpecker CI (server + agent)
|
||||
|
||||
Self-hosted CI for the co-op's repos. **As of 2026-11-04 the CI is installed but
|
||||
NOT working end to end** — read the *Current state* section before trusting it for
|
||||
anything.
|
||||
|
||||
Two Cloudron apps, added 2026-11-03:
|
||||
|
||||
| App | FQDN | Cloudron id | Image |
|
||||
|---|---|---|---|
|
||||
| Woodpecker **server** | `ci.inference.coop` | `4984f95e-0c0a-4af3-9ce1-4ca4f3e8483d` | `cloudron/org.woodpecker_ci.cloudronapp:202610300219490000` (store, v2.22.1 / upstream 3.20.1) |
|
||||
| Woodpecker **agent** | `ci-agent.inference.coop` | `5ac63873-cf2c-4ad9-acd6-81dc0e016c83` | `registry.inference.coop/co-op/woodpecker-agent:20261104-e070275` (**custom**) |
|
||||
|
||||
- The **server** is the stock Cloudron app. It uses the `mysql` addon for storage
|
||||
and exposes `GRPC_PORT` 9000 to the agent. Its config lives in
|
||||
`/app/data/env.sh` (`WOODPECKER_GITEA`, `WOODPECKER_ADMIN`,
|
||||
`WOODPECKER_AGENT_SECRET`, `WOODPECKER_OPEN`, …). **No env vars are set through
|
||||
the app's Cloudron env** (`env-keys` is empty) — everything is in that file.
|
||||
- The **agent** is a custom app. Its config is set as Cloudron env (names only):
|
||||
`WOODPECKER_SERVER` = `ci.inference.coop:9000`, `WOODPECKER_GRPC_SECURE=true`,
|
||||
`WOODPECKER_BACKEND=local`, `WOODPECKER_MAX_WORKFLOWS=2`,
|
||||
`WOODPECKER_AGENT_CONFIG_FILE=/app/data/agent.conf`, plus the shared
|
||||
`WOODPECKER_AGENT_SECRET`.
|
||||
|
||||
## Why the agent image is custom (and why the backend is `local`)
|
||||
|
||||
Cloudron apps have **no Docker socket**, so the default `docker` backend cannot
|
||||
run and the agent uses the **`local` backend**: each pipeline step executes as a
|
||||
shell command *inside the agent container*, and a step's `image:` field selects
|
||||
the **shell**, not a container image. Consequence: **every tool a pipeline step
|
||||
needs must be baked into the agent image**, and the Cloudron rootfs is read-only
|
||||
at run time, so nothing can install at run time.
|
||||
|
||||
The image is `FROM woodpeckerci/woodpecker-agent:v3.20.1-alpine` (the **alpine**
|
||||
variant — the default `scratch` image has no shell). It adds `bash`, `git`,
|
||||
**`git-lfs`**, `tar`, `ca-certificates`, `python3`, `node`/`npm`, the `cloudron`
|
||||
CLI, and a pinned MkDocs toolchain at `/opt/docs-venv`. It **runs as root**
|
||||
(`/app/data` on this Cloudron is root-owned, as for the other custom apps here).
|
||||
|
||||
## Building and pinning the agent image
|
||||
|
||||
- **Source repo:** `/home/ubuntu/repos/woodpecker-agent` on the agent host.
|
||||
⚠️ **This repo has no git remote and has never been pushed to Gitea** — the CI
|
||||
agent's build source exists only on the agent host. Treat that as a gap to close.
|
||||
- **Tag convention:** `<yyyymmdd>-<gitsha>` (e.g. `20261104-e070275`). Never a
|
||||
floating tag.
|
||||
- **Build:** `~/.hermes/scripts/cloudron_builder_build.sh registry.inference.coop/co-op/woodpecker-agent <tag> <dir>`
|
||||
(the script carries the registry auth triple, which is not optional).
|
||||
- **Deploy:** write the new tag into `CloudronManifest.json`'s `dockerImage`,
|
||||
commit, build `--no-cache`, then `cloudron update --app 5ac63873-…` **without**
|
||||
`--image` (the manifest's pin wins). Verify by the running image tag **and**
|
||||
something only the new build produces — never by exit code alone.
|
||||
- `publishing-requirements.txt` mirrors the docs repo's
|
||||
`publishing/requirements.txt`; **keep the two in step** when the docs toolchain
|
||||
changes.
|
||||
|
||||
## When a pipeline runs (and when it does not)
|
||||
|
||||
- **Manual-trigger only.** Nothing here fires on push; a pipeline is started by
|
||||
hand from the Woodpecker UI.
|
||||
- **The pipeline is not yet green end to end.**
|
||||
- **`git-lfs` is required by the clone step.** Woodpecker runs `git lfs fetch`
|
||||
after cloning, so a clone step fails with
|
||||
`git: 'lfs' is not a git command` on any image without it — this was added to
|
||||
the agent image on 2026-11-04. If a rebuild ever drops it, clones break again.
|
||||
|
||||
## Current state (2026-11-04): installed, agent connected, cannot run jobs
|
||||
|
||||
- The server is up — `https://ci.inference.coop/` serves the Woodpecker UI.
|
||||
- The agent app is `running`/`healthy` and its health endpoint answers — **but
|
||||
that is liveness, not capability.** The log shows the agent's **gRPC job-collection
|
||||
long-poll failing continuously**: the server answers the agent's RPC with
|
||||
**HTTP 500** (an nginx `text/html` 500 body), so the agent retries with backoff
|
||||
and **never receives work**.
|
||||
- A healthy agent that cannot collect jobs is the exact trap to avoid reading as
|
||||
progress. Score a CI worker by **the route that delivers it work**, not by its
|
||||
health RPC.
|
||||
|
||||
**Unresolved / needs an operator:** the server-side gRPC 500. Until the server
|
||||
answers the agent's stream, no pipeline can execute regardless of repository
|
||||
configuration. This is a server-side (or platform-proxy) problem, not an agent
|
||||
image problem.
|
||||
|
||||
## Enabling a repo (known gate)
|
||||
|
||||
- Repos are connected through the **Gitea** source (`WOODPECKER_GITEA`). Enabling
|
||||
a repo requires the repo owner to log in to Woodpecker with **Gitea OAuth** and
|
||||
activate it. Standing up that OAuth login source is an **operator step**
|
||||
(creating the Gitea OAuth application needs `write:user`, which the bot's Gitea
|
||||
token does not have).
|
||||
- Reading run/pipeline state is done from the server's **MySQL** database, not
|
||||
from a public API.
|
||||
- Note: the whole point of this CI is to take the docs build and custom-app builds
|
||||
off the agent host — see `co-op/support#30` (moving `REGISTRY_BUILD_TOKEN` to CI).
|
||||
Reference in new issue
Block a user