docs: infrastructure audit — app inventory, backups, CI, provider pricing, agent fleet

Audit of the operator docs against live reality (2026-11-04): - app-inventory.md: all 19 Cloudron apps with id, fqdn and registry image pin. Corrects the portal path (code/member-portal, not co-op/member-portal) and the panel path (local/coop.inference.admin-panel, not code/admin-panel). - maintenance/backups.md: the two backup systems; the deliberate per-user restic cache split and the --group-by "" retention fix, with why. - maintenance/woodpecker-ci.md: CI server + agent, image build/pin, manual-only trigger, git-lfs requirement, and its current not-yet-green state (agent's job-collection long-poll gets HTTP 500 from the server). - provider-pricing/: index + GreenPT, Tinfoil (local EHBP proxy architecture) and the Metis /v1/systemone decision-model route; regolo.md updated to record the gateway prices were applied. - agent-fleet.md: steward, infrastructure-bot and socialbot (Unix users, gateway units, teams); member-bot tier 2 planned but not yet provisioned. - README: file list; operator-token-runbook: correct the membership_sweep cadence (3x daily, not nightly); migration doc: post-migration addendum (16->19 apps, DNS split resolved, old box still on with apps stopped).
Merge pull request 'operator-token-runbook: credential rotation policy and reminder (co-op/support#35)' (#3 ) from rotation-policy into main
2026-11-04 14:46:12 +00:00 · 2026-11-04 14:37:22 +00:00 · 2026-11-04 14:36:57 +00:00
14 changed files with 925 additions and 60 deletions
@@ -24,12 +24,15 @@ Read it on the web here: <https://git.inference.coop/co-op/ops-docs>
 | Path | Contents |
 |---|---|
 | `operator-token-runbook.md` | The Cloudron token model (agent vs portal, admin vs webhook), how to rotate each without breaking the stack |
-| `membership-allowances.md` | How a member's monthly allowance renews, how a lapse is handled, and the join-day anchoring rule |
+| `app-inventory.md` | Every Cloudron app: id, fqdn, registry image path and pin (read from the running apps) |
 | `agent-fleet.md` | The three agents (steward, infrastructure-bot, socialbot): Unix users, gateway units, teams, and the planned member-bot tier |
 | `maintenance/chat-task-models.md` | The small pinned model used for chat titles/tags/suggestions |
 | `maintenance/librechat-sandbox.md` | Operating the LibreChat evaluation instance (config, overlay image, key sync, pitfalls) |
 | `maintenance/backups.md` | The two backup systems (agent-host restic + Cloudron app backup): schedules, retention, the cache split and the grouping fix |
 | `maintenance/woodpecker-ci.md` | The self-hosted CI server and agent: build/pin, manual-only trigger, git-lfs, and its not-yet-green state |
 | `incidents/` | Post-incident write-ups, one file per incident |
 | `migration/` | The Cloudron server-migration record: the plan, the DNS change sheet, and the captured pre-migration state |
-| `provider-pricing/` | What our model providers charge, and when it changed |
+| `provider-pricing/` | What each model provider charges, and when it changed (GreenPT, Tinfoil, Regolo, and the decision-model route) |
 ## The migration material
@@ -0,0 +1,65 @@
 # The agent fleet
 The co-op runs **more than one agent**, each as its **own Unix user** on the agent
 host (`ov-0f018e`, `179.237.103.165`) with its **own Hermes gateway unit** and its
 **own LiteLLM team**. Any document that describes "the agent" as a single thing is
 now wrong.
 ## The three agents today
 | Agent | Unix user | systemd unit | Gateway team (budget) | Model | Presence |
 |---|---|---|---|---|---|
 | **Inferencebot** (steward) | `ubuntu` | `hermes-gateway.service` | *(co-op account team)* | per `~/.hermes/config.yaml` | Matrix `@inferencebot:inference.coop`; digests, newsletters, membership/key plumbing |
 | **infrastructure-bot** | `infrabot` | `hermes-gateway-infrabot.service` | `agent:infrastructure-bot` ($5/30d) | `greenpt/glm-5.3-flash` | Matrix `#infrastructure` (answers only when mentioned); infrastructure scope |
 | **socialbot** | `socialbot` | `hermes-gateway-socialbot.service` | `agent:socialbot` ($5/30d) | `greenpt/glm-5.3-flash` | Matrix `#social-drafts`; drafts the co-op's AT Protocol (Bluesky) account `inference.coop` |
 All three gateways run on the one host; each is a separate process, a separate
 `.env`, and a separate set of rooms. There is **one LiteLLM gateway** (the Cloudron
 app) that all three draw on — the agent host runs no gateway of its own.
 ### Who is who (do not conflate them)
 - **Inferencebot** is the **steward** and the fleet administrator: full Cloudron
  token, gateway master key, member-data plumbing. It is NOT a member-facing chat
  persona.
 - **infrastructure-bot** is a **lower-privileged** infrastructure agent. Per its
  own SOUL it has **no sudo, no Cloudron admin token, and no member data**; when a
  task needs more privilege it hands up to the steward rather than working around
  it. Its gateway key has a small monthly budget and a flash-tier allowlist.
 - **socialbot** has **no Cloudron, portal or member-data access** and no power over
  Matrix rooms; its only writable surface is the `#social-drafts` room, and it
  **never publishes an unapproved draft** — silence is its correct failure mode.
 ## Member bots — TIER 2 (planned, not yet provisioned)
 The intended path for **member-facing** bots is a second tier: each member bot is a
 **Hermes profile under the `infrabot` Unix user**, sharing **one** gateway process
 (the infrabot one), with its own gateway team/key. This is tracked as **co-op/support#33**.
 **Status (2026-11-04): not yet provisioned.** `/home/infrabot/.hermes/profiles/`
 holds only a `.deleted` directory with two probe files — there are **no active
 member-bot profiles**. Record this honestly: the design exists, the instances do
 not.
 ## Why one Unix user per agent — and the limit of it
 **Profile isolation is organizational, not security.** Run one OS user (or a
 rootless container) per agent from day one: **on a shared Unix user, sibling
 Hermes profiles can read each other's `.env`.** So tier-2 member bots sharing the
 `infrabot` user are isolated from *members* but **not from each other** — that
 sharing is accepted for member bots, not extended to the privileged agents.
 ## Backup coverage is automatic — but prove it
 The nightly restic run (see `maintenance/backups.md`) enumerates `/home/*` **as
 root** and stages every agent home, so a new agent is covered without editing any
 list. Coverage is still **unproven until that agent's files show up in a
 snapshot**.
 ## Gateway teams (member vs agent)
 From `GET https://gateway.inference.coop/team/list` (2026-11-04): 97 teams — one
 `member:<email>` team per member (mostly $15/30d), `member:info@inference.coop`
 ($135/30d, the co-op account), the two `agent:*` teams above, and
 `sandbox:librechat-test` ($3). Agent teams carry a **small budget and a flash-tier
 model allowlist** enforced at the gateway.
@@ -0,0 +1,82 @@
 # Cloudron app inventory
 The co-op's Cloudron server carries **19 apps**. Before 2026-11-02 it carried 16;
 the three added since the migration are the docs site, the CI server and the CI
 agent (marked **NEW** below).
 **Source of truth:** read it from the running apps, not from memory or a sibling
 deploy. `GET https://my.inference.coop/api/v1/apps` returns every app with its
 `id`, `fqdn`, install state and — inside `manifest` — `dockerImage`. The app
 record's own top-level `image` field is **null**; the CLI's `list` output does
 not carry the image at all. The registry paths are **per-app and not uniform**
 (there are three different namespaces in use — see below), and a path copied from
 a remembered deploy of a *different* app points the update at a repository that
 does not hold it.
 *Verified against the live API 2026-11-04.*
 ## The 19 apps
 | FQDN | App id | Label | Image (pin) | Install |
 |---|---|---|---|---|
 | `inference.coop` | `55f5f904-3233-44de-ad7b-5dc7c8533b75` | — | `cloudron/io.cloudron.surfer:202611021342220000` | store (Surfer — marketing site) |
 | `chat.inference.coop` | `4023ccbf-d57a-4500-97df-14de9cf3516e` | AI Chat | `registry.inference.coop/co-op/librechat-custom:20261031-2a98892` | **custom** (LibreChat overlay) |
 | `chat-legacy.inference.coop` | `50ae6c97-e851-4d17-b013-51ca7a4e5be0` | Legacy AI Chat | `cloudron/com.openwebui.cloudronapp:202611030420300000` | store (Open WebUI) |
 | `portal.inference.coop` | `d4f05e4b-75c6-492e-9faf-7afb0712fc94` | — | `registry.inference.coop/code/member-portal:20261104-d076729` | **custom** |
 | `dashboard.inference.coop` | `daf8750e-2683-4104-83ea-f77d1b4601e8` | Dashboard | `registry.inference.coop/code/member-dashboard:20261103-3453131` | **custom** |
 | `panel.inference.coop` | `ae187071-7a43-4a6e-8a02-986609a66a1a` | Admin Panel | `local/coop.inference.admin-panel:0.1.0-1793634279942` | **custom** |
 | `gateway.inference.coop` | `adababd4-2427-4bdd-9bf7-6bdad0302610` | LiteLLM | `registry.inference.coop/local/ai.coop.litellm:1.0.0-20261103-d6cfe78` | **custom** |
 | `matrix.inference.coop` | `aa6f4597-6206-4509-b32e-c89750b2c1d8` | Matrix Chat | `cloudron/org.matrix.synapse:202611030719520000` | store (Synapse) |
 | `git.inference.coop` | `c91c13aa-30ed-4af9-9599-7a87ab4f221d` | Code | `cloudron/io.gitea.cloudronapp:202611011719500000` | store (Gitea) |
 | `registry.inference.coop` | `c257ba3c-6737-4375-ad51-ea19586bf3f0` | — | `cloudron/io.cloudron.container-registry:202610221511140000` | store (container registry) |
 | `vault.inference.coop` | `cb838bcf-f2e2-4670-b117-dca04b0ee957` | — | `cloudron/com.github.bitwardenrs:202611020820270000` | store (Vaultwarden) |
 | `forum.inference.coop` | `8b1a972c-08af-4e92-852c-d573c37b01ad` | Forum | `cloudron/org.loomio.cloudronapp:202611010321240000` | store (Loomio) |
 | `newsletter.inference.coop` | `eed2217c-b165-430e-bc29-7ccfef9b11e4` | — | `cloudron/app.listmonk.cloudronapp:202610220221130000` | store (Listmonk) |
 | `search.inference.coop` | `d7c73fff-ee42-4db0-98c0-4ce7a8ab35cc` | — | `cloudron/io.github.ascimoo.searx:202611012319290000` | store (SearXNG) |
 | `cloud.inference.coop` | `fd261d67-b790-424f-9e82-f74883ee8585` | — | `cloudron/com.nextcloud.cloudronapp:202610211606270000` | store (Nextcloud) |
 | `email.inference.coop` | `fbb256b9-ead3-41d8-bee0-d5ea87149436` | — | `cloudron/io.cloudron.mail:202610261602550000` | store (Cloudron mail UI) |
 | `docs.inference.coop` | `85b7ce06-e2af-49a4-bd4f-a9c2e692e3ab` | Docs | `cloudron/io.cloudron.surfer:202611021342220000` | store (Surfer) — **NEW 2026-11-03** |
 | `ci.inference.coop` | `4984f95e-0c0a-4af3-9ce1-4ca4f3e8483d` | Woodpecker | `cloudron/org.woodpecker_ci.cloudronapp:202610300219490000` | store (Woodpecker server) — **NEW 2026-11-03** |
 | `ci-agent.inference.coop` | `5ac63873-cf2c-4ad9-acd6-81dc0e016c83` | — | `registry.inference.coop/co-op/woodpecker-agent:20261104-e070275` | **custom** (Woodpecker agent) — **NEW 2026-11-03** |
 (`my.inference.coop` is the Cloudron dashboard hostname — the platform itself, not
 an app object — so it does not appear here.)
 ## Registry namespaces — they are NOT uniform
 Five custom apps pull from four different namespaces. Deriving one path from
 another app's pattern deploys the wrong (or a missing) image:
 | App | Registry path |
 |---|---|
 | chat (LibreChat overlay) | `registry.inference.coop/co-op/librechat-custom` |
 | CI agent | `registry.inference.coop/co-op/woodpecker-agent` |
 | member portal | `registry.inference.coop/code/member-portal` |
 | member dashboard | `registry.inference.coop/code/member-dashboard` |
 | gateway (LiteLLM) | `registry.inference.coop/local/ai.coop.litellm` |
 | admin panel | `local/coop.inference.admin-panel` |
 **Correcting a stale memory:** the portal is `code/member-portal` (not
 `co-op/member-portal`), and the admin panel is `local/coop.inference.admin-panel`
 (not `code/admin-panel`). Both were cited wrongly in earlier notes; the values
 above are read from the running apps.
 ## Store vs custom, and what that decides
 Six apps are **custom** installs (packaged by us, `appStoreId == ""`): chat,
 CI agent, portal, dashboard, panel, gateway. Custom apps get **no automatic
 update** (Cloudron says "not available for custom apps") — their protection is the
 `manifest.dockerImage` pin, which an update deploys, so a custom app must never be
 pinned to a floating tag. The rest carry a real `appStoreId` and update through
 the store.
 ## Notes that are easy to get wrong
 - **`enableAutomaticUpdate` exists on every app object and means nothing for a
  custom one.** Judge update behaviour by the install SOURCE (`appStoreId`), not
  by that flag.
 - **Public collectives reach `co-op/*` and `code/*` repos; the only private repos
  are `ops/inferencebot` and `ops/security`.**
 - **`info@inference.coop` is the co-op's gateway team account** (`member:info@inference.coop`,
  budget $135/30d) and also the owner account of several provider keys — see the
  token runbook.
@@ -0,0 +1,137 @@
 # Backups
 There are **two independent backup systems**, on two different machines, with two
 different targets and two different passphrases. Confusing them is the most likely
 mistake — they do not back each other up, and neither one covers the other's host.
 | | Agent-host backup | Cloudron app backup |
 |---|---|---|
 | What | The agent host's state (all agent homes, `~/.hermes`, `~/.ssh`, `~/repos`, gateway unit files) | Every Cloudron app's `/app/data` + addon databases, plus platform config |
 | Machine | Agent host `ov-0f018e` (`179.237.103.165`) | The Cloudron server `179.237.86.116` |
 | Tool | `restic` (client-side encrypted, run nightly by a systemd timer) | Cloudron's built-in backup (dashboard-configured) |
 | Target | Infomaniak **Swiss Backup** (S3-compatible) | Infomaniak **Swiss Backup** (S3-compatible, `s3-v4-compat`) |
 | Schedule | 04:30 UTC + up to 5 min jitter (`agent-backup.timer`) | daily 23:00 UTC (`00 00 23 * * *`) |
 | Retention | 7 daily / 4 weekly / 3 monthly, no grouping | 7 days (`keepWithinSecs: 604800`) |
 | Config lives | `~/.hermes/scripts/backup.sh`, unit at `/etc/systemd/system/agent-backup.*` | Cloudron dashboard (no write API — see below) |
 | Last verified | 2026-11-04 (13 snapshots in the repo) | 2026-11-02 (runs 2026-10-30 → 11-03 all `normal`, ~5.4 GB) |
 Nothing on the Cloudron **host OS** is backed up by either system (netplan, sshd,
 `authorized_keys`, hand-edited `/etc`) — build the box, do not expect to restore
 it. This was the migration's central caveat and it still holds.
 ---
 ## 1. Agent-host backup (restic)
 `agent-backup.timer` → `agent-backup.service` → `~/.hermes/scripts/backup.sh`.
 - **The unit runs as `User=root` on purpose.** The agent homes under `/home/*`
  are mode `0700`, so only root can enumerate them; `ubuntu` sees `EACCES` and a
  non-root `[ -d .../.hermes ]` test reports **false for a home that exists** —
  which would silently skip that agent. The script therefore discovers agents by
  enumerating `/home/*` **as root** and stages each one; a new agent is covered
  without editing any list. Coverage is still unproven until its files show up in
  a snapshot.
 - **It backs up:** `/home/ubuntu/.hermes` (config, `.env`, skills, cron, memory,
  logs, state DBs), `/home/ubuntu/.ssh`, `/home/ubuntu/repos`, every other
  agent's `/home/<agent>/.hermes`, and `/etc/systemd/system/hermes-gateway*.service`.
  It **excludes** rebuildable bulk (`tools/`, `hermes-agent/`, `installs/`,
  `cache/`, `lsp/`, model caches) and the live SQLite files — consistent copies of
  those are staged first with `sqlite3 .backup`.
 - **Failure is visible, not silent.** On failure the script appends a note to
  `~/.hermes/cron/updates_notes.jsonl`, which the daily stewarding digest reads.
  Exit codes: 0 ok, 1 config, 2 backup, 3 prune.
 ### Deliberate-but-surprising: the PER-USER restic cache split
 `RESTIC_CACHE_DIR` defaults to **`/var/cache/restic`** (root's run), while manual
 runs by `ubuntu` use **`~/.cache/restic-manual`**. This is intentional — **do not
 "tidy" it back into one shared `~/.cache/restic`.**
 A shared cache directory let each user create files the other could not read;
 restic then **retried every snapshot with exponential backoff** (30s, 1m25s, …)
 until the run crawled or timed out. The visible symptom (2026-11-03) was a prune
 that "failed" **three nights running** — which looked exactly like retention being
 broken, when it was a permissions/retry storm. Splitting the cache per user ended
 it. The lock and log also live **under `$HOME/.hermes/`** (not `/tmp`): with
 `fs.protected_regular`, a root run is denied a sticky-dir file owned by another
 user (`EACCES`), and under `set -e` that killed the root run *silently*.
 ### Deliberate-but-surprising: `--group-by ""` on retention
 Retention is:
 ```
 restic forget --tag nightly --group-by "" \
  --keep-daily 7 --keep-weekly 4 --keep-monthly 3 --prune
 ```
 **`--group-by ""` (no grouping) is required, not cosmetic.** restic's default is
 `--group-by host,paths`, which splits the nightly snapshots into one group per
 distinct **paths set**. The backup glob `/etc/systemd/system/hermes-gateway*.service`
 grew from one unit to three when the infrabot and socialbot gateways were added
 (2026-10-30), so the paths set changed mid-history: snapshots before 2026-10-30
 formed one group and snapshots after formed another. Each group then sat **under**
 the keep-7-daily limit *on its own*, so **nothing was ever expired and the repo
 could never shrink**. Empty grouping applies one policy across all nightly
 snapshots. `--group-by host` would also collapse today's groups, but the box has
 already migrated once and a hostname change would reintroduce the same split — no
 grouping is immune to both path-set drift and a hostname change.
 **Current state (honest):** the repo still holds **13 snapshots** — the pre-fix
 accumulation. The script on disk carries the fix, and a `--dry-run` (2026-11-04)
 confirms it **keeps 8 and removes 5**. The next timer run (2026-11-05 04:31 UTC)
 applies it. Until it runs, treat "13 snapshots" as expected, not as the fix
 having failed.
 ### Other mechanics
 - Weekly `restic check` on Sundays (repo metadata); full data check monthly.
 - The passphrase (`RESTIC_PASSWORD`) lives in `~/.hermes/.env` **and** the co-op
  Vault. A backup whose key exists only on the backed-up box is not a backup.
 - S3 credentials come from `SWB_S3_ACCESS_KEY` / `SWB_S3_SECRET_KEY` (mapped to
  `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` in the script).
 ## 2. Cloudron app backup
 - Configured in the **dashboard** (*Settings → Backups*). There is **no write
  API**: `GET /api/v1/backup_sites` works, but `PUT`/`POST` on it and
  `POST /backups` all 404. Schedule and retention are dashboard-only.
 - **Two sites exist:** the live one (`s3-v4-compat`, daily 23:00, 7-day
  retention, id `03cfbdca-88fc-4ccc-97c3-7e1b58cf892d`) and an inactive
  `filesystem` site with schedule `never`. Do not be alarmed by the second one.
 - **A run reads `state=normal` even when it failed.** The failure's real error
  text is in `GET /api/v1/tasks` (task id `backup_<siteId>`, field `error.message`),
  not in the run list. `/events` needs a `?since=` range and `/system/logs` does
  not exist — a 404 there is not "no failures".
 - `/backup_sites` carries `schedule` as a 6-field cron and `retention.keepWithinSecs`.
  Check retention **before** claiming it needs changing.
 - **Quota math:** a failure whose upload pushed the retained set over the
  provider ceiling is the signature. The old box's `EntityTooLarge` failures came
  from a 4.82 GB run twice daily against the ceiling; the ceiling was raised and
  the schedule moved to once daily. Per-run size still grows (mostly registry
  image blobs) — watch the number.
 - **Two boxes on one S3 bucket interleave and prune each other's backups.** The
  old box's schedule must stay **disabled** while it is kept for rollback.
 - **Cross-check `siteId` per run:** runs on an old site id are pre-migration
  leftovers, kept `state=normal` forever and never pruned by the current site's
  retention.
 ## Restoring
 - **Cloudron** restores apps per-app, from the **newest** box-backup entry:
  `GET /apps/:id/backups` → pick by `creationTime` → `POST /apps/:id/restore`
  with `{"backupId": ...}`. Restore apps **sequentially** (parallel restores
  exhaust disk). `repair` recreates a container but restores **no data** — a
  repaired app comes up empty.
 - Full procedure, triage and the post-restore checker: the `cloudron-ops` skill
  and `migration/cloudron-server-migration.md`.
 - **Agent host:** `restic restore latest --target /` (or a specific snapshot) with
  `RESTIC_PASSWORD` and the S3 keys in the environment, run as root.
 ## The credential that carries the backups
 The agent-host `RESTIC_PASSWORD` and the Cloudron backup passphrase are **two
 different secrets**, both held by the GM and recorded in the token runbook's
 inventory. The Cloudron backup passphrase + S3 `secretAccessKey` are the
 chicken-and-egg pair a bare-metal rebuild needs first — they must be held off-box.
@@ -0,0 +1,94 @@
 # Woodpecker CI (server + agent)
 Self-hosted CI for the co-op's repos. **As of 2026-11-04 the CI is installed but
 NOT working end to end** — read the *Current state* section before trusting it for
 anything.
 Two Cloudron apps, added 2026-11-03:
 | App | FQDN | Cloudron id | Image |
 |---|---|---|---|
 | Woodpecker **server** | `ci.inference.coop` | `4984f95e-0c0a-4af3-9ce1-4ca4f3e8483d` | `cloudron/org.woodpecker_ci.cloudronapp:202610300219490000` (store, v2.22.1 / upstream 3.20.1) |
 | Woodpecker **agent** | `ci-agent.inference.coop` | `5ac63873-cf2c-4ad9-acd6-81dc0e016c83` | `registry.inference.coop/co-op/woodpecker-agent:20261104-e070275` (**custom**) |
 - The **server** is the stock Cloudron app. It uses the `mysql` addon for storage
  and exposes `GRPC_PORT` 9000 to the agent. Its config lives in
  `/app/data/env.sh` (`WOODPECKER_GITEA`, `WOODPECKER_ADMIN`,
  `WOODPECKER_AGENT_SECRET`, `WOODPECKER_OPEN`, …). **No env vars are set through
  the app's Cloudron env** (`env-keys` is empty) — everything is in that file.
 - The **agent** is a custom app. Its config is set as Cloudron env (names only):
  `WOODPECKER_SERVER` = `ci.inference.coop:9000`, `WOODPECKER_GRPC_SECURE=true`,
  `WOODPECKER_BACKEND=local`, `WOODPECKER_MAX_WORKFLOWS=2`,
  `WOODPECKER_AGENT_CONFIG_FILE=/app/data/agent.conf`, plus the shared
  `WOODPECKER_AGENT_SECRET`.
 ## Why the agent image is custom (and why the backend is `local`)
 Cloudron apps have **no Docker socket**, so the default `docker` backend cannot
 run and the agent uses the **`local` backend**: each pipeline step executes as a
 shell command *inside the agent container*, and a step's `image:` field selects
 the **shell**, not a container image. Consequence: **every tool a pipeline step
 needs must be baked into the agent image**, and the Cloudron rootfs is read-only
 at run time, so nothing can install at run time.
 The image is `FROM woodpeckerci/woodpecker-agent:v3.20.1-alpine` (the **alpine**
 variant — the default `scratch` image has no shell). It adds `bash`, `git`,
 **`git-lfs`**, `tar`, `ca-certificates`, `python3`, `node`/`npm`, the `cloudron`
 CLI, and a pinned MkDocs toolchain at `/opt/docs-venv`. It **runs as root**
 (`/app/data` on this Cloudron is root-owned, as for the other custom apps here).
 ## Building and pinning the agent image
 - **Source repo:** `/home/ubuntu/repos/woodpecker-agent` on the agent host.
  ⚠️ **This repo has no git remote and has never been pushed to Gitea** — the CI
  agent's build source exists only on the agent host. Treat that as a gap to close.
 - **Tag convention:** `<yyyymmdd>-<gitsha>` (e.g. `20261104-e070275`). Never a
  floating tag.
 - **Build:** `~/.hermes/scripts/cloudron_builder_build.sh registry.inference.coop/co-op/woodpecker-agent <tag> <dir>`
  (the script carries the registry auth triple, which is not optional).
 - **Deploy:** write the new tag into `CloudronManifest.json`'s `dockerImage`,
  commit, build `--no-cache`, then `cloudron update --app 5ac63873-…` **without**
  `--image` (the manifest's pin wins). Verify by the running image tag **and**
  something only the new build produces — never by exit code alone.
 - `publishing-requirements.txt` mirrors the docs repo's
  `publishing/requirements.txt`; **keep the two in step** when the docs toolchain
  changes.
 ## When a pipeline runs (and when it does not)
 - **Manual-trigger only.** Nothing here fires on push; a pipeline is started by
  hand from the Woodpecker UI.
 - **The pipeline is not yet green end to end.**
 - **`git-lfs` is required by the clone step.** Woodpecker runs `git lfs fetch`
  after cloning, so a clone step fails with
  `git: 'lfs' is not a git command` on any image without it — this was added to
  the agent image on 2026-11-04. If a rebuild ever drops it, clones break again.
 ## Current state (2026-11-04): installed, agent connected, cannot run jobs
 - The server is up — `https://ci.inference.coop/` serves the Woodpecker UI.
 - The agent app is `running`/`healthy` and its health endpoint answers — **but
  that is liveness, not capability.** The log shows the agent's **gRPC job-collection
  long-poll failing continuously**: the server answers the agent's RPC with
  **HTTP 500** (an nginx `text/html` 500 body), so the agent retries with backoff
  and **never receives work**.
 - A healthy agent that cannot collect jobs is the exact trap to avoid reading as
  progress. Score a CI worker by **the route that delivers it work**, not by its
  health RPC.
 **Unresolved / needs an operator:** the server-side gRPC 500. Until the server
 answers the agent's stream, no pipeline can execute regardless of repository
 configuration. This is a server-side (or platform-proxy) problem, not an agent
 image problem.
 ## Enabling a repo (known gate)
 - Repos are connected through the **Gitea** source (`WOODPECKER_GITEA`). Enabling
  a repo requires the repo owner to log in to Woodpecker with **Gitea OAuth** and
  activate it. Standing up that OAuth login source is an **operator step**
  (creating the Gitea OAuth application needs `write:user`, which the bot's Gitea
  token does not have).
 - Reading run/pipeline state is done from the server's **MySQL** database, not
  from a public API.
 - Note: the whole point of this CI is to take the docs build and custom-app builds
  off the agent host — see `co-op/support#30` (moving `REGISTRY_BUILD_TOKEN` to CI).
@@ -1,54 +0,0 @@
 # Membership allowance renewal and lapse
 Operator-facing record of the decision taken **2026-11-04** on how a member's
 monthly allowance renews, and how a lapse is handled. The member-facing summary
 is `co-op/support#44`; the code lives in `co-op/member-portal`.
 ## The rule
 - **Renewal is decided by asking Open Collective, on our own schedule.** A
  scheduled reconciliation asks whether the member has a *live recurring
  contribution* (order status `ACTIVE` or `PROCESSING` — a payment in flight is
  not a missed payment). Renewal is a decision of the sweep, not a side effect of
  webhook delivery, so a lost webhook costs a delayed renewal, never a silently
  skipped month.
 - **Anchor to the member's join day, never the payment date.** Open Collective
  aligns a whole collective's charges onto one common billing day, so anchoring
  to a charge lets an early charge move the boundary and skip a cycle. The
  member's join day is stored (`members.anchor_day`) and the reset is pinned to
  its next occurrence.
 - **Grace before deactivation.** One reminder email per cycle; deactivate only
  after `PAYMENT_GRACE_DAYS` unpaid.
 - **Deactivate by removing group access and stopping renewal, keeping the
  account**, so a returning payer resumes cleanly at the next cycle.
 - **Log every decision** (`allowance_log`) with the Open Collective evidence —
  renewed / reminded / deactivated, plus the payment date or its absence — so a
  dispute is answered from a record, not reconstructed.
 ## Scope: monthly only
 Verified **2026-11-04** that there are **no annual contributors**: 104 orders on
 Open Collective, 87 MONTHLY and 17 ONETIME, zero annual, and no order has a
 charge gap over 45 days. Only the monthly case is implemented. One-time
 contributions are not an allowance cycle — they top up the non-expiring credit
 balance and stay in the existing credits model.
 ## Operating it
 - `ALLOWANCE_POLICY` = `report` (default) | `on`. `report` computes every
  decision and changes nothing; `on` performs renewals and sends the reminder.
 - `ALLOWANCE_DEACTIVATE` = `off` (default) | `on`. Gated separately so renewals
  and reminders can run before anyone loses access.
 - `PAYMENT_GRACE_DAYS` (default `7`).
 - Manual check: `POST /admin/allowance?mode=report` (`X-Admin-Token`) is a no-op
  report; `mode=test&to=<addr>` emails one sample reminder; `mode=send` requires
  `ALLOWANCE_POLICY=on`.
 ## The bug that prompted it
 A member who joined on the 15th was charged on the 2nd (Open Collective's common
 billing day). The reset rule re-anchored his allowance to the new payment date
 and pushed his renewal from the 14th to the 2nd of the next month — skipping the
 reset he was owed while he had already paid. The fix is both changes above:
 anchor to the join day, and decide renewal on a schedule rather than on the
 payment.
@@ -508,3 +508,57 @@ access removed 2026-11-02 — retirement begun; app restored and safe in backups
 **Outstanding:** tonight 23:00 UTC first backup (verify `normal`, ~4.9 GiB); Cloudron subscription
 page check on the new dashboard; my primary model route re-heals on its own (fallback worked all
 night); old box keep powered-on/apps-stopped/backups-disabled for a few days, then decommission.
 ---
 # POST-MIGRATION ADDENDUM — 2026-11-04
 Recorded by the infra-docs audit. This section is current state; everything above
 is the as-it-happened record.
 ## App count: 16 → 19 (three apps added since cutover)
 The migration restored **16 apps**. Since then the server carries **19** — the
 three new ones are the docs site, the CI server and the CI agent. See
 `app-inventory.md` for the full list. Anywhere above that says "16 apps" is the
 state at cutover, not today.
 | New since 2026-11-02 | FQDN | Cloudron id |
 |---|---|---|
 | Docs site (Surfer) | `docs.inference.coop` | `85b7ce06-e2af-49a4-bd4f-a9c2e692e3ab` |
 | Woodpecker CI server | `ci.inference.coop` | `4984f95e-0c0a-4af3-9ce1-4ca4f3e8483d` |
 | Woodpecker CI agent (custom) | `ci-agent.inference.coop` | `5ac63873-cf2c-4ad9-acd6-81dc0e016c83` |
 ## The DNS "split" is resolved — no hostname is served by the old box
 At one point `chat.inference.coop` and `portal.inference.coop` were observed still
 resolving to the **old** box (`179.199.135.72`) while `gateway`/`my`/`docs` were on
 the **new** box (`179.237.86.116`). **As of 2026-11-04 there is no such split:**
 every hostname — `chat`, `portal`, `gateway`, `my`, `docs`, `ci`, `ci-agent`,
 `matrix`, `git`, `www` and the apex — resolves to the **new** box.
 The zone is **wildcarded**, so a resolving name proves nothing by itself. The
 decisive fact: a **made-up** hostname (`zzznope.inference.coop`) also answers the
 **new** box `179.237.86.116` at the authoritative nameservers (`ns1.yoursrs.com`).
 That means the **wildcard itself now points at the new box**, so no name is being
 served by the old box via the wildcard either.
 ## The old box still exists (rollback), and serves nothing
 - `179.199.135.72` is **still powered on** and answers on 22/80/443/587/993, **but
  it hosts no running app.** A request with `Host: chat.inference.coop` returns
  Cloudron's **"app down / Not Found" fallback page**, and its TLS certificate has
  an **empty subject** — the signature of a Cloudron whose vhost's app is stopped,
  exactly as documented ("powered on with apps stopped").
 - **No DNS reference points at it**, so it receives no real traffic.
 - **Decommission is still pending** — this is the one open item from the cutover
  that has not happened. Its backup schedule must remain **disabled** while it is
  kept (two boxes on one S3 bucket interleave and prune each other's backups).
 ## Still unresolved / for the operator
 - **Decommission the old box** (`179.199.135.72`) when rollback is no longer
  wanted, after confirming the new box's backups are proven.
 - **Member mailbox contents** — the long-standing unknown from §7 (whether
  members' mail is inside the Cloudron backup) still needs a real IMAP login to
  confirm.
@@ -3,6 +3,85 @@
 How the co-op's Cloudron credentials are organized, and how to rotate them
 without breaking the stack.
 ## Credential rotation policy
 **Cadence: 12 months for every credential.** One interval, no exceptions for now.
 Twelve months is a starting point, not a principle — revisit it once the token
 list is stable (co-op/support#35).
 **Rotate on a trigger immediately, off the calendar, when any of these happens:**
 - **Offboarding** — anyone who had access to the credential leaves or changes
  role, or the device/account it lived on is lost.
 - **Suspected exposure** — the value was seen somewhere it should not be: a log,
  a chat, a commit, a screenshot.
 - **After any incident, or any diagnostic session on the agent host in which a
  secret could have been read.** Reading a live secret to debug with it is itself
  the trigger; it does not have to be a confirmed breach.
 - **After any restore** — the credential travels in the backup, so a restored
  host re-materialises whatever values the snapshot held.
 **The GM controls all tokens for now** (operator, 2026-11-04), so this record
 needs **no owner column**: every row is rotated by the GM. Provider keys are
 minted on the provider account, but the value held in the co-op's environment is
 still the GM's to rotate.
 ### Last rotated
 One row per credential, or per credential group — a group rotates together, so it
 dates together. A blank cell means **no rotation has been recorded**, not "not
 rotated". Dates are UTC, `YYYY-MM-DD`.
 The reminder reads this exact table: `scripts/rotation_due.py` in this repository
 (deployed on the agent host at `~/.hermes/scripts/rotation_due.py`) prints a line
 only when a due date is within 30 days, and prints nothing otherwise. Keep the
 table machine-legible — one `YYYY-MM-DD` date per row.
 | Credential or group | Last rotated (UTC) |
 |---|---|
 | Open Collective token (`OC_PERSONAL_TOKEN`) | 2026-10-27 |
 | `LITELLM_MASTER_KEY` / `GW_MASTER_KEY` (one value, four holders) | |
 | Cloudron API tokens — portal + agent (both on `inference-bot`) | |
 | `REGISTRY_BUILD_TOKEN` (on `inference-admin`; moves to CI, co-op/support#30) | |
 | Gitea token (`inference-bot`) | |
 | `WEBHOOK_TOKEN` (portal env + Open Collective webhook URL) | |
 | `ADMIN_TOKEN` (portal + admin panel + agent host) | |
 | `BROKER_SECRET` and `PORTAL_SECRET` | |
 | Matrix tokens — portal `MATRIX_BOT_TOKEN` + agent `MATRIX_ADMIN_TOKEN` (one account, `@inference-admin`) | |
 | Matrix token — steward agent `@inferencebot` | |
 | Loomio API keys (`LOOMIO_API_KEY`, `LOOMIO_B3_KEY`) | |
 | Listmonk API (`LISTMONK_API_USER`, `LISTMONK_API_TOKEN`) | |
 | Provider keys — Tinfoil, GreenPT, Regolo, and the GreenPT fallback | |
 | Restic repository passphrase; Infomaniak Swiss Backup S3 keys | |
 ### Accounts behind the credentials
 A credential belongs to an **account**, which is what a rotation is actually
 performed against — ask the provider, do not read the variable name. Two
 credentials on one account are a single hazard: rotating one and not the other is
 the silent break.
 | Credential | Account | Basis |
 |---|---|---|
 | `LITELLM_MASTER_KEY` / `GW_MASTER_KEY` | self-issued by the gateway | one literal, four holders |
 | Cloudron API tokens — portal and agent | `inference-bot` | verified (co-op/support#35) |
 | `REGISTRY_BUILD_TOKEN` | `inference-admin` | verified (co-op/support#35) |
 | `MATRIX_BOT_TOKEN` (portal) + `MATRIX_ADMIN_TOKEN` (agent) | `@inference-admin:inference.coop` — two tokens, one account | verified, homeserver `whoami` 2026-11-04 |
 | Steward agent's `MATRIX_ACCESS_TOKEN` | `@inferencebot:inference.coop` | verified, homeserver `whoami` 2026-11-04 |
 | `OC_PERSONAL_TOKEN` | `info@inference.coop` | from the 2026-10-27 rotation record (inference) |
 | `TINFOIL_API_KEY` | `info@inference.coop` | operator-confirmed |
 | `GREENPT_API_KEY` (gateway) and `GREENPT_FALLBACK_API_KEY` (agent) | the GreenPT account — the **same value** in both | verified: the "fallback" isolates nothing |
 | `REGOLO_API_KEY` | Regolo account | not yet identified |
 | Gitea token | `inference-bot` | inferred, not verified |
 | `BROKER_SECRET`, `PORTAL_SECRET`, `WEBHOOK_TOKEN`, `ADMIN_TOKEN`, `LOOMIO_*`, `LISTMONK_*` | self-issued / service account | — |
 | Restic passphrase; Infomaniak Swiss Backup S3 keys | self-issued; the Infomaniak account | — |
 The two couplings to watch: **one account with several credentials** (the
 `@inference-admin` Matrix pair, the two Cloudron tokens, the four holders of the
 master key) and **one value with several holders** (the GreenPT key, which is the
 only such case found). Full account map, including the hash comparison behind the
 GreenPT finding, is held with the security working notes.
 ## The two-token split
 There are **two separate** Cloudron API tokens, deliberately decoupled so
@@ -109,8 +188,9 @@ container). The `PYTHONPATH` env survives restores (it's in app config).
 The portal and gateway run their own housekeeping via the Cloudron scheduler
 addon, declared in each app's `CloudronManifest.json` under `addons.scheduler`:
- **portal**: `membership_sweep` (nightly 3am, `/app/code/sweep.sh` →
+- **portal**: `membership_sweep` (three times daily — `0 3,11,19 * * *` UTC per
-  `POST /admin/sweep` with the **`ADMIN_TOKEN`** header — NOT `WEBHOOK_TOKEN`,
+  the live manifest; earlier notes said "nightly at 3am" and are stale. Runs
  `/app/code/sweep.sh` → `POST /admin/sweep` with the **`ADMIN_TOKEN`** header — NOT `WEBHOOK_TOKEN`,
  which returns 403 on `/admin/*` since the 2026-10-28 cutover), `rag_refresh`
  (nightly 3:30am, `/app/code/rag_refresh.py` — pulls docs from the Gitea raw
  API and refreshes the chat knowledge base; needs `OWUI_API_KEY` +
@@ -202,7 +282,7 @@ row that names the same credential: several are held by more than one app.
 | `WEBHOOK_TOKEN` | Secret in the Open Collective webhook URL | Portal app env, and the webhook URL on Open Collective | GM | Rotating means re-creating the webhook on OC as well |
 | `LITELLM_MASTER_KEY` | Create gateway teams and keys for members | Portal app env (copy of the gateway's) | GM | Three holders: portal, gateway, steward agent. Rotate all together |
 | `LITELLM_DB_*` | Read spend logs from the gateway database | Portal app env | Cloudron (addon) | Postgres credentials for the gateway's database |
-| `MATRIX_BOT_TOKEN` | `@inference-bot:inference.coop` admin account: invites, space membership | Portal app env | GM | The account has no password. Paired with the steward agent's `MATRIX_LOCAL_BOT_TOKEN`: same account, separate token on a separate device (verified). Revoke or rotate both together |
+| `MATRIX_BOT_TOKEN` | `@inference-admin:inference.coop` admin account: invites, space membership | Portal app env | GM | The account has no password. Paired with the steward agent's `MATRIX_ADMIN_TOKEN`: same account, separate token on a separate device (verified by homeserver `whoami`, 2026-11-04). Revoke or rotate both together |
 | `LOOMIO_API_KEY`, `LOOMIO_B3_KEY` | Forum (Loomio) membership sync | Portal app env | GM | |
 | `LISTMONK_API_USER`, `LISTMONK_API_TOKEN` | Newsletter list sync | Portal app env | GM | |
 | `OWUI_API_KEY`, `OWUI_JWT_SECRET` | Open WebUI knowledge refresh and SSO | Portal app env | GM | Chat moved to LibreChat in October; confirm whether still needed, and retire if not |
@@ -236,7 +316,7 @@ granularity for the public half. Highest privilege first.
 | Credential | Purpose | Where it lives | Who rotates | Notes |
 |---|---|---|---|---|
 | Gateway master key (`GW_MASTER_KEY`) | Admin over every gateway team, key and spend record | Steward agent `.env` | GM | **Third holder**, with the portal and the gateway. The row most in need of coordinated rotation |
-| `MATRIX_LOCAL_BOT_TOKEN` | The `@inference-bot:inference.coop` automation account: room administration in every co-op room | Steward agent `.env` | GM | Same account as the portal's `MATRIX_BOT_TOKEN`, but a separate token on a separate device (verified). Revoking one does not revoke the other; revoke or rotate both together |
+| `MATRIX_ADMIN_TOKEN` | The `@inference-admin:inference.coop` automation account: room administration in every co-op room | Steward agent `.env` | GM | Same account as the portal's `MATRIX_BOT_TOKEN`, but a separate token on a separate device (verified by homeserver `whoami`, 2026-11-04). Revoking one does not revoke the other; revoke or rotate both together |
 | Cloudron agent token | Cloudron API | Steward agent `.env` | GM | See "The two-token split" above |
 | `REGISTRY_BUILD_TOKEN` (and its username) | Image build and push for the custom apps | Steward agent `.env` | GM | Moves to CI (co-op/support#30); the agent stops holding it then |
 | Gitea token | Commits, issues, PRs as `inference-bot` | Steward agent `.env` | GM | |
@@ -0,0 +1,44 @@
 # Provider pricing
 What our model providers charge, and where the authoritative numbers live.
 ## The authority
 **The gateway's own per-model costs are the authority** — they are what the
 response headers (`x-litellm-response-cost`) and member billing both read. They
 live in the LiteLLM config on the gateway's persistent volume
 (`/app/data/config.yaml`), under each model's `model_info`, as **USD per token**
 (the comments in that file quote per-1M rates). Read them live with
 `GET https://gateway.inference.coop/model/info` (master key), or read the config
 straight from the running app.
 **The member-facing mirror** is `co-op/docs/models.md`. When a rate changes,
 update **both** the gateway config and `models.md` — the config is authoritative,
 `models.md` is the mirror.
 A missing cost field makes a paid feature free; a stale one bills wrong; a missing
 `cache_read_input_token_cost` bills cache hits at the full input rate. All three
 have happened here.
 ## The pages
 | File | Provider | Notes |
 |---|---|---|
 | `greenpt.md` | GreenPT | renewable energy; the default model lives here |
 | `tinfoil.md` | Tinfoil | TEE; the local-proxy architecture lives here |
 | `regolo.md` | Regolo | renewable + ZDR; includes our measured usage |
 | `decision-model.md` | GreenPT "Metis" via `/v1/systemone` | a pass-through **route**, not a chat model |
 ## The routing rule
 Members reach models through one gateway, which fronts all providers. The
 per-request cost the gateway records is the number to trust; do not re-derive a
 price from a provider's public page (they drift, and some are login-walled).
 ## A note on currencies
 Regolo quotes in **EUR**; our gateway figures convert at **EUR × 1.16 → USD**
 (the rate our numbers use — `regolo.md` documents the derivation). GreenPT's own
 listed rates appear in **EUR with the same numerals the gateway carries** —
 whether an EUR→USD conversion is intended there is **unverified**; confirm it
 against the GreenPT invoice before treating the figures as final.
@@ -0,0 +1,58 @@
 # The decision-model route — GreenPT "Metis" (`/v1/systemone`)
 The gateway exposes a **decision model** — GreenPT's **Metis** (System One API) —
 as a **pass-through route**, not a chat model.
 ## What it is, and why it is a route and not a model
 Metis is a **decision** endpoint, not a chat completion: it takes a `state` plus up
 to 32 typed questions and returns **probabilities / choices / scores**. It
 generates no text (`output_tokens` is always 0), so **it cannot be a `model_list`
 entry** — LiteLLM's chat pipeline has nothing to parse. It is therefore exposed as
 a pass-through:
 ```
 path:    /v1/systemone
 target:  https://api.greenpt.ai/v1/systemone
 auth:    true        # members call it with their normal co-op API key
 ```
 - **It is API-only and deliberately NOT a chat-picker model.** It never appears in
  the chat model list. Do not add it there.
 - **Members authenticate with their own co-op key**, never the provider key — the
  route's `Authorization` header injects the provider key server-side.
 - Hosted by GreenPT (policy-based privacy, EU, 100% renewable), same
  `GREENPT_API_KEY` the green chat models use.
 ## Pricing and the metering fix
 - **$0.12 per 1M input tokens; output is free.**
 Metering is **exact, from the token count GreenPT reports back**. Because Metis is
 not a `model_list` entry, it is priced through the
 `coop_passthrough_cost_per_token` map, keyed by the route path
 (`/v1/systemone: 0.00000012`). `patch_passthrough_token_cost.py` (built into the
 image) reads `usage.input_tokens` from the Metis response body at logging time and
 writes the exact figure to member spend.
 **Why this exists — record it, it will recur:** **generic pass-through routes
 originally logged nothing.** A route that is not a `model_list` entry gets no
 automatic pricing, so its calls were metered as $0 — a paid feature silently free.
 The pass-through accounting had to be fixed so spend lands against the **calling
 member's key**. If another pass-through route is ever added, it needs the same
 treatment or it bills nothing.
 `cost_per_request: 0.0001` on the route is now only a **fallback** — it applies if
 a response arrives without a usable token count, so a call is never silently free.
 Keep the two in sync.
 ## Access control gotcha (community edition)
 The route is opened to any authenticated member key via
 `coop_passthrough_open_to_keys` (consumed at build time by
 `patch_passthrough_key_access.py`). **LiteLLM's own gate for `auth=true`
 pass-through routes (`allowed_passthrough_routes`) is an Enterprise-only field**,
 so on this community-edition proxy it can never be granted to member keys. The
 only alternative, `auth: false`, would expose the route publicly and forward the
 co-op's GreenPT key to anonymous callers. **Authentication is unchanged** — the
 endpoint still requires a valid key.
@@ -0,0 +1,44 @@
 # GreenPT pricing
 GreenPT is OpenAI-compatible, EU-hosted on **100% renewable energy**. LiteLLM
 talks to it **directly** (no TEE proxy). Billing across the gateway:
 - `api_base`: `https://api.greenpt.ai/v1`
 - key: `GREENPT_API_KEY` (gateway app env)
 ## What we serve, and what it costs
 Rates as recorded in the gateway's authorititative config comment, **USD per 1M
 tokens** (the gateway's `model_info` carries the per-token equivalents):
 | our model | input | output | cached input |
 |---|---|---|---|
 | `greenpt/deepseek-v4.1-flash` **(DEFAULT model)** | $0.22 | $1.10 | $0.011 |
 | `greenpt/glm-5.3-flash` | $0.11 | $0.44 | $0.022 |
 | `greenpt/glm-5.3` | $1.10 | $4.40 | $0.275 |
 - **`greenpt/deepseek-v4.1-flash` is the gateway's default model.** It is the same
  model family as Tinfoil's `tinfoil/deepseek-v4-1-flash`, but on renewable energy
  and **not** marked "Experimental" by the provider (Tinfoil's counterpart is), so
  it is the default and Tinfoil's is its private counterpart. The policy trade is
  **policy-based privacy (GreenPT) vs TEE (Tinfoil)** — say so plainly rather than
  implying GreenPT is confidential computing.
 - `greenpt/glm-5.3` is the strongest **agentic** model in the green tier (1M context).
 - Note the **dotted slug** `glm-5.3-flash` on GreenPT, distinct from Tinfoil's
  `glm-5-3-flash` (hyphenated). Getting a model slug wrong is a silent 404 at call
  time.
 ## Historical
 - **Removed 2026-10:** `green-l` / `green-r` — GreenPT now marks them "not
  recommended".
 - The config comment previously read "€0.22 in / €1.10 out" alongside the dollar
  figures — see the currency note in `README.md`; the intended FX is unverified.
 ## The fallback key
 `GREENPT_FALLBACK_API_KEY` (agent host) is the **same value** as the gateway's
 `GREENPT_API_KEY` — verified by hash comparison. It powers the steward agent's
 direct-to-provider fallback during a gateway outage. **Rotating one without the
 other silently breaks the fallback** (a dedicated key is the cleaner end state).
 See the token runbook.
@@ -97,3 +97,21 @@ At these volumes the mispricing is small: apertus's input roughly 2.5x-ing on a
 **$0.82 lifetime** is about **$1/month** of exposure, and qwen's cut moves ~$0.6/month in members'
 favour. So this is **billing accuracy, not urgency** — but accuracy still matters, because the same
 numbers feed the cost headers members can now read.
 ## UPDATE 2026-11-04 — the gateway was updated; both stale sections above are now resolved
 Checked against the live gateway config (`/app/data/config.yaml`, `model_info`):
 - `regolo/apertus-70b` = **1.16e-06 / 3.944e-06** per token
  ($1.16 / $3.94 per 1M) — the new 2026-11-01 rate. The "~70 cent/M loss while the
  gateway still bills $0.46" paragraph above no longer applies.
 - `regolo/qwen3.5-122b` = **6.96e-07 / 3.944e-06** ($0.70 / $3.94 per 1M) — the new
  rate. The "over-bills members by ~46 cents/M" paragraph no longer applies.
 - `regolo/qwen-image` now carries **`input_cost_per_pixel: 1.74e-8`** — the EUR
  figure converted at ×1.16, i.e. the "image model is priced in EUR-as-USD" open
  inconsistency above is **fixed**. It is no longer priced at the raw EUR number.
 So the two "Until the gateway is updated" tables and the image-model section above
 are **historical**. The usage measurement ($4.38/month, ~1.1% of inference spend)
 and the subscription analysis stand. Keep this file dated-section-per-change; do
 not delete the history, it shows the reasoning.
@@ -0,0 +1,84 @@
 # Tinfoil pricing and architecture
 Tinfoil serves models inside **TEE-protected enclaves**. The co-op does not
 self-host it — Tinfoil is a hosted cloud service; models run in Tinfoil's TEEs.
 Tinfoil is the co-op's **architecturally-private** tier (contrast: GreenPT and
 Regolo are the renewable tier, with policy-based privacy, not TEE).
 ## The architecture — read this before touching the Tinfoil route
 LiteLLM does **not** talk to Tinfoil's enclave directly. Every Tinfoil model in the
 gateway config points at:
 ```
 api_base: http://127.0.0.1:3301/v1
 ```
 That is a **local proxy** running **inside the gateway container**
 (`/app/data/tinfoil-proxy -b 127.0.0.1 -p 3301`). LiteLLM speaks **plaintext
 OpenAI** to the proxy; the proxy **verifies the enclave attestation and wraps the
 request/response bodies in EHBP (HPKE)** before forwarding to the Tinfoil
 enclave. The provider key is set via the environment variable `TINFOIL_API_KEY`.
 Consequences:
 - **The proxy is load-bearing.** If it is not running, every Tinfoil model fails;
  LiteLLM's own config still looks correct. Check the process, not just the config.
 - **`http://127.0.0.1:3301` is intentional, not a misconfiguration.** Do not
  "fix" it to a public Tinfoil URL — the encryption happens in the proxy.
 - The proxy binary lives on the persistent volume (`/app/data/tinfoil-proxy`), so
  it survives restarts; it is started by the app's `start.sh`.
 ## What we serve, and what it costs
 Rates as recorded in the gateway config, **USD per 1M tokens** unless noted:
 | our model | input | output | cached input |
 |---|---|---|---|
 | `tinfoil/deepseek-v4-1-flash` | $0.65 | $1.45 | $0.13 |
 | `tinfoil/gpt-oss-120b` | $0.15 | $0.60 | — |
 | `tinfoil/glm-5.3` | $1.80 | $5.75 | $0.45 |
 | `tinfoil/whisper-large-v3-turbo` (STT) | $0 / 1M | — | **$0.01 per request** |
 | `tinfoil/voxtral-tts` (TTS) | $0 / 1M | — | **$0.002 per request** |
 | `tinfoil/nomic-embed-text` (embeddings) | $0.05 | $0 | — |
 - **`tinfoil/glm-5.3` replaced `tinfoil/glm-5-3-flash`** (Tinfoil deprecated the
  flash variant on 2026-10-09). Text-only (no vision), reasoning + tool calling.
 - **`tinfoil/deepseek-v4-1-flash` and the glm-flash variant carry an
  "Experimental: may be deprecated with minimal notice" flag from Tinfoil** —
  which is part of why the GreenPT counterpart is the default.
 ### Audio is billed per REQUEST, and needs a build-time patch
 Tinfoil bills the audio models **per request** (its model list carries a
 `requestPrice`), not per token. Reading only the two token prices made TTS look
 free and left STT billed off an older catalog. **LiteLLM cannot price per request
 on the audio paths**, so `patch_audio_request_cost.py` (built into the image via
 the Dockerfile) makes the cost calculator honour `input_cost_per_request`.
 **Re-check that build-log line after every LiteLLM upgrade.**
 One non-obvious load-bearing detail: for TTS, `input_cost_per_character: 0` is
 **required**. Without it the TTS cost branch raises `ValueError("cost for tts call
 is None")` and aborts the whole calculation, so the spend write is skipped and
 speech bills nothing — whatever `input_cost_per_request` says.
 **Audio keep-alive** (the Cloudron scheduler task `audio_keepalive`, every 10 min)
 pings the Tinfoil audio models so voice mode does not pay the ~24s TEE cold-start
 after idle. It is **TTS-only by design**: the STT warm ping was disabled
 2026-10-11 because it costs $0.01/request — roughly $52/month at 6 pings/hour.
 `docs.inference.coop`-style "keep-alive economics" detail is in the script.
 ### The embedding model is operator-only
 `tinfoil/nomic-embed-text` (768-dim) is **not** a member-facing model — it is
 scoped to the co-op's own admin team (`model_info.team_id`) for the docs-RAG, so
 member keys cannot call it. The portal also filters it out of the chat-facing
 model list by its `mode: embedding`.
 ## Fallbacks
 The router treats `tinfoil/deepseek-v4-1-flash` as the fallback for
 `greenpt/deepseek-v4.1-flash`, and `tinfoil/gpt-oss-120b` as the fallback for
 `tinfoil/deepseek-v4-1-flash`. The timeout is deliberately generous (180s) with
 `TimeoutErrorRetries: 0` — DeepSeek generations routinely exceed 30s, and a
 timed-out generation is **already billed** by Tinfoil, so re-hitting it re-bills.
@@ -0,0 +1,156 @@
 #!/usr/bin/env python3
 """Rotation-due watchdog for the co-op credential rotation policy.
 Reads the `last rotated` table from the operator token runbook and prints a
 reminder ONLY when a credential is due -- i.e. within 30 days of its due date
 (last rotation + 12 months), or already past it. Prints NOTHING when nothing is
 due, so it can drive a watchdog cron where empty output means "no message".
 Deterministic: the output depends only on the table and the current date. It
 prints no timestamps of its own, so when nothing is due it produces no output at
 all and stays silent across runs.
 Table source (first that applies):
  1. --file PATH
  2. $ROTATION_RUNBOOK
  3. the runbook fetched from Gitea's main branch (authoritative, always current)
  4. the local clone ~/repos/ops-docs/operator-token-runbook.md
 Exit codes: 0 ok (including "nothing due", no output), 2 could not read the
 runbook or found no dated rows -- a loud failure so a watcher is not silently
 blind.
 """
 import argparse
 import calendar
 import datetime as dt
 import os
 import re
 import sys
 import urllib.request
 RUNBOOK_URL = "https://git.inference.coop/co-op/ops-docs/raw/branch/main/operator-token-runbook.md"
 LOCAL_FALLBACK = os.path.expanduser("~/repos/ops-docs/operator-token-runbook.md")
 DEFAULT_WITHIN_DAYS = 30
 SECTION_RE = re.compile(r"^#{1,6}\s+.*last\s+rotated.*$", re.IGNORECASE)
 HEADING_RE = re.compile(r"^#{1,6}\s+")
 SEP_RE = re.compile(r"[\s|:\-]+")
 DATE_RE = re.compile(r"\b(\d{4})-(\d{2})-(\d{2})\b")
 def fetch_text(args):
    if args.file:
        try:
            with open(args.file, encoding="utf-8") as fh:
                return fh.read()
        except OSError as exc:
            print(f"rotation_due: could not read --file {args.file} ({exc})", file=sys.stderr)
            raise SystemExit(2)
    env = os.environ.get("ROTATION_RUNBOOK")
    if env and os.path.exists(env):
        try:
            with open(env, encoding="utf-8") as fh:
                return fh.read()
        except OSError as exc:
            print(f"rotation_due: could not read $ROTATION_RUNBOOK {env} ({exc})", file=sys.stderr)
            raise SystemExit(2)
    try:
        req = urllib.request.Request(RUNBOOK_URL, headers={"User-Agent": "rotation_due/1.0"})
        with urllib.request.urlopen(req, timeout=20) as resp:
            return resp.read().decode("utf-8")
    except Exception as exc:  # noqa: BLE001 - fall back, then fail loudly
        if os.path.exists(LOCAL_FALLBACK):
            with open(LOCAL_FALLBACK, encoding="utf-8") as fh:
                return fh.read()
        print(
            f"rotation_due: could not read the runbook from {RUNBOOK_URL} ({exc}) "
            f"and no local copy at {LOCAL_FALLBACK}",
            file=sys.stderr,
        )
        raise SystemExit(2)
 def extract_section(text):
    """Return the lines of the first 'last rotated' section."""
    out, inside = [], False
    for line in text.splitlines():
        if SECTION_RE.match(line):
            inside = True
            continue
        if inside and HEADING_RE.match(line):
            break
        if inside:
            out.append(line)
    return out
 def parse_rows(section_lines):
    """Yield (name, last_rotated_date) for table rows that carry a date."""
    for line in section_lines:
        if not line.lstrip().startswith("|"):
            continue
        stripped = line.strip().strip("|")
        if not stripped or SEP_RE.fullmatch(line.strip()):
            continue
        cells = [c.strip() for c in stripped.split("|")]
        if not cells:
            continue
        match = DATE_RE.search(line)
        if not match:
            continue
        year, month, day = (int(g) for g in match.groups())
        try:
            last = dt.date(year, month, day)
        except ValueError:
            continue
        yield cells[0], last
 def add_months(date, months):
    month_index = date.month - 1 + months
    year = date.year + month_index // 12
    month = month_index % 12 + 1
    day = min(date.day, calendar.monthrange(year, month)[1])
    return dt.date(year, month, day)
 def main():
    ap = argparse.ArgumentParser(description="Remind when a credential rotation is due.")
    ap.add_argument("--file", help="read the runbook from this path instead of fetching it")
    ap.add_argument("--within-days", type=int, default=DEFAULT_WITHIN_DAYS,
                    help="days before the due date to start reminding (default 30)")
    ap.add_argument("--as-of", help="evaluate as of this date (YYYY-MM-DD); default today UTC")
    args = ap.parse_args()
    section = extract_section(fetch_text(args))
    if not section:
        print("rotation_due: no 'last rotated' section found in the runbook", file=sys.stderr)
        return 2
    parsed = list(parse_rows(section))
    if not parsed:
        print("rotation_due: the 'last rotated' section has no rows carrying a date", file=sys.stderr)
        return 2
    as_of = dt.date.fromisoformat(args.as_of) if args.as_of else dt.datetime.now(dt.timezone.utc).date()
    due = []
    for name, last in parsed:
        due_date = add_months(last, 12)
        days = (due_date - as_of).days
        if days <= args.within_days:
            due.append((days, name, due_date))
    if not due:
        return 0  # nothing due -> no output at all
    due.sort()
    for days, name, due_date in due:
        if days < 0:
            print(f"credential rotation OVERDUE: {name} -- was due {due_date.isoformat()} ({-days} days ago)")
        else:
            print(f"credential rotation due soon: {name} -- due {due_date.isoformat()} (in {days} days)")
    return 0
 if __name__ == "__main__":
    sys.exit(main())