upgrade window 2026-11-09: executed (gateway 1.108.3, chat overlay 1.4.8) — record the outcome, the 400 MB task-limit OOM and its --no-backup remedy, and refresh the live image pins

runbook: re-check the upgrade window against live state — correct the rollback tag, mark the prebuilt candidate stale, extend the marker list to ten, and flag the failing backups as the open risk
ops: document the member provisioning boundary (org-backed, credit-pack, manual, hidden-email) and the daily paid-but-unprovisioned watchdog
2026-11-09 05:04:27 +00:00 · 2026-11-09 04:17:06 +00:00 · 2026-11-07 00:27:15 +00:00 · 2026-11-06 04:11:11 +00:00 · 2026-11-06 04:11:01 +00:00 · 2026-11-05 19:10:26 +00:00
19 changed files with 1467 additions and 3 deletions
@@ -24,11 +24,17 @@ Read it on the web here: <https://git.inference.coop/co-op/ops-docs>
 | Path | Contents |
 |---|---|
 | `operator-token-runbook.md` | The Cloudron token model (agent vs portal, admin vs webhook), how to rotate each without breaking the stack |
 | `app-inventory.md` | Every Cloudron app: id, fqdn, registry image path and pin (read from the running apps) |
 | `agent-fleet.md` | The three agents (steward, infrastructure-bot, socialbot): Unix users, gateway units, teams, and the planned member-bot tier |
 | `maintenance/chat-task-models.md` | The small pinned model used for chat titles/tags/suggestions |
 | `maintenance/librechat-sandbox.md` | Operating the LibreChat evaluation instance (config, overlay image, key sync, pitfalls) |
 | `maintenance/backups.md` | The two backup systems (agent-host restic + Cloudron app backup): schedules, retention, the cache split and the grouping fix |
 | `maintenance/woodpecker-ci.md` | The self-hosted CI server and agent: build/pin, manual-only trigger, git-lfs, and its not-yet-green state |
 | `maintenance/provisioning-boundary.md` | Which Open Collective contributors auto-provision as members and which are deliberately excluded (org-backed, credit-pack, manual, hidden-email), plus the daily paid-but-unprovisioned watchdog |
 | `maintenance/2026-11-08-litellm-1.108.0-librechat-1.4.8.md` | The Sunday-evening upgrade-window runbook (gateway → 1.108.0, chat overlay → appstore 1.4.8): candidates, order, exact commands, per-step verification, rollback, window estimate |
 | `incidents/` | Post-incident write-ups, one file per incident |
 | `migration/` | The Cloudron server-migration record: the plan, the DNS change sheet, and the captured pre-migration state |
-| `provider-pricing/` | What our model providers charge, and when it changed |
+| `provider-pricing/` | What each model provider charges, and when it changed (GreenPT, Tinfoil, Regolo, and the decision-model route) |
 ## The migration material
@@ -52,6 +58,21 @@ answers proves nothing about whether a record exists. At least two listed
 records turned out not to exist, and the real record inventory is only visible
 in the registrar's own list.
 ## Known open items — recorded, not hidden
 These are genuinely unresolved; a tidy document that omits them would be wrong.
 - **The old box `179.199.135.72` is still alive and internet-exposed** (22/80/443/587/993
  open), holding old data and config. Nothing in DNS points at it, so it serves no
  member traffic — but it needs an operator decision: **shut it down or firewall it**.
  Do not act on it without the operator's word. See `migration/cloudron-server-migration.md` (addendum).
 - **CI is not green.** The Woodpecker agent is healthy but the server answers its
  job-collection stream with HTTP 500, so no pipeline can run. See `maintenance/woodpecker-ci.md`.
 - **The CI agent image source (`/home/ubuntu/repos/woodpecker-agent`) has no git
  remote** — it exists only on the agent host.
 - **Member mailbox contents after the migration** are still unconfirmed (needs a
  real IMAP login).
 ## No secrets here
 Audited 2026-11-01, including the full commit history: no credentials, tokens,
@@ -0,0 +1,95 @@
 # The agent fleet
 The co-op runs **more than one agent**, each as its **own Unix user** on the agent
 host (`ov-0f018e`, `179.237.103.165`) with its **own Hermes gateway unit** and its
 **own LiteLLM team**. Any document that describes "the agent" as a single thing is
 now wrong.
 ## The three agents today
 | Agent | Unix user | systemd unit | Gateway team (budget) | Model | Presence |
 |---|---|---|---|---|---|
 | **Inferencebot** (steward) | `ubuntu` | `hermes-gateway.service` | *(co-op account team)* | per `~/.hermes/config.yaml` | Matrix `@inferencebot:inference.coop`; digests, newsletters, membership/key plumbing |
 | **infrastructure-bot** | `infrabot` | `hermes-gateway-infrabot.service` | `agent:infrastructure-bot` ($5/30d) | `greenpt/glm-5.3-flash` | Matrix `#infrastructure` (answers only when mentioned); infrastructure scope |
 | **socialbot** | `socialbot` | `hermes-gateway-socialbot.service` | `agent:socialbot` ($5/30d) | `greenpt/glm-5.3-flash` | Matrix `#social-drafts`; drafts the co-op's AT Protocol (Bluesky) account `inference.coop` |
 All three gateways run on the one host; each is a separate process, a separate
 `.env`, and a separate set of rooms. There is **one LiteLLM gateway** (the Cloudron
 app) that all three draw on — the agent host runs no gateway of its own.
 ### Who is who (do not conflate them)
 - **Inferencebot** is the **steward** and the fleet administrator: full Cloudron
  token, gateway master key, member-data plumbing. It is NOT a member-facing chat
  persona.
 - **infrastructure-bot** is a **lower-privileged** infrastructure agent. Per its
  own SOUL it has **no sudo, no Cloudron admin token, and no member data**; when a
  task needs more privilege it hands up to the steward rather than working around
  it. Its gateway key has a small monthly budget and a flash-tier allowlist.
 - **socialbot** has **no Cloudron, portal or member-data access** and no power over
  Matrix rooms; its only writable surface is the `#social-drafts` room, and it
  **never publishes an unapproved draft** — silence is its correct failure mode.
 ## Member bots — TIER 2 (planned, not yet provisioned)
 The intended path for **member-facing** bots is a second tier: each member bot is a
 **Hermes profile under the `infrabot` Unix user**, sharing **one** gateway process
 (the infrabot one), with its own gateway team/key. This is tracked as **co-op/support#33**.
 **Status (2026-11-04): not yet provisioned.** `/home/infrabot/.hermes/profiles/`
 holds only a `.deleted` directory with two probe files — there are **no active
 member-bot profiles**. Record this honestly: the design exists, the instances do
 not.
 ## Why one Unix user per agent — and the limit of it
 **Profile isolation is organizational, not security.** Run one OS user (or a
 rootless container) per agent from day one: **on a shared Unix user, sibling
 Hermes profiles can read each other's `.env`.** So tier-2 member bots sharing the
 `infrabot` user are isolated from *members* but **not from each other** — that
 sharing is accepted for member bots, not extended to the privileged agents.
 ## Backup coverage is automatic — but prove it
 The nightly restic run (see `maintenance/backups.md`) enumerates `/home/*` **as
 root** and stages every agent home, so a new agent is covered without editing any
 list. Coverage is still **unproven until that agent's files show up in a
 snapshot**.
 ## Gateway teams (member vs agent)
 From `GET https://gateway.inference.coop/team/list` (2026-11-04): 97 teams — one
 `member:<email>` team per member (mostly $15/30d), `member:info@inference.coop`
 ($135/30d, the co-op account), the two `agent:*` teams above, and
 `sandbox:librechat-test` ($3). Agent teams carry a **small budget and a flash-tier
 model allowlist** enforced at the gateway.
 ## Matrix homeserver accounts
 Beyond the agent bots above, the homeserver carries two automation accounts.
 Also recorded here: the room account retired in October 2026, moved out of the
 member-facing docs.
 - **`@inference-admin:inference.coop` — the room and server administrator.** A
  Synapse server admin, used for provisioning rooms, managing membership, and
  retiring rooms. It is the administrator of record for the co-op's Matrix rooms
  — general, community, design, skillshare, updates and infrastructure — and
  holds the portal's member-invite credential (`MATRIX_BOT_TOKEN`). It **never
  speaks in a member room**; a message from it is a mistake, not a feature.
 - **`@inference-bot:inference.coop` is retired (2026-10-29).** It was the
  original room-administration account (it created the space and most of the
  rooms) and held a Synapse server-admin flag. Its room power passed to
  `@inference-admin`; its admin flag was dropped, it was deactivated, and every
  device and token was deleted. Nothing should use it again.
 - **`@ops_audit:inference.coop`** — a server-admin account created for the
  October 2026 security review. Retained only until its tokens are revoked.
 Two loose ends, recorded so nobody re-investigates them from scratch: the
 retired account remains joined to two rooms it cannot leave (they reject every
 write — `500` on leave), and those rooms are also why they can't be deleted; and
 its *name* is still the tempting one, so check `whoami` rather than the name
 before trusting any account that looks like it.
 **House rule: power belongs to `@inference-admin`, voice belongs to `@inferencebot`.**
 Don't create an account whose name differs from an existing one by a character or
 two — that lookalike confusion is what the rule exists to undo.
@@ -0,0 +1,82 @@
 # Cloudron app inventory
 The co-op's Cloudron server carries **19 apps**. Before 2026-11-02 it carried 16;
 the three added since the migration are the docs site, the CI server and the CI
 agent (marked **NEW** below).
 **Source of truth:** read it from the running apps, not from memory or a sibling
 deploy. `GET https://my.inference.coop/api/v1/apps` returns every app with its
 `id`, `fqdn`, install state and — inside `manifest` — `dockerImage`. The app
 record's own top-level `image` field is **null**; the CLI's `list` output does
 not carry the image at all. The registry paths are **per-app and not uniform**
 (there are three different namespaces in use — see below), and a path copied from
 a remembered deploy of a *different* app points the update at a repository that
 does not hold it.
 *Verified against the live API 2026-11-04.*
 ## The 19 apps
 | FQDN | App id | Label | Image (pin) | Install |
 |---|---|---|---|---|
 | `inference.coop` | `55f5f904-3233-44de-ad7b-5dc7c8533b75` | — | `cloudron/io.cloudron.surfer:202611021342220000` | store (Surfer — marketing site) |
 | `chat.inference.coop` | `4023ccbf-d57a-4500-97df-14de9cf3516e` | AI Chat | `registry.inference.coop/co-op/librechat-custom:20261106-c2aec74` | **custom** (LibreChat overlay) |
 | `chat-legacy.inference.coop` | `50ae6c97-e851-4d17-b013-51ca7a4e5be0` | Legacy AI Chat | `cloudron/com.openwebui.cloudronapp:202611040420410000` | store (Open WebUI) |
 | `portal.inference.coop` | `d4f05e4b-75c6-492e-9faf-7afb0712fc94` | — | `registry.inference.coop/code/member-portal:20261108-2954fe60` | **custom** |
 | `dashboard.inference.coop` | `daf8750e-2683-4104-83ea-f77d1b4601e8` | Dashboard | `registry.inference.coop/code/member-dashboard:20261108-1e6e9e48` | **custom** |
 | `panel.inference.coop` | `ae187071-7a43-4a6e-8a02-986609a66a1a` | Admin Panel | `registry.inference.coop/code/admin-panel:20261108-b23cc67` | **custom** |
 | `gateway.inference.coop` | `adababd4-2427-4bdd-9bf7-6bdad0302610` | LiteLLM | `registry.inference.coop/local/ai.coop.litellm:20261109-c43d45c` | **custom** |
 | `matrix.inference.coop` | `aa6f4597-6206-4509-b32e-c89750b2c1d8` | Matrix Chat | `cloudron/org.matrix.synapse:202611030719520000` | store (Synapse) |
 | `git.inference.coop` | `c91c13aa-30ed-4af9-9599-7a87ab4f221d` | Code | `cloudron/io.gitea.cloudronapp:202611011719500000` | store (Gitea) |
 | `registry.inference.coop` | `c257ba3c-6737-4375-ad51-ea19586bf3f0` | — | `cloudron/io.cloudron.container-registry:202610221511140000` | store (container registry) |
 | `vault.inference.coop` | `cb838bcf-f2e2-4670-b117-dca04b0ee957` | — | `cloudron/com.github.bitwardenrs:202611020820270000` | store (Vaultwarden) |
 | `forum.inference.coop` | `8b1a972c-08af-4e92-852c-d573c37b01ad` | Forum | `cloudron/org.loomio.cloudronapp:202611050321160000` | store (Loomio) |
 | `newsletter.inference.coop` | `eed2217c-b165-430e-bc29-7ccfef9b11e4` | — | `cloudron/app.listmonk.cloudronapp:202610220221130000` | store (Listmonk) |
 | `search.inference.coop` | `d7c73fff-ee42-4db0-98c0-4ce7a8ab35cc` | — | `cloudron/io.github.ascimoo.searx:202611012319290000` | store (SearXNG) |
 | `cloud.inference.coop` | `fd261d67-b790-424f-9e82-f74883ee8585` | — | `cloudron/com.nextcloud.cloudronapp:202610211606270000` | store (Nextcloud) |
 | `email.inference.coop` | `fbb256b9-ead3-41d8-bee0-d5ea87149436` | — | `cloudron/io.cloudron.mail:202610261602550000` | store (Cloudron mail UI) |
 | `docs.inference.coop` | `85b7ce06-e2af-49a4-bd4f-a9c2e692e3ab` | Docs | `cloudron/io.cloudron.surfer:202611021342220000` | store (Surfer) — **NEW 2026-11-03** |
 | `ci.inference.coop` | `4984f95e-0c0a-4af3-9ce1-4ca4f3e8483d` | Woodpecker | `cloudron/org.woodpecker_ci.cloudronapp:202610300219490000` | store (Woodpecker server) — **NEW 2026-11-03** |
 | `ci-agent.inference.coop` | `5ac63873-cf2c-4ad9-acd6-81dc0e016c83` | — | `registry.inference.coop/co-op/woodpecker-agent:20261104-e070275` | **custom** (Woodpecker agent) — **NEW 2026-11-03** |
 (`my.inference.coop` is the Cloudron dashboard hostname — the platform itself, not
 an app object — so it does not appear here.)
 ## Registry namespaces — they are NOT uniform
 Five custom apps pull from four different namespaces. Deriving one path from
 another app's pattern deploys the wrong (or a missing) image:
 | App | Registry path |
 |---|---|
 | chat (LibreChat overlay) | `registry.inference.coop/co-op/librechat-custom` |
 | CI agent | `registry.inference.coop/co-op/woodpecker-agent` |
 | member portal | `registry.inference.coop/code/member-portal` |
 | member dashboard | `registry.inference.coop/code/member-dashboard` |
 | gateway (LiteLLM) | `registry.inference.coop/local/ai.coop.litellm` |
 | admin panel | `local/coop.inference.admin-panel` |
 **Correcting a stale memory:** the portal is `code/member-portal` (not
 `co-op/member-portal`), and the admin panel is `local/coop.inference.admin-panel`
 (not `code/admin-panel`). Both were cited wrongly in earlier notes; the values
 above are read from the running apps.
 ## Store vs custom, and what that decides
 Six apps are **custom** installs (packaged by us, `appStoreId == ""`): chat,
 CI agent, portal, dashboard, panel, gateway. Custom apps get **no automatic
 update** (Cloudron says "not available for custom apps") — their protection is the
 `manifest.dockerImage` pin, which an update deploys, so a custom app must never be
 pinned to a floating tag. The rest carry a real `appStoreId` and update through
 the store.
 ## Notes that are easy to get wrong
 - **`enableAutomaticUpdate` exists on every app object and means nothing for a
  custom one.** Judge update behaviour by the install SOURCE (`appStoreId`), not
  by that flag.
 - **Public collectives reach `co-op/*` and `code/*` repos; the only private repos
  are `ops/inferencebot` and `ops/security`.**
 - **`info@inference.coop` is the co-op's gateway team account** (`member:info@inference.coop`,
  budget $135/30d) and also the owner account of several provider keys — see the
  token runbook.
@@ -0,0 +1,26 @@
 # Allowance shortfall remediation — manual-remediation ledger (2026-11-04/05)
 The portal's shortfall mechanism (`derive_shortfalls` in
 `member-portal/app/main.py`) credits a member's missing allowance whenever OC
 realigned a charge onto the collective billing day and truncated a paid cycle.
 Each credit is keyed on the OC charge reference (`tx_…` publicId) so a re-run
 never credits twice.
 ## Entries in `SHORTFALL_REMEDIATED` (code constant, no PII by design)
 | OC charge reference | Why it is excluded from automatic shortfall crediting |
 |---|---|
 | `[redacted member reference]` | The truncated cycle this charge created was already made whole by a **manual allowance reset on 2026-11-04**, performed outside the mechanism (before the automatic path shipped). The member joined mid-October; OC moved their next charge to the 2 Nov common billing day, so their cycle ran ~17 days. The reset re-anchored the allowance to the member's own join day, which is the policy now in code. Later charges of this member form new cycles and are credited normally by the automatic path. |
 The member's identity is in the portal `credit_ledger` / `members` tables
 (the credit row itself) — the public portal repository deliberately carries
 only the charge reference (audit H2, 2026-11-05: the constant used to embed
 the member's email and billing timeline; the portal repo is public).
 ## Why the exclusion is keyed on the charge, not the member
 `derive_shortfalls` reads charge history from OC's order list; the charge that
 truncated the cycle is the idempotence key everywhere else in the flow
 (`credit_reference_exists`). Keying the manual-remediation exclusion on the
 same charge means a sweep re-run can neither double-credit the charge nor
 skip a genuinely new truncated cycle for the same member.
@@ -0,0 +1,95 @@
 # Incident 2026-11-05 — gateway/panel briefly deployed with the portal's manifest; gateway LiteLLM database wiped and restored
 Self-reported by the audit-fix session (inferencebot), 2026-11-05 ~18:00 UTC.
 Damage fully recovered the same hour. Nathan: see "What you need to know" at
 the end.
 ## What happened
 During the audit fix-order deploys, three apps (member-portal, admin-panel,
 gateway) were updated in one shell loop that stayed `cd`'d into the
 **member-portal repo**. `cloudron update` reads `CloudronManifest.json` from
 the CURRENT DIRECTORY, and the portal manifest carries a `dockerImage` pin —
 which wins over any `--image` argument. So:
 1. The **admin-panel** app received the portal's manifest (portal image
   `20261105-ca3b8db`, portal manifest id `coop.inference.member-portal`,
   portal addons) for ~10 minutes.
 2. The **gateway** app received the portal's manifest too. The portal manifest
   has **no postgresql addon** → Cloudron **dropped the gateway's Postgres
   database** (the LiteLLM key/team/spend store). The panel recovery and the
   gateway image recovery ran ~10 and ~25 minutes later; the gateway's addon
   was re-created on recovery as an EMPTY database with the same name.
 3. Result: from ~16:29 to ~17:20 UTC every member key 401'd (184 keys gone
   from the gateway's view), chat was down for the membership. The portal app
   itself was never affected (its own manifest was the one deployed).
 ## Impact
 - Gateway member inference: **down ~50 minutes** (16:29–17:20 UTC). Master-key
  surfaces (/v1/models with the master key) worked, which made the outage look
  partial at first glance.
 - Gateway LiteLLM Postgres: keys 184→0, teams 100→1, SpendLogs 125k→17 at the
  moment of detection.
 - Spend rows for live traffic between the 16:28:12 backup and the database
  wipe (a few minutes) are unrecoverable; everything after the wipe was
  refused calls (no spend to lose).
 - Panel: wrong image for ~10 min, no data loss (stateless; no durable addon
  data).
 - No member data (portal SQLite) was ever at risk — that lives in the portal
  app's own localstorage volume, which was never touched.
 ## Detection
 Post-deploy verification battery failed exactly where it should: the
 member-scope chat call returned 401, the panel's container md5 matched the
 PORTAL's main.py, and `/key/list` returned 0 keys. All three surfaced within
 minutes of the recovery deploy.
 ## Resolution (all verified)
 1. Redeployed panel and gateway **from their own repo directories** with their
   correct manifests/images (16:47, 16:52 UTC).
 2. Gateway database confirmed wiped (0 keys) → located the deploy-time app
   snapshot `app_…_v1.0.0_5aad8953` (16:28:12 UTC — taken seconds before the
   damage, contains the addon dump).
 3. `cloudron restore --app <gateway> --backup …_5aad8953` (17:2x UTC) →
   **184 keys, 100 teams, 125,106 spend rows restored**; /v1/models 200.
 4. Re-applied the intended gateway image update `1.0.0-20261105-e161db9` from
   the litellm repo dir (17:3x UTC) → verified DB still intact after the
   redeploy (184/100/125,110 rows — the +4 spend rows are live traffic billing
   again), member-scope chat completes for real (tinfoil/glm-5.3, tokens
   billed), contactEmail + corrected start.sh live, audio_keepalive scheduler
   restored.
 5. Panel pin bumped to its own new image `20261105-0713551` and deployed from
   its own repo dir; container md5 matches the repo; panel 200.
 ## Root cause
 Running `cloudron update` for multiple apps from a shared shell whose working
 directory belonged to a DIFFERENT app's repo. The CLI's behavior (CWD manifest
 wins; `--image` silently ignored when a `dockerImage` pin exists) is documented
 in the co-op skill, but nothing in the loop enforced "one app, one repo, one
 CWD".
 ## Prevention (recorded in the deploy skill as a hard rule)
 **Never batch `cloudron update` for different apps in one shell without
 `cd`-ing to each app's own repo per command, and never run an update for an
 app while CWD holds another app's `CloudronManifest.json`.** Verify after
 EVERY deploy with both facts: the app's own `manifest.dockerImage` from the
 API (never the CLI's output) and a container-content check (md5/marker).
 Additionally: a manifest change that alters the `addons` block of an
 app with a database addon drops/reprovisions that addon's data — treat addon
 blocks as data-bearing, not metadata.
 ## What you need to know (Nathan)
 - Chat was down for members ~50 minutes this afternoon (UTC); it is verified
  working again with a real member key and real billed tokens.
 - The gateway database was restored from the automatic pre-deploy snapshot;
  only a few minutes' worth of spend rows from before the outage are missing.
  No member key, team, or balance data was lost.
 - The portal's member database was never touched.
 - The audit fix order otherwise completed; a separate summary has the full
  stage-by-stage record.
@@ -0,0 +1,100 @@
 # Incident 2026-11-05 — the portal's copied LiteLLM DB credential rotted after the gateway restore; all-time spend silently read $0
 Companion to [`2026-11-05-gateway-addon-loss.md`](2026-11-05-gateway-addon-loss.md)
 (the deploy mistake that made the gateway restore necessary). This document
 covers the second-order failure that restore caused. No spend data was lost at
 any point; the failure was in the portal's **read path**, and it failed in a
 way that looked like clean data.
 ## What broke
 The portal aggregates all-time spend, model usage and the leaderboard by
 querying LiteLLM's Postgres directly, using a **copy** of the gateway's
 postgres addon credential: the portal app's `LITELLM_DB_*` env mirrors the
 gateway's `CLOUDRON_POSTGRESQL_*` runtime env. Two things then combined:
 1. **The copy rotted.** Restoring a Cloudron app re-provisions its postgres
   addon and **rotates its credential**. When the gateway was restored
   (16:32:35 UTC, see timeline), the gateway got the new credential re-injected
   at runtime — but the portal's copy went stale at that exact moment.
 2. **The failure degraded silently.** The portal's aggregation catches a DB
   auth failure, logs a WARNING, and degrades to an empty result — all-time
   spend read `$0.00`. The panel and member dashboard rendered that $0 while
   the *cycle* figures (read from LiteLLM's API, unaffected) still looked
   plausible, so nothing on screen looked broken. The daily digest had no
   spend-consistency check, so it also reported nothing.
 ## Timeline (UTC, 2026-11-05 — restore/configure times from the platform eventlog)
 - **16:29:01** — during the audit-fix deploys, the gateway received the
  portal's manifest; its postgres addon was dropped. (Sibling incident.)
 - **16:32:35** — the gateway was restored from the pre-deploy snapshot to
  recover the sibling incident. The restore re-provisioned the postgres addon
  and **rotated its credential** — the moment this incident began.
 - **16:32:35 → 18:51** — every portal aggregation query against LiteLLM's
  Postgres failed authentication and degraded to empty: panel + dashboard
  all-time spend read $0. LiteLLM itself kept serving and billing throughout;
  the portal log carried only WARNING lines ("all-time spend query failed"),
  easy to read as noise.
 - **18:51:22** — the portal's `LITELLM_DB_PASSWORD` was re-synced from the
  gateway's runtime env (value staged file→argv through the Cloudron API;
  never printed, never committed). Aggregation restored immediately.
 - **Same day, verification:** the source of truth (LiteLLM's own Postgres, read
  gateway-side) held 125,385 rows / **$655.13** all-time; member teams 97 /
  $652.88 — **no gap through the incident window**, because the gateway's own
  writes were never affected. The member dashboard was verified restored (a
  sample member's all-time figure matched the source exactly).
 - **Same day, panel verification (final):** panel running image
  `code/admin-panel:20261105-0713551` (matches the repo pin; code marker
  confirmed in the container). Its data path is `GET /api/overview` → portal
  `GET /admin/overview` (server-side, `X-Admin-Token: ADMIN_TOKEN`) — replayed
  live from the panel container, HTTP 200. Displayed all-time total
  **$653.08** with the co-op's own-accounts split **$170.73**, both matching an
  independent SQL read of the source exactly; four sample members' all-time
  figures matched the source to the cent. The full-table total ($655.33–655.41
  while verifying — the ledger accrues live) decomposes exactly: members +
  operators $653.08 + agent teams $1.18 (shown in the panel's Agents section) +
  sandbox/unattributed rows $1.07 (displayed nowhere, accepted knowingly).
 ## Fix
 1. **Immediate:** re-sync the portal's `LITELLM_DB_PASSWORD` from the gateway's
   runtime env (18:51:22). Verified by re-reading the portal's aggregation
   against the source, not by the absence of log lines.
 2. **Standing guard (added the same day):** the daily digest
   (cron job `95e5b8fece86`) gained a **SPEND CONSISTENCY** section: it runs
   `~/.hermes/scripts/spend_consistency_check.py`, which compares the portal's
   reported all-time spend (`/admin/overview` `totals.total_all_time_spend` +
   the `agents[]` rows) against LiteLLM's own SpendLogs total — read from the
   **gateway** app, the credential owner, so the source does not depend on the
   copied credential. A $0 portal figure, a missing field, a mismatch beyond a
   $5 tolerance (the deliberately-unattributed sandbox rows are ~$1), or an
   unreadable source is a FAILED check reported near the top of the digest —
   never as clean. All three failure branches were exercised (unreadable / $0 /
   mismatch) and fire with exit 1; the live comparison passes with both numbers
   printed. Do not substitute LiteLLM's `/global/spend` for the source: it
   returns a ~30-day figure (~$576 when the table holds ~$655), not all-time.
 3. **Practice (recorded here because it bit):** after any `app.restore` on this
   box, diff the credential **fingerprint** (`sha256`[:8] of the value — never
   the value) between the owner app and every app holding a copy, and re-sync
   any that differ.
 ## Architectural dependency — read before rotating anything
 **The portal does not hold its own LiteLLM DB credential. It aggregates from
 LiteLLM's Postgres using a COPY of the gateway's addon credential.** They are
 two copies of one secret with no shared source, and Cloudron rotates the addon
 secret on every restore (and on any addon re-provision). Therefore:
 - **A gateway DB credential rotation or restore MUST be followed by syncing the
  portal's copy.** The rotation itself announces nothing; the only symptoms are
  the portal's WARNING lines and $0 figures downstream.
 - **Long-term fix (flagged, deliberately not done now):** both apps should
  source the credential from ONE place — a shared secret location — or the
  portal should not hold a DB credential at all (e.g. the gateway exposes a
  read-only aggregation endpoint the portal calls). Either removes the
  two-copies-of-one-secret failure mode.
 - Until that lands, the digest's SPEND CONSISTENCY check is the drift alarm:
  if it ever reports FAILED with a $0 or mismatched portal figure, check the
  portal's `LITELLM_DB_PASSWORD` against the gateway's current addon credential
  first, before theorising about code.
@@ -0,0 +1,358 @@
 # Upgrade window — 2026-11-08 (Sunday evening): LiteLLM 1.103.2 → 1.108.0, LibreChat overlay 1.4.5 → 1.4.8
 Both upgrades were **prepared and validated in advance** (2026-11-06); nothing has
 been deployed. This runbook is written to be followed by any operator during the
 member-watched maintenance window. Read the gotchas in §6 before touching anything —
 this app pair has a history of deploys that look successful while changing nothing.
 **Candidates (already built and validated, sitting in the registry):**
 | App | Candidate | Rollback (currently deployed) |
 |---|---|---|
 | LiteLLM gateway (`adababd4-2427-4bdd-9bf7-6bdad0302610`, gateway.inference.coop) | branch `upgrade-v1.108.0` in `code/litellm`; prebuilt image `registry.inference.coop/local/ai.coop.litellm:1.0.0-20261106-9e25578` (STALE — see §0 and §3.1) | image `registry.inference.coop/local/ai.coop.litellm:20261107-201000-407e751e2` — the tag running tonight, verified present in the registry. `1.0.0-20261105-e161db9` is two deploys older and is NOT the rollback target |
 | LibreChat chat (`4023ccbf-d57a-4500-97df-14de9cf3516e`, chat.inference.coop) | branch `bump-appstore-1.4.8` in `co-op/librechat-custom`; prebuilt overlay `registry.inference.coop/co-op/librechat-custom:20261106-c2aec74` | overlay `registry.inference.coop/co-op/librechat-custom:20261031-2a98892` (verified present) |
 **Prep evidence (what was already verified, so Sunday only re-confirms it):**
 - LiteLLM 1.108.0 candidate: all 8 build patches verified inside the running candidate
  (file markers + build-log markers, zero WARNs); litellm reports 1.108.0.
 - `patch_image_cost.py` (image-gen spend): a real image generation through the
  candidate produced a **nonzero success SpendLogs row** — twice (fake OpenAI-shaped
  endpoint and the real Regolo Qwen-Image API): `spend 0.0011403264` for 256×256 at
  `$1.74e-8/px`, `call_type aimage_generation`, `status success`.
 - `patch_passthrough_token_cost.py` (Metis metering): a real `/v1/systemone` call
  logged `[coop-passthrough-cost] input_tokens=148 rate=1.2e-07 -> response_cost=1.776e-05`
  and the SpendLogs row matched exactly (148 × $0.12/1M), tokens recorded.
 - Member path: temp key generated with master auth → `/v1/models` (11 models) → real
  chat completion (greenpt/glm-5.3-flash, "pong", usage 19/3 tokens, spend row written).
 - Migrations: 36 new between 1.103.2 and 1.108.0, **all additive** (no unguarded
  DROP/DELETE/UPDATE). A fresh DB ran the full 220-migration chain cleanly; the live
  ledger (1.103.2) will apply only the 36 new ones.
 - `budget_reset_at`: column unchanged on `LiteLLM_TeamTable`; a direct-DB write
  (the portal's exact pattern) plus API read-back verified on the candidate.
 - Portal API compatibility: `/team/list` (bare list) and `/key/list`
  (`{keys, total_count, …}`) shapes are **unchanged**; a bare `/team/list` call still
  returns ALL teams (pagination only applies when `page`/`page_size` are passed —
  verified with 11 teams); `/team/update` still drops `budget_reset_at`, so the
  portal's direct-DB write remains required. SpendLogs gains only an additive
  `billing_agent_id` column — digest queries are unaffected.
 - LibreChat 1.4.8 (appstore `ai.librechat.cloudronapp`): changelog 1.4.5 → 1.4.8 is
  tooling only — uv 0.13.6/0.13.7 and Node.js 22.23.4 (security fix, 1.4.7); the
  LibreChat upstream stays 0.8.8. Every overlay patch anchor (title tag, manifest
  link, theme-color metas, webmanifest colors, "MCP Servers" / "Every AI for
  Everyone." strings, `index.*.js`, `sw.js`) was verified against the real 1.4.8
  image — **no overlay porting needed**; the candidate overlay built with all
  markers (UI strings patched in 3 files, index/manifest branding applied).
 ## 0. State re-check — 2026-11-09 04:15 UTC (read live before the window)
 Verified against the live box on the night of the window. Anything here that
 contradicts the prep notes above is newer and wins.
 - **Gateway:** image `registry.inference.coop/local/ai.coop.litellm:20261107-201000-407e751e2`
  (deployed 2026-11-07 20:10 UTC), in-container version **1.103.2**, 8 expected models
  + extras, healthy. **The rollback target is therefore THIS tag, not
  `1.0.0-20261105-e161db9`** (see §3.3).
 - **Chat:** overlay `registry.inference.coop/co-op/librechat-custom:20261031-2a98892`,
  appstore 1.4.5, healthy. `master` is 2 commits ahead of `bump-appstore-1.4.8`, but both
  touch only `librechat.yaml` — a persistent-volume file that is NOT baked into the image,
  so pinning the Nov-06 candidate does not revert them.
 - **Registry:** all four tags this plan needs are present —
  `local/ai.coop.litellm:{1.0.0-20261106-9e25578, 20261107-201000-407e751e2}` and
  `co-op/librechat-custom:{20261106-c2aec74, 20261031-2a98892}` (listed from the registry
  app's own storage).
 - **`upgrade-v1.108.0` is 5 commits BEHIND `main`.** main gained
  `patch_transcription_response_format.py`, `patch_audio_default_mp3.py` and
  `audio_keepalive.py` (Dockerfile RUN/chmod lines) plus the STT/embedding model additions
  on 2026-11-06/07, all after the candidate branch was cut. Merging the branch into main
  (as §3 does) carries both sides — but the **prebuilt candidate image does NOT contain
  them** (see the §3.1 fallback note and the extended marker list).
 - **Version target:** the branch pins **1.108.0**; latest stable upstream is **1.108.3**
  (2026-11-07). The prep validates 1.108.0 only — ship 1.108.0 tonight unless someone
  re-runs the anchor/migration validation against 1.108.3 first.
 - **Backups are NOT healthy.** Last successful Cloudron backup: **2026-11-08 21:18 UTC**
  (box, encrypted v2, state normal). The scheduled **2026-11-09 04:00 UTC run FAILED**
  (`tarPack pipeline error … aborted` — the same signature as the Infomaniak S3 quota
  rejections of 2026-11-06/07/08). The pre-flight backups in §2.1 are this window's undo
  and must be proven to land before anything is deployed.
 ## 1. Order: LiteLLM first, then LibreChat
 LibreChat's post-upgrade checks (key sync, a real chat completion) run **through the
 gateway** — doing the gateway first means those checks also exercise the new
 gateway, and each upgrade stays an isolated abort point: if LiteLLM fails, roll it
 back and skip LibreChat (chat keeps serving on the old overlay); if LibreChat fails,
 the new gateway is already proven.
 ## 2. Pre-flight (both apps, ~5 min)
 ```bash
 set -a; . ~/.hermes/.env; set +a
 CR() { cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" "$@"; }
 # 2.1 Fresh update-rollback backups (DB + volume snapshot each)
 CR backup create --app adababd4-2427-4bdd-9bf7-6bdad0302610
 CR backup create --app 4023ccbf-d57a-4500-97df-14de9cf3516e
 CR backup list --app adababd4-2427-4bdd-9bf7-6bdad0302610 | tail -3   # note newest id
 CR backup list --app 4023ccbf-d57a-4500-97df-14de9cf3516e | tail -3
 # 2.1b PROVE the backup landed BEFORE deploying. The backup pipeline is failing on the
 #      Infomaniak S3 plan quota (last success 2026-11-08 21:18 UTC; the 04:00 UTC run on
 #      2026-11-09 failed again). Re-run `CR backup list --app <id>` and require a NEW id
 #      with state normal — an unchanged newest id, or no new entry, means this window has
 #      NO undo. Decide before deploying, not after.
 # 2.2 Record the member-visible baseline for after: number of keys and teams
 #     (use the MASTER key — it is in /app/data/.master_key inside the gateway:
 #      CR exec --app gateway.inference.coop -- cat /app/data/.master_key
 #      then GET /key/list and /team/list with it; count both. NEVER paste the key.)
 ```
 ## 3. LiteLLM 1.103.2 → 1.108.0 (~20–35 min)
 ```bash
 cd ~/repos/litellm                       # ONE APP = ONE REPO DIR. Never run an update
 git fetch origin && git checkout main && git pull   # for another app from this cwd
 git merge upgrade-v1.108.0               # branch is pushed; approve the merge in Gitea
                                         # if you prefer the PR path
 git log --oneline -1                     # note the sha
 ```
 3.1 **Deploy (on-server build — the repo manifest has no `dockerImage` pin, so
 `cloudron update` builds and swaps; do NOT pass `--image`):**
 ```bash
 CR update --app adababd4-2427-4bdd-9bf7-6bdad0302610
 ```
 - The build output must contain **all TEN** patch markers (each is one line in the
  build log; a missing line = that patch silently did not apply — STOP and check):
  1. `INFO: patched budget-exceeded messages in …/auth_checks.py`
  2. `INFO: patched image-cost registration in …/router.py`
  3. `[coop-patch] applied to …/cost_calculator.py`
  4. `[coop-patch] applied audio shared pricing to …/router.py`
  5. `[coop-patch] applied: provider response headers are no longer forwarded …`
  6. `expose-cost-headers: appended 20 cost header names to expose_headers …`
  7. `[coop-patch] applied: passthrough-key-access -> …/route_checks.py`
  8. `[coop-patch] applied: passthrough-token-cost -> …/success_handler.py`
  9. `[coop-patch] applied skip-verbose_json to …` (patch_transcription_response_format.py)
  10. `[coop-patch] applied TTS default mp3 to …` (patch_audio_default_mp3.py)
 - `audio_keepalive.py` must be copied and `chmod +x`'d in the build (start.sh calls it for
  the TTS warm path). Markers 9–10 and this file are the 2026-11-06/07 audio work; missing
  any of them means that work silently reverts to the pre-Nov-06 behaviour.
 - **Boot**: LiteLLM runs `prisma migrate deploy` (the 36 additive migrations) before
  binding port 4000. Expect `502`/`ECONNREFUSED` on health checks for a few minutes
  (10–20 in the bad case). **Never restart mid-wait** — a restart resets the
  migration clock. Once `/health/liveliness` returns 200, continue immediately
  ("up" ≠ "migrated"; the checks below are the proof).
 - **DO NOT use the prebuilt candidate as a flake fallback tonight.** It exists
  (`…ai.coop.litellm:1.0.0-20261106-9e25578`, and `--image` would work because the repo
  manifest carries no `dockerImage` field), but it was built from the branch tree and
  therefore predates `patch_transcription_response_format.py`, `patch_audio_default_mp3.py`,
  `audio_keepalive.py` and the STT/embedding model additions. Deploying it reverts the
  2026-11-06/07 audio and model work. If the on-server build flakes, fix the build or build
  the merged tree — the candidate image is not usable tonight.
 3.2 **Verify (in order — a failing early step means stop, not continue):**
 ```bash
 # 1. version inside the running container
 CR exec --app gateway.inference.coop -- \
  /app/.venv/bin/python3 -c "from importlib.metadata import version; print(version('litellm'))"
 #    → 1.108.0
 # 2. patch markers inside the RUNNING container (not the build log alone)
 CR exec --app gateway.inference.coop -- sh -c \
  'grep -c "CO-OP PATCH (2026-10-19)" /app/.venv/lib/python3.13/site-packages/litellm/router.py; \
   grep -c "x-litellm-response-cost-input" /app/.venv/lib/python3.13/site-packages/litellm/proxy/proxy_server.py'
 #    → 1 and 1
 # 2b. supply-chain IoC check (standing policy on every LiteLLM image change):
 CR exec --app gateway.inference.coop -- sh -c \
  'ls /app/.venv/lib/python3.13/site-packages/litellm_init.pth 2>/dev/null; \
   ls /root/.config/sysmon 2>/dev/null; \
   ls /tmp/pglog 2>/dev/null; echo IOC-CLEAN'
 #    → must print ONLY "IOC-CLEAN" (no such files). Any hit = stop, quarantine, rotate
 #      the gateway env keys (master + providers live in that env).
 # 3. MEMBER-SCOPE key path (master-key tests alone prove nothing — master bypasses
 #    exactly the tables a broken migration empties):
 #    POST /key/generate  {"alias":"upgrade-verify-temp"}   (master auth) → temp key
 #    GET  /v1/models with the TEMP key                     → 11 models
 #    POST /v1/chat/completions {"model":"greenpt/glm-5.3-flash",
 #         "messages":[{"role":"user","content":"Reply with the single word: pong"}],
 #         "max_tokens":16}                                  → content + usage
 #    GET  /key/list (master)                               → same count as §2.2
 #    POST /key/delete {"keys":["<temp>"]}                  → 200
 # 4. CORS preflight (chat origin must be allowed):
 #    OPTIONS /v1/models with Origin: https://chat.inference.coop → 200 + allow-origin
 # 5. Cloudron state
 curl -sS -H "Authorization: Bearer $CLOUDRON_TOKEN" \
  https://my.inference.coop/api/v1/apps/adababd4-2427-4bdd-9bf7-6bdad0302610 \
  | python3 -c "import sys,json; a=json.load(sys.stdin); a=a.get('app',a); print(a['health'], a['runState'], a['manifest'].get('dockerImage'))"
 #    → healthy running registry.inference.coop/local/ai.coop.litellm:<new tag>
 # 6. Watch 10–15 min of real member traffic before closing:
 #    SpendLogs rows arriving with status 'success' (portal admin panel or
 #    GET /spend/logs with master auth). Failure rows with call_type = a route
 #    path and spend 0 are 401-refusals, not duplicates.
 ```
 3.3 **Rollback (one command; the additive migrations make old-code-on-new-schema safe):**
 ```bash
 CR update --app adababd4-2427-4bdd-9bf7-6bdad0302610 \
  --image registry.inference.coop/local/ai.coop.litellm:20261107-201000-407e751e2
 ```
 If member keys 401 after any image change, the schema is the suspect: read the
 exact error text from a member key call — a missing-relation error names the failing
 path; do NOT reissue keys. (Full recipe: `code/litellm` README, "Verify after
 upgrading".)
 ## 4. LibreChat overlay 1.4.5 → 1.4.8 (~10–15 min)
 ```bash
 cd ~/repos/librechat-custom              # again: one app = its own repo dir
 git fetch origin && git checkout master && git pull
 git merge bump-appstore-1.4.8            # overlay code is UNCHANGED apart from the
                                         # FROM pin — appstore 1.4.8 carries the same
                                         # LibreChat 0.8.8; only Node/uv/base bumped
 set -a; . ~/.hermes/.env; set +a
 # The candidate overlay is already built: pin + deploy via the repo's release tool
 ./release.sh pin 20261106-c2aec74        # writes dockerImage into CloudronManifest.json, commits, pushes
 ./release.sh deploy                      # cloudron update --force from THIS dir, then verify
 ```
 4.1 **Verify** (release.sh verify runs the served-HTML checks; confirm the rest by hand):
 ```bash
 ./release.sh verify                      # title = Inference Co-op, x-coop-build meta present
 # key sync task ran (manifest scheduler `librechat_key_sync`, every minute):
 CR logs --app chat.inference.coop --lines 100 | grep -i "sync" | tail -5
 #    → the sync script prints its own md5 prefix each run; it must be the new build's
 # RAG service: the appstore package runs rag_api alongside; confirm no crash loop
 curl -sS -o /dev/null -w "%{http_code}\n" https://chat.inference.coop/   # 200 (SSO shell)
 # login via Cloudron SSO in a browser, then send one real chat completion through
 # LibreChat (this exercises chat → NEW gateway → provider end-to-end)
 ```
 4.2 **Rollback:**
 ```bash
 cd ~/repos/librechat-custom
 ./release.sh pin 20261031-2a98892 && ./release.sh deploy
 ```
 ## 5. Window estimate
 | Phase | Time |
 |---|---|
 | Pre-flight + backups | 5 min |
 | LiteLLM build+swap (on-server build) | 5–10 min |
 | LiteLLM boot migrations + member-scope verification | 10–20 min |
 | LibreChat pin+deploy+verify (incl. an SSO login + one chat) | 10–15 min |
 | Post-watch on SpendLogs / close-out | 10 min |
 | **Total** | **~45–60 min** |
 ## 6. Gotchas that have each bitten this stack once
 1. **One app = one `cd`, immediately before its update.** `cloudron update` reads
   `CloudronManifest.json` from the CURRENT directory; updating app B from repo A's
   dir ships A's manifest to B — on 2026-11-05 that dropped the gateway's postgres
   addon and wiped the LiteLLM DB (~50 min member outage, restored from the
   pre-deploy snapshot). The pre-deploy backups in §2 are the undo; addons-block
   changes are DATA-BEARING.
 2. **A pinned manifest beats `--image`, silently.** LibreChat's repo manifest pins
   `dockerImage` — always change the pin (release.sh pin), never pass `--image` to
   it. LiteLLM's repo manifest has NO pin — `--image` works there (used for its
   rollback and build-flake fallback above).
 3. **"App is updated." proves nothing.** Verify the running image tag via the API
   AND a container-content marker (§3.2 steps 1–2, §4.1) before trusting any check.
 4. **Don't restart the gateway mid-migration-wait** — it resets the boot clock.
 5. **Do not "fix" `/team/update` dropping `budget_reset_at` today** — the portal's
   direct-DB write is the known, verified workaround; it still works on 1.108.0.
 6. After both apps are up, `app-inventory.md` in this repo should be updated with
   the new pins in the same session (it mirrors live state).
 7. **The prebuilt candidate is not the tree you are deploying.** The branch was cut five
   commits behind `main`; the merge supplies the rest, the prebuilt image does not. Verify
   the TEN build markers of §3.1 inside the running container, not the old eight.
 8. **A stale rollback reference is worse than none** — the runbook pointed at a Nov-05
   image while the box was running a Nov-07 build, so "roll back" would have silently
   dropped two days of member-visible fixes. Re-read the running tag (§0/§2.2) immediately
   before any rollback, and roll back to what was actually running.
 9. **The backup pipeline is itself an open incident.** Assume no undo until a pre-flight
   backup shows `state: normal` with a new id.
 ## 7. If something looks wrong
 - Member keys 401 with a missing-relation error → schema/migration issue; see
  §3.3 and the `code/litellm` README triage section. Restore path: the §2 backup,
  then re-pin the old image.
 - Chat serves a stale shell after the overlay deploy → wait ~60 s (mid-swap) and
  re-check; release.sh verify already retries fetches.
 - Anything else: halt at the failing step, note the exact command + output, roll
  back that app only, and leave the other app on its new version if it verified.
 ## 8. Executed — 2026-11-09, 04:41–05:05 UTC (what happened, and where it differed)
 Order followed as planned: LiteLLM first, then LibreChat. Two deliberate changes:
 1. **Target 1.108.3, not 1.108.0** — latest stable, and it pins the *same*
   `litellm-proxy-extras==0.4.110`, so the migration set is exactly the one validated here
   (36 new, all additive). All ten build-patch anchors were re-validated against the real
   1.108.3 sources first; `patch_expose_cost_headers.py`'s 1.108.x anchor applied.
 2. **A pre-built, pre-verified image was deployed, not an on-server build** —
   `20261109-c43d45c`, built from `upgrade-v1.108.3` (cut on top of `main`, so it carries the
   2026-11-06/07 audio patches the original candidate predated) and exercised first as a
   throwaway app (`gateway-dryrun.inference.coop`, its own Postgres/Redis, no member data).
 **The one thing that did not go as written — the pre-update snapshot OOMs.**
 A default `cloudron update` died at `Backup - Snapshotting app gateway.inference.coop`:
 task 1430 → `Task 1430 ran out of memory or terminated`, box log `Memory peak: 401.8M`,
 `oom-kill`. Cloudron's app-task service runs with a **hard 400 MB cgroup limit**
 (`startTask … "memoryLimit":400` from the system log), and this gateway's snapshot now
 exceeds it. Production was never at risk — the app kept serving the old image; only
 `installationState` went to `error`.
 **Workaround used: `--no-backup`**, with the 04:41 pre-flight app backups as the undo —
 which is what §2.1 is for. Next time: take the app backups first, then
 `cloudron update … --no-backup`.
 **Also learned the hard way:** `--wait-timeout` is a GLOBAL option, so
 `cloudron --wait-timeout 45m update …` works and `cloudron update --wait-timeout …` errors out.
 ### What was verified live afterwards
 - **gateway** `…ai.coop.litellm:20261109-c43d45c` — litellm **1.108.3**, extras 0.4.110;
  **all ten** patch markers present in the *running* container; `audio_keepalive.py`
  executable; IoC check clean; migrations **185 → 221**, none failed or rolled back;
  keys 210 / teams 111 unchanged; temp member key → `/v1/models` (15 models) → real
  completion → **success SpendLogs row**; CORS preflight 200. Member traffic within ten
  minutes: 73 success rows.
 - **chat** `…/librechat-custom:20261106-c2aec74` (appstore 1.4.8 / LibreChat 0.8.8) —
  `release.sh verify` all checks passed, key sync running every minute (108 member keys,
  revision matches the repo), Meili resync normal, served title still `Inference Co‑op`.
 ### Migrations were rehearsed, not merely run
 Before the window, the live gateway database was restored from the 2026-11-08 stopgap
 snapshot into a local PostgreSQL **18.6** (prod's exact version) and the image's own
 entrypoint (`python -m prisma migrate deploy`) was run against it: **185 → 221 applied in
 12 s**, row counts identical (210 keys / 111 teams / 138,422 spend rows), **no table lost**
 (90 → 110, the 20 additions all new-feature tables). That is what made `--no-backup`
 acceptable on the real box.
 ### Not rehearsed
 The LibreChat leg — a second chat app means ~2 GB of data plus its Meili/rag stack, for a
 change that is tooling-only (Node/uv; LibreChat upstream unchanged). Mitigation is the
 one-command rollback: `./release.sh pin 20261031-2a98892 && ./release.sh deploy`.
@@ -0,0 +1,137 @@
 # Backups
 There are **two independent backup systems**, on two different machines, with two
 different targets and two different passphrases. Confusing them is the most likely
 mistake — they do not back each other up, and neither one covers the other's host.
 | | Agent-host backup | Cloudron app backup |
 |---|---|---|
 | What | The agent host's state (all agent homes, `~/.hermes`, `~/.ssh`, `~/repos`, gateway unit files) | Every Cloudron app's `/app/data` + addon databases, plus platform config |
 | Machine | Agent host `ov-0f018e` (`179.237.103.165`) | The Cloudron server `179.237.86.116` |
 | Tool | `restic` (client-side encrypted, run nightly by a systemd timer) | Cloudron's built-in backup (dashboard-configured) |
 | Target | Infomaniak **Swiss Backup** (S3-compatible) | Infomaniak **Swiss Backup** (S3-compatible, `s3-v4-compat`) |
 | Schedule | 04:30 UTC + up to 5 min jitter (`agent-backup.timer`) | daily 23:00 UTC (`00 00 23 * * *`) |
 | Retention | 7 daily / 4 weekly / 3 monthly, no grouping | 7 days (`keepWithinSecs: 604800`) |
 | Config lives | `~/.hermes/scripts/backup.sh`, unit at `/etc/systemd/system/agent-backup.*` | Cloudron dashboard (no write API — see below) |
 | Last verified | 2026-11-04 (13 snapshots in the repo) | 2026-11-02 (runs 2026-10-30 → 11-03 all `normal`, ~5.4 GB) |
 Nothing on the Cloudron **host OS** is backed up by either system (netplan, sshd,
 `authorized_keys`, hand-edited `/etc`) — build the box, do not expect to restore
 it. This was the migration's central caveat and it still holds.
 ---
 ## 1. Agent-host backup (restic)
 `agent-backup.timer` → `agent-backup.service` → `~/.hermes/scripts/backup.sh`.
 - **The unit runs as `User=root` on purpose.** The agent homes under `/home/*`
  are mode `0700`, so only root can enumerate them; `ubuntu` sees `EACCES` and a
  non-root `[ -d .../.hermes ]` test reports **false for a home that exists** —
  which would silently skip that agent. The script therefore discovers agents by
  enumerating `/home/*` **as root** and stages each one; a new agent is covered
  without editing any list. Coverage is still unproven until its files show up in
  a snapshot.
 - **It backs up:** `/home/ubuntu/.hermes` (config, `.env`, skills, cron, memory,
  logs, state DBs), `/home/ubuntu/.ssh`, `/home/ubuntu/repos`, every other
  agent's `/home/<agent>/.hermes`, and `/etc/systemd/system/hermes-gateway*.service`.
  It **excludes** rebuildable bulk (`tools/`, `hermes-agent/`, `installs/`,
  `cache/`, `lsp/`, model caches) and the live SQLite files — consistent copies of
  those are staged first with `sqlite3 .backup`.
 - **Failure is visible, not silent.** On failure the script appends a note to
  `~/.hermes/cron/updates_notes.jsonl`, which the daily stewarding digest reads.
  Exit codes: 0 ok, 1 config, 2 backup, 3 prune.
 ### Deliberate-but-surprising: the PER-USER restic cache split
 `RESTIC_CACHE_DIR` defaults to **`/var/cache/restic`** (root's run), while manual
 runs by `ubuntu` use **`~/.cache/restic-manual`**. This is intentional — **do not
 "tidy" it back into one shared `~/.cache/restic`.**
 A shared cache directory let each user create files the other could not read;
 restic then **retried every snapshot with exponential backoff** (30s, 1m25s, …)
 until the run crawled or timed out. The visible symptom (2026-11-03) was a prune
 that "failed" **three nights running** — which looked exactly like retention being
 broken, when it was a permissions/retry storm. Splitting the cache per user ended
 it. The lock and log also live **under `$HOME/.hermes/`** (not `/tmp`): with
 `fs.protected_regular`, a root run is denied a sticky-dir file owned by another
 user (`EACCES`), and under `set -e` that killed the root run *silently*.
 ### Deliberate-but-surprising: `--group-by ""` on retention
 Retention is:
 ```
 restic forget --tag nightly --group-by "" \
  --keep-daily 7 --keep-weekly 4 --keep-monthly 3 --prune
 ```
 **`--group-by ""` (no grouping) is required, not cosmetic.** restic's default is
 `--group-by host,paths`, which splits the nightly snapshots into one group per
 distinct **paths set**. The backup glob `/etc/systemd/system/hermes-gateway*.service`
 grew from one unit to three when the infrabot and socialbot gateways were added
 (2026-10-30), so the paths set changed mid-history: snapshots before 2026-10-30
 formed one group and snapshots after formed another. Each group then sat **under**
 the keep-7-daily limit *on its own*, so **nothing was ever expired and the repo
 could never shrink**. Empty grouping applies one policy across all nightly
 snapshots. `--group-by host` would also collapse today's groups, but the box has
 already migrated once and a hostname change would reintroduce the same split — no
 grouping is immune to both path-set drift and a hostname change.
 **Current state (honest):** the repo still holds **13 snapshots** — the pre-fix
 accumulation. The script on disk carries the fix, and a `--dry-run` (2026-11-04)
 confirms it **keeps 8 and removes 5**. The next timer run (2026-11-05 04:31 UTC)
 applies it. Until it runs, treat "13 snapshots" as expected, not as the fix
 having failed.
 ### Other mechanics
 - Weekly `restic check` on Sundays (repo metadata); full data check monthly.
 - The passphrase (`RESTIC_PASSWORD`) lives in `~/.hermes/.env` **and** the co-op
  Vault. A backup whose key exists only on the backed-up box is not a backup.
 - S3 credentials come from `SWB_S3_ACCESS_KEY` / `SWB_S3_SECRET_KEY` (mapped to
  `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` in the script).
 ## 2. Cloudron app backup
 - Configured in the **dashboard** (*Settings → Backups*). There is **no write
  API**: `GET /api/v1/backup_sites` works, but `PUT`/`POST` on it and
  `POST /backups` all 404. Schedule and retention are dashboard-only.
 - **Two sites exist:** the live one (`s3-v4-compat`, daily 23:00, 7-day
  retention, id `03cfbdca-88fc-4ccc-97c3-7e1b58cf892d`) and an inactive
  `filesystem` site with schedule `never`. Do not be alarmed by the second one.
 - **A run reads `state=normal` even when it failed.** The failure's real error
  text is in `GET /api/v1/tasks` (task id `backup_<siteId>`, field `error.message`),
  not in the run list. `/events` needs a `?since=` range and `/system/logs` does
  not exist — a 404 there is not "no failures".
 - `/backup_sites` carries `schedule` as a 6-field cron and `retention.keepWithinSecs`.
  Check retention **before** claiming it needs changing.
 - **Quota math:** a failure whose upload pushed the retained set over the
  provider ceiling is the signature. The old box's `EntityTooLarge` failures came
  from a 4.82 GB run twice daily against the ceiling; the ceiling was raised and
  the schedule moved to once daily. Per-run size still grows (mostly registry
  image blobs) — watch the number.
 - **Two boxes on one S3 bucket interleave and prune each other's backups.** The
  old box's schedule must stay **disabled** while it is kept for rollback.
 - **Cross-check `siteId` per run:** runs on an old site id are pre-migration
  leftovers, kept `state=normal` forever and never pruned by the current site's
  retention.
 ## Restoring
 - **Cloudron** restores apps per-app, from the **newest** box-backup entry:
  `GET /apps/:id/backups` → pick by `creationTime` → `POST /apps/:id/restore`
  with `{"backupId": ...}`. Restore apps **sequentially** (parallel restores
  exhaust disk). `repair` recreates a container but restores **no data** — a
  repaired app comes up empty.
 - Full procedure, triage and the post-restore checker: the `cloudron-ops` skill
  and `migration/cloudron-server-migration.md`.
 - **Agent host:** `restic restore latest --target /` (or a specific snapshot) with
  `RESTIC_PASSWORD` and the S3 keys in the environment, run as root.
 ## The credential that carries the backups
 The agent-host `RESTIC_PASSWORD` and the Cloudron backup passphrase are **two
 different secrets**, both held by the GM and recorded in the token runbook's
 inventory. The Cloudron backup passphrase + S3 `secretAccessKey` are the
 chicken-and-egg pair a bare-metal rebuild needs first — they must be held off-box.
@@ -0,0 +1,15 @@
 # Test rows in the credit ledger — deliberately kept (audit L7, 2026-11-05)
 The portal `credit_ledger` carries a handful of historical test rows from the
 early build ("test credits (will revert)", "admin test - collective expense",
 and the matching reverts). They are **net zero** against member balances and
 every one is paired with its reversal.
 Decision: leave them in place. Rewriting the money ledger — even to remove
 test noise — destroys the audit trail's property of "what's in the table is
 what happened"; the pollution is cosmetic and far outweighed by the cost of
 ever being wrong about a ledger edit. This note is the record; no code or data
 change accompanies it.
 If the ledger ever grows a reporting consumer that must exclude them, filter
 by the `reason` text at query time rather than deleting rows.
@@ -0,0 +1,52 @@
 # LibreChat key feed (`/admin/librechat/keys`) — design decision (audit M2)
 Audited 2026-11-05. Status: **kept as-is, with monitoring**; a per-member
 fetch was considered and deliberately rejected.
 ## What the route is
 `GET https://portal.inference.coop/admin/librechat/keys`, authed by a single
 static token (`LIBRECHAT_SYNC_TOKEN`, held by the portal and the LibreChat
 sandbox app). It returns every ACTIVE member's decrypted chat key so the
 sandbox's in-app key-sync (`librechat_key_sync`, every 10 min inside the
 LibreChat app; `sync-keys.js`) can seed LibreChat's per-user key store —
 members never see an API-key prompt, and chat bills to each member's own
 LiteLLM team/budget.
 ## Blast radius (why the audit flagged it)
 The sync token unlocks every member's chat key in one response. Anyone holding
 the token (sandbox compromise, token leak from the sandbox env) reads the whole
 membership's keys. It is a deliberate exception to the admin token — members'
 key storage is decrypted in the portal only for this feed and for serving a
 member's own dashboard.
 ## Why a per-member fetch was rejected (the riskier path)
 The sandbox is the ONLY consumer, and it does not know which member it is
 seeding at the API layer — it seeds ALL members in one pass from one identity.
 Scoping the feed per member means changing the sandbox's identity model
 (per-member credentials or a member list with per-member auth), touching
 `sync-keys.js` and the seeding loop. A wrong change there stops key seeding
 for every member at once — the exact regression the audit order forbids
 ("nothing breaks"). The feed also runs on a fixed cadence, so the practical
 exposure window is bounded by how fast misuse would show.
 ## What is in place instead (2026-11-05)
 1. **Access logging on the route**: every fetch logs a WARNING line
   `librechat key feed served: N member key(s) to <ip> (ua=…)`. Count only —
   never key material, never member emails (the portal's log redactor strips
   emails anyway).
 2. **Daily-digest watch** (cron job 95e5b8fece86, §7 anomaly list): the
   sandbox polls ~every 10 min (~144 serves/day expected). The digest counts
   `librechat key feed served` lines and flags (a) a rate far above that
   cadence, or (b) any 401 burst on the route — either is a suspected token
   misuse and becomes a decision item.
 ## What a real fix looks like (if ever needed)
 Move the sandbox to fetch per member: a new portal route keyed by the member
 identity the sandbox already knows at seed time, plus a sandbox-side change to
 seed from that route. Only worth doing when the sandbox is already being
 modified for another reason — never as a standalone risky change.
@@ -0,0 +1,75 @@
 # Member provisioning boundary — who auto-provisions from Open Collective
 Which Open Collective contributors become portal members automatically, and
 which are deliberately excluded. The portal defines the member set in
 `fetch_members()` (member-portal repo, `app/main.py`), acts on it in the
 webhook and the nightly `reconcile_memberships()` sweep, and exposes two admin
 overrides: `POST /admin/org-link` and `POST /admin/provision`. A daily
 watchdog (below) is the net for anything that slips past both.
 ## Auto-provisioned
 **Individual OC accounts holding BACKER/ADMIN on the Membership tier (or no
 tier — pre-tier contributors are grandfathered) whose email is visible to the
 admin token.** The webhook provisions on first payment; the nightly sweep
 (03:00 scheduler task) provisions anything missed within 24h and deactivates
 lapsed members. `fetch_members()` matches the tier by NAME
 (`"membership"`, case-insensitive): renaming the tier on Open Collective
 silently empties the member set — the sweep's fail-safe skips reconciliation
 rather than deactivating everyone, but provisioning also stops.
 ## Deliberately excluded
 1. **Org-backed memberships.** An ORGANIZATION account never exposes an email
   (the `... on Individual { email }` fragment matches nothing, and OC returns
   org emails as null even to collective admins), so reconcile cannot map one
   and does not try. Org-backed memberships become members only when LINKED:
   `POST /admin/org-link {"org_slug", "email", "name"}` runs the full
   provisioning pipeline and stores `members.org_slug`; the sweep keeps linked
   accounts active while the org holds BACKER/ADMIN on the Membership tier and
   deactivates them when the org's dues lapse (fail-safe: if the org fetch
   itself errors, linked accounts stay active). An unlinked org's first
   payment emails info@ with the exact linking command.
 2. **Non-membership tiers (Credit pack).** A one-time purchase grants BACKER
   role but is NOT a membership — `fetch_members()` filters it out. This is
   why a contributor can pay, appear in the public backer list, and still have
   no portal member row: it is a credit-pack purchase, not a lapsed
   provisioning. The money is not lost: the webhook's held-credit-pack flow
   records a hold, notifies the buyer, and the operator grants, refunds or
   transfers it via `/admin/credit-packs-held`; a hold granted to a named
   member auto-applies if that person later becomes a member.
 3. **Manual members.** The `MANUAL_MEMBERS` env allowlist (founder/free tier,
   the co-op's own accounts) holds members with no OC membership BY DESIGN:
   never auto-provisioned from OC, and exempt from lapsed-membership
   deactivation. Check for orphaned LiteLLM `member:*` teams after any
   `MANUAL_MEMBERS` change.
 4. **Hidden-email / anonymous contributors.** An individual whose email is
   null even in the ADMIN view cannot be mapped by reconcile. They are not
   silently dropped — the daily watchdog reports them with their public dues,
   and provisioning is manual: `POST /admin/provision {"email", "name"}` on
   the operator's instruction. Incognito accounts publish an
   `incognito-xxxxxxxx` slug, but the ADMIN member query usually still reveals
   their email — always check the admin view before treating one as
   unmappable (that is what resolved the mmoya case).
 ## The watchdog: `oc_unmapped_check.py`
 Daily no-agent cron on the agent host (07:20, job `d0fd1126cc80`), running
 `~/.hermes/scripts/oc_unmapped_check.py` — the same silent-until-findings
 pattern as the spend-consistency guard. It reads the collective's backer list
 through the portal env's admin token and the portal rows read-only via
 `cloudron exec`, maps them by email, slug or org_slug, and prints ONLY
 member-ish backers (the exact set `fetch_members()` treats as members) with no
 portal row at all: slug, display name, whether the admin view sees an email,
 dues (amount/frequency/status from the public order list), days as a backer,
 and the matching action (org-link vs /admin/provision vs investigate-the-sweep).
 Backer nodes with no account object (deleted/incognito-hidden accounts) are
 attributed via same-day public orders and stay silent when every same-day
 payer is already mapped. Excluded from findings: credit-pack-only backers
 (the held-credits flow owns them) and backers matching an inactive portal row
 (the sweep re-provisions/reactivates those within 24h). Empty output = clean;
 any fetch failure prints FAILED — a check that cannot look never reads as
 clean.
@@ -0,0 +1,94 @@
 # Woodpecker CI (server + agent)
 Self-hosted CI for the co-op's repos. **As of 2026-11-04 the CI is installed but
 NOT working end to end** — read the *Current state* section before trusting it for
 anything.
 Two Cloudron apps, added 2026-11-03:
 | App | FQDN | Cloudron id | Image |
 |---|---|---|---|
 | Woodpecker **server** | `ci.inference.coop` | `4984f95e-0c0a-4af3-9ce1-4ca4f3e8483d` | `cloudron/org.woodpecker_ci.cloudronapp:202610300219490000` (store, v2.22.1 / upstream 3.20.1) |
 | Woodpecker **agent** | `ci-agent.inference.coop` | `5ac63873-cf2c-4ad9-acd6-81dc0e016c83` | `registry.inference.coop/co-op/woodpecker-agent:20261104-e070275` (**custom**) |
 - The **server** is the stock Cloudron app. It uses the `mysql` addon for storage
  and exposes `GRPC_PORT` 9000 to the agent. Its config lives in
  `/app/data/env.sh` (`WOODPECKER_GITEA`, `WOODPECKER_ADMIN`,
  `WOODPECKER_AGENT_SECRET`, `WOODPECKER_OPEN`, …). **No env vars are set through
  the app's Cloudron env** (`env-keys` is empty) — everything is in that file.
 - The **agent** is a custom app. Its config is set as Cloudron env (names only):
  `WOODPECKER_SERVER` = `ci.inference.coop:9000`, `WOODPECKER_GRPC_SECURE=true`,
  `WOODPECKER_BACKEND=local`, `WOODPECKER_MAX_WORKFLOWS=2`,
  `WOODPECKER_AGENT_CONFIG_FILE=/app/data/agent.conf`, plus the shared
  `WOODPECKER_AGENT_SECRET`.
 ## Why the agent image is custom (and why the backend is `local`)
 Cloudron apps have **no Docker socket**, so the default `docker` backend cannot
 run and the agent uses the **`local` backend**: each pipeline step executes as a
 shell command *inside the agent container*, and a step's `image:` field selects
 the **shell**, not a container image. Consequence: **every tool a pipeline step
 needs must be baked into the agent image**, and the Cloudron rootfs is read-only
 at run time, so nothing can install at run time.
 The image is `FROM woodpeckerci/woodpecker-agent:v3.20.1-alpine` (the **alpine**
 variant — the default `scratch` image has no shell). It adds `bash`, `git`,
 **`git-lfs`**, `tar`, `ca-certificates`, `python3`, `node`/`npm`, the `cloudron`
 CLI, and a pinned MkDocs toolchain at `/opt/docs-venv`. It **runs as root**
 (`/app/data` on this Cloudron is root-owned, as for the other custom apps here).
 ## Building and pinning the agent image
 - **Source repo:** `/home/ubuntu/repos/woodpecker-agent` on the agent host.
  ⚠️ **This repo has no git remote and has never been pushed to Gitea** — the CI
  agent's build source exists only on the agent host. Treat that as a gap to close.
 - **Tag convention:** `<yyyymmdd>-<gitsha>` (e.g. `20261104-e070275`). Never a
  floating tag.
 - **Build:** `~/.hermes/scripts/cloudron_builder_build.sh registry.inference.coop/co-op/woodpecker-agent <tag> <dir>`
  (the script carries the registry auth triple, which is not optional).
 - **Deploy:** write the new tag into `CloudronManifest.json`'s `dockerImage`,
  commit, build `--no-cache`, then `cloudron update --app 5ac63873-…` **without**
  `--image` (the manifest's pin wins). Verify by the running image tag **and**
  something only the new build produces — never by exit code alone.
 - `publishing-requirements.txt` mirrors the docs repo's
  `publishing/requirements.txt`; **keep the two in step** when the docs toolchain
  changes.
 ## When a pipeline runs (and when it does not)
 - **Manual-trigger only.** Nothing here fires on push; a pipeline is started by
  hand from the Woodpecker UI.
 - **The pipeline is not yet green end to end.**
 - **`git-lfs` is required by the clone step.** Woodpecker runs `git lfs fetch`
  after cloning, so a clone step fails with
  `git: 'lfs' is not a git command` on any image without it — this was added to
  the agent image on 2026-11-04. If a rebuild ever drops it, clones break again.
 ## Current state (2026-11-04): installed, agent connected, cannot run jobs
 - The server is up — `https://ci.inference.coop/` serves the Woodpecker UI.
 - The agent app is `running`/`healthy` and its health endpoint answers — **but
  that is liveness, not capability.** The log shows the agent's **gRPC job-collection
  long-poll failing continuously**: the server answers the agent's RPC with
  **HTTP 500** (an nginx `text/html` 500 body), so the agent retries with backoff
  and **never receives work**.
 - A healthy agent that cannot collect jobs is the exact trap to avoid reading as
  progress. Score a CI worker by **the route that delivers it work**, not by its
  health RPC.
 **Unresolved / needs an operator:** the server-side gRPC 500. Until the server
 answers the agent's stream, no pipeline can execute regardless of repository
 configuration. This is a server-side (or platform-proxy) problem, not an agent
 image problem.
 ## Enabling a repo (known gate)
 - Repos are connected through the **Gitea** source (`WOODPECKER_GITEA`). Enabling
  a repo requires the repo owner to log in to Woodpecker with **Gitea OAuth** and
  activate it. Standing up that OAuth login source is an **operator step**
  (creating the Gitea OAuth application needs `write:user`, which the bot's Gitea
  token does not have).
 - Reading run/pipeline state is done from the server's **MySQL** database, not
  from a public API.
 - Note: the whole point of this CI is to take the docs build and custom-app builds
  off the agent host — see `co-op/support#30` (moving `REGISTRY_BUILD_TOKEN` to CI).
@@ -508,3 +508,68 @@ access removed 2026-11-02 — retirement begun; app restored and safe in backups
 **Outstanding:** tonight 23:00 UTC first backup (verify `normal`, ~4.9 GiB); Cloudron subscription
 page check on the new dashboard; my primary model route re-heals on its own (fallback worked all
 night); old box keep powered-on/apps-stopped/backups-disabled for a few days, then decommission.
 ---
 # POST-MIGRATION ADDENDUM — 2026-11-04
 Recorded by the infra-docs audit. This section is current state; everything above
 is the as-it-happened record.
 ## App count: 16 → 19 (three apps added since cutover)
 The migration restored **16 apps**. Since then the server carries **19** — the
 three new ones are the docs site, the CI server and the CI agent. See
 `app-inventory.md` for the full list. Anywhere above that says "16 apps" is the
 state at cutover, not today.
 | New since 2026-11-02 | FQDN | Cloudron id |
 |---|---|---|
 | Docs site (Surfer) | `docs.inference.coop` | `85b7ce06-e2af-49a4-bd4f-a9c2e692e3ab` |
 | Woodpecker CI server | `ci.inference.coop` | `4984f95e-0c0a-4af3-9ce1-4ca4f3e8483d` |
 | Woodpecker CI agent (custom) | `ci-agent.inference.coop` | `5ac63873-cf2c-4ad9-acd6-81dc0e016c83` |
 ## The DNS cutover is COMPLETE — no hostname is served by the old box
 At one point `chat.inference.coop` and `portal.inference.coop` were *reported* as
 still resolving to the **old** box (`179.199.135.72`). **That observation was
 stale and the split does not exist.** Verified 2026-11-04 against an external
 resolver (Cloudflare, `@1.1.1.1`): **every** hostname — apex, `my`, `chat`,
 `portal`, `gateway`, `docs`, `ci`, `newsletter`, `git` — resolves to the **new**
 box `179.237.86.116` with **both A and AAAA**
 (`2001:1600:18:206::89`). No hostname resolves to the old box, and nothing in DNS
 points at it. `chat.inference.coop` serves LibreChat and `portal.inference.coop`
 returns the portal from the new box.
 The zone is **wildcarded**, so a resolving name proves nothing on its own — but a
 made-up hostname (`zzznope.inference.coop`) also answers the **new** box at the
 authoritative nameservers, so the wildcard itself now points at the new box.
 ## ⚠️ UNRESOLVED (operator decision): the old box is still alive and internet-exposed
 `179.199.135.72` is **powered on, reachable from the internet, and unaltered.** This
 is not "a spare kept for rollback" — it is a **decommissioned server still running
 and still exposed.**
 - **Open to the internet (verified 2026-11-04):** ports **22, 80, 443, 587, 993**
  answer. (3306/5432/8080 are closed.)
 - It answers HTTPS **for any Host header** with a **generic Cloudron-served page**
  (its fallback / "app down" page, empty-subject certificate) rather than a real
  app — because its apps are stopped.
 - **It still holds whatever old data and configuration it had** — the old Cloudron
  platform, its app data, and its host config are all present on the disk.
 - **Nothing in DNS points to it**, so it serves **no member traffic** — the exposure
  is the open ports and the retained data/config, not live traffic.
 **Decision needed from the operator:** either **shut it down**, or **firewall it**
 (close all but what is genuinely needed) if it is being kept as a rollback. It is
 NOT the steward's to act on: **do not shut it down or change anything on it without
 the operator's word.** Its backup schedule must stay **disabled** while it is kept
 (two boxes on one S3 bucket interleave and prune each other's backups).
 ## Still unresolved / for the operator
 - **The old box's exposure** — see above. Shut down or firewall.
 - **Member mailbox contents** — the long-standing unknown from §7 (whether
  members' mail is inside the Cloudron backup) still needs a real IMAP login to
  confirm.
@@ -188,8 +188,9 @@ container). The `PYTHONPATH` env survives restores (it's in app config).
 The portal and gateway run their own housekeeping via the Cloudron scheduler
 addon, declared in each app's `CloudronManifest.json` under `addons.scheduler`:
- **portal**: `membership_sweep` (nightly 3am, `/app/code/sweep.sh` →
+- **portal**: `membership_sweep` (three times daily — `0 3,11,19 * * *` UTC per
-  `POST /admin/sweep` with the **`ADMIN_TOKEN`** header — NOT `WEBHOOK_TOKEN`,
+  the live manifest; earlier notes said "nightly at 3am" and are stale. Runs
  `/app/code/sweep.sh` → `POST /admin/sweep` with the **`ADMIN_TOKEN`** header — NOT `WEBHOOK_TOKEN`,
  which returns 403 on `/admin/*` since the 2026-10-28 cutover), `rag_refresh`
  (nightly 3:30am, `/app/code/rag_refresh.py` — pulls docs from the Gitea raw
  API and refreshes the chat knowledge base; needs `OWUI_API_KEY` +
@@ -0,0 +1,44 @@
 # Provider pricing
 What our model providers charge, and where the authoritative numbers live.
 ## The authority
 **The gateway's own per-model costs are the authority** — they are what the
 response headers (`x-litellm-response-cost`) and member billing both read. They
 live in the LiteLLM config on the gateway's persistent volume
 (`/app/data/config.yaml`), under each model's `model_info`, as **USD per token**
 (the comments in that file quote per-1M rates). Read them live with
 `GET https://gateway.inference.coop/model/info` (master key), or read the config
 straight from the running app.
 **The member-facing mirror** is `co-op/docs/models.md`. When a rate changes,
 update **both** the gateway config and `models.md` — the config is authoritative,
 `models.md` is the mirror.
 A missing cost field makes a paid feature free; a stale one bills wrong; a missing
 `cache_read_input_token_cost` bills cache hits at the full input rate. All three
 have happened here.
 ## The pages
 | File | Provider | Notes |
 |---|---|---|
 | `greenpt.md` | GreenPT | renewable energy; the default model lives here |
 | `tinfoil.md` | Tinfoil | TEE; the local-proxy architecture lives here |
 | `regolo.md` | Regolo | renewable + ZDR; includes our measured usage |
 | `decision-model.md` | GreenPT "Metis" via `/v1/systemone` | a pass-through **route**, not a chat model |
 ## The routing rule
 Members reach models through one gateway, which fronts all providers. The
 per-request cost the gateway records is the number to trust; do not re-derive a
 price from a provider's public page (they drift, and some are login-walled).
 ## A note on currencies
 Regolo quotes in **EUR**; our gateway figures convert at **EUR × 1.16 → USD**
 (the rate our numbers use — `regolo.md` documents the derivation). GreenPT's own
 listed rates appear in **EUR with the same numerals the gateway carries** —
 whether an EUR→USD conversion is intended there is **unverified**; confirm it
 against the GreenPT invoice before treating the figures as final.
@@ -0,0 +1,58 @@
 # The decision-model route — GreenPT "Metis" (`/v1/systemone`)
 The gateway exposes a **decision model** — GreenPT's **Metis** (System One API) —
 as a **pass-through route**, not a chat model.
 ## What it is, and why it is a route and not a model
 Metis is a **decision** endpoint, not a chat completion: it takes a `state` plus up
 to 32 typed questions and returns **probabilities / choices / scores**. It
 generates no text (`output_tokens` is always 0), so **it cannot be a `model_list`
 entry** — LiteLLM's chat pipeline has nothing to parse. It is therefore exposed as
 a pass-through:
 ```
 path:    /v1/systemone
 target:  https://api.greenpt.ai/v1/systemone
 auth:    true        # members call it with their normal co-op API key
 ```
 - **It is API-only and deliberately NOT a chat-picker model.** It never appears in
  the chat model list. Do not add it there.
 - **Members authenticate with their own co-op key**, never the provider key — the
  route's `Authorization` header injects the provider key server-side.
 - Hosted by GreenPT (policy-based privacy, EU, 100% renewable), same
  `GREENPT_API_KEY` the green chat models use.
 ## Pricing and the metering fix
 - **$0.12 per 1M input tokens; output is free.**
 Metering is **exact, from the token count GreenPT reports back**. Because Metis is
 not a `model_list` entry, it is priced through the
 `coop_passthrough_cost_per_token` map, keyed by the route path
 (`/v1/systemone: 0.00000012`). `patch_passthrough_token_cost.py` (built into the
 image) reads `usage.input_tokens` from the Metis response body at logging time and
 writes the exact figure to member spend.
 **Why this exists — record it, it will recur:** **generic pass-through routes
 originally logged nothing.** A route that is not a `model_list` entry gets no
 automatic pricing, so its calls were metered as $0 — a paid feature silently free.
 The pass-through accounting had to be fixed so spend lands against the **calling
 member's key**. If another pass-through route is ever added, it needs the same
 treatment or it bills nothing.
 `cost_per_request: 0.0001` on the route is now only a **fallback** — it applies if
 a response arrives without a usable token count, so a call is never silently free.
 Keep the two in sync.
 ## Access control gotcha (community edition)
 The route is opened to any authenticated member key via
 `coop_passthrough_open_to_keys` (consumed at build time by
 `patch_passthrough_key_access.py`). **LiteLLM's own gate for `auth=true`
 pass-through routes (`allowed_passthrough_routes`) is an Enterprise-only field**,
 so on this community-edition proxy it can never be granted to member keys. The
 only alternative, `auth: false`, would expose the route publicly and forward the
 co-op's GreenPT key to anonymous callers. **Authentication is unchanged** — the
 endpoint still requires a valid key.
@@ -0,0 +1,44 @@
 # GreenPT pricing
 GreenPT is OpenAI-compatible, EU-hosted on **100% renewable energy**. LiteLLM
 talks to it **directly** (no TEE proxy). Billing across the gateway:
 - `api_base`: `https://api.greenpt.ai/v1`
 - key: `GREENPT_API_KEY` (gateway app env)
 ## What we serve, and what it costs
 Rates as recorded in the gateway's authorititative config comment, **USD per 1M
 tokens** (the gateway's `model_info` carries the per-token equivalents):
 | our model | input | output | cached input |
 |---|---|---|---|
 | `greenpt/deepseek-v4.1-flash` **(DEFAULT model)** | $0.22 | $1.10 | $0.011 |
 | `greenpt/glm-5.3-flash` | $0.11 | $0.44 | $0.022 |
 | `greenpt/glm-5.3` | $1.10 | $4.40 | $0.275 |
 - **`greenpt/deepseek-v4.1-flash` is the gateway's default model.** It is the same
  model family as Tinfoil's `tinfoil/deepseek-v4-1-flash`, but on renewable energy
  and **not** marked "Experimental" by the provider (Tinfoil's counterpart is), so
  it is the default and Tinfoil's is its private counterpart. The policy trade is
  **policy-based privacy (GreenPT) vs TEE (Tinfoil)** — say so plainly rather than
  implying GreenPT is confidential computing.
 - `greenpt/glm-5.3` is the strongest **agentic** model in the green tier (1M context).
 - Note the **dotted slug** `glm-5.3-flash` on GreenPT, distinct from Tinfoil's
  `glm-5-3-flash` (hyphenated). Getting a model slug wrong is a silent 404 at call
  time.
 ## Historical
 - **Removed 2026-10:** `green-l` / `green-r` — GreenPT now marks them "not
  recommended".
 - The config comment previously read "€0.22 in / €1.10 out" alongside the dollar
  figures — see the currency note in `README.md`; the intended FX is unverified.
 ## The fallback key
 `GREENPT_FALLBACK_API_KEY` (agent host) is the **same value** as the gateway's
 `GREENPT_API_KEY` — verified by hash comparison. It powers the steward agent's
 direct-to-provider fallback during a gateway outage. **Rotating one without the
 other silently breaks the fallback** (a dedicated key is the cleaner end state).
 See the token runbook.
@@ -97,3 +97,21 @@ At these volumes the mispricing is small: apertus's input roughly 2.5x-ing on a
 **$0.82 lifetime** is about **$1/month** of exposure, and qwen's cut moves ~$0.6/month in members'
 favour. So this is **billing accuracy, not urgency** — but accuracy still matters, because the same
 numbers feed the cost headers members can now read.
 ## UPDATE 2026-11-04 — the gateway was updated; both stale sections above are now resolved
 Checked against the live gateway config (`/app/data/config.yaml`, `model_info`):
 - `regolo/apertus-70b` = **1.16e-06 / 3.944e-06** per token
  ($1.16 / $3.94 per 1M) — the new 2026-11-01 rate. The "~70 cent/M loss while the
  gateway still bills $0.46" paragraph above no longer applies.
 - `regolo/qwen3.5-122b` = **6.96e-07 / 3.944e-06** ($0.70 / $3.94 per 1M) — the new
  rate. The "over-bills members by ~46 cents/M" paragraph no longer applies.
 - `regolo/qwen-image` now carries **`input_cost_per_pixel: 1.74e-8`** — the EUR
  figure converted at ×1.16, i.e. the "image model is priced in EUR-as-USD" open
  inconsistency above is **fixed**. It is no longer priced at the raw EUR number.
 So the two "Until the gateway is updated" tables and the image-model section above
 are **historical**. The usage measurement ($4.38/month, ~1.1% of inference spend)
 and the subscription analysis stand. Keep this file dated-section-per-change; do
 not delete the history, it shows the reasoning.
@@ -0,0 +1,84 @@
 # Tinfoil pricing and architecture
 Tinfoil serves models inside **TEE-protected enclaves**. The co-op does not
 self-host it — Tinfoil is a hosted cloud service; models run in Tinfoil's TEEs.
 Tinfoil is the co-op's **architecturally-private** tier (contrast: GreenPT and
 Regolo are the renewable tier, with policy-based privacy, not TEE).
 ## The architecture — read this before touching the Tinfoil route
 LiteLLM does **not** talk to Tinfoil's enclave directly. Every Tinfoil model in the
 gateway config points at:
 ```
 api_base: http://127.0.0.1:3301/v1
 ```
 That is a **local proxy** running **inside the gateway container**
 (`/app/data/tinfoil-proxy -b 127.0.0.1 -p 3301`). LiteLLM speaks **plaintext
 OpenAI** to the proxy; the proxy **verifies the enclave attestation and wraps the
 request/response bodies in EHBP (HPKE)** before forwarding to the Tinfoil
 enclave. The provider key is set via the environment variable `TINFOIL_API_KEY`.
 Consequences:
 - **The proxy is load-bearing.** If it is not running, every Tinfoil model fails;
  LiteLLM's own config still looks correct. Check the process, not just the config.
 - **`http://127.0.0.1:3301` is intentional, not a misconfiguration.** Do not
  "fix" it to a public Tinfoil URL — the encryption happens in the proxy.
 - The proxy binary lives on the persistent volume (`/app/data/tinfoil-proxy`), so
  it survives restarts; it is started by the app's `start.sh`.
 ## What we serve, and what it costs
 Rates as recorded in the gateway config, **USD per 1M tokens** unless noted:
 | our model | input | output | cached input |
 |---|---|---|---|
 | `tinfoil/deepseek-v4-1-flash` | $0.65 | $1.45 | $0.13 |
 | `tinfoil/gpt-oss-120b` | $0.15 | $0.60 | — |
 | `tinfoil/glm-5.3` | $1.80 | $5.75 | $0.45 |
 | `tinfoil/whisper-large-v3-turbo` (STT) | $0 / 1M | — | **$0.01 per request** |
 | `tinfoil/voxtral-tts` (TTS) | $0 / 1M | — | **$0.002 per request** |
 | `tinfoil/nomic-embed-text` (embeddings) | $0.05 | $0 | — |
 - **`tinfoil/glm-5.3` replaced `tinfoil/glm-5-3-flash`** (Tinfoil deprecated the
  flash variant on 2026-10-09). Text-only (no vision), reasoning + tool calling.
 - **`tinfoil/deepseek-v4-1-flash` and the glm-flash variant carry an
  "Experimental: may be deprecated with minimal notice" flag from Tinfoil** —
  which is part of why the GreenPT counterpart is the default.
 ### Audio is billed per REQUEST, and needs a build-time patch
 Tinfoil bills the audio models **per request** (its model list carries a
 `requestPrice`), not per token. Reading only the two token prices made TTS look
 free and left STT billed off an older catalog. **LiteLLM cannot price per request
 on the audio paths**, so `patch_audio_request_cost.py` (built into the image via
 the Dockerfile) makes the cost calculator honour `input_cost_per_request`.
 **Re-check that build-log line after every LiteLLM upgrade.**
 One non-obvious load-bearing detail: for TTS, `input_cost_per_character: 0` is
 **required**. Without it the TTS cost branch raises `ValueError("cost for tts call
 is None")` and aborts the whole calculation, so the spend write is skipped and
 speech bills nothing — whatever `input_cost_per_request` says.
 **Audio keep-alive** (the Cloudron scheduler task `audio_keepalive`, every 10 min)
 pings the Tinfoil audio models so voice mode does not pay the ~24s TEE cold-start
 after idle. It is **TTS-only by design**: the STT warm ping was disabled
 2026-10-11 because it costs $0.01/request — roughly $52/month at 6 pings/hour.
 `docs.inference.coop`-style "keep-alive economics" detail is in the script.
 ### The embedding model is operator-only
 `tinfoil/nomic-embed-text` (768-dim) is **not** a member-facing model — it is
 scoped to the co-op's own admin team (`model_info.team_id`) for the docs-RAG, so
 member keys cannot call it. The portal also filters it out of the chat-facing
 model list by its `mode: embedding`.
 ## Fallbacks
 The router treats `tinfoil/deepseek-v4-1-flash` as the fallback for
 `greenpt/deepseek-v4.1-flash`, and `tinfoil/gpt-oss-120b` as the fallback for
 `tinfoil/deepseek-v4-1-flash`. The timeout is deliberately generous (180s) with
 `TimeoutErrorRetries: 0` — DeepSeek generations routinely exceed 30s, and a
 timed-out generation is **already billed** by Tinfoil, so re-hitting it re-bills.
Author	SHA1	Message	Date
inference-bot	2c38f4d2c3	upgrade window 2026-11-09: executed (gateway 1.108.3, chat overlay 1.4.8) — record the outcome, the 400 MB task-limit OOM and its --no-backup remedy, and refresh the live image pins	2026-11-09 05:04:27 +00:00
inference-bot	52e3c9d502	runbook: re-check the upgrade window against live state — correct the rollback tag, mark the prebuilt candidate stale, extend the marker list to ten, and flag the failing backups as the open risk	2026-11-09 04:17:06 +00:00
inference-bot	5a4492e725	ops: document the member provisioning boundary (org-backed, credit-pack, manual, hidden-email) and the daily paid-but-unprovisioned watchdog	2026-11-07 00:27:15 +00:00
inference-bot	0f40dcc519	runbook: add the standing supply-chain IoC check to the gateway verify steps	2026-11-06 04:11:11 +00:00
inference-bot	816961fb2d	runbook: 2026-11-08 upgrade window — LiteLLM 1.103.2→1.108.0 + LibreChat overlay 1.4.5→1.4.8 Prepared and validated on 2026-11-06; nothing deployed. Candidates: - code/litellm branch upgrade-v1.108.0 → image 1.0.0-20261106-9e25578 (8/8 patches verified in the built image and in a live sandbox run; image-gen spend row verified nonzero twice — fake + real Regolo probe; 36 additive migrations; budget_reset_at + portal API shapes unchanged). - co-op/librechat-custom branch bump-appstore-1.4.8 → overlay 20261106-c2aec74 (appstore 1.4.8 = tooling bumps on the same 0.8.8 upstream; all overlay anchors verified against the real 1.4.8 image — no porting needed). Runbook: LiteLLM first, member-scope verification, rollback = previous tag per app, window estimate 45-60 min.	2026-11-06 04:11:01 +00:00
inference-bot	fcd4ac6943	incident: portal's copied LiteLLM DB credential rotted after the 2026-11-05 gateway restore; all-time spend silently read $0 — timeline, fix, the digest spend-consistency guard, and the two-copies-of-one-secret dependency flagged for the next rotation	2026-11-05 19:10:26 +00:00
inference-bot	bb10d93412	incident: 2026-11-05 gateway addon loss during multi-app deploy — wiped LiteLLM DB, restored from the pre-deploy snapshot the same hour Full mechanics, impact, detection and prevention in the file. Portal member data was never at risk.	2026-11-05 16:36:05 +00:00
inference-bot	4d3f539a84	ops: record the decision to keep the historical ledger test rows (audit L7 — net zero; rewriting the money ledger is worse than the noise)	2026-11-05 16:26:17 +00:00
inference-bot	3bf152b58a	ops: record the LibreChat key-feed design decision (audit M2) — kept with monitoring; per-member fetch rejected as the riskier path	2026-11-05 16:21:52 +00:00
inference-bot	b1a1c70e8a	ops: record the allowance-shortfall manual-remediation ledger The portal's SHORTFALL_REMEDIATED constant now carries only the OC charge reference (de-PII'd from the public member-portal repo); the member identity and the timeline of the manual reset live here, keyed on the charge, plus the rationale for charging-keyed exclusion.	2026-11-05 15:59:13 +00:00
ops-bot	a9edaa3e4b	agent-fleet: record the Matrix homeserver accounts moved off the member docs The member-facing infrastructure page no longer documents who holds room power or which automation accounts exist. Record it here instead: @inference-admin as administrator of record, the retired @inference-bot account and its two un-leavable rooms, and the @ops_audit review account. Also repairs a heading glitch carried over from the docs page: a line began with a bare '#infrastructure', which Markdown parsed as an <h1>. The room names are now written inline so no line starts with '#'.	2026-11-04 21:29:29 +00:00
inference-bot	835aed2852	Record the DNS cutover as complete; flag the old box as an unresolved exposure (#5 )	2026-11-04 14:49:04 +00:00
inference-bot	6f2a6d01ed	docs: record the DNS cutover as complete; flag the old box as an unresolved internet exposure - Migration addendum: replace the 'DNS split' wording with the verified state — every hostname resolves to the new box (A+AAAA) per Cloudflare's public resolver; nothing points at the old box. - Reframe the old box: not a benign rollback spare but a decommissioned server still alive and internet-exposed (22/80/443/587/993 open), holding old data and config, serving no member traffic. UNRESOLVED — operator decision to shut it down or firewall it. Do not act on it without the operator's word. - README: add a 'Known open items' section listing the exposure, the CI failure, the unpushed CI-agent repo and the unconfirmed member mailbox contents.	2026-11-04 14:48:53 +00:00
inference-bot	6e777be842	Infrastructure audit: app inventory, backups, CI, provider pricing, agent fleet (#4 )	2026-11-04 14:47:18 +00:00
inference-bot	2eb88f55b4	docs: infrastructure audit — app inventory, backups, CI, provider pricing, agent fleet Audit of the operator docs against live reality (2026-11-04): - app-inventory.md: all 19 Cloudron apps with id, fqdn and registry image pin. Corrects the portal path (code/member-portal, not co-op/member-portal) and the panel path (local/coop.inference.admin-panel, not code/admin-panel). - maintenance/backups.md: the two backup systems; the deliberate per-user restic cache split and the --group-by "" retention fix, with why. - maintenance/woodpecker-ci.md: CI server + agent, image build/pin, manual-only trigger, git-lfs requirement, and its current not-yet-green state (agent's job-collection long-poll gets HTTP 500 from the server). - provider-pricing/: index + GreenPT, Tinfoil (local EHBP proxy architecture) and the Metis /v1/systemone decision-model route; regolo.md updated to record the gateway prices were applied. - agent-fleet.md: steward, infrastructure-bot and socialbot (Unix users, gateway units, teams); member-bot tier 2 planned but not yet provisioned. - README: file list; operator-token-runbook: correct the membership_sweep cadence (3x daily, not nightly); migration doc: post-migration addendum (16->19 apps, DNS split resolved, old box still on with apps stopped).	2026-11-04 14:46:12 +00:00
inference-bot	8b33d823c1	Merge pull request 'operator-token-runbook: credential rotation policy and reminder (co-op/support#35)' (#3 ) from rotation-policy into main	2026-11-04 14:37:22 +00:00