upgrade window 2026-11-09: executed (gateway 1.108.3, chat overlay 1.4.8) — record the outcome, the 400 MB task-limit OOM and its --no-backup remedy, and refresh the live image pins

Prepared and validated on 2026-11-06; nothing deployed. Candidates:
- code/litellm branch upgrade-v1.108.0 → image 1.0.0-20261106-9e25578
  (8/8 patches verified in the built image and in a live sandbox run;
  image-gen spend row verified nonzero twice — fake + real Regolo probe;
  36 additive migrations; budget_reset_at + portal API shapes unchanged).
- co-op/librechat-custom branch bump-appstore-1.4.8 → overlay 20261106-c2aec74
  (appstore 1.4.8 = tooling bumps on the same 0.8.8 upstream; all overlay
  anchors verified against the real 1.4.8 image — no porting needed).
Runbook: LiteLLM first, member-scope verification, rollback = previous tag
per app, window estimate 45-60 min.

README.md

@@ -30,6 +30,8 @@ Read it on the web here: <https://git.inference.coop/co-op/ops-docs>
 | `maintenance/librechat-sandbox.md` | Operating the LibreChat evaluation instance (config, overlay image, key sync, pitfalls) |
 | `maintenance/backups.md` | The two backup systems (agent-host restic + Cloudron app backup): schedules, retention, the cache split and the grouping fix |
 | `maintenance/woodpecker-ci.md` | The self-hosted CI server and agent: build/pin, manual-only trigger, git-lfs, and its not-yet-green state |
+| `maintenance/provisioning-boundary.md` | Which Open Collective contributors auto-provision as members and which are deliberately excluded (org-backed, credit-pack, manual, hidden-email), plus the daily paid-but-unprovisioned watchdog |
+| `maintenance/2026-11-08-litellm-1.108.0-librechat-1.4.8.md` | The Sunday-evening upgrade-window runbook (gateway → 1.108.0, chat overlay → appstore 1.4.8): candidates, order, exact commands, per-step verification, rollback, window estimate |
 | `incidents/` | Post-incident write-ups, one file per incident |
 | `migration/` | The Cloudron server-migration record: the plan, the DNS change sheet, and the captured pre-migration state |
 | `provider-pricing/` | What each model provider charges, and when it changed (GreenPT, Tinfoil, Regolo, and the decision-model route) |
@@ -56,6 +58,21 @@ answers proves nothing about whether a record exists. At least two listed
 records turned out not to exist, and the real record inventory is only visible
 in the registrar's own list.
 
+## Known open items — recorded, not hidden
+
+These are genuinely unresolved; a tidy document that omits them would be wrong.
+
+- **The old box `179.199.135.72` is still alive and internet-exposed** (22/80/443/587/993
+  open), holding old data and config. Nothing in DNS points at it, so it serves no
+  member traffic — but it needs an operator decision: **shut it down or firewall it**.
+  Do not act on it without the operator's word. See `migration/cloudron-server-migration.md` (addendum).
+- **CI is not green.** The Woodpecker agent is healthy but the server answers its
+  job-collection stream with HTTP 500, so no pipeline can run. See `maintenance/woodpecker-ci.md`.
+- **The CI agent image source (`/home/ubuntu/repos/woodpecker-agent`) has no git
+  remote** — it exists only on the agent host.
+- **Member mailbox contents after the migration** are still unconfirmed (needs a
+  real IMAP login).
+
 ## No secrets here
 
 Audited 2026-11-01, including the full commit history: no credentials, tokens,

agent-fleet.md

@@ -63,3 +63,33 @@ From `GET https://gateway.inference.coop/team/list` (2026-11-04): 97 teams — o
 ($135/30d, the co-op account), the two `agent:*` teams above, and
 `sandbox:librechat-test` ($3). Agent teams carry a **small budget and a flash-tier
 model allowlist** enforced at the gateway.
+
+## Matrix homeserver accounts
+
+Beyond the agent bots above, the homeserver carries two automation accounts.
+Also recorded here: the room account retired in October 2026, moved out of the
+member-facing docs.
+
+- **`@inference-admin:inference.coop` — the room and server administrator.** A
+  Synapse server admin, used for provisioning rooms, managing membership, and
+  retiring rooms. It is the administrator of record for the co-op's Matrix rooms
+  — general, community, design, skillshare, updates and infrastructure — and
+  holds the portal's member-invite credential (`MATRIX_BOT_TOKEN`). It **never
+  speaks in a member room**; a message from it is a mistake, not a feature.
+- **`@inference-bot:inference.coop` is retired (2026-10-29).** It was the
+  original room-administration account (it created the space and most of the
+  rooms) and held a Synapse server-admin flag. Its room power passed to
+  `@inference-admin`; its admin flag was dropped, it was deactivated, and every
+  device and token was deleted. Nothing should use it again.
+- **`@ops_audit:inference.coop`** — a server-admin account created for the
+  October 2026 security review. Retained only until its tokens are revoked.
+
+Two loose ends, recorded so nobody re-investigates them from scratch: the
+retired account remains joined to two rooms it cannot leave (they reject every
+write — `500` on leave), and those rooms are also why they can't be deleted; and
+its *name* is still the tempting one, so check `whoami` rather than the name
+before trusting any account that looks like it.
+
+**House rule: power belongs to `@inference-admin`, voice belongs to `@inferencebot`.**
+Don't create an account whose name differs from an existing one by a character or
+two — that lookalike confusion is what the rule exists to undo.

app-inventory.md

@@ -20,17 +20,17 @@ does not hold it.
 | FQDN | App id | Label | Image (pin) | Install |
 |---|---|---|---|---|
 | `inference.coop` | `55f5f904-3233-44de-ad7b-5dc7c8533b75` | — | `cloudron/io.cloudron.surfer:202611021342220000` | store (Surfer — marketing site) |
-| `chat.inference.coop` | `4023ccbf-d57a-4500-97df-14de9cf3516e` | AI Chat | `registry.inference.coop/co-op/librechat-custom:20261031-2a98892` | **custom** (LibreChat overlay) |
+| `chat.inference.coop` | `4023ccbf-d57a-4500-97df-14de9cf3516e` | AI Chat | `registry.inference.coop/co-op/librechat-custom:20261106-c2aec74` | **custom** (LibreChat overlay) |
-| `chat-legacy.inference.coop` | `50ae6c97-e851-4d17-b013-51ca7a4e5be0` | Legacy AI Chat | `cloudron/com.openwebui.cloudronapp:202611030420300000` | store (Open WebUI) |
+| `chat-legacy.inference.coop` | `50ae6c97-e851-4d17-b013-51ca7a4e5be0` | Legacy AI Chat | `cloudron/com.openwebui.cloudronapp:202611040420410000` | store (Open WebUI) |
-| `portal.inference.coop` | `d4f05e4b-75c6-492e-9faf-7afb0712fc94` | — | `registry.inference.coop/code/member-portal:20261104-d076729` | **custom** |
+| `portal.inference.coop` | `d4f05e4b-75c6-492e-9faf-7afb0712fc94` | — | `registry.inference.coop/code/member-portal:20261108-2954fe60` | **custom** |
-| `dashboard.inference.coop` | `daf8750e-2683-4104-83ea-f77d1b4601e8` | Dashboard | `registry.inference.coop/code/member-dashboard:20261103-3453131` | **custom** |
+| `dashboard.inference.coop` | `daf8750e-2683-4104-83ea-f77d1b4601e8` | Dashboard | `registry.inference.coop/code/member-dashboard:20261108-1e6e9e48` | **custom** |
-| `panel.inference.coop` | `ae187071-7a43-4a6e-8a02-986609a66a1a` | Admin Panel | `local/coop.inference.admin-panel:0.1.0-1793634279942` | **custom** |
+| `panel.inference.coop` | `ae187071-7a43-4a6e-8a02-986609a66a1a` | Admin Panel | `registry.inference.coop/code/admin-panel:20261108-b23cc67` | **custom** |
-| `gateway.inference.coop` | `adababd4-2427-4bdd-9bf7-6bdad0302610` | LiteLLM | `registry.inference.coop/local/ai.coop.litellm:1.0.0-20261103-d6cfe78` | **custom** |
+| `gateway.inference.coop` | `adababd4-2427-4bdd-9bf7-6bdad0302610` | LiteLLM | `registry.inference.coop/local/ai.coop.litellm:20261109-c43d45c` | **custom** |
 | `matrix.inference.coop` | `aa6f4597-6206-4509-b32e-c89750b2c1d8` | Matrix Chat | `cloudron/org.matrix.synapse:202611030719520000` | store (Synapse) |
 | `git.inference.coop` | `c91c13aa-30ed-4af9-9599-7a87ab4f221d` | Code | `cloudron/io.gitea.cloudronapp:202611011719500000` | store (Gitea) |
 | `registry.inference.coop` | `c257ba3c-6737-4375-ad51-ea19586bf3f0` | — | `cloudron/io.cloudron.container-registry:202610221511140000` | store (container registry) |
 | `vault.inference.coop` | `cb838bcf-f2e2-4670-b117-dca04b0ee957` | — | `cloudron/com.github.bitwardenrs:202611020820270000` | store (Vaultwarden) |
-| `forum.inference.coop` | `8b1a972c-08af-4e92-852c-d573c37b01ad` | Forum | `cloudron/org.loomio.cloudronapp:202611010321240000` | store (Loomio) |
+| `forum.inference.coop` | `8b1a972c-08af-4e92-852c-d573c37b01ad` | Forum | `cloudron/org.loomio.cloudronapp:202611050321160000` | store (Loomio) |
 | `newsletter.inference.coop` | `eed2217c-b165-430e-bc29-7ccfef9b11e4` | — | `cloudron/app.listmonk.cloudronapp:202610220221130000` | store (Listmonk) |
 | `search.inference.coop` | `d7c73fff-ee42-4db0-98c0-4ce7a8ab35cc` | — | `cloudron/io.github.ascimoo.searx:202611012319290000` | store (SearXNG) |
 | `cloud.inference.coop` | `fd261d67-b790-424f-9e82-f74883ee8585` | — | `cloudron/com.nextcloud.cloudronapp:202610211606270000` | store (Nextcloud) |

incidents/2026-11-04-allowance-shortfall-remediation.md

@@ -0,0 +1,26 @@
+# Allowance shortfall remediation — manual-remediation ledger (2026-11-04/05)
+
+The portal's shortfall mechanism (`derive_shortfalls` in
+`member-portal/app/main.py`) credits a member's missing allowance whenever OC
+realigned a charge onto the collective billing day and truncated a paid cycle.
+Each credit is keyed on the OC charge reference (`tx_…` publicId) so a re-run
+never credits twice.
+
+## Entries in `SHORTFALL_REMEDIATED` (code constant, no PII by design)
+
+| OC charge reference | Why it is excluded from automatic shortfall crediting |
+|---|---|
+| `[redacted member reference]` | The truncated cycle this charge created was already made whole by a **manual allowance reset on 2026-11-04**, performed outside the mechanism (before the automatic path shipped). The member joined mid-October; OC moved their next charge to the 2 Nov common billing day, so their cycle ran ~17 days. The reset re-anchored the allowance to the member's own join day, which is the policy now in code. Later charges of this member form new cycles and are credited normally by the automatic path. |
+
+The member's identity is in the portal `credit_ledger` / `members` tables
+(the credit row itself) — the public portal repository deliberately carries
+only the charge reference (audit H2, 2026-11-05: the constant used to embed
+the member's email and billing timeline; the portal repo is public).
+
+## Why the exclusion is keyed on the charge, not the member
+
+`derive_shortfalls` reads charge history from OC's order list; the charge that
+truncated the cycle is the idempotence key everywhere else in the flow
+(`credit_reference_exists`). Keying the manual-remediation exclusion on the
+same charge means a sweep re-run can neither double-credit the charge nor
+skip a genuinely new truncated cycle for the same member.

incidents/2026-11-05-gateway-addon-loss.md

@@ -0,0 +1,95 @@
+# Incident 2026-11-05 — gateway/panel briefly deployed with the portal's manifest; gateway LiteLLM database wiped and restored
+
+Self-reported by the audit-fix session (inferencebot), 2026-11-05 ~18:00 UTC.
+Damage fully recovered the same hour. Nathan: see "What you need to know" at
+the end.
+
+## What happened
+
+During the audit fix-order deploys, three apps (member-portal, admin-panel,
+gateway) were updated in one shell loop that stayed `cd`'d into the
+**member-portal repo**. `cloudron update` reads `CloudronManifest.json` from
+the CURRENT DIRECTORY, and the portal manifest carries a `dockerImage` pin —
+which wins over any `--image` argument. So:
+
+1. The **admin-panel** app received the portal's manifest (portal image
+   `20261105-ca3b8db`, portal manifest id `coop.inference.member-portal`,
+   portal addons) for ~10 minutes.
+2. The **gateway** app received the portal's manifest too. The portal manifest
+   has **no postgresql addon** → Cloudron **dropped the gateway's Postgres
+   database** (the LiteLLM key/team/spend store). The panel recovery and the
+   gateway image recovery ran ~10 and ~25 minutes later; the gateway's addon
+   was re-created on recovery as an EMPTY database with the same name.
+3. Result: from ~16:29 to ~17:20 UTC every member key 401'd (184 keys gone
+   from the gateway's view), chat was down for the membership. The portal app
+   itself was never affected (its own manifest was the one deployed).
+
+## Impact
+
+- Gateway member inference: **down ~50 minutes** (16:29–17:20 UTC). Master-key
+  surfaces (/v1/models with the master key) worked, which made the outage look
+  partial at first glance.
+- Gateway LiteLLM Postgres: keys 184→0, teams 100→1, SpendLogs 125k→17 at the
+  moment of detection.
+- Spend rows for live traffic between the 16:28:12 backup and the database
+  wipe (a few minutes) are unrecoverable; everything after the wipe was
+  refused calls (no spend to lose).
+- Panel: wrong image for ~10 min, no data loss (stateless; no durable addon
+  data).
+- No member data (portal SQLite) was ever at risk — that lives in the portal
+  app's own localstorage volume, which was never touched.
+
+## Detection
+
+Post-deploy verification battery failed exactly where it should: the
+member-scope chat call returned 401, the panel's container md5 matched the
+PORTAL's main.py, and `/key/list` returned 0 keys. All three surfaced within
+minutes of the recovery deploy.
+
+## Resolution (all verified)
+
+1. Redeployed panel and gateway **from their own repo directories** with their
+   correct manifests/images (16:47, 16:52 UTC).
+2. Gateway database confirmed wiped (0 keys) → located the deploy-time app
+   snapshot `app_…_v1.0.0_5aad8953` (16:28:12 UTC — taken seconds before the
+   damage, contains the addon dump).
+3. `cloudron restore --app <gateway> --backup …_5aad8953` (17:2x UTC) →
+   **184 keys, 100 teams, 125,106 spend rows restored**; /v1/models 200.
+4. Re-applied the intended gateway image update `1.0.0-20261105-e161db9` from
+   the litellm repo dir (17:3x UTC) → verified DB still intact after the
+   redeploy (184/100/125,110 rows — the +4 spend rows are live traffic billing
+   again), member-scope chat completes for real (tinfoil/glm-5.3, tokens
+   billed), contactEmail + corrected start.sh live, audio_keepalive scheduler
+   restored.
+5. Panel pin bumped to its own new image `20261105-0713551` and deployed from
+   its own repo dir; container md5 matches the repo; panel 200.
+
+## Root cause
+
+Running `cloudron update` for multiple apps from a shared shell whose working
+directory belonged to a DIFFERENT app's repo. The CLI's behavior (CWD manifest
+wins; `--image` silently ignored when a `dockerImage` pin exists) is documented
+in the co-op skill, but nothing in the loop enforced "one app, one repo, one
+CWD".
+
+## Prevention (recorded in the deploy skill as a hard rule)
+
+**Never batch `cloudron update` for different apps in one shell without
+`cd`-ing to each app's own repo per command, and never run an update for an
+app while CWD holds another app's `CloudronManifest.json`.** Verify after
+EVERY deploy with both facts: the app's own `manifest.dockerImage` from the
+API (never the CLI's output) and a container-content check (md5/marker).
+Additionally: a manifest change that alters the `addons` block of an
+app with a database addon drops/reprovisions that addon's data — treat addon
+blocks as data-bearing, not metadata.
+
+## What you need to know (Nathan)
+
+- Chat was down for members ~50 minutes this afternoon (UTC); it is verified
+  working again with a real member key and real billed tokens.
+- The gateway database was restored from the automatic pre-deploy snapshot;
+  only a few minutes' worth of spend rows from before the outage are missing.
+  No member key, team, or balance data was lost.
+- The portal's member database was never touched.
+- The audit fix order otherwise completed; a separate summary has the full
+  stage-by-stage record.

incidents/2026-11-05-portal-alltime-spend-zero.md

@@ -0,0 +1,100 @@
+# Incident 2026-11-05 — the portal's copied LiteLLM DB credential rotted after the gateway restore; all-time spend silently read $0
+
+Companion to [`2026-11-05-gateway-addon-loss.md`](2026-11-05-gateway-addon-loss.md)
+(the deploy mistake that made the gateway restore necessary). This document
+covers the second-order failure that restore caused. No spend data was lost at
+any point; the failure was in the portal's **read path**, and it failed in a
+way that looked like clean data.
+
+## What broke
+
+The portal aggregates all-time spend, model usage and the leaderboard by
+querying LiteLLM's Postgres directly, using a **copy** of the gateway's
+postgres addon credential: the portal app's `LITELLM_DB_*` env mirrors the
+gateway's `CLOUDRON_POSTGRESQL_*` runtime env. Two things then combined:
+
+1. **The copy rotted.** Restoring a Cloudron app re-provisions its postgres
+   addon and **rotates its credential**. When the gateway was restored
+   (16:32:35 UTC, see timeline), the gateway got the new credential re-injected
+   at runtime — but the portal's copy went stale at that exact moment.
+2. **The failure degraded silently.** The portal's aggregation catches a DB
+   auth failure, logs a WARNING, and degrades to an empty result — all-time
+   spend read `$0.00`. The panel and member dashboard rendered that $0 while
+   the *cycle* figures (read from LiteLLM's API, unaffected) still looked
+   plausible, so nothing on screen looked broken. The daily digest had no
+   spend-consistency check, so it also reported nothing.
+
+## Timeline (UTC, 2026-11-05 — restore/configure times from the platform eventlog)
+
+- **16:29:01** — during the audit-fix deploys, the gateway received the
+  portal's manifest; its postgres addon was dropped. (Sibling incident.)
+- **16:32:35** — the gateway was restored from the pre-deploy snapshot to
+  recover the sibling incident. The restore re-provisioned the postgres addon
+  and **rotated its credential** — the moment this incident began.
+- **16:32:35 → 18:51** — every portal aggregation query against LiteLLM's
+  Postgres failed authentication and degraded to empty: panel + dashboard
+  all-time spend read $0. LiteLLM itself kept serving and billing throughout;
+  the portal log carried only WARNING lines ("all-time spend query failed"),
+  easy to read as noise.
+- **18:51:22** — the portal's `LITELLM_DB_PASSWORD` was re-synced from the
+  gateway's runtime env (value staged file→argv through the Cloudron API;
+  never printed, never committed). Aggregation restored immediately.
+- **Same day, verification:** the source of truth (LiteLLM's own Postgres, read
+  gateway-side) held 125,385 rows / **$655.13** all-time; member teams 97 /
+  $652.88 — **no gap through the incident window**, because the gateway's own
+  writes were never affected. The member dashboard was verified restored (a
+  sample member's all-time figure matched the source exactly).
+- **Same day, panel verification (final):** panel running image
+  `code/admin-panel:20261105-0713551` (matches the repo pin; code marker
+  confirmed in the container). Its data path is `GET /api/overview` → portal
+  `GET /admin/overview` (server-side, `X-Admin-Token: ADMIN_TOKEN`) — replayed
+  live from the panel container, HTTP 200. Displayed all-time total
+  **$653.08** with the co-op's own-accounts split **$170.73**, both matching an
+  independent SQL read of the source exactly; four sample members' all-time
+  figures matched the source to the cent. The full-table total ($655.33–655.41
+  while verifying — the ledger accrues live) decomposes exactly: members +
+  operators $653.08 + agent teams $1.18 (shown in the panel's Agents section) +
+  sandbox/unattributed rows $1.07 (displayed nowhere, accepted knowingly).
+
+## Fix
+
+1. **Immediate:** re-sync the portal's `LITELLM_DB_PASSWORD` from the gateway's
+   runtime env (18:51:22). Verified by re-reading the portal's aggregation
+   against the source, not by the absence of log lines.
+2. **Standing guard (added the same day):** the daily digest
+   (cron job `95e5b8fece86`) gained a **SPEND CONSISTENCY** section: it runs
+   `~/.hermes/scripts/spend_consistency_check.py`, which compares the portal's
+   reported all-time spend (`/admin/overview` `totals.total_all_time_spend` +
+   the `agents[]` rows) against LiteLLM's own SpendLogs total — read from the
+   **gateway** app, the credential owner, so the source does not depend on the
+   copied credential. A $0 portal figure, a missing field, a mismatch beyond a
+   $5 tolerance (the deliberately-unattributed sandbox rows are ~$1), or an
+   unreadable source is a FAILED check reported near the top of the digest —
+   never as clean. All three failure branches were exercised (unreadable / $0 /
+   mismatch) and fire with exit 1; the live comparison passes with both numbers
+   printed. Do not substitute LiteLLM's `/global/spend` for the source: it
+   returns a ~30-day figure (~$576 when the table holds ~$655), not all-time.
+3. **Practice (recorded here because it bit):** after any `app.restore` on this
+   box, diff the credential **fingerprint** (`sha256`[:8] of the value — never
+   the value) between the owner app and every app holding a copy, and re-sync
+   any that differ.
+
+## Architectural dependency — read before rotating anything
+
+**The portal does not hold its own LiteLLM DB credential. It aggregates from
+LiteLLM's Postgres using a COPY of the gateway's addon credential.** They are
+two copies of one secret with no shared source, and Cloudron rotates the addon
+secret on every restore (and on any addon re-provision). Therefore:
+
+- **A gateway DB credential rotation or restore MUST be followed by syncing the
+  portal's copy.** The rotation itself announces nothing; the only symptoms are
+  the portal's WARNING lines and $0 figures downstream.
+- **Long-term fix (flagged, deliberately not done now):** both apps should
+  source the credential from ONE place — a shared secret location — or the
+  portal should not hold a DB credential at all (e.g. the gateway exposes a
+  read-only aggregation endpoint the portal calls). Either removes the
+  two-copies-of-one-secret failure mode.
+- Until that lands, the digest's SPEND CONSISTENCY check is the drift alarm:
+  if it ever reports FAILED with a $0 or mismatched portal figure, check the
+  portal's `LITELLM_DB_PASSWORD` against the gateway's current addon credential
+  first, before theorising about code.

maintenance/2026-11-08-litellm-1.108.0-librechat-1.4.8.md

@@ -0,0 +1,358 @@
+# Upgrade window — 2026-11-08 (Sunday evening): LiteLLM 1.103.2 → 1.108.0, LibreChat overlay 1.4.5 → 1.4.8
+
+Both upgrades were **prepared and validated in advance** (2026-11-06); nothing has
+been deployed. This runbook is written to be followed by any operator during the
+member-watched maintenance window. Read the gotchas in §6 before touching anything —
+this app pair has a history of deploys that look successful while changing nothing.
+
+**Candidates (already built and validated, sitting in the registry):**
+
+| App | Candidate | Rollback (currently deployed) |
+|---|---|---|
+| LiteLLM gateway (`adababd4-2427-4bdd-9bf7-6bdad0302610`, gateway.inference.coop) | branch `upgrade-v1.108.0` in `code/litellm`; prebuilt image `registry.inference.coop/local/ai.coop.litellm:1.0.0-20261106-9e25578` (STALE — see §0 and §3.1) | image `registry.inference.coop/local/ai.coop.litellm:20261107-201000-407e751e2` — the tag running tonight, verified present in the registry. `1.0.0-20261105-e161db9` is two deploys older and is NOT the rollback target |
+| LibreChat chat (`4023ccbf-d57a-4500-97df-14de9cf3516e`, chat.inference.coop) | branch `bump-appstore-1.4.8` in `co-op/librechat-custom`; prebuilt overlay `registry.inference.coop/co-op/librechat-custom:20261106-c2aec74` | overlay `registry.inference.coop/co-op/librechat-custom:20261031-2a98892` (verified present) |
+
+**Prep evidence (what was already verified, so Sunday only re-confirms it):**
+
+- LiteLLM 1.108.0 candidate: all 8 build patches verified inside the running candidate
+  (file markers + build-log markers, zero WARNs); litellm reports 1.108.0.
+- `patch_image_cost.py` (image-gen spend): a real image generation through the
+  candidate produced a **nonzero success SpendLogs row** — twice (fake OpenAI-shaped
+  endpoint and the real Regolo Qwen-Image API): `spend 0.0011403264` for 256×256 at
+  `$1.74e-8/px`, `call_type aimage_generation`, `status success`.
+- `patch_passthrough_token_cost.py` (Metis metering): a real `/v1/systemone` call
+  logged `[coop-passthrough-cost] input_tokens=148 rate=1.2e-07 -> response_cost=1.776e-05`
+  and the SpendLogs row matched exactly (148 × $0.12/1M), tokens recorded.
+- Member path: temp key generated with master auth → `/v1/models` (11 models) → real
+  chat completion (greenpt/glm-5.3-flash, "pong", usage 19/3 tokens, spend row written).
+- Migrations: 36 new between 1.103.2 and 1.108.0, **all additive** (no unguarded
+  DROP/DELETE/UPDATE). A fresh DB ran the full 220-migration chain cleanly; the live
+  ledger (1.103.2) will apply only the 36 new ones.
+- `budget_reset_at`: column unchanged on `LiteLLM_TeamTable`; a direct-DB write
+  (the portal's exact pattern) plus API read-back verified on the candidate.
+- Portal API compatibility: `/team/list` (bare list) and `/key/list`
+  (`{keys, total_count, …}`) shapes are **unchanged**; a bare `/team/list` call still
+  returns ALL teams (pagination only applies when `page`/`page_size` are passed —
+  verified with 11 teams); `/team/update` still drops `budget_reset_at`, so the
+  portal's direct-DB write remains required. SpendLogs gains only an additive
+  `billing_agent_id` column — digest queries are unaffected.
+- LibreChat 1.4.8 (appstore `ai.librechat.cloudronapp`): changelog 1.4.5 → 1.4.8 is
+  tooling only — uv 0.13.6/0.13.7 and Node.js 22.23.4 (security fix, 1.4.7); the
+  LibreChat upstream stays 0.8.8. Every overlay patch anchor (title tag, manifest
+  link, theme-color metas, webmanifest colors, "MCP Servers" / "Every AI for
+  Everyone." strings, `index.*.js`, `sw.js`) was verified against the real 1.4.8
+  image — **no overlay porting needed**; the candidate overlay built with all
+  markers (UI strings patched in 3 files, index/manifest branding applied).
+
+## 0. State re-check — 2026-11-09 04:15 UTC (read live before the window)
+
+Verified against the live box on the night of the window. Anything here that
+contradicts the prep notes above is newer and wins.
+
+- **Gateway:** image `registry.inference.coop/local/ai.coop.litellm:20261107-201000-407e751e2`
+  (deployed 2026-11-07 20:10 UTC), in-container version **1.103.2**, 8 expected models
+  + extras, healthy. **The rollback target is therefore THIS tag, not
+  `1.0.0-20261105-e161db9`** (see §3.3).
+- **Chat:** overlay `registry.inference.coop/co-op/librechat-custom:20261031-2a98892`,
+  appstore 1.4.5, healthy. `master` is 2 commits ahead of `bump-appstore-1.4.8`, but both
+  touch only `librechat.yaml` — a persistent-volume file that is NOT baked into the image,
+  so pinning the Nov-06 candidate does not revert them.
+- **Registry:** all four tags this plan needs are present —
+  `local/ai.coop.litellm:{1.0.0-20261106-9e25578, 20261107-201000-407e751e2}` and
+  `co-op/librechat-custom:{20261106-c2aec74, 20261031-2a98892}` (listed from the registry
+  app's own storage).
+- **`upgrade-v1.108.0` is 5 commits BEHIND `main`.** main gained
+  `patch_transcription_response_format.py`, `patch_audio_default_mp3.py` and
+  `audio_keepalive.py` (Dockerfile RUN/chmod lines) plus the STT/embedding model additions
+  on 2026-11-06/07, all after the candidate branch was cut. Merging the branch into main
+  (as §3 does) carries both sides — but the **prebuilt candidate image does NOT contain
+  them** (see the §3.1 fallback note and the extended marker list).
+- **Version target:** the branch pins **1.108.0**; latest stable upstream is **1.108.3**
+  (2026-11-07). The prep validates 1.108.0 only — ship 1.108.0 tonight unless someone
+  re-runs the anchor/migration validation against 1.108.3 first.
+- **Backups are NOT healthy.** Last successful Cloudron backup: **2026-11-08 21:18 UTC**
+  (box, encrypted v2, state normal). The scheduled **2026-11-09 04:00 UTC run FAILED**
+  (`tarPack pipeline error … aborted` — the same signature as the Infomaniak S3 quota
+  rejections of 2026-11-06/07/08). The pre-flight backups in §2.1 are this window's undo
+  and must be proven to land before anything is deployed.
+
+## 1. Order: LiteLLM first, then LibreChat
+
+LibreChat's post-upgrade checks (key sync, a real chat completion) run **through the
+gateway** — doing the gateway first means those checks also exercise the new
+gateway, and each upgrade stays an isolated abort point: if LiteLLM fails, roll it
+back and skip LibreChat (chat keeps serving on the old overlay); if LibreChat fails,
+the new gateway is already proven.
+
+## 2. Pre-flight (both apps, ~5 min)
+
+```bash
+set -a; . ~/.hermes/.env; set +a
+CR() { cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" "$@"; }
+
+# 2.1 Fresh update-rollback backups (DB + volume snapshot each)
+CR backup create --app adababd4-2427-4bdd-9bf7-6bdad0302610
+CR backup create --app 4023ccbf-d57a-4500-97df-14de9cf3516e
+CR backup list --app adababd4-2427-4bdd-9bf7-6bdad0302610 | tail -3   # note newest id
+CR backup list --app 4023ccbf-d57a-4500-97df-14de9cf3516e | tail -3
+
+# 2.1b PROVE the backup landed BEFORE deploying. The backup pipeline is failing on the
+#      Infomaniak S3 plan quota (last success 2026-11-08 21:18 UTC; the 04:00 UTC run on
+#      2026-11-09 failed again). Re-run `CR backup list --app <id>` and require a NEW id
+#      with state normal — an unchanged newest id, or no new entry, means this window has
+#      NO undo. Decide before deploying, not after.
+
+# 2.2 Record the member-visible baseline for after: number of keys and teams
+#     (use the MASTER key — it is in /app/data/.master_key inside the gateway:
+#      CR exec --app gateway.inference.coop -- cat /app/data/.master_key
+#      then GET /key/list and /team/list with it; count both. NEVER paste the key.)
+```
+
+## 3. LiteLLM 1.103.2 → 1.108.0 (~20–35 min)
+
+```bash
+cd ~/repos/litellm                       # ONE APP = ONE REPO DIR. Never run an update
+git fetch origin && git checkout main && git pull   # for another app from this cwd
+git merge upgrade-v1.108.0               # branch is pushed; approve the merge in Gitea
+                                         # if you prefer the PR path
+git log --oneline -1                     # note the sha
+```
+
+3.1 **Deploy (on-server build — the repo manifest has no `dockerImage` pin, so
+`cloudron update` builds and swaps; do NOT pass `--image`):**
+
+```bash
+CR update --app adababd4-2427-4bdd-9bf7-6bdad0302610
+```
+
+- The build output must contain **all TEN** patch markers (each is one line in the
+  build log; a missing line = that patch silently did not apply — STOP and check):
+
+  1. `INFO: patched budget-exceeded messages in …/auth_checks.py`
+  2. `INFO: patched image-cost registration in …/router.py`
+  3. `[coop-patch] applied to …/cost_calculator.py`
+  4. `[coop-patch] applied audio shared pricing to …/router.py`
+  5. `[coop-patch] applied: provider response headers are no longer forwarded …`
+  6. `expose-cost-headers: appended 20 cost header names to expose_headers …`
+  7. `[coop-patch] applied: passthrough-key-access -> …/route_checks.py`
+  8. `[coop-patch] applied: passthrough-token-cost -> …/success_handler.py`
+  9. `[coop-patch] applied skip-verbose_json to …` (patch_transcription_response_format.py)
+  10. `[coop-patch] applied TTS default mp3 to …` (patch_audio_default_mp3.py)
+
+- `audio_keepalive.py` must be copied and `chmod +x`'d in the build (start.sh calls it for
+  the TTS warm path). Markers 9–10 and this file are the 2026-11-06/07 audio work; missing
+  any of them means that work silently reverts to the pre-Nov-06 behaviour.
+
+- **Boot**: LiteLLM runs `prisma migrate deploy` (the 36 additive migrations) before
+  binding port 4000. Expect `502`/`ECONNREFUSED` on health checks for a few minutes
+  (10–20 in the bad case). **Never restart mid-wait** — a restart resets the
+  migration clock. Once `/health/liveliness` returns 200, continue immediately
+  ("up" ≠ "migrated"; the checks below are the proof).
+- **DO NOT use the prebuilt candidate as a flake fallback tonight.** It exists
+  (`…ai.coop.litellm:1.0.0-20261106-9e25578`, and `--image` would work because the repo
+  manifest carries no `dockerImage` field), but it was built from the branch tree and
+  therefore predates `patch_transcription_response_format.py`, `patch_audio_default_mp3.py`,
+  `audio_keepalive.py` and the STT/embedding model additions. Deploying it reverts the
+  2026-11-06/07 audio and model work. If the on-server build flakes, fix the build or build
+  the merged tree — the candidate image is not usable tonight.
+
+3.2 **Verify (in order — a failing early step means stop, not continue):**
+
+```bash
+# 1. version inside the running container
+CR exec --app gateway.inference.coop -- \
+  /app/.venv/bin/python3 -c "from importlib.metadata import version; print(version('litellm'))"
+#    → 1.108.0
+
+# 2. patch markers inside the RUNNING container (not the build log alone)
+CR exec --app gateway.inference.coop -- sh -c \
+  'grep -c "CO-OP PATCH (2026-10-19)" /app/.venv/lib/python3.13/site-packages/litellm/router.py; \
+   grep -c "x-litellm-response-cost-input" /app/.venv/lib/python3.13/site-packages/litellm/proxy/proxy_server.py'
+#    → 1 and 1
+
+# 2b. supply-chain IoC check (standing policy on every LiteLLM image change):
+CR exec --app gateway.inference.coop -- sh -c \
+  'ls /app/.venv/lib/python3.13/site-packages/litellm_init.pth 2>/dev/null; \
+   ls /root/.config/sysmon 2>/dev/null; \
+   ls /tmp/pglog 2>/dev/null; echo IOC-CLEAN'
+#    → must print ONLY "IOC-CLEAN" (no such files). Any hit = stop, quarantine, rotate
+#      the gateway env keys (master + providers live in that env).
+
+# 3. MEMBER-SCOPE key path (master-key tests alone prove nothing — master bypasses
+#    exactly the tables a broken migration empties):
+#    POST /key/generate  {"alias":"upgrade-verify-temp"}   (master auth) → temp key
+#    GET  /v1/models with the TEMP key                     → 11 models
+#    POST /v1/chat/completions {"model":"greenpt/glm-5.3-flash",
+#         "messages":[{"role":"user","content":"Reply with the single word: pong"}],
+#         "max_tokens":16}                                  → content + usage
+#    GET  /key/list (master)                               → same count as §2.2
+#    POST /key/delete {"keys":["<temp>"]}                  → 200
+
+# 4. CORS preflight (chat origin must be allowed):
+#    OPTIONS /v1/models with Origin: https://chat.inference.coop → 200 + allow-origin
+
+# 5. Cloudron state
+curl -sS -H "Authorization: Bearer $CLOUDRON_TOKEN" \
+  https://my.inference.coop/api/v1/apps/adababd4-2427-4bdd-9bf7-6bdad0302610 \
+  | python3 -c "import sys,json; a=json.load(sys.stdin); a=a.get('app',a); print(a['health'], a['runState'], a['manifest'].get('dockerImage'))"
+#    → healthy running registry.inference.coop/local/ai.coop.litellm:<new tag>
+
+# 6. Watch 10–15 min of real member traffic before closing:
+#    SpendLogs rows arriving with status 'success' (portal admin panel or
+#    GET /spend/logs with master auth). Failure rows with call_type = a route
+#    path and spend 0 are 401-refusals, not duplicates.
+```
+
+3.3 **Rollback (one command; the additive migrations make old-code-on-new-schema safe):**
+
+```bash
+CR update --app adababd4-2427-4bdd-9bf7-6bdad0302610 \
+  --image registry.inference.coop/local/ai.coop.litellm:20261107-201000-407e751e2
+```
+
+If member keys 401 after any image change, the schema is the suspect: read the
+exact error text from a member key call — a missing-relation error names the failing
+path; do NOT reissue keys. (Full recipe: `code/litellm` README, "Verify after
+upgrading".)
+
+## 4. LibreChat overlay 1.4.5 → 1.4.8 (~10–15 min)
+
+```bash
+cd ~/repos/librechat-custom              # again: one app = its own repo dir
+git fetch origin && git checkout master && git pull
+git merge bump-appstore-1.4.8            # overlay code is UNCHANGED apart from the
+                                         # FROM pin — appstore 1.4.8 carries the same
+                                         # LibreChat 0.8.8; only Node/uv/base bumped
+set -a; . ~/.hermes/.env; set +a
+
+# The candidate overlay is already built: pin + deploy via the repo's release tool
+./release.sh pin 20261106-c2aec74        # writes dockerImage into CloudronManifest.json, commits, pushes
+./release.sh deploy                      # cloudron update --force from THIS dir, then verify
+```
+
+4.1 **Verify** (release.sh verify runs the served-HTML checks; confirm the rest by hand):
+
+```bash
+./release.sh verify                      # title = Inference Co-op, x-coop-build meta present
+
+# key sync task ran (manifest scheduler `librechat_key_sync`, every minute):
+CR logs --app chat.inference.coop --lines 100 | grep -i "sync" | tail -5
+#    → the sync script prints its own md5 prefix each run; it must be the new build's
+
+# RAG service: the appstore package runs rag_api alongside; confirm no crash loop
+curl -sS -o /dev/null -w "%{http_code}\n" https://chat.inference.coop/   # 200 (SSO shell)
+
+# login via Cloudron SSO in a browser, then send one real chat completion through
+# LibreChat (this exercises chat → NEW gateway → provider end-to-end)
+```
+
+4.2 **Rollback:**
+
+```bash
+cd ~/repos/librechat-custom
+./release.sh pin 20261031-2a98892 && ./release.sh deploy
+```
+
+## 5. Window estimate
+
+| Phase | Time |
+|---|---|
+| Pre-flight + backups | 5 min |
+| LiteLLM build+swap (on-server build) | 5–10 min |
+| LiteLLM boot migrations + member-scope verification | 10–20 min |
+| LibreChat pin+deploy+verify (incl. an SSO login + one chat) | 10–15 min |
+| Post-watch on SpendLogs / close-out | 10 min |
+| **Total** | **~45–60 min** |
+
+## 6. Gotchas that have each bitten this stack once
+
+1. **One app = one `cd`, immediately before its update.** `cloudron update` reads
+   `CloudronManifest.json` from the CURRENT directory; updating app B from repo A's
+   dir ships A's manifest to B — on 2026-11-05 that dropped the gateway's postgres
+   addon and wiped the LiteLLM DB (~50 min member outage, restored from the
+   pre-deploy snapshot). The pre-deploy backups in §2 are the undo; addons-block
+   changes are DATA-BEARING.
+2. **A pinned manifest beats `--image`, silently.** LibreChat's repo manifest pins
+   `dockerImage` — always change the pin (release.sh pin), never pass `--image` to
+   it. LiteLLM's repo manifest has NO pin — `--image` works there (used for its
+   rollback and build-flake fallback above).
+3. **"App is updated." proves nothing.** Verify the running image tag via the API
+   AND a container-content marker (§3.2 steps 1–2, §4.1) before trusting any check.
+4. **Don't restart the gateway mid-migration-wait** — it resets the boot clock.
+5. **Do not "fix" `/team/update` dropping `budget_reset_at` today** — the portal's
+   direct-DB write is the known, verified workaround; it still works on 1.108.0.
+6. After both apps are up, `app-inventory.md` in this repo should be updated with
+   the new pins in the same session (it mirrors live state).
+7. **The prebuilt candidate is not the tree you are deploying.** The branch was cut five
+   commits behind `main`; the merge supplies the rest, the prebuilt image does not. Verify
+   the TEN build markers of §3.1 inside the running container, not the old eight.
+8. **A stale rollback reference is worse than none** — the runbook pointed at a Nov-05
+   image while the box was running a Nov-07 build, so "roll back" would have silently
+   dropped two days of member-visible fixes. Re-read the running tag (§0/§2.2) immediately
+   before any rollback, and roll back to what was actually running.
+9. **The backup pipeline is itself an open incident.** Assume no undo until a pre-flight
+   backup shows `state: normal` with a new id.
+
+## 7. If something looks wrong
+
+- Member keys 401 with a missing-relation error → schema/migration issue; see
+  §3.3 and the `code/litellm` README triage section. Restore path: the §2 backup,
+  then re-pin the old image.
+- Chat serves a stale shell after the overlay deploy → wait ~60 s (mid-swap) and
+  re-check; release.sh verify already retries fetches.
+- Anything else: halt at the failing step, note the exact command + output, roll
+  back that app only, and leave the other app on its new version if it verified.
+
+## 8. Executed — 2026-11-09, 04:41–05:05 UTC (what happened, and where it differed)
+
+Order followed as planned: LiteLLM first, then LibreChat. Two deliberate changes:
+
+1. **Target 1.108.3, not 1.108.0** — latest stable, and it pins the *same*
+   `litellm-proxy-extras==0.4.110`, so the migration set is exactly the one validated here
+   (36 new, all additive). All ten build-patch anchors were re-validated against the real
+   1.108.3 sources first; `patch_expose_cost_headers.py`'s 1.108.x anchor applied.
+2. **A pre-built, pre-verified image was deployed, not an on-server build** —
+   `20261109-c43d45c`, built from `upgrade-v1.108.3` (cut on top of `main`, so it carries the
+   2026-11-06/07 audio patches the original candidate predated) and exercised first as a
+   throwaway app (`gateway-dryrun.inference.coop`, its own Postgres/Redis, no member data).
+
+**The one thing that did not go as written — the pre-update snapshot OOMs.**
+A default `cloudron update` died at `Backup - Snapshotting app gateway.inference.coop`:
+task 1430 → `Task 1430 ran out of memory or terminated`, box log `Memory peak: 401.8M`,
+`oom-kill`. Cloudron's app-task service runs with a **hard 400 MB cgroup limit**
+(`startTask … "memoryLimit":400` from the system log), and this gateway's snapshot now
+exceeds it. Production was never at risk — the app kept serving the old image; only
+`installationState` went to `error`.
+**Workaround used: `--no-backup`**, with the 04:41 pre-flight app backups as the undo —
+which is what §2.1 is for. Next time: take the app backups first, then
+`cloudron update … --no-backup`.
+
+**Also learned the hard way:** `--wait-timeout` is a GLOBAL option, so
+`cloudron --wait-timeout 45m update …` works and `cloudron update --wait-timeout …` errors out.
+
+### What was verified live afterwards
+
+- **gateway** `…ai.coop.litellm:20261109-c43d45c` — litellm **1.108.3**, extras 0.4.110;
+  **all ten** patch markers present in the *running* container; `audio_keepalive.py`
+  executable; IoC check clean; migrations **185 → 221**, none failed or rolled back;
+  keys 210 / teams 111 unchanged; temp member key → `/v1/models` (15 models) → real
+  completion → **success SpendLogs row**; CORS preflight 200. Member traffic within ten
+  minutes: 73 success rows.
+- **chat** `…/librechat-custom:20261106-c2aec74` (appstore 1.4.8 / LibreChat 0.8.8) —
+  `release.sh verify` all checks passed, key sync running every minute (108 member keys,
+  revision matches the repo), Meili resync normal, served title still `Inference Co‑op`.
+
+### Migrations were rehearsed, not merely run
+
+Before the window, the live gateway database was restored from the 2026-11-08 stopgap
+snapshot into a local PostgreSQL **18.6** (prod's exact version) and the image's own
+entrypoint (`python -m prisma migrate deploy`) was run against it: **185 → 221 applied in
+12 s**, row counts identical (210 keys / 111 teams / 138,422 spend rows), **no table lost**
+(90 → 110, the 20 additions all new-feature tables). That is what made `--no-backup`
+acceptable on the real box.
+
+### Not rehearsed
+
+The LibreChat leg — a second chat app means ~2 GB of data plus its Meili/rag stack, for a
+change that is tooling-only (Node/uv; LibreChat upstream unchanged). Mitigation is the
+one-command rollback: `./release.sh pin 20261031-2a98892 && ./release.sh deploy`.

maintenance/ledger-test-rows.md

@@ -0,0 +1,15 @@
+# Test rows in the credit ledger — deliberately kept (audit L7, 2026-11-05)
+
+The portal `credit_ledger` carries a handful of historical test rows from the
+early build ("test credits (will revert)", "admin test - collective expense",
+and the matching reverts). They are **net zero** against member balances and
+every one is paired with its reversal.
+
+Decision: leave them in place. Rewriting the money ledger — even to remove
+test noise — destroys the audit trail's property of "what's in the table is
+what happened"; the pollution is cosmetic and far outweighed by the cost of
+ever being wrong about a ledger edit. This note is the record; no code or data
+change accompanies it.
+
+If the ledger ever grows a reporting consumer that must exclude them, filter
+by the `reason` text at query time rather than deleting rows.

maintenance/librechat-key-feed.md

@@ -0,0 +1,52 @@
+# LibreChat key feed (`/admin/librechat/keys`) — design decision (audit M2)
+
+Audited 2026-11-05. Status: **kept as-is, with monitoring**; a per-member
+fetch was considered and deliberately rejected.
+
+## What the route is
+
+`GET https://portal.inference.coop/admin/librechat/keys`, authed by a single
+static token (`LIBRECHAT_SYNC_TOKEN`, held by the portal and the LibreChat
+sandbox app). It returns every ACTIVE member's decrypted chat key so the
+sandbox's in-app key-sync (`librechat_key_sync`, every 10 min inside the
+LibreChat app; `sync-keys.js`) can seed LibreChat's per-user key store —
+members never see an API-key prompt, and chat bills to each member's own
+LiteLLM team/budget.
+
+## Blast radius (why the audit flagged it)
+
+The sync token unlocks every member's chat key in one response. Anyone holding
+the token (sandbox compromise, token leak from the sandbox env) reads the whole
+membership's keys. It is a deliberate exception to the admin token — members'
+key storage is decrypted in the portal only for this feed and for serving a
+member's own dashboard.
+
+## Why a per-member fetch was rejected (the riskier path)
+
+The sandbox is the ONLY consumer, and it does not know which member it is
+seeding at the API layer — it seeds ALL members in one pass from one identity.
+Scoping the feed per member means changing the sandbox's identity model
+(per-member credentials or a member list with per-member auth), touching
+`sync-keys.js` and the seeding loop. A wrong change there stops key seeding
+for every member at once — the exact regression the audit order forbids
+("nothing breaks"). The feed also runs on a fixed cadence, so the practical
+exposure window is bounded by how fast misuse would show.
+
+## What is in place instead (2026-11-05)
+
+1. **Access logging on the route**: every fetch logs a WARNING line
+   `librechat key feed served: N member key(s) to <ip> (ua=…)`. Count only —
+   never key material, never member emails (the portal's log redactor strips
+   emails anyway).
+2. **Daily-digest watch** (cron job 95e5b8fece86, §7 anomaly list): the
+   sandbox polls ~every 10 min (~144 serves/day expected). The digest counts
+   `librechat key feed served` lines and flags (a) a rate far above that
+   cadence, or (b) any 401 burst on the route — either is a suspected token
+   misuse and becomes a decision item.
+
+## What a real fix looks like (if ever needed)
+
+Move the sandbox to fetch per member: a new portal route keyed by the member
+identity the sandbox already knows at seed time, plus a sandbox-side change to
+seed from that route. Only worth doing when the sandbox is already being
+modified for another reason — never as a standalone risky change.

maintenance/provisioning-boundary.md

@@ -0,0 +1,75 @@
+# Member provisioning boundary — who auto-provisions from Open Collective
+
+Which Open Collective contributors become portal members automatically, and
+which are deliberately excluded. The portal defines the member set in
+`fetch_members()` (member-portal repo, `app/main.py`), acts on it in the
+webhook and the nightly `reconcile_memberships()` sweep, and exposes two admin
+overrides: `POST /admin/org-link` and `POST /admin/provision`. A daily
+watchdog (below) is the net for anything that slips past both.
+
+## Auto-provisioned
+
+**Individual OC accounts holding BACKER/ADMIN on the Membership tier (or no
+tier — pre-tier contributors are grandfathered) whose email is visible to the
+admin token.** The webhook provisions on first payment; the nightly sweep
+(03:00 scheduler task) provisions anything missed within 24h and deactivates
+lapsed members. `fetch_members()` matches the tier by NAME
+(`"membership"`, case-insensitive): renaming the tier on Open Collective
+silently empties the member set — the sweep's fail-safe skips reconciliation
+rather than deactivating everyone, but provisioning also stops.
+
+## Deliberately excluded
+
+1. **Org-backed memberships.** An ORGANIZATION account never exposes an email
+   (the `... on Individual { email }` fragment matches nothing, and OC returns
+   org emails as null even to collective admins), so reconcile cannot map one
+   and does not try. Org-backed memberships become members only when LINKED:
+   `POST /admin/org-link {"org_slug", "email", "name"}` runs the full
+   provisioning pipeline and stores `members.org_slug`; the sweep keeps linked
+   accounts active while the org holds BACKER/ADMIN on the Membership tier and
+   deactivates them when the org's dues lapse (fail-safe: if the org fetch
+   itself errors, linked accounts stay active). An unlinked org's first
+   payment emails info@ with the exact linking command.
+
+2. **Non-membership tiers (Credit pack).** A one-time purchase grants BACKER
+   role but is NOT a membership — `fetch_members()` filters it out. This is
+   why a contributor can pay, appear in the public backer list, and still have
+   no portal member row: it is a credit-pack purchase, not a lapsed
+   provisioning. The money is not lost: the webhook's held-credit-pack flow
+   records a hold, notifies the buyer, and the operator grants, refunds or
+   transfers it via `/admin/credit-packs-held`; a hold granted to a named
+   member auto-applies if that person later becomes a member.
+
+3. **Manual members.** The `MANUAL_MEMBERS` env allowlist (founder/free tier,
+   the co-op's own accounts) holds members with no OC membership BY DESIGN:
+   never auto-provisioned from OC, and exempt from lapsed-membership
+   deactivation. Check for orphaned LiteLLM `member:*` teams after any
+   `MANUAL_MEMBERS` change.
+
+4. **Hidden-email / anonymous contributors.** An individual whose email is
+   null even in the ADMIN view cannot be mapped by reconcile. They are not
+   silently dropped — the daily watchdog reports them with their public dues,
+   and provisioning is manual: `POST /admin/provision {"email", "name"}` on
+   the operator's instruction. Incognito accounts publish an
+   `incognito-xxxxxxxx` slug, but the ADMIN member query usually still reveals
+   their email — always check the admin view before treating one as
+   unmappable (that is what resolved the mmoya case).
+
+## The watchdog: `oc_unmapped_check.py`
+
+Daily no-agent cron on the agent host (07:20, job `d0fd1126cc80`), running
+`~/.hermes/scripts/oc_unmapped_check.py` — the same silent-until-findings
+pattern as the spend-consistency guard. It reads the collective's backer list
+through the portal env's admin token and the portal rows read-only via
+`cloudron exec`, maps them by email, slug or org_slug, and prints ONLY
+member-ish backers (the exact set `fetch_members()` treats as members) with no
+portal row at all: slug, display name, whether the admin view sees an email,
+dues (amount/frequency/status from the public order list), days as a backer,
+and the matching action (org-link vs /admin/provision vs investigate-the-sweep).
+Backer nodes with no account object (deleted/incognito-hidden accounts) are
+attributed via same-day public orders and stay silent when every same-day
+payer is already mapped. Excluded from findings: credit-pack-only backers
+(the held-credits flow owns them) and backers matching an inactive portal row
+(the sweep re-provisions/reactivates those within 24h). Empty output = clean;
+any fetch failure prints FAILED — a check that cannot look never reads as
+clean.

migration/cloudron-server-migration.md

@@ -529,36 +529,47 @@ state at cutover, not today.
 | Woodpecker CI server | `ci.inference.coop` | `4984f95e-0c0a-4af3-9ce1-4ca4f3e8483d` |
 | Woodpecker CI agent (custom) | `ci-agent.inference.coop` | `5ac63873-cf2c-4ad9-acd6-81dc0e016c83` |
 
-## The DNS "split" is resolved — no hostname is served by the old box
+## The DNS cutover is COMPLETE — no hostname is served by the old box
 
-At one point `chat.inference.coop` and `portal.inference.coop` were observed still
+At one point `chat.inference.coop` and `portal.inference.coop` were *reported* as
-resolving to the **old** box (`179.199.135.72`) while `gateway`/`my`/`docs` were on
+still resolving to the **old** box (`179.199.135.72`). **That observation was
-the **new** box (`179.237.86.116`). **As of 2026-11-04 there is no such split:**
+stale and the split does not exist.** Verified 2026-11-04 against an external
-every hostname — `chat`, `portal`, `gateway`, `my`, `docs`, `ci`, `ci-agent`,
+resolver (Cloudflare, `@1.1.1.1`): **every** hostname — apex, `my`, `chat`,
-`matrix`, `git`, `www` and the apex — resolves to the **new** box.
+`portal`, `gateway`, `docs`, `ci`, `newsletter`, `git` — resolves to the **new**
+box `179.237.86.116` with **both A and AAAA**
+(`2001:1600:18:206::89`). No hostname resolves to the old box, and nothing in DNS
+points at it. `chat.inference.coop` serves LibreChat and `portal.inference.coop`
+returns the portal from the new box.
 
-The zone is **wildcarded**, so a resolving name proves nothing by itself. The
+The zone is **wildcarded**, so a resolving name proves nothing on its own — but a
-decisive fact: a **made-up** hostname (`zzznope.inference.coop`) also answers the
+made-up hostname (`zzznope.inference.coop`) also answers the **new** box at the
-**new** box `179.237.86.116` at the authoritative nameservers (`ns1.yoursrs.com`).
+authoritative nameservers, so the wildcard itself now points at the new box.
-That means the **wildcard itself now points at the new box**, so no name is being
-served by the old box via the wildcard either.
 
-## The old box still exists (rollback), and serves nothing
+## ⚠️ UNRESOLVED (operator decision): the old box is still alive and internet-exposed
 
-- `179.199.135.72` is **still powered on** and answers on 22/80/443/587/993, **but
+`179.199.135.72` is **powered on, reachable from the internet, and unaltered.** This
-  it hosts no running app.** A request with `Host: chat.inference.coop` returns
+is not "a spare kept for rollback" — it is a **decommissioned server still running
-  Cloudron's **"app down / Not Found" fallback page**, and its TLS certificate has
+and still exposed.**
-  an **empty subject** — the signature of a Cloudron whose vhost's app is stopped,
+
-  exactly as documented ("powered on with apps stopped").
+- **Open to the internet (verified 2026-11-04):** ports **22, 80, 443, 587, 993**
-- **No DNS reference points at it**, so it receives no real traffic.
+  answer. (3306/5432/8080 are closed.)
-- **Decommission is still pending** — this is the one open item from the cutover
+- It answers HTTPS **for any Host header** with a **generic Cloudron-served page**
-  that has not happened. Its backup schedule must remain **disabled** while it is
+  (its fallback / "app down" page, empty-subject certificate) rather than a real
-  kept (two boxes on one S3 bucket interleave and prune each other's backups).
+  app — because its apps are stopped.
+- **It still holds whatever old data and configuration it had** — the old Cloudron
+  platform, its app data, and its host config are all present on the disk.
+- **Nothing in DNS points to it**, so it serves **no member traffic** — the exposure
+  is the open ports and the retained data/config, not live traffic.
+
+**Decision needed from the operator:** either **shut it down**, or **firewall it**
+(close all but what is genuinely needed) if it is being kept as a rollback. It is
+NOT the steward's to act on: **do not shut it down or change anything on it without
+the operator's word.** Its backup schedule must stay **disabled** while it is kept
+(two boxes on one S3 bucket interleave and prune each other's backups).
 
 ## Still unresolved / for the operator
 
-- **Decommission the old box** (`179.199.135.72`) when rollback is no longer
+- **The old box's exposure** — see above. Shut down or firewall.
-  wanted, after confirming the new box's backups are proven.
 - **Member mailbox contents** — the long-standing unknown from §7 (whether
   members' mail is inside the Cloudron backup) still needs a real IMAP login to
   confirm.