Document gateway v1.84.0 pin, CORS env rename, and 2026-10-02 outage incident

2026-10-01 23:35:39 -06:00
2 changed files with 46 additions and 32 deletions
@@ -55,22 +55,6 @@ You don't just use the co-op; you help run it. Decisions about models, pricing,
 privacy, and direction happen on our **Loomio forum**:
 - **[forum.inference.coop](https://forum.inference.coop)** — where members discuss and decide.
 - **[matrix.inference.coop](https://matrix.inference.coop)** — our Matrix chat server. Log in
  with your co-op account (Cloudron SSO); your Matrix ID is `@username:inference.coop`.
  Every member is invited to the **Inference Cooperative** space automatically — check your
  invites on first login.
  **Using it from any Matrix client** — the server speaks the standard Matrix protocol,
  so you can connect from any client:
  - **Element** (web, desktop, iOS, Android): on the sign-in screen, click *Edit* next to
    "Homeserver" and enter `https://matrix.inference.coop`, then sign in with your co-op
    account. (On [app.element.io](https://app.element.io) or the desktop/mobile apps alike.)
  - **FluffyChat, Cinny, SchildiChat** or any other client: look for the "custom homeserver"
    / "other server" option in the client's login screen and enter the same address.
  - Your co-op account and your existing Matrix account on another server (e.g.
    matrix.org or a personal homeserver) are separate identities — keep whichever you
    like for co-op rooms; both can join. Invites go to your co-op account by default.
 - See the [Charter](charter.md) for how governance is structured.
 - See [Open questions](open-questions.md) for the decisions currently on the table.
@@ -32,7 +32,6 @@ single sign-on, and per-app isolation.
 | **Admin Panel** | Admin-only member overview, spend, balance management | panel.inference.coop | [code/admin-panel](https://git.inference.coop/code/admin-panel) |
 | **SearXNG** | Self-hosted web search (feeds the chat's search) | search.inference.coop | — |
 | **Loomio** | Member governance | forum.inference.coop | — |
 | **Matrix (Synapse)** | Member chat server (federated; SSO login via Cloudron) | matrix.inference.coop | — |
 | **Gitea** | Git hosting (code + docs) | git.inference.coop | — |
 | **Surfer** | Static hosting (landing page, thanks page) | inference.coop | [co-op/website](https://git.inference.coop/co-op/website) |
 | **Listmonk** | Member newsletter (auto-synced from membership) | newsletter.inference.coop | — |
@@ -42,21 +41,6 @@ single sign-on, and per-app isolation.
 Open Collective (billing and fiscal sponsorship) is the one external service in
 the flow — see [opencollective.com/inference-cooperative](https://opencollective.com/inference-cooperative).
 ## Matrix
 Members get a federated Matrix account on our self-hosted Synapse
 (matrix.inference.coop, user IDs `@username:inference.coop`) using their
 Cloudron SSO login. The portal invites every active member — and Nathan's
 federated account — to the private **Inference Cooperative** Space (with a
 **General** room) on every membership sync, so new members land in the space
 automatically. Invites honor rate limits; external/federated members can be
 added via the portal's `MATRIX_EXTRA_MXIDS` env.
 Federation is enabled (delegated via `.well-known/matrix/server` on the bare
 domain, port 443), so members can also join from any homeserver they already
 use. The `@inference-bot:inference.coop` admin account (used by the portal
 for invites) has no password — token lives in the portal env.
 ## How membership is wired
 Membership flows through the member portal, which is the single source of truth
@@ -174,3 +158,49 @@ through it; the other providers are called directly.
 Maintaining the co-op's infrastructure — rotating Cloudron tokens, editing the
 portal's environment, and troubleshooting the provisioning pipeline — is
 documented in the [operator runbook](operator-token-runbook.md).
 ## Gateway version and security update policy
 The LiteLLM gateway is pinned to an **exact upstream tag** in
 [`code/litellm`](https://git.inference.coop/code/litellm) (currently
 `v1.84.0`). Two reasons:
 1. **CVE fix line.** A member security review (Oct 2026) found the prior pin
   (`main-v1.74.0-stable`) sat below the fix line for two published
   vulnerabilities: CVE-2026-35029 (authorization bypass on
   `/config/update`, fixed in 1.83.0, with documented in-the-wild probing)
   and CVE-2026-59822 (MCP session auth bypass, fixed in 1.84.0, listed in
   the CISA Known Exploited Vulnerabilities catalog). `v1.84.0` is at or
   above both fix lines.
 2. **Floating-tag drift.** `main-v1.74.0-stable` is a *floating* tag — the
   registry silently moved it to a newer build. A rebuild intended as a
   no-op therefore pulled different bits and triggered a full schema
   migration, which is how the 2026-10-02 outage happened (see below).
 Rule: **never pin a floating tag** (`main-*` / `latest`); always the exact
 version tag. When upgrading, expect the first boot after the upgrade to take
 10–15 minutes (prisma migration with internal retries) — plan the restart
 window accordingly.
 CORS: the gateway answers browser preflights only for
 `https://chat.inference.coop` (via LiteLLM's native
 `LITELLM_CORS_ORIGINS`). The old `LITELLM_CORS_ALLOWED_ORIGINS` name
 (our v1.74.0-era sed-patch variable) is dead — v1.84.0 reads the native
 name only.
 ## Incident log
 **2026-10-02 — gateway outage after security-upgrade rebuild.** A Cloudron
 update triggered by the CVE report rebuilt the app image; the floating
 `main-v1.74.0-stable` tag had moved to newer bits (v1.84.0-era), so the
 rebuild changed the running version and kicked off a full prisma DB
 migration that takes 10–15 minutes per boot with internal timeouts/retries.
 The gateway was down ~50 minutes (04:50–05:34 UTC); no data loss. Repairs:
 version pinned to the exact tag in the repo (PR
 [code/litellm#1](https://git.inference.coop/code/litellm/pulls/1) — pending
 merge), `LITELLM_CORS_ORIGINS` set as a Cloudron env var (the upgrade had
 silently reverted CORS to `*`), packaging synced to mirror the live
 deployment. Also during the incident: the domain's authoritative
 nameservers (yoursrs.com) were intermittently unreachable, briefly failing
 resolution of some hostnames (git, portal) — transient registrar-side
 issue, self-recovered, no records were wrong.