Document gateway v1.84.0 pin, CORS env rename, and 2026-10-02 outage incident

README.md

@@ -20,7 +20,6 @@ funded through our [Open Collective](https://opencollective.com/inference-cooper
 
 - **Website:** https://inference.coop
 - **AI Chat:** https://chat.inference.coop
-- **Matrix chat:** https://matrix.inference.coop
 - **Member Dashboard:** https://dashboard.inference.coop
 - **Governance forum:** https://forum.inference.coop
 - **Source code:** https://git.inference.coop

api-access.md

@@ -57,7 +57,7 @@ endpoint configuration:
 - **API type / protocol:** "OpenAI compatible" (or `openai-completions` where a
   protocol string must be named) — not "Ollama", "Anthropic", or "Responses"
 - **API key:** your member key (the `sk-…` string)
-- **Model:** one from the [model list](models.md), e.g. `tinfoil/deepseek-v4-1-flash` or `greenpt/glm-5.3-flash`
+- **Model:** one from the [model list](models.md), e.g. `tinfoil/deepseek-v4-1-flash` or `greenpt/green-r`
 
 A typical provider entry looks like:
 
@@ -72,7 +72,8 @@ A typical provider entry looks like:
         { "id": "tinfoil/deepseek-v4-1-flash" },
         { "id": "tinfoil/gpt-oss-120b" },
         { "id": "tinfoil/glm-5-3-flash" },
-        { "id": "greenpt/glm-5.3-flash" }
+        { "id": "greenpt/green-r" },
+        { "id": "greenpt/green-l" }
       ]
     }
   }

getting-started.md

@@ -55,30 +55,6 @@ You don't just use the co-op; you help run it. Decisions about models, pricing,
 privacy, and direction happen on our **Loomio forum**:
 
 - **[forum.inference.coop](https://forum.inference.coop)** — where members discuss and decide.
-- **[matrix.inference.coop](https://matrix.inference.coop)** — our Matrix chat server. Log in
-  with your co-op account (Cloudron SSO); your Matrix ID is `@username:inference.coop`.
-  Every member is invited to the **Inference Cooperative** space automatically — check your
-  invites on first login.
-
-  **Using it from any Matrix client** — the server speaks the standard Matrix protocol,
-  so you can connect from any client:
-
-  - **Element** (web, desktop, iOS, Android): on the sign-in screen, click *Edit* next to
-    "Homeserver" and enter `https://matrix.inference.coop`, then sign in with your co-op
-    account. (On [app.element.io](https://app.element.io) or the desktop/mobile apps alike.)
-  - **FluffyChat, Cinny, SchildiChat** or any other client: look for the "custom homeserver"
-    / "other server" option in the client's login screen and enter the same address.
-  - Your co-op account and your existing Matrix account on another server (e.g.
-    matrix.org or a personal homeserver) are separate identities. If you'd
-    rather use an existing account for co-op rooms, set it on your
-    [dashboard](https://dashboard.inference.coop) under **Matrix chat** (one
-    extra account) — invitations will go there instead. Both work either way.
-
-  **Members-only, readable history.** The space is invite-only and unencrypted
-  so every member can read the full history (including messages from before
-  they joined). Invites are restricted to admins — so while any member is
-  welcome to bring a collaborator or guest, an admin sends the invite rather
-  than members handing the space to outsiders directly.
 - See the [Charter](charter.md) for how governance is structured.
 - See [Open questions](open-questions.md) for the decisions currently on the table.
 

infrastructure.md

@@ -32,7 +32,6 @@ single sign-on, and per-app isolation.
 | **Admin Panel** | Admin-only member overview, spend, balance management | panel.inference.coop | [code/admin-panel](https://git.inference.coop/code/admin-panel) |
 | **SearXNG** | Self-hosted web search (feeds the chat's search) | search.inference.coop | — |
 | **Loomio** | Member governance | forum.inference.coop | — |
-| **Matrix (Synapse)** | Member chat server (federated; SSO login via Cloudron) | matrix.inference.coop | — |
 | **Gitea** | Git hosting (code + docs) | git.inference.coop | — |
 | **Surfer** | Static hosting (landing page, thanks page) | inference.coop | [co-op/website](https://git.inference.coop/co-op/website) |
 | **Listmonk** | Member newsletter (auto-synced from membership) | newsletter.inference.coop | — |
@@ -42,21 +41,6 @@ single sign-on, and per-app isolation.
 Open Collective (billing and fiscal sponsorship) is the one external service in
 the flow — see [opencollective.com/inference-cooperative](https://opencollective.com/inference-cooperative).
 
-## Matrix
-
-Members get a federated Matrix account on our self-hosted Synapse
-(matrix.inference.coop, user IDs `@username:inference.coop`) using their
-Cloudron SSO login. The portal invites every active member — and Nathan's
-federated account — to the private **Inference Cooperative** Space (with a
-**General** room) on every membership sync, so new members land in the space
-automatically. Invites honor rate limits; external/federated members can be
-added via the portal's `MATRIX_EXTRA_MXIDS` env.
-
-Federation is enabled (delegated via `.well-known/matrix/server` on the bare
-domain, port 443), so members can also join from any homeserver they already
-use. The `@inference-bot:inference.coop` admin account (used by the portal
-for invites) has no password — token lives in the portal env.
-
 ## How membership is wired
 
 Membership flows through the member portal, which is the single source of truth
@@ -174,3 +158,49 @@ through it; the other providers are called directly.
 Maintaining the co-op's infrastructure — rotating Cloudron tokens, editing the
 portal's environment, and troubleshooting the provisioning pipeline — is
 documented in the [operator runbook](operator-token-runbook.md).
+
+## Gateway version and security update policy
+
+The LiteLLM gateway is pinned to an **exact upstream tag** in
+[`code/litellm`](https://git.inference.coop/code/litellm) (currently
+`v1.84.0`). Two reasons:
+
+1. **CVE fix line.** A member security review (Oct 2026) found the prior pin
+   (`main-v1.74.0-stable`) sat below the fix line for two published
+   vulnerabilities: CVE-2026-35029 (authorization bypass on
+   `/config/update`, fixed in 1.83.0, with documented in-the-wild probing)
+   and CVE-2026-59822 (MCP session auth bypass, fixed in 1.84.0, listed in
+   the CISA Known Exploited Vulnerabilities catalog). `v1.84.0` is at or
+   above both fix lines.
+2. **Floating-tag drift.** `main-v1.74.0-stable` is a *floating* tag — the
+   registry silently moved it to a newer build. A rebuild intended as a
+   no-op therefore pulled different bits and triggered a full schema
+   migration, which is how the 2026-10-02 outage happened (see below).
+
+Rule: **never pin a floating tag** (`main-*` / `latest`); always the exact
+version tag. When upgrading, expect the first boot after the upgrade to take
+10–15 minutes (prisma migration with internal retries) — plan the restart
+window accordingly.
+
+CORS: the gateway answers browser preflights only for
+`https://chat.inference.coop` (via LiteLLM's native
+`LITELLM_CORS_ORIGINS`). The old `LITELLM_CORS_ALLOWED_ORIGINS` name
+(our v1.74.0-era sed-patch variable) is dead — v1.84.0 reads the native
+name only.
+
+## Incident log
+
+**2026-10-02 — gateway outage after security-upgrade rebuild.** A Cloudron
+update triggered by the CVE report rebuilt the app image; the floating
+`main-v1.74.0-stable` tag had moved to newer bits (v1.84.0-era), so the
+rebuild changed the running version and kicked off a full prisma DB
+migration that takes 10–15 minutes per boot with internal timeouts/retries.
+The gateway was down ~50 minutes (04:50–05:34 UTC); no data loss. Repairs:
+version pinned to the exact tag in the repo (PR
+[code/litellm#1](https://git.inference.coop/code/litellm/pulls/1) — pending
+merge), `LITELLM_CORS_ORIGINS` set as a Cloudron env var (the upgrade had
+silently reverted CORS to `*`), packaging synced to mirror the live
+deployment. Also during the incident: the domain's authoritative
+nameservers (yoursrs.com) were intermittently unreachable, briefly failing
+resolution of some hostnames (git, portal) — transient registrar-side
+issue, self-recovered, no records were wrong.

models.md

@@ -37,15 +37,12 @@ that **you choose** which axis matters more for a given task.
 
 **How to pick, quick guide:**
 
-- **Default** → `greenpt/deepseek-v4.1-flash`: DeepSeek's efficient multimodal
+- **Everyday chat and drafting** → a fast model: `tinfoil/glm-5-3-flash`,
-  reasoning model on renewable energy, and the model the chat starts on. For the
+  `greenpt/glm-5.3-flash`, or `greenpt/green-l`. These handle the bulk of what
-  TEE-protected (private) version of the same model, use `tinfoil/deepseek-v4-1-flash`.
+  people use chat for, at a fraction of flagship cost.
-- **Everyday chat and drafting** → a fast model: `tinfoil/glm-5-3-flash` or
+- **Hard reasoning, coding, agentic work** → a flagship: `tinfoil/glm-5-3` or
-  `greenpt/glm-5.3-flash`. These handle the bulk of what people use chat for,
+  `greenpt/glm-5.3` (same underlying model, different provider trade-off),
-  at a fraction of flagship cost.
+  or `tinfoil/deepseek-v4-1-flash` for long documents and tool-driven tasks.
-- **Hard reasoning, coding, agentic work** → a flagship: `greenpt/glm-5.3`
-  (the GLM flagship), or `tinfoil/deepseek-v4-1-flash` for long documents and
-  tool-driven tasks.
 - **Images in, text out (vision)** → `tinfoil/glm-5-3-flash`,
   `tinfoil/deepseek-v4-1-flash`, or the Apertus models — see the ✅ in the
   table below.
@@ -55,40 +52,36 @@ that **you choose** which axis matters more for a given task.
 
 ## Current models
 
-Pricing is per 1 million tokens, input / output / **cached input** (what the
+Pricing is per 1 million tokens, input / output (what the co-op pays the
-co-op pays the provider — your own cost is your monthly allowance, not
+provider — your own cost is your monthly allowance, not per-token billing).
-per-token billing). The cached-input rate applies when a model reuses a
+**Vision** means the model accepts images as input (screenshots, photos,
-shared prompt prefix (multi-turn conversations, agent loops); most providers
+documents to reason about) — all models output text. None of our current
-charge a steep discount for it. Models with `—` don't offer a distinct
+models generate images.
-cache-read price. **Vision** means the model accepts images as input
-(screenshots, photos, documents to reason about) — all models output text.
-None of our current models generate images.
 
-The emoji in the model name marks its **value tier** (the same icons used in the chat picker):
+| Model | Provider | Privacy | Green | Vision | Context | Pricing (in / out) | Notes |
-
-- 🔒 **Private** — Tinfoil: architecturally private (TEE enclaves)
-- 🌱 **Green** — GreenPT: 100% renewable energy
-- 📖 **Public** — PublicAI: publicly developed, sovereign open models
-
-| Model | Provider | Privacy | Green | Vision | Context | Pricing (in / out / cached) | Notes |
 |---|---|---|---|---|---|---|---|
-| `🌱 greenpt/deepseek-v4.1-flash` | GreenPT | policy | 100% renewable | ✅ | 1M | $0.22 / $1.10 / $0.011 | **Default.** DeepSeek's multimodal reasoning model, renewable energy; TEE twin below. |
+| `tinfoil/glm-5-3-flash` | Tinfoil | architectural | undisclosed | ✅ | 1M | $0.40 / $1.25 | Fast multimodal MoE; great everyday default with image input. |
-| `🔒 tinfoil/deepseek-v4-1-flash` | Tinfoil | architectural | undisclosed | ✅ | 1M | $0.65 / $1.45 / $0.13 | TEE-protected version of the default; image input, tool calling, long docs/agents. |
+| `tinfoil/glm-5-3` | Tinfoil | architectural | undisclosed | — | 1M | $1.80 / $5.75 | GLM flagship; strongest agentic model, private tier. |
-| `🔒 tinfoil/glm-5-3-flash` | Tinfoil | architectural | undisclosed | ✅ | 1M | $0.40 / $1.25 / $0.10 | Fast multimodal MoE; great everyday default with image input. |
+| `tinfoil/deepseek-v4-1-flash` | Tinfoil | architectural | undisclosed | ✅ | 1M | $0.65 / $1.45 | Efficient MoE, image input, tool calling; great for long docs and agents. |
-| `🔒 tinfoil/gpt-oss-120b` | Tinfoil | architectural | undisclosed | — | 131K | $0.15 / $0.60 / — | Lightweight open-weight reasoning; cheapest private tier. |
+| `tinfoil/gpt-oss-120b` | Tinfoil | architectural | undisclosed | — | 131K | $0.15 / $0.60 | Lightweight open-weight reasoning; cheapest private tier. |
-| `🌱 greenpt/glm-5.3-flash` | GreenPT | policy | 100% renewable | — | 1M | $0.11 / $0.44 / $0.022 | Fast GLM model, renewable energy. |
+| `tinfoil/kimi-k3` | Tinfoil | architectural | undisclosed | ✅ (image + video) | 262K | $4.00 / $20.00 | Moonshot's flagship multimodal MoE; premium tier. |
-| `🌱 greenpt/glm-5.3` | GreenPT | policy | 100% renewable | — | 1M | $1.10 / $4.40 / — | GLM flagship — strongest agentic model in the green tier. |
+| `greenpt/green-r` | GreenPT | policy | 100% renewable | — | 128K | $0.35 / $0.95 | Reasoning-focused, renewable energy. |
-| `📖 publicai/apertus-v1.5-8b` | PublicAI | policy | undisclosed | ✅ | 262K | $0.10 / $0.20 / — | Fully open sovereign model, small/fast, image understanding. |
+| `greenpt/green-l` | GreenPT | policy | 100% renewable | — | 128K | $0.25 / $0.80 | Lightweight, renewable energy. |
-| `📖 publicai/apertus-v1.5-70b` | PublicAI | policy | undisclosed | ✅ | 262K | $0.82 / $2.92 / — | Fully open sovereign model, large/capable, image understanding. |
+| `greenpt/glm-5.3-flash` | GreenPT | policy | 100% renewable | — | 1M | $0.11 / $0.44 | Fast GLM model, renewable energy. |
+| `greenpt/glm-5.3` | GreenPT | policy | 100% renewable | — | 1M | $1.10 / $4.40 | GLM flagship — strongest agentic model in the green tier. |
+| `publicai/apertus-v1.5-8b` | PublicAI | policy | undisclosed | ✅ | 262K | $0.10 / $0.20 | Fully open sovereign model, small/fast, image understanding. |
+| `publicai/apertus-v1.5-70b` | PublicAI | policy | undisclosed | ✅ | 262K | $0.82 / $2.92 | Fully open sovereign model, large/capable, image understanding. |
 
 Vision is verified working through the co-op gateway (base64 image input via
 `/v1/chat/completions`, standard OpenAI format). In the chat UI, just attach
 an image to your message while using a ✅ model.
 
-**Same model, two providers:** `glm-5.3` (GreenPT) is the Z.ai GLM flagship
+**Same model, two providers:** `glm-5-3` (Tinfoil) and `glm-5.3` (GreenPT)
-offered on 100% renewable energy at $1.10/$4.40. We don't currently serve the
+are the same Z.ai model offered by two providers — Tinfoil's is
-Tinfoil (TEE) version of the full GLM-5.3 — for private-tier GLM, use the
+architecturally private at $1.80/$5.75; GreenPT's is 100% renewable at
-`tinfoil/glm-5-3-flash` light model instead.
+$1.10/$4.40. That price/privacy trade-off in a single comparison is the
+co-op's whole idea in miniature: you pick which axis matters for the task
+at hand.
 
 ### Audio models
 
@@ -99,8 +92,8 @@ model picker to avoid confusion.
 
 | Model | Provider | Use | Pricing |
 |---|---|---|---|
-| `🔒 tinfoil/whisper-large-v3-turbo` | Tinfoil | Speech-to-text | $0.05 / 1M input tokens |
+| `tinfoil/whisper-large-v3-turbo` | Tinfoil | Speech-to-text | $0.05 / 1M input tokens |
-| `🔒 tinfoil/voxtral-tts` | Tinfoil | Text-to-speech | listed $0 (verify on invoice) |
+| `tinfoil/voxtral-tts` | Tinfoil | Text-to-speech | listed $0 (verify on invoice) |
 
 Both run inside Tinfoil's enclaves, so voice data gets the same architectural
 privacy as the private chat models. TTS voices available: `neutral_female`,
@@ -154,9 +147,6 @@ window, improved tool use, and instruction-following. It's the *publicly
 developed* counterpart to the private (Tinfoil) and renewable (GreenPT)
 offerings — sovereignty as a third axis of member choice.
 
-For more information on open-source AI and the Apertus models, see the
-[European Open Source AI Index](https://osai-index.eu/database/).
-
 *Source: [apertus-ai.org](https://www.apertus-ai.org/) and the
 [Swiss AI Hugging Face](https://huggingface.co/swiss-ai) model cards.*