From 278c1fda1a63ccda4076019de43b28b27331ac1c Mon Sep 17 00:00:00 2001 From: docs-bot Date: Wed, 4 Nov 2026 21:29:06 +0000 Subject: [PATCH] infrastructure: make the page member-facing; lighten the light-mode menu-bar stripe - condense the page to what a member needs: where services run (Infomaniak), what backups mean to them, the .coop domain via domains.coop, mail via Scaleway, the Matrix space and the accounts they may meet - drop operator detail (migration narration, DNS record inventory, scheduled-task internals, admin-only components, internal account/credential identifiers) to co-op/ops-docs; fix a heading glitch where a line began with '#infrastructure' - brand.css: light-mode .md-tabs uses the header's pale wash #e8f5e8 (was the saturated --green-deep) with a warm hairline; dark scheme unchanged --- infrastructure.md | 335 +++++++++++++++---------------------------- publishing/brand.css | 10 +- 2 files changed, 122 insertions(+), 223 deletions(-) diff --git a/infrastructure.md b/infrastructure.md index 859080d..af39ce0 100644 --- a/infrastructure.md +++ b/infrastructure.md @@ -7,11 +7,11 @@ and contributors who want to work on it. Members reach the co-op through two front doors: the **chat** (LibreChat) and the **API** (the LiteLLM gateway). Both route through a single gateway to -multiple LLM providers — so members choose what they value for each task: -architecturally private inference in **TEE-protected enclaves** (Tinfoil), -100% renewable energy (GreenPT, Regolo). A thin layer of custom middleware — -the member portal — ties Open -Collective membership to Cloudron accounts and LiteLLM keys. +multiple LLM providers, so members can choose what they value for each task: +architecturally private inference in **TEE-protected enclaves** (Tinfoil), or +100% renewable energy (GreenPT, Regolo). A thin layer of custom middleware — the +member portal — ties Open Collective membership to Cloudron accounts and LiteLLM +keys. ``` ┌─> Tinfoil (TEE) 🔒 private @@ -28,228 +28,133 @@ single sign-on, and per-app isolation. ## Where it runs -The co-op's services run on a server the co-op owns and pays for directly, at -**Infomaniak** — a Swiss sovereign cloud provider whose data centres are in -Geneva, run on renewable energy, and certified to ISO 14001, ISO 50001 and -ISO 27001. The stack moved there from a personal Hostinger account on -**2026-11-02**; nothing about how members use the co-op changed in the move. +The co-op's member-facing services run on a server the co-op owns and pays for +directly, at **Infomaniak** — a Swiss sovereign cloud provider whose data centres +are in Geneva, run on renewable energy, and are certified to ISO 14001, ISO +50001 and ISO 27001. -One thing runs elsewhere: the co-op's **steward bot** — the automation behind the -digests and the newsletter drafts — runs on a separate machine, not on the -Cloudron server, and was not part of the migration. - -### DNS - -`inference.coop` is managed by hand at **domains.coop** (nameservers -`ns1/ns2.yoursrs.com`). The zone is deliberately small — **eleven records**. Most -of the co-op hangs off just a few of them: the apex (`inference.coop`), `www`, -and a wildcard (`*.inference.coop`) point at the server, so each application -hostname — chat, portal, dashboard, gateway, git, newsletter, vault and the rest -— is served by the wildcard rather than having a record of its own. The DNS -change sheet listing all eleven records is public, in -[`co-op/runbooks`](https://git.inference.coop/co-op/runbooks). +The co-op's **automation runs on a separate server**, also hosted on Infomaniak +and deliberately kept apart from the member-facing applications. The steward bot +— the automation behind the digests and the newsletter drafts — and the co-op's +other agents live there. Keeping them on their own host means an agent problem +stays an agent problem: it cannot reach the chat, the gateway or member data. ### Backups -Cloudron writes **encrypted** backups to **Infomaniak's object storage** (the -"Infomaniak Encrypted S3" site, in Switzerland) every night, and keeps the last -**seven days**. The steward-bot host backs up its own selected home directories -to the same object storage. +Cloudron writes **encrypted** backups to Infomaniak's object storage in +Switzerland every night and keeps the last **seven days**, so a failed upgrade +or a lost server can be restored. The automation host backs up its own data to +the same object storage. + +### Domain and DNS + +The co-op's domain is **inference.coop** — a `.coop` domain. The co-op registers +and manages its domains with **domains.coop**, the domain provider run by the +cooperative movement, and holds them in the co-op's own name rather than in any +individual's. DNS itself is deliberately simple, and the co-op's own nameservers +answer it. ## Components -| Component | Purpose | URL | Code | -|-----------|---------|-----|------| -| **LibreChat** | Member-facing chat (models, web search, image generation, voice) | chat.inference.coop | [co-op/librechat-custom](https://git.inference.coop/co-op/librechat-custom) | -| **Open WebUI** | Legacy chat instance — read-only reference (old history + file attachments) | chat-legacy.inference.coop | — | -| **LiteLLM** | AI gateway — keys, metering, model routing | gateway.inference.coop | [code/litellm](https://git.inference.coop/code/litellm) | -| **Member Portal** | Membership middleware — onboarding, key provisioning, Loomio sync | portal.inference.coop | [code/member-portal](https://git.inference.coop/code/member-portal) | -| **Member Dashboard** | Member-facing usage view + API key management | dashboard.inference.coop | [code/member-dashboard](https://git.inference.coop/code/member-dashboard) | -| **Admin Panel** | Admin-only member overview, spend, balance management | panel.inference.coop | [code/admin-panel](https://git.inference.coop/code/admin-panel) | -| **SearXNG** | Self-hosted web search (feeds the chat's search) | search.inference.coop | — | -| **Loomio** | Member governance | forum.inference.coop | — | -| **Matrix (Synapse)** | Member chat server (federated; SSO login via Cloudron) | matrix.inference.coop | — | -| **Gitea** | Git hosting (code + docs) | git.inference.coop | — | -| **Surfer** | Static hosting (landing page, thanks page) | inference.coop | [co-op/website](https://git.inference.coop/co-op/website) | -| **Listmonk** | Member newsletter (auto-synced from membership) | newsletter.inference.coop | — | -| **Vaultwarden** | Password manager (admin credentials) | vault.inference.coop | — | -| **SnappyMail** | Webmail | email.inference.coop | — | +| Component | Purpose | URL | +|-----------|---------|-----| +| **LibreChat** | Member-facing chat (models, web search, image generation, voice) | chat.inference.coop | +| **Open WebUI** | Legacy chat instance — read-only reference (old history and file attachments) | chat-legacy.inference.coop | +| **LiteLLM** | AI gateway — keys, metering and model routing, for chat and API | gateway.inference.coop | +| **Member portal** | Membership middleware — onboarding, key provisioning, Loomio sync | portal.inference.coop | +| **Member dashboard** | Your usage view and API keys | dashboard.inference.coop | +| **Loomio** | Member governance | forum.inference.coop | +| **Matrix (Synapse)** | Member chat server, federated, SSO login via Cloudron | matrix.inference.coop | +| **Gitea** | Git hosting for the co-op's code and this documentation | git.inference.coop | +| **Listmonk** | Member newsletter (synced from membership) | newsletter.inference.coop | +| **SnappyMail** | Webmail | email.inference.coop | + +The co-op's software behind all of these is open source — see [The +code](#the-code) below for where each piece lives. Open Collective (billing and fiscal sponsorship) is the one external service in -the flow — see [opencollective.com/inference-cooperative](https://opencollective.com/inference-cooperative). +the flow — see +[opencollective.com/inference-cooperative](https://opencollective.com/inference-cooperative). ## Mail -The `inference.coop` mail domain is the co-op's, and the mail itself runs on the -co-op's Cloudron server. +The co-op runs its own `@inference.coop` mail. **Incoming** mail arrives directly +on the co-op's server, and webmail and mail apps reach it over the usual +encrypted ports (IMAP and SMTP over TLS). **Outgoing** mail is sent through +**Scaleway**, an EU-based email service, authenticated for our domain. -**Incoming** mail is received directly by that server — the domain's MX record -points at it, it accepts mail on the standard SMTP port, and webmail and mail -apps reach it over the usual encrypted ports (IMAP and SMTP over TLS). If mail -to your `@inference.coop` address stops arriving, that is a fault on our side to -fix, not a limitation of how the co-op is set up. - -**Outgoing** mail travels by two different paths, on purpose: - -- The **newsletter** (Listmonk) sends through **Scaleway Transactional Email**, - an EU-based SMTP relay, rather than from our own server. Our domain is - authenticated to that relay with its own DKIM key, and a real message has been - checked end to end: SPF, DKIM and DMARC all pass under a `p=reject` policy — - so anything claiming to be from us that the relay hasn't authenticated is - rejected outright rather than quietly filed away. -- The **member portal's** transactional email — welcome messages, budget - notices, the confirmation the nightly membership sweep sends — still sends from - the co-op's own server, through the mail service Cloudron runs for us. - -Both paths send as **Inference Cooperative ``**. - -Why split them: sending the newsletter through a relay lets a provider with -established sending IPs carry a list this size, instead of leaning on one -freshly-migrated server's reputation. Receiving we keep in-house, on our own -hardware. +You can reach your mailbox through webmail at email.inference.coop, or add it to +a standard mail app on your phone or computer. If mail to your `@inference.coop` +address stops arriving, that is a fault on our side to fix. ## Matrix -Members get a federated Matrix account on our self-hosted Synapse -(matrix.inference.coop, user IDs `@username:inference.coop`) using their -Cloudron SSO login. The portal invites every active member — and Nathan's -federated account — to the private **Inference Cooperative** Space (with a -**General** room) on every membership sync, so new members land in the space -automatically. Invites honor rate limits; external/federated members can be -added via the portal's `MATRIX_EXTRA_MXIDS` env. +Members get a federated Matrix account on the co-op's self-hosted Synapse +(matrix.inference.coop, user IDs `@username:inference.coop`), signing in with the +same Cloudron SSO as the chat. The co-op invites every active member to the +private **Inference Cooperative** space (with a **General** room) automatically, +so new members land there as they join. Federation is on, so you can also join +from any homeserver you already use. -Federation is enabled (delegated via `.well-known/matrix/server` on the bare -domain, port 443), so members can also join from any homeserver they already -use. +### Accounts you may meet -### Which account is which +A few accounts in the space are automation rather than people: -Several automated accounts touch this homeserver, and two of them once differed -by a single character — which caused real confusion, including a bot appearing to -announce itself. The distinction is deliberate: +- **`@inferencebot` — the steward.** It posts the digests, newsletter notes and + reminders. It holds no more power than posting needs. +- **`@inference-admin`** — the administrator of record for the co-op's rooms + (#general, #community, #design, #skillshare, #updates and #infrastructure) and + their membership. It **never speaks in a member room** — a message from it is + a mistake, not a feature. -- **`@inferencebot:inference.coop` — the steward.** The bot members actually meet: - it posts digests, newsletter notes and reminders. It is *not* a Synapse server - admin, and holds no room power beyond what it needs to post. -- **`@inference-admin:inference.coop` — the room and server administrator.** A - Synapse server admin, used for provisioning rooms, managing membership, and - retiring rooms. It **never speaks in a member room**; a message from it is a - mistake, not a feature. -- **`@infrastructure-bot:inference.coop` — the infrastructure working group's bot.** - Answers infrastructure questions in **#infrastructure** when mentioned. -- **`@ops_audit:inference.coop`** — a server-admin account created for the October - 2026 security review. Retained only until its tokens are revoked. +The house rule: **power belongs to `@inference-admin`, voice belongs to +`@inferencebot`.** Please don't create an account whose name differs from an +existing one by a character or two — spotting lookalikes is exactly what this +convention is for. -**House rule: power belongs to `@inference-admin`, voice belongs to `@inferencebot`.** -Please don't create another account whose name differs from an existing one by a -character or two; that is the specific confusion this section exists to undo. +## Membership -**`@inference-bot:inference.coop` is retired (2026-10-29).** It was the original -room-administration account (it created the space and most of the rooms) and held a -Synapse server-admin flag. Its room power went to `@inference-admin`, which is now -administrator of record for #general, #community, #design, #skillshare, #updates and -#infrastructure, **and holds the portal's member-invite credential** -(`MATRIX_BOT_TOKEN`) under its own token. The old account's admin flag was dropped, -it was deactivated, and every device and token was deleted. Nothing should use it -again. - -Two loose ends, recorded so nobody re-investigates them from scratch: it remains -joined to two rooms it cannot leave (they reject every write — `500` on leave), and -those rooms are also why they can't be deleted; and the old account's *name* is still -the tempting one, so check `whoami` rather than the name before trusting any account -that looks like it. - -## How the apps run themselves - -The portal and gateway are self-maintaining: their recurring housekeeping is -built into the apps as scheduled tasks, not run from any member's or operator's -personal machine. - -- **Member portal** — two scheduled jobs (Cloudron scheduler): - - *Membership sweep* (nightly, 3am) — reconciles members against the live - Open Collective list: provisions new members who signed up while a webhook - was missed, deactivates lapsed members, and syncs Loomio + Matrix - membership. A missed webhook self-heals within 24 hours. - - *Docs knowledge refresh* (nightly, 3:30am) — pulls this documentation from - the public Git repo and updates the shared knowledge base behind the chat's - models, so answers grounded in the docs stay current. Changes to the docs - appear in the chat within a day, no manual step. -- **Gateway** — an *audio keep-alive* (every 10 minutes) sends a tiny text-to- - speech ping to the private enclave so voice mode responds quickly even after - long quiet periods, without compromising the end-to-end encryption model. - -Scheduled tasks are declared in each app's `CloudronManifest.json` (see the -`code/` repos) and survive redeploys. If a task fails, it shows up in the -app's logs on the admin panel; the portal also exposes a manual trigger for -the membership sweep. - -## How membership is wired - -Membership flows through the member portal, which is the single source of truth -for "who is a member": +Membership runs through the member portal, the single source of truth for who is +a member: 1. A member contributes on Open Collective. -2. Open Collective fires a webhook to the portal, which provisions a Cloudron - account, a LiteLLM key, and a welcome email. -3. A nightly sweep reconciles against the live Open Collective list, so a missed - webhook self-heals within 24 hours. +2. Open Collective notifies the portal, which provisions a Cloudron account, a + LiteLLM key, and a welcome email. +3. A nightly reconciliation against the live Open Collective list means a missed + notification self-heals within a day. -The portal database is authoritative for member emails and balances; LiteLLM's -team budget is the source of truth for a member's allowance, shared between chat -and API. - -### The bot account on Open Collective - -You may notice **"Inference Co-op Bot"** listed as an admin of the collective on -Open Collective. This is infrastructure, not a member or a person: - -- **What it is:** a dedicated Open Collective account owned by the co-op. It - holds a **Personal Token** that the member portal uses to read membership - data — the roster, tiers (needed to route credit-pack purchases correctly), - and contributor emails (including guests, which only admins can see). -- **What it can do:** read member and tier data via Open Collective's API. - It cannot move money, issue refunds, or change the collective — Open - Collective restricts those to dashboard actions by human admins. Its API - access was verified read-mostly during the October 2026 security review. -- **Why it has admin:** reading guest contributor emails requires admin on - Open Collective; without it, membership reconciliation and credit-pack - routing can't function. -- **Who supervises it:** all of its code is open source in - [code/member-portal](https://git.inference.coop/code/member-portal) — every - API call it makes is documented there. Nathan Schneider (collective admin) - oversees its use and can revoke or rotate its token at any time from the - Open Collective dashboard. - -Nothing on Open Collective is charged, transferred, or refunded by automation: -payments always come from a human's own contribution action, and any automated -system only *reads* the resulting records. +You may see **"Inference Co-op Bot"** listed as an admin of the collective on +Open Collective. It is infrastructure, not a person or a member: a dedicated +account the portal uses, **read-only**, to check the member roster and tiers. It +cannot move money, issue refunds or change the collective — payments always come +from a human's own contribution, and automation only reads the result. Its code +is open source in +[code/member-portal](https://git.inference.coop/code/member-portal), and a +collective admin can revoke its access at any time. ## The code -The cooperative's software is open source, in the [`code`](https://git.inference.coop/code) -organization on Gitea: +The co-op's software is open source, in the [`code`](https://git.inference.coop/code) +and [`co-op`](https://git.inference.coop/co-op) organizations on Gitea: - [**code/member-portal**](https://git.inference.coop/code/member-portal) — membership middleware (Python). -- [**code/member-dashboard**](https://git.inference.coop/code/member-dashboard) — member-facing usage + API keys (Python). -- [**code/admin-panel**](https://git.inference.coop/code/admin-panel) — admin overview (Python). +- [**code/member-dashboard**](https://git.inference.coop/code/member-dashboard) — member dashboard: usage and API keys. - [**code/litellm**](https://git.inference.coop/code/litellm) — the gateway packaged as a Cloudron app. - -Plus the `co-op` organization: - - [**co-op/website**](https://git.inference.coop/co-op/website) — landing page and static site. - [**co-op/docs**](https://git.inference.coop/co-op/docs) — this documentation. -- [**co-op/design-assets**](https://git.inference.coop/co-op/design-assets) — logos, fonts, brand materials. + +The chat, portal and other apps live in `code/` and `co-op/` too; everything is +browsable at [git.inference.coop](https://git.inference.coop). ## Contributing (the Git is yours, too) -Everything the co-op runs is open source and lives at -[git.inference.coop](https://git.inference.coop) — the website, this -documentation, the member portal, the dashboard, the admin panel, and the -gateway. As a member, you can browse all of it and propose changes. +Everything the co-op runs is open source, so as a member you can browse all of +it and propose changes. **Who can sign in.** Git access is tied to your membership: you log into the Git with the same single sign-on you use for the chat. Members are in automatically -(through the `members` group); when a membership lapses, Git access lapses with it. +(through the `members` group); when a membership lapses, Git access lapses with +it. **How to contribute.** Anyone can read the repos without an account. To make a change, fork the repo, edit in your fork, and open a **pull request**: @@ -260,51 +165,39 @@ change, fork the repo, edit in your fork, and open a **pull request**: 4. Open a **pull request** back to the original repo, describing your change. **All changes are reviewed.** Nothing merges directly into `main` — every change -comes in as a pull request that an admin reviews and approves first. You don't -need to be able to create repos or organizations to contribute; you just fork, -edit, and ask for review. +comes in as a pull request an admin reviews and approves first. You don't need +to be able to create repos or organizations to contribute; you just fork, edit +and ask for review. -Questions about the Git itself? Write to info@inference.coop. +Questions? Write to info@inference.coop. ## Models and providers -The co-op serves models from multiple providers (Tinfoil for privacy, GreenPT -for renewable energy, PublicAI coming soon). See the [model list](models.md) for -the current catalog — that's the single source of truth for which models are -available and how they're named. - -One infra detail worth knowing: a local Tinfoil proxy sidecar handles the -enclave encryption on our side of the gateway — only Tinfoil traffic goes -through it; the other providers are called directly. +The co-op serves models from multiple providers — Tinfoil for architectural +privacy, GreenPT and Regolo for renewable energy. See the [model list](models.md), +the single source of truth for which models are available and how they're named. ## Security posture (plain language) -- **Token-gated control plane.** The portal is publicly reachable, but every - sensitive surface requires a secret token (admin, broker, webhook). All fail - closed — unauthenticated requests get `401`. -- **Separated secrets.** Chat, broker, admin, and model-layer secrets are +- **Member isolation.** Each API key is scoped to its owner's team and budget; a + member can only ever act on their own keys. +- **Separated secrets.** Chat, portal, admin and model-layer secrets are independent and independently rotatable, so a leak has bounded scope. -- **Member isolation.** Each API key is scoped to its owner's LiteLLM team and - budget; a member can only ever act on their own keys. -- **Rate limiting** on control-plane endpoints; the inference path is capped by - each member's budget instead. -- **No secrets in version control.** Credentials live in Cloudron env vars, never - in the repos. +- **No secrets in version control.** Credentials live in the hosting platform's + environment, never in the repos. - **Accepted trade-offs** (documented, not hidden): member keys are stored in - plaintext in the portal database (functionally necessary), and the gateway - must stay publicly reachable for API access (its admin UI and docs are - disabled; only the `/v1/*` API is exposed). + the portal database so the co-op can serve them, and the API gateway must stay + publicly reachable. Its admin surfaces are disabled; only the public API is + exposed. ## For operators -Maintaining the co-op's infrastructure — rotating Cloudron tokens, editing the -portal's environment, and troubleshooting the provisioning pipeline — is -documented in the [operator -runbook](https://git.inference.coop/co-op/ops-docs/src/branch/main/operator-token-runbook.md). -All operator-facing material (that runbook, the maintenance notes, and incident -write-ups) lives in the git-only -[`co-op/ops-docs`](https://git.inference.coop/co-op/ops-docs) repository, which -is deliberately not published on this site. +Maintaining the co-op's infrastructure — token rotation, app configuration and +troubleshooting — is documented in the git-only +[`co-op/ops-docs`](https://git.inference.coop/co-op/ops-docs) repository, which is +deliberately not published on this site. It is the home for the operator runbook, +the maintenance notes, incident write-ups, and the record of past server +migrations. Anyone may read it; operators are the audience. ## Incident log diff --git a/publishing/brand.css b/publishing/brand.css index a2e34f0..2355f28 100644 --- a/publishing/brand.css +++ b/publishing/brand.css @@ -185,15 +185,21 @@ font-weight: 600; } -/* Tabs and the search field pick up the brand green. */ +/* The menu-bar (tab) strip. In light it is the same pale green WASH as the + * header — #e8f5e8, the homepage's "open source" pill — not a saturated fill: + * the deep greens are accents (links, active navigation, rules), and a large + * --green-deep bar clashed with the rest of the light palette. A warm hairline + * separates the wash from the cream page below it. Dark is unchanged. */ .md-tabs { - background-color: var(--green-deep); + background-color: #e8f5e8; + border-bottom: 1px solid var(--border); } /* In dark, keep the tab row the same green as the header bar so the text on it * clears 4.5:1 against green-mid (green-deep against near-black text is only * ~4.2:1 — marginal for the small tab labels). */ [data-md-color-scheme="slate"] .md-tabs { background-color: var(--green-mid); + border-bottom: none; } .md-search__input { background-color: rgba(255, 255, 255, 0.92);