inference-bot
deb53eabc8
Reviewed by dckc (review 6, APPROVED) and cross-checked against the gateway upgrade runbook. Merged by push: this repo's approval gate is restricted to the Owners team, which currently holds only machine accounts, and Gitea will not accept their API reviews as approvals — see the PR discussion.
248 lines
13 KiB
Markdown
248 lines
13 KiB
Markdown
# Infrastructure
|
|
|
|
How the Inference Cooperative is built, for members who want to understand it —
|
|
and contributors who want to work on it.
|
|
|
|
## The big picture
|
|
|
|
Members reach the co-op through two front doors: the **chat** (LibreChat) and
|
|
the **API** (the LiteLLM gateway). Both route through a single gateway to
|
|
multiple LLM providers — so members choose what they value for each task:
|
|
architecturally private inference in **TEE-protected enclaves** (Tinfoil),
|
|
100% renewable energy (GreenPT, Regolo). A thin layer of custom middleware —
|
|
the member portal — ties Open
|
|
Collective membership to Cloudron accounts and LiteLLM keys.
|
|
|
|
```
|
|
┌─> Tinfoil (TEE) 🔒 private
|
|
Members ──> LibreChat (chat) ──┐ │
|
|
├──> LiteLLM ──┼─> GreenPT (renewable) 🌱 green
|
|
Members ──> your own tools ────┘ gateway │
|
|
(API) (keys, └─> Regolo (renewable) 🌱 green
|
|
metering,
|
|
routing)
|
|
```
|
|
|
|
The whole stack runs on [Cloudron](https://cloudron.io), which handles hosting,
|
|
single sign-on, and per-app isolation.
|
|
|
|
## Components
|
|
|
|
| Component | Purpose | URL | Code |
|
|
|-----------|---------|-----|------|
|
|
| **LibreChat** | Member-facing chat (models, web search, image generation, voice) | chat.inference.coop | [co-op/librechat-custom](https://git.inference.coop/co-op/librechat-custom) |
|
|
| **Open WebUI** | Legacy chat instance — read-only reference (old history + file attachments) | chat-legacy.inference.coop | — |
|
|
| **LiteLLM** | AI gateway — keys, metering, model routing | gateway.inference.coop | [code/litellm](https://git.inference.coop/code/litellm) |
|
|
| **Member Portal** | Membership middleware — onboarding, key provisioning, Loomio sync | portal.inference.coop | [code/member-portal](https://git.inference.coop/code/member-portal) |
|
|
| **Member Dashboard** | Member-facing usage view + API key management | dashboard.inference.coop | [code/member-dashboard](https://git.inference.coop/code/member-dashboard) |
|
|
| **Admin Panel** | Admin-only member overview, spend, balance management | panel.inference.coop | [code/admin-panel](https://git.inference.coop/code/admin-panel) |
|
|
| **SearXNG** | Self-hosted web search (feeds the chat's search) | search.inference.coop | — |
|
|
| **Loomio** | Member governance | forum.inference.coop | — |
|
|
| **Matrix (Synapse)** | Member chat server (federated; SSO login via Cloudron) | matrix.inference.coop | — |
|
|
| **Gitea** | Git hosting (code + docs) | git.inference.coop | — |
|
|
| **Surfer** | Static hosting (landing page, thanks page) | inference.coop | [co-op/website](https://git.inference.coop/co-op/website) |
|
|
| **Listmonk** | Member newsletter (auto-synced from membership) | newsletter.inference.coop | — |
|
|
| **Vaultwarden** | Password manager (admin credentials) | vault.inference.coop | — |
|
|
| **SnappyMail** | Webmail | mail.inference.coop | — |
|
|
|
|
Open Collective (billing and fiscal sponsorship) is the one external service in
|
|
the flow — see [opencollective.com/inference-cooperative](https://opencollective.com/inference-cooperative).
|
|
|
|
## Matrix
|
|
|
|
Members get a federated Matrix account on our self-hosted Synapse
|
|
(matrix.inference.coop, user IDs `@username:inference.coop`) using their
|
|
Cloudron SSO login. The portal invites every active member — and Nathan's
|
|
federated account — to the private **Inference Cooperative** Space (with a
|
|
**General** room) on every membership sync, so new members land in the space
|
|
automatically. Invites honor rate limits; external/federated members can be
|
|
added via the portal's `MATRIX_EXTRA_MXIDS` env.
|
|
|
|
Federation is enabled (delegated via `.well-known/matrix/server` on the bare
|
|
domain, port 443), so members can also join from any homeserver they already
|
|
use.
|
|
|
|
### Which account is which
|
|
|
|
Several automated accounts touch this homeserver, and two of them once differed
|
|
by a single character — which caused real confusion, including a bot appearing to
|
|
announce itself. The distinction is deliberate:
|
|
|
|
- **`@inferencebot:inference.coop` — the steward.** The bot members actually meet:
|
|
it posts digests, newsletter notes and reminders. It is *not* a Synapse server
|
|
admin, and holds no room power beyond what it needs to post.
|
|
- **`@inference-admin:inference.coop` — the room and server administrator.** A
|
|
Synapse server admin, used for provisioning rooms, managing membership, and
|
|
retiring rooms. It **never speaks in a member room**; a message from it is a
|
|
mistake, not a feature.
|
|
- **`@infrastructure-bot:inference.coop` — the infrastructure working group's bot.**
|
|
Answers infrastructure questions in **#infrastructure** when mentioned.
|
|
- **`@ops_audit:inference.coop`** — a server-admin account created for the October
|
|
2026 security review. Retained only until its tokens are revoked.
|
|
|
|
**House rule: power belongs to `@inference-admin`, voice belongs to `@inferencebot`.**
|
|
Please don't create another account whose name differs from an existing one by a
|
|
character or two; that is the specific confusion this section exists to undo.
|
|
|
|
**`@inference-bot:inference.coop` is retired (2026-10-29).** It was the original
|
|
room-administration account (it created the space and most of the rooms) and held a
|
|
Synapse server-admin flag. Its room power went to `@inference-admin`, which is now
|
|
administrator of record for #general, #community, #design, #skillshare, #updates and
|
|
#infrastructure, **and holds the portal's member-invite credential**
|
|
(`MATRIX_BOT_TOKEN`) under its own token. The old account's admin flag was dropped,
|
|
it was deactivated, and every device and token was deleted. Nothing should use it
|
|
again.
|
|
|
|
Two loose ends, recorded so nobody re-investigates them from scratch: it remains
|
|
joined to two rooms it cannot leave (they reject every write — `500` on leave), and
|
|
those rooms are also why they can't be deleted; and the old account's *name* is still
|
|
the tempting one, so check `whoami` rather than the name before trusting any account
|
|
that looks like it.
|
|
|
|
## How the apps run themselves
|
|
|
|
The portal and gateway are self-maintaining: their recurring housekeeping is
|
|
built into the apps as scheduled tasks, not run from any member's or operator's
|
|
personal machine.
|
|
|
|
- **Member portal** — two scheduled jobs (Cloudron scheduler):
|
|
- *Membership sweep* (nightly, 3am) — reconciles members against the live
|
|
Open Collective list: provisions new members who signed up while a webhook
|
|
was missed, deactivates lapsed members, and syncs Loomio + Matrix
|
|
membership. A missed webhook self-heals within 24 hours.
|
|
- *Docs knowledge refresh* (nightly, 3:30am) — pulls this documentation from
|
|
the public Git repo and updates the shared knowledge base behind the chat's
|
|
models, so answers grounded in the docs stay current. Changes to the docs
|
|
appear in the chat within a day, no manual step.
|
|
- **Gateway** — an *audio keep-alive* (every 10 minutes) sends a tiny text-to-
|
|
speech ping to the private enclave so voice mode responds quickly even after
|
|
long quiet periods, without compromising the end-to-end encryption model.
|
|
|
|
Scheduled tasks are declared in each app's `CloudronManifest.json` (see the
|
|
`code/` repos) and survive redeploys. If a task fails, it shows up in the
|
|
app's logs on the admin panel; the portal also exposes a manual trigger for
|
|
the membership sweep.
|
|
|
|
## How membership is wired
|
|
|
|
Membership flows through the member portal, which is the single source of truth
|
|
for "who is a member":
|
|
|
|
1. A member contributes on Open Collective.
|
|
2. Open Collective fires a webhook to the portal, which provisions a Cloudron
|
|
account, a LiteLLM key, and a welcome email.
|
|
3. A nightly sweep reconciles against the live Open Collective list, so a missed
|
|
webhook self-heals within 24 hours.
|
|
|
|
The portal database is authoritative for member emails and balances; LiteLLM's
|
|
team budget is the source of truth for a member's allowance, shared between chat
|
|
and API.
|
|
|
|
### The bot account on Open Collective
|
|
|
|
You may notice **"Inference Co-op Bot"** listed as an admin of the collective on
|
|
Open Collective. This is infrastructure, not a member or a person:
|
|
|
|
- **What it is:** a dedicated Open Collective account owned by the co-op. It
|
|
holds a **Personal Token** that the member portal uses to read membership
|
|
data — the roster, tiers (needed to route credit-pack purchases correctly),
|
|
and contributor emails (including guests, which only admins can see).
|
|
- **What it can do:** read member and tier data via Open Collective's API.
|
|
It cannot move money, issue refunds, or change the collective — Open
|
|
Collective restricts those to dashboard actions by human admins. Its API
|
|
access was verified read-mostly during the October 2026 security review.
|
|
- **Why it has admin:** reading guest contributor emails requires admin on
|
|
Open Collective; without it, membership reconciliation and credit-pack
|
|
routing can't function.
|
|
- **Who supervises it:** all of its code is open source in
|
|
[code/member-portal](https://git.inference.coop/code/member-portal) — every
|
|
API call it makes is documented there. Nathan Schneider (collective admin)
|
|
oversees its use and can revoke or rotate its token at any time from the
|
|
Open Collective dashboard.
|
|
|
|
Nothing on Open Collective is charged, transferred, or refunded by automation:
|
|
payments always come from a human's own contribution action, and any automated
|
|
system only *reads* the resulting records.
|
|
|
|
## The code
|
|
|
|
The cooperative's software is open source, in the [`code`](https://git.inference.coop/code)
|
|
organization on Gitea:
|
|
|
|
- [**code/member-portal**](https://git.inference.coop/code/member-portal) — membership middleware (Python).
|
|
- [**code/member-dashboard**](https://git.inference.coop/code/member-dashboard) — member-facing usage + API keys (Python).
|
|
- [**code/admin-panel**](https://git.inference.coop/code/admin-panel) — admin overview (Python).
|
|
- [**code/litellm**](https://git.inference.coop/code/litellm) — the gateway packaged as a Cloudron app.
|
|
|
|
Plus the `co-op` organization:
|
|
|
|
- [**co-op/website**](https://git.inference.coop/co-op/website) — landing page and static site.
|
|
- [**co-op/docs**](https://git.inference.coop/co-op/docs) — this documentation.
|
|
- [**co-op/design-assets**](https://git.inference.coop/co-op/design-assets) — logos, fonts, brand materials.
|
|
|
|
## Contributing (the Git is yours, too)
|
|
|
|
Everything the co-op runs is open source and lives at
|
|
[git.inference.coop](https://git.inference.coop) — the website, this
|
|
documentation, the member portal, the dashboard, the admin panel, and the
|
|
gateway. As a member, you can browse all of it and propose changes.
|
|
|
|
**Who can sign in.** Git access is tied to your membership: you log into the Git
|
|
with the same single sign-on you use for the chat. Members are in automatically
|
|
(through the `members` group); when a membership lapses, Git access lapses with it.
|
|
|
|
**How to contribute.** Anyone can read the repos without an account. To make a
|
|
change, fork the repo, edit in your fork, and open a **pull request**:
|
|
|
|
1. Log into [git.inference.coop](https://git.inference.coop) with your co-op account.
|
|
2. **Fork** the repo you want to change (e.g. `co-op/docs`).
|
|
3. Make your edits in your fork and commit them.
|
|
4. Open a **pull request** back to the original repo, describing your change.
|
|
|
|
**All changes are reviewed.** Nothing merges directly into `main` — every change
|
|
comes in as a pull request that an admin reviews and approves first. You don't
|
|
need to be able to create repos or organizations to contribute; you just fork,
|
|
edit, and ask for review.
|
|
|
|
Questions about the Git itself? Write to info@inference.coop.
|
|
|
|
## Models and providers
|
|
|
|
The co-op serves models from multiple providers (Tinfoil for privacy, GreenPT
|
|
for renewable energy, PublicAI coming soon). See the [model list](models.md) for
|
|
the current catalog — that's the single source of truth for which models are
|
|
available and how they're named.
|
|
|
|
One infra detail worth knowing: a local Tinfoil proxy sidecar handles the
|
|
enclave encryption on our side of the gateway — only Tinfoil traffic goes
|
|
through it; the other providers are called directly.
|
|
|
|
## Security posture (plain language)
|
|
|
|
- **Token-gated control plane.** The portal is publicly reachable, but every
|
|
sensitive surface requires a secret token (admin, broker, webhook). All fail
|
|
closed — unauthenticated requests get `401`.
|
|
- **Separated secrets.** Chat, broker, admin, and model-layer secrets are
|
|
independent and independently rotatable, so a leak has bounded scope.
|
|
- **Member isolation.** Each API key is scoped to its owner's LiteLLM team and
|
|
budget; a member can only ever act on their own keys.
|
|
- **Rate limiting** on control-plane endpoints; the inference path is capped by
|
|
each member's budget instead.
|
|
- **No secrets in version control.** Credentials live in Cloudron env vars, never
|
|
in the repos.
|
|
- **Accepted trade-offs** (documented, not hidden): member keys are stored in
|
|
plaintext in the portal database (functionally necessary), and the gateway
|
|
must stay publicly reachable for API access (its admin UI and docs are
|
|
disabled; only the `/v1/*` API is exposed).
|
|
|
|
## For operators
|
|
|
|
Maintaining the co-op's infrastructure — rotating Cloudron tokens, editing the
|
|
portal's environment, and troubleshooting the provisioning pipeline — is
|
|
documented in the [operator runbook](operator-token-runbook.md).
|
|
|
|
## Incident log
|
|
|
|
Write-ups for outages live in [`incidents/`](incidents/). One so far:
|
|
[2026-10-02 — gateway outage after a security-upgrade rebuild](incidents/2026-10-02-gateway-cve-upgrade.md).
|