Adds guides/what-happens-with-your-data.md covering what the gateway, chat server and each provider keep, why the routing path differs from a marketplace router, and what the co-op does not claim. Links it from Models, API access, Getting started, Guest access, Infrastructure, the docs index, README and privacy policy. Fixes Getting started's temporary-chat claim: temporary chats are server-backed and deleted about an hour after creation, not never stored.
13 KiB
Infrastructure
How the Inference Cooperative is built, for members who want to understand it — and contributors who want to work on it.
The big picture
Members reach the co-op through two front doors: the chat (LibreChat) and the API (the LiteLLM gateway). Both route through a single gateway to multiple LLM providers, so members can choose what they value for each task: architecturally private inference in TEE-protected enclaves (Tinfoil), or 100% renewable energy (GreenPT, Regolo). A thin layer of custom middleware — the member portal — ties Open Collective membership to Cloudron accounts and LiteLLM keys.
┌─> Tinfoil (TEE) 🔒 private
Members ──> LibreChat (chat) ──┐ │
├──> LiteLLM ──┼─> GreenPT (renewable) 🌱 green
Members ──> your own tools ────┘ gateway │
(API) (keys, └─> Regolo (renewable) 🌱 green
metering,
routing)
The whole stack runs on Cloudron, which handles hosting, single sign-on, and per-app isolation.
Where it runs
The co-op's member-facing services run on a server the co-op owns and pays for directly, at Infomaniak — a Swiss sovereign cloud provider whose data centres are in Geneva, run on renewable energy, and are certified to ISO 14001, ISO 50001 and ISO 27001.
The co-op's automation runs on a separate server, also hosted on Infomaniak and deliberately kept apart from the member-facing applications. The steward bot — the automation behind the digests and the newsletter drafts — and the co-op's other agents live there. Keeping them on their own host means an agent problem stays an agent problem: it cannot reach the chat, the gateway or member data. Access to that host is SSH-key only — no agent or bot account has a password. The steward can act inside the bot accounts it maintains but cannot take root; root is a separate human operator account, so host-level changes stay with a person.
Backups
Cloudron writes encrypted backups to Infomaniak's object storage in Switzerland every night and keeps the last seven days, so a failed upgrade or a lost server can be restored. The automation host keeps its own encrypted backup as well, with credentials kept apart from the agent's environment. That backup runs as root from a root-owned script, so the agent can trigger it but cannot alter the backup code or read the backup keys.
Domain and DNS
The co-op's domain is inference.coop — a .coop domain. The co-op registers
and manages its domains with domains.coop, the domain provider run by the
cooperative movement, and holds them in the co-op's own name rather than in any
individual's. DNS itself is deliberately simple, and the co-op's own nameservers
answer it.
Components
| Component | Purpose | URL |
|---|---|---|
| LibreChat | Member-facing chat (models, web search, image generation, voice) | chat.inference.coop |
| Open WebUI | Legacy chat instance — read-only reference (old history and file attachments) | chat-legacy.inference.coop |
| LiteLLM | AI gateway — keys, metering and model routing, for chat and API | gateway.inference.coop |
| Member portal | Membership middleware — onboarding, key provisioning, Loomio sync | portal.inference.coop |
| Member dashboard | Your usage view and API keys | dashboard.inference.coop |
| Loomio | Member governance | forum.inference.coop |
| Matrix (Synapse) | Member chat server, federated, SSO login via Cloudron | matrix.inference.coop |
| Gitea | Git hosting for the co-op's code and this documentation | git.inference.coop |
| Listmonk | Member newsletter (synced from membership) | newsletter.inference.coop |
| SnappyMail | Webmail | email.inference.coop |
The co-op's software behind all of these is open source — see The code below for where each piece lives.
Open Collective (billing and fiscal sponsorship) is the one external service in the membership flow. See opencollective.com/inference-cooperative. Web search uses external services too; see Chat retention and web search.
The co-op runs its own @inference.coop mail. Incoming mail arrives directly
on the co-op's server, and webmail and mail apps reach it over the usual
encrypted ports (IMAP and SMTP over TLS). Outgoing mail is sent through
Scaleway, an EU-based email service, authenticated for our domain.
You can reach your mailbox through webmail at email.inference.coop, or add it to
a standard mail app on your phone or computer. If mail to your @inference.coop
address stops arriving, that is a fault on our side to fix.
Matrix
Members get a federated Matrix account on the co-op's self-hosted Synapse
(matrix.inference.coop, user IDs @username:inference.coop), signing in with the
same Cloudron SSO as the chat. The co-op invites every active member to the
private Inference Cooperative space (with a General room) automatically,
so new members land there as they join. Federation is on, so you can also join
from any homeserver you already use.
Accounts you may meet
A few accounts in the space are automation rather than people:
@inferencebot— the steward. It posts the digests, newsletter notes and reminders. It holds no more power than posting needs.@inference-admin— the administrator of record for the co-op's rooms (#general, #community, #design, #skillshare, #updates and #infrastructure) and their membership. It never speaks in a member room — a message from it is a mistake, not a feature.
The house rule: power belongs to @inference-admin, voice belongs to
@inferencebot. Please don't create an account whose name differs from an
existing one by a character or two — spotting lookalikes is exactly what this
convention is for.
Membership
Membership runs through the member portal, the single source of truth for who is a member:
- A member contributes on Open Collective.
- Open Collective notifies the portal, which provisions a Cloudron account, a LiteLLM key, and a welcome email.
- A nightly reconciliation against the live Open Collective list means a missed notification self-heals within a day.
You may see "Inference Co-op Bot" listed as an admin of the collective on Open Collective. It is infrastructure, not a person or a member: a dedicated account the portal uses, read-only, to check the member roster and tiers. It cannot move money, issue refunds or change the collective — payments always come from a human's own contribution, and automation only reads the result. Its code is open source in code/member-portal, and a collective admin can revoke its access at any time.
The code
The co-op's software is open source, in the code
and co-op organizations on Gitea:
- code/member-portal — membership middleware (Python).
- code/member-dashboard — member dashboard: usage and API keys.
- code/litellm — the gateway packaged as a Cloudron app.
- co-op/website — landing page and static site.
- co-op/docs — this documentation.
The chat, portal and other apps live in code/ and co-op/ too; everything is
browsable at git.inference.coop. Everything the co-op runs
is open source, so as a member you can browse all of
it and propose changes.
Who can sign in. Git access is tied to your membership: you log into the Git
with the same single sign-on you use for the chat. Members are in automatically
(through the members group); when a membership lapses, Git access lapses with
it.
How to contribute. Anyone can read the repos without an account. To make a change, fork the repo, edit in your fork, and open a pull request:
- Log into git.inference.coop with your co-op account.
- Fork the repo you want to change (e.g.
co-op/docs). - Make your edits in your fork and commit them.
- Open a pull request back to the original repo, describing your change.
All changes are reviewed. Nothing merges directly into main — every change
comes in as a pull request an admin reviews and approves first. You don't need
to be able to create repos or organizations to contribute; you just fork, edit
and ask for review.
Models and providers
The co-op serves models from multiple providers — Tinfoil for architectural privacy, GreenPT and Regolo for renewable energy. See the model list, the single source of truth for which models are available and how they're named.
Chat retention and web search
Saved chats live in the chat database and its search index until the member deletes them. Web-search tool calls are part of that record: the query and the results returned are stored with the message.
Temporary chats are server-backed — messages must reach the server while the
chat is running — so they are not client-only. The chat is configured to delete
them from the database about one hour after creation (interface.temporaryChatRetention: 1
in the chat configuration; one hour is the minimum this LibreChat version
accepts). They are hidden from history immediately, and the database enforces the
expiry automatically. Temporary chats created before this setting took effect
keep their original 30-day expiry until they age out.
Backups are a separate layer. Cloudron backups are whole-app snapshots and cannot exclude individual chats, so a temporary chat present when a backup runs can remain recoverable from that backup until the backup itself expires (see Backups).
The member-facing summary across all three layers — gateway, chat server and providers — is What happens with your data.
Web search in the chat uses DuckDuckGo and Brave through the co-op's self-hosted SearXNG instance. Queries are sent from the co-op server, so the upstream engines see the query but not the member's IP or identity. The SearXNG instance is also publicly reachable at search.inference.coop, and its logs can contain query text when an upstream engine request fails; those log lines carry no member identity or IP, and are retained only briefly. The reverse proxy does not log search queries.
LibreChat reads the pages behind the results through the co-op's own
page-fetch service (fetch.inference.coop), a small app on our Cloudron. It
runs with no access log and keeps no copy of URLs or page text; it refuses
private, loopback, and internal addresses. The
privacy policy describes it.
Security posture (plain language)
- Member isolation. Each API key is scoped to its owner's team and budget; a member can only ever act on their own keys.
- Separated secrets. Chat, portal, admin and model-layer secrets are independent and independently rotatable, so a leak has bounded scope.
- No secrets in version control. Credentials live in the hosting platform's environment, never in the repos.
- Least-privilege automation. Every agent runs as its own Unix user; bots have
no
sudo, the steward can act only within the bot accounts it maintains, and root belongs to a separate key-only operator account. Host-level changes are therefore deliberate human actions, not automation. - Accepted trade-offs (documented, not hidden): member keys are stored in the portal database so the co-op can serve them, and the API gateway must stay publicly reachable. Its admin surfaces are disabled; only the public API is exposed.
- Operational detail lives with operators. Token scopes, backup handling,
CI trust and host access are recorded in
co-op/ops-docs, not on this page.
For operators
Maintaining the co-op's infrastructure — token rotation, app configuration and
troubleshooting — is documented in the git-only
co-op/ops-docs repository, which is
deliberately not published on this site. It is the home for the operator runbook,
the maintenance notes, incident write-ups, and the record of past server
migrations. Anyone may read it; operators are the audience.
Incident log
Write-ups for outages live in the operator docs, in
incidents/.