Explain the two stages: SearXNG queries DuckDuckGo and Brave, then Keenable fetches the top result pages. Name the SearXNG instance URL and disclose Keenable's logging, retention, processing location, and training use. Correct the claim that web search has no external dependency.
214 lines
10 KiB
Markdown
214 lines
10 KiB
Markdown
# Infrastructure
|
|
|
|
How the Inference Cooperative is built, for members who want to understand it —
|
|
and contributors who want to work on it.
|
|
|
|
## The big picture
|
|
|
|
Members reach the co-op through two front doors: the **chat** (LibreChat) and
|
|
the **API** (the LiteLLM gateway). Both route through a single gateway to
|
|
multiple LLM providers, so members can choose what they value for each task:
|
|
architecturally private inference in **TEE-protected enclaves** (Tinfoil), or
|
|
100% renewable energy (GreenPT, Regolo). A thin layer of custom middleware — the
|
|
member portal — ties Open Collective membership to Cloudron accounts and LiteLLM
|
|
keys.
|
|
|
|
```
|
|
┌─> Tinfoil (TEE) 🔒 private
|
|
Members ──> LibreChat (chat) ──┐ │
|
|
├──> LiteLLM ──┼─> GreenPT (renewable) 🌱 green
|
|
Members ──> your own tools ────┘ gateway │
|
|
(API) (keys, └─> Regolo (renewable) 🌱 green
|
|
metering,
|
|
routing)
|
|
```
|
|
|
|
The whole stack runs on [Cloudron](https://cloudron.io), which handles hosting,
|
|
single sign-on, and per-app isolation.
|
|
|
|
## Where it runs
|
|
|
|
The co-op's member-facing services run on a server the co-op owns and pays for
|
|
directly, at **Infomaniak** — a Swiss sovereign cloud provider whose data centres
|
|
are in Geneva, run on renewable energy, and are certified to ISO 14001, ISO
|
|
50001 and ISO 27001.
|
|
|
|
The co-op's **automation runs on a separate server**, also hosted on Infomaniak
|
|
and deliberately kept apart from the member-facing applications. The steward bot
|
|
— the automation behind the digests and the newsletter drafts — and the co-op's
|
|
other agents live there. Keeping them on their own host means an agent problem
|
|
stays an agent problem: it cannot reach the chat, the gateway or member data.
|
|
|
|
### Backups
|
|
|
|
Cloudron writes **encrypted** backups to Infomaniak's object storage in
|
|
Switzerland every night and keeps the last **seven days**, so a failed upgrade
|
|
or a lost server can be restored. The automation host backs up its own data to
|
|
the same object storage.
|
|
|
|
### Domain and DNS
|
|
|
|
The co-op's domain is **inference.coop** — a `.coop` domain. The co-op registers
|
|
and manages its domains with **domains.coop**, the domain provider run by the
|
|
cooperative movement, and holds them in the co-op's own name rather than in any
|
|
individual's. DNS itself is deliberately simple, and the co-op's own nameservers
|
|
answer it.
|
|
|
|
## Components
|
|
|
|
| Component | Purpose | URL |
|
|
|-----------|---------|-----|
|
|
| **LibreChat** | Member-facing chat (models, web search, image generation, voice) | chat.inference.coop |
|
|
| **Open WebUI** | Legacy chat instance — read-only reference (old history and file attachments) | chat-legacy.inference.coop |
|
|
| **LiteLLM** | AI gateway — keys, metering and model routing, for chat and API | gateway.inference.coop |
|
|
| **Member portal** | Membership middleware — onboarding, key provisioning, Loomio sync | portal.inference.coop |
|
|
| **Member dashboard** | Your usage view and API keys | dashboard.inference.coop |
|
|
| **Loomio** | Member governance | forum.inference.coop |
|
|
| **Matrix (Synapse)** | Member chat server, federated, SSO login via Cloudron | matrix.inference.coop |
|
|
| **Gitea** | Git hosting for the co-op's code and this documentation | git.inference.coop |
|
|
| **Listmonk** | Member newsletter (synced from membership) | newsletter.inference.coop |
|
|
| **SnappyMail** | Webmail | email.inference.coop |
|
|
|
|
The co-op's software behind all of these is open source — see [The
|
|
code](#the-code) below for where each piece lives.
|
|
|
|
Open Collective (billing and fiscal sponsorship) is the one external service in
|
|
the membership flow. See
|
|
[opencollective.com/inference-cooperative](https://opencollective.com/inference-cooperative).
|
|
Web search uses external services: DuckDuckGo and Brave receive queries, and
|
|
Keenable fetches page content. See [Web search](#web-search).
|
|
|
|
### Web search
|
|
|
|
LibreChat's web search runs in two stages. A self-hosted **SearXNG** instance at
|
|
[search.inference.coop](https://search.inference.coop) sends the query to
|
|
DuckDuckGo and Brave. It returns candidate results. LibreChat sends the top
|
|
result URLs to **Keenable**, a third-party page-fetch service.
|
|
Keenable extracts text from each page. The model uses that text as context for
|
|
the answer. SearXNG runs on co-op infrastructure. Keenable is an external
|
|
service, and the [privacy policy](../governance/privacy-policy.md#5-search)
|
|
describes its logging and retention.
|
|
|
|
## Mail
|
|
|
|
The co-op runs its own `@inference.coop` mail. **Incoming** mail arrives directly
|
|
on the co-op's server, and webmail and mail apps reach it over the usual
|
|
encrypted ports (IMAP and SMTP over TLS). **Outgoing** mail is sent through
|
|
**Scaleway**, an EU-based email service, authenticated for our domain.
|
|
|
|
You can reach your mailbox through webmail at email.inference.coop, or add it to
|
|
a standard mail app on your phone or computer. If mail to your `@inference.coop`
|
|
address stops arriving, that is a fault on our side to fix.
|
|
|
|
## Matrix
|
|
|
|
Members get a federated Matrix account on the co-op's self-hosted Synapse
|
|
(matrix.inference.coop, user IDs `@username:inference.coop`), signing in with the
|
|
same Cloudron SSO as the chat. The co-op invites every active member to the
|
|
private **Inference Cooperative** space (with a **General** room) automatically,
|
|
so new members land there as they join. Federation is on, so you can also join
|
|
from any homeserver you already use.
|
|
|
|
### Accounts you may meet
|
|
|
|
A few accounts in the space are automation rather than people:
|
|
|
|
- **`@inferencebot` — the steward.** It posts the digests, newsletter notes and
|
|
reminders. It holds no more power than posting needs.
|
|
- **`@inference-admin`** — the administrator of record for the co-op's rooms
|
|
(#general, #community, #design, #skillshare, #updates and #infrastructure) and
|
|
their membership. It **never speaks in a member room** — a message from it is
|
|
a mistake, not a feature.
|
|
|
|
The house rule: **power belongs to `@inference-admin`, voice belongs to
|
|
`@inferencebot`.** Please don't create an account whose name differs from an
|
|
existing one by a character or two — spotting lookalikes is exactly what this
|
|
convention is for.
|
|
|
|
## Membership
|
|
|
|
Membership runs through the member portal, the single source of truth for who is
|
|
a member:
|
|
|
|
1. A member contributes on Open Collective.
|
|
2. Open Collective notifies the portal, which provisions a Cloudron account, a
|
|
LiteLLM key, and a welcome email.
|
|
3. A nightly reconciliation against the live Open Collective list means a missed
|
|
notification self-heals within a day.
|
|
|
|
You may see **"Inference Co-op Bot"** listed as an admin of the collective on
|
|
Open Collective. It is infrastructure, not a person or a member: a dedicated
|
|
account the portal uses, **read-only**, to check the member roster and tiers. It
|
|
cannot move money, issue refunds or change the collective — payments always come
|
|
from a human's own contribution, and automation only reads the result. Its code
|
|
is open source in
|
|
[code/member-portal](https://git.inference.coop/code/member-portal), and a
|
|
collective admin can revoke its access at any time.
|
|
|
|
## The code
|
|
|
|
The co-op's software is open source, in the [`code`](https://git.inference.coop/code)
|
|
and [`co-op`](https://git.inference.coop/co-op) organizations on Gitea:
|
|
|
|
- [**code/member-portal**](https://git.inference.coop/code/member-portal) — membership middleware (Python).
|
|
- [**code/member-dashboard**](https://git.inference.coop/code/member-dashboard) — member dashboard: usage and API keys.
|
|
- [**code/litellm**](https://git.inference.coop/code/litellm) — the gateway packaged as a Cloudron app.
|
|
- [**co-op/website**](https://git.inference.coop/co-op/website) — landing page and static site.
|
|
- [**co-op/docs**](https://git.inference.coop/co-op/docs) — this documentation.
|
|
|
|
The chat, portal and other apps live in `code/` and `co-op/` too; everything is
|
|
browsable at [git.inference.coop](https://git.inference.coop). Everything the co-op runs
|
|
is open source, so as a member you can browse all of
|
|
it and propose changes.
|
|
|
|
**Who can sign in.** Git access is tied to your membership: you log into the Git
|
|
with the same single sign-on you use for the chat. Members are in automatically
|
|
(through the `members` group); when a membership lapses, Git access lapses with
|
|
it.
|
|
|
|
**How to contribute.** Anyone can read the repos without an account. To make a
|
|
change, fork the repo, edit in your fork, and open a **pull request**:
|
|
|
|
1. Log into [git.inference.coop](https://git.inference.coop) with your co-op account.
|
|
2. **Fork** the repo you want to change (e.g. `co-op/docs`).
|
|
3. Make your edits in your fork and commit them.
|
|
4. Open a **pull request** back to the original repo, describing your change.
|
|
|
|
**All changes are reviewed.** Nothing merges directly into `main` — every change
|
|
comes in as a pull request an admin reviews and approves first. You don't need
|
|
to be able to create repos or organizations to contribute; you just fork, edit
|
|
and ask for review.
|
|
|
|
## Models and providers
|
|
|
|
The co-op serves models from multiple providers — Tinfoil for architectural
|
|
privacy, GreenPT and Regolo for renewable energy. See the [model list](../guides/models.md),
|
|
the single source of truth for which models are available and how they're named.
|
|
|
|
## Security posture (plain language)
|
|
|
|
- **Member isolation.** Each API key is scoped to its owner's team and budget; a
|
|
member can only ever act on their own keys.
|
|
- **Separated secrets.** Chat, portal, admin and model-layer secrets are
|
|
independent and independently rotatable, so a leak has bounded scope.
|
|
- **No secrets in version control.** Credentials live in the hosting platform's
|
|
environment, never in the repos.
|
|
- **Accepted trade-offs** (documented, not hidden): member keys are stored in
|
|
the portal database so the co-op can serve them, and the API gateway must stay
|
|
publicly reachable. Its admin surfaces are disabled; only the public API is
|
|
exposed.
|
|
|
|
## For operators
|
|
|
|
Maintaining the co-op's infrastructure — token rotation, app configuration and
|
|
troubleshooting — is documented in the git-only
|
|
[`co-op/ops-docs`](https://git.inference.coop/co-op/ops-docs) repository, which is
|
|
deliberately not published on this site. It is the home for the operator runbook,
|
|
the maintenance notes, incident write-ups, and the record of past server
|
|
migrations. Anyone may read it; operators are the audience.
|
|
|
|
## Incident log
|
|
|
|
Write-ups for outages live in the operator docs, in
|
|
[`incidents/`](https://git.inference.coop/co-op/ops-docs/src/branch/main/incidents).
|