Split docs into README TOC + getting-started, api-access, infrastructure guides
This commit is contained in:
1 parent
39b6bfa770
commit
4cccc521e8
4 files changed
+298
-172
No files matched your search
@@ -0,0 +1,112 @@
|
||||
# Infrastructure
|
||||
|
||||
How the Inference Cooperative is built, for members who want to understand it —
|
||||
and contributors who want to work on it.
|
||||
|
||||
## The big picture
|
||||
|
||||
Members reach the co-op through two front doors: the **chat** (Open WebUI) and
|
||||
the **API** (the LiteLLM gateway). Both route through a single gateway to cloud
|
||||
LLM providers running inside **TEE-protected enclaves** (Tinfoil), so inference
|
||||
is private end-to-end. A thin layer of custom middleware — the member portal —
|
||||
ties Open Collective membership to Cloudron accounts and LiteLLM keys.
|
||||
|
||||
```
|
||||
Members ──> Open WebUI (chat) ─┐
|
||||
├──> LiteLLM gateway ──> Tinfoil (TEE) ──> models
|
||||
Members ──> your own tools ────┘
|
||||
(API)
|
||||
```
|
||||
|
||||
The whole stack runs on [Cloudron](https://cloudron.io), which handles hosting,
|
||||
single sign-on, and per-app isolation.
|
||||
|
||||
## Components
|
||||
|
||||
| Component | Purpose | URL | Code |
|
||||
|-----------|---------|-----|------|
|
||||
| **Open WebUI** | Member-facing chat (web search, file uploads) | chat.inference.coop | — |
|
||||
| **LiteLLM** | AI gateway — keys, metering, model routing | gateway.inference.coop | [code/litellm](https://git.inference.coop/code/litellm) |
|
||||
| **Member Portal** | Membership middleware — onboarding, key provisioning, Loomio sync | portal.inference.coop | [code/member-portal](https://git.inference.coop/code/member-portal) |
|
||||
| **Member Dashboard** | Member-facing usage view + API key management | dashboard.inference.coop | [code/member-dashboard](https://git.inference.coop/code/member-dashboard) |
|
||||
| **Admin Panel** | Admin-only member overview, spend, balance management | panel.inference.coop | [code/admin-panel](https://git.inference.coop/code/admin-panel) |
|
||||
| **SearXNG** | Self-hosted web search (feeds the chat's search) | search.inference.coop | — |
|
||||
| **Loomio** | Member governance | forum.inference.coop | — |
|
||||
| **Gitea** | Git hosting (code + docs) | git.inference.coop | — |
|
||||
| **Surfer** | Static hosting (landing page, thanks page) | inference.coop | [co-op/website](https://git.inference.coop/co-op/website) |
|
||||
| **Listmonk** | Member newsletter (auto-synced from membership) | newsletter.inference.coop | — |
|
||||
| **Vaultwarden** | Password manager (admin credentials) | vault.inference.coop | — |
|
||||
| **SnappyMail** | Webmail | mail.inference.coop | — |
|
||||
|
||||
Open Collective (billing and fiscal sponsorship) is the one external service in
|
||||
the flow — see [opencollective.com/inference-cooperative](https://opencollective.com/inference-cooperative).
|
||||
|
||||
## How membership is wired
|
||||
|
||||
Membership flows through the member portal, which is the single source of truth
|
||||
for "who is a member":
|
||||
|
||||
1. A member contributes on Open Collective.
|
||||
2. Open Collective fires a webhook to the portal, which provisions a Cloudron
|
||||
account, a LiteLLM key, and a welcome email.
|
||||
3. A nightly sweep reconciles against the live Open Collective list, so a missed
|
||||
webhook self-heals within 24 hours.
|
||||
|
||||
The portal database is authoritative for member emails and balances; LiteLLM's
|
||||
team budget is the source of truth for a member's allowance, shared between chat
|
||||
and API.
|
||||
|
||||
## The code
|
||||
|
||||
The cooperative's software is open source, in the [`code`](https://git.inference.coop/code)
|
||||
organization on Gitea:
|
||||
|
||||
- [**code/member-portal**](https://git.inference.coop/code/member-portal) — membership middleware (Python).
|
||||
- [**code/member-dashboard**](https://git.inference.coop/code/member-dashboard) — member-facing usage + API keys (Python).
|
||||
- [**code/admin-panel**](https://git.inference.coop/code/admin-panel) — admin overview (Python).
|
||||
- [**code/litellm**](https://git.inference.coop/code/litellm) — the gateway packaged as a Cloudron app.
|
||||
|
||||
Plus the `co-op` organization:
|
||||
|
||||
- [**co-op/website**](https://git.inference.coop/co-op/website) — landing page and static site.
|
||||
- [**co-op/docs**](https://git.inference.coop/co-op/docs) — this documentation.
|
||||
- [**co-op/design-assets**](https://git.inference.coop/co-op/design-assets) — logos, fonts, brand materials.
|
||||
|
||||
## Models and privacy
|
||||
|
||||
Inference runs through [Tinfoil](https://tinfoil.sh/inference), which provides
|
||||
**architectural** privacy: models run inside hardware enclaves (TEEs), and
|
||||
request/response bodies are encrypted end-to-end (EHBP), so even Tinfoil's own
|
||||
infrastructure can't read them. This is verifiable via remote attestation — not
|
||||
just a policy promise. A local Tinfoil proxy sidecar handles the encryption on
|
||||
our side of the gateway.
|
||||
|
||||
Three models are exposed:
|
||||
|
||||
- **DeepSeek V4.1 Flash** — default, agentic tasks.
|
||||
- **GPT-OSS 120B** — lightweight fallback.
|
||||
- **GLM-5.3 Flash** — fast, efficient.
|
||||
|
||||
The honest caveat: your **chat history** is stored on our server so you can
|
||||
revisit it, and that stored history is not encrypted in a way that prevents us
|
||||
from technically reading it. We commit not to. The full distinction — what's
|
||||
architecturally private versus what's a policy commitment — is in the
|
||||
[Privacy Policy](privacy-policy.md).
|
||||
|
||||
## Security posture (plain language)
|
||||
|
||||
- **Token-gated control plane.** The portal is publicly reachable, but every
|
||||
sensitive surface requires a secret token (admin, broker, webhook). All fail
|
||||
closed — unauthenticated requests get `401`.
|
||||
- **Separated secrets.** Chat, broker, admin, and model-layer secrets are
|
||||
independent and independently rotatable, so a leak has bounded scope.
|
||||
- **Member isolation.** Each API key is scoped to its owner's LiteLLM team and
|
||||
budget; a member can only ever act on their own keys.
|
||||
- **Rate limiting** on control-plane endpoints; the inference path is capped by
|
||||
each member's budget instead.
|
||||
- **No secrets in version control.** Credentials live in Cloudron env vars, never
|
||||
in the repos.
|
||||
- **Accepted trade-offs** (documented, not hidden): member keys are stored in
|
||||
plaintext in the portal database (functionally necessary), and the gateway
|
||||
must stay publicly reachable for API access (its admin UI and docs are
|
||||
disabled; only the `/v1/*` API is exposed).
|
||||
Reference in new issue
Block a user