71 lines
4.1 KiB
Markdown
71 lines
4.1 KiB
Markdown
# Inference Cooperative
|
||
|
||
**Private AI, governed together.**
|
||
|
||
The Inference Cooperative is a member-governed project providing private AI inference. We are fiscally sponsored by [Metagov](https://metagov.org), a nonprofit, and funded through our [Open Collective](https://opencollective.com/inference-cooperative).
|
||
|
||
- **Website:** https://inference.coop
|
||
- **AI Chat:** https://chat.inference.coop
|
||
- **Open Collective:** https://opencollective.com/inference-cooperative
|
||
- **Contact:** info@inference.coop
|
||
|
||
## The model
|
||
|
||
The Inference Cooperative is a **member-governed** AI inference utility. Members contribute on a sliding scale (currently $10–20/month) and, in return, get access to private AI inference through a shared, cooperatively governed infrastructure.
|
||
|
||
Because we are legally under a nonprofit—[Metagov](https://metagov.org/) is our fiscal sponsor—we are member-governed rather than member-owned: members direct the project through governance, while the nonprofit holds the legal and financial structure.
|
||
|
||
### Membership
|
||
|
||
- **Sliding scale dues:** $10–20/month
|
||
- **Access:** private AI inference through the cooperative's gateway
|
||
- **Governance:** members participate in decisions about models, priorities, and direction
|
||
|
||
## The stack
|
||
|
||
The Inference Cooperative runs a self-hosted stack on [Cloudron](https://cloudron.io), with all inference routed through a single gateway to cloud-based LLM providers.
|
||
|
||
### Components
|
||
|
||
| Component | Purpose | URL |
|
||
|-----------|---------|-----|
|
||
| **Open WebUI** | Member-facing chat interface (with web search) | chat.inference.coop |
|
||
| **LiteLLM** | AI gateway — keys, metering, model routing | gateway.inference.coop |
|
||
| **Member Portal** | Membership middleware — onboarding, key provisioning, Loomio sync | portal.inference.coop |
|
||
| **SearXNG** | Self-hosted web search (feeds the chat's search tool) | search.inference.coop |
|
||
| **Gitea** | Git hosting (code + docs) | git.inference.coop |
|
||
| **Loomio** | Member governance | forum.inference.coop |
|
||
| **Open Collective** | Membership billing + fiscal sponsorship | opencollective.com/inference-cooperative |
|
||
|
||
The cooperative's software is open source and lives in the [`code`](https://git.inference.coop/code) organization on our Gitea instance.
|
||
|
||
### Models
|
||
|
||
The gateway currently exposes three models (all served through Tinfoil's TEE-protected enclaves):
|
||
|
||
- **DeepSeek V4 Flash** — default model, best for agentic tasks (1M context, tool calling)
|
||
- **GPT-OSS 120B** — lightweight fallback
|
||
- **GLM-5.3 Flash** — fast, efficient MoE model
|
||
|
||
### Features
|
||
|
||
- **Web search** — enabled by default, backed by a self-hosted SearXNG instance (no external search API or scraper key required).
|
||
- **File uploads (RAG)** — works out of the box via Open WebUI's bundled local embedding model (`sentence-transformers/all-MiniLM-L6-v2`), no external embedding API.
|
||
|
||
### Privacy
|
||
|
||
Privacy is a core value. Inference now runs through [Tinfoil](https://tinfoil.sh/inference), which provides **architectural** privacy: models run inside hardware enclaves (TEEs), and request/response bodies are encrypted end-to-end with the Encrypted HTTP Body Protocol (EHBP), so even Tinfoil's own infrastructure cannot read them. This is verifiable via remote attestation — not just a policy promise.
|
||
|
||
Because Tinfoil requires EHBP-encrypted request bodies (it rejects plaintext with `426 EHBP_REQUIRED`), the LiteLLM gateway routes through a local [Tinfoil proxy](https://github.com/tinfoilsh/tinfoil-proxy) sidecar, which verifies the enclave attestation and handles the encryption. LiteLLM talks plaintext OpenAI to the proxy; the proxy encrypts and forwards to the enclave.
|
||
|
||
|
||
## Governance
|
||
|
||
Members govern the project through [Loomio](https://www.loomio.com) and the Open Collective. See the [Charter](charter.md) for the cooperative's values, membership terms, and governance structure, and [Open Questions](open-questions.md) for the decisions currently open for member discussion.
|
||
|
||
During the pilot phase, Nathan Schneider serves as Managing Director and has sole final discretion on decisions.
|
||
|
||
---
|
||
|
||
*This documentation lives in a public git repository. Contributions welcome.*
|