inference-bot
b11552fa49
The calculation lived only in code comments. Adds docs/energy-metrics.md and wires it into the README, per the issue's proposed scope. No calculation changes. Contents: the input (completion_tokens, output tokens only, grouped on model_group) and why output-only understates total compute; the factor table with its three provenance tiers (measured / same-family / estimated) and the fact that the tiers are never exposed; the formulas; NULL-not-zero and the reason (the first backfill claimed every day used no energy); the implied-grid-intensity tripwire and the 1000x carbon error it exists to catch; the 03:15 schedule and that --backfill REWRITES history. States plainly what is measured and what is not: GreenPT is measured, glm-5.3 is assumed from its flash sibling, and all Tinfoil and Regolo traffic currently falls to the proxy estimate — with the consequence that the 250 gCO2e/kWh proxy OVERSTATES Regolo, which runs on renewable energy. README: "Two jobs" -> "Three", the metrics_snapshot task documented, and a short Energy & carbon metrics subsection linking the new doc. Also corrects "listing both tasks" in the post-change verification step.
130 lines
5.9 KiB
Markdown
130 lines
5.9 KiB
Markdown
# Member Portal — membership middleware for the Inference Cooperative
|
|
|
|
Reconciles three systems into one membership lifecycle:
|
|
|
|
| System | Role | Source of truth for |
|
|
|--------|------|---------------------|
|
|
| Open Collective | Billing | Who is a *paying* member |
|
|
| Cloudron | Identity/SSO | Who can *log in* |
|
|
| LiteLLM | Inference | Who can *use models*, and how much |
|
|
|
|
## Architecture
|
|
|
|
```
|
|
Open Collective (billing)
|
|
│ webhook
|
|
▼
|
|
┌─────────────────────────────┐
|
|
│ Member Portal (this app) │
|
|
│ - OC webhook handler │
|
|
│ - Cloudron user management │
|
|
│ - LiteLLM key management │
|
|
│ - key-injection proxy │
|
|
└─────────────────────────────┘
|
|
▲
|
|
│ X-User-Email header (LibreChat forwards identity)
|
|
│
|
|
LibreChat ──→ Member Portal ──→ LiteLLM ──→ backend
|
|
```
|
|
|
|
## The three responsibilities
|
|
|
|
1. **Webhook handler** (`POST /webhook/opencollective`) — listens for Open
|
|
Collective membership events and activates/deactivates members.
|
|
2. **Key injector** (`/v1/*`) — reads the member's email from the
|
|
`X-User-Email` header, looks up their LiteLLM key, and forwards the
|
|
request to the gateway with that key.
|
|
3. **Admin** (`/health`, `/`) — health and status.
|
|
|
|
## Configuration (environment variables)
|
|
|
|
| Variable | Purpose |
|
|
|----------|---------|
|
|
| `CLOUDRON_API` | Cloudron API origin (auto-set by Cloudron) |
|
|
| `CLOUDRON_TOKEN` | Cloudron API token (Read+Write) |
|
|
| `MEMBERS_GROUP_ID` | Cloudron group ID for members (grants chat access) |
|
|
| `LITELLM_BASE` | LiteLLM gateway URL |
|
|
| `LITELLM_MASTER_KEY` | LiteLLM master key |
|
|
| `OPENCOLLECTIVE_WEBHOOK_SECRET` | Optional webhook signature secret |
|
|
| `PORTAL_KEK` | **Required.** Base64-encoded 32-byte key-encryption key. Member chat keys (`members.key_token`) are stored AES-256-GCM-encrypted at rest (`enc1:` prefix) and decrypted in memory only when served (dashboard, LibreChat key feed, LiteLLM management). Generate with `python3 -c "import base64,os;print(base64.b64encode(os.urandom(32)).decode())"`. If unset, the portal runs but refuses to encrypt new keys, keeps legacy plaintext, and logs CRITICAL. Rotating it requires re-encrypting every row (decrypt with the old key, encrypt with the new). |
|
|
|
|
## Tier budgets
|
|
|
|
Governance decision (set in Loomio). Defaults:
|
|
|
|
- `free` — $2/month of tokens
|
|
- `member` — $15/month
|
|
- `supporter` — $30/month
|
|
|
|
## LibreChat wiring
|
|
|
|
Point LibreChat's custom endpoint at this portal (not directly at LiteLLM),
|
|
and add the identity header:
|
|
|
|
```yaml
|
|
endpoints:
|
|
custom:
|
|
- name: "Inference Cooperative"
|
|
apiKey: "${LITELLM_KEY}"
|
|
baseURL: "https://portal.inference.coop/v1"
|
|
headers:
|
|
X-User-Email: "{{LIBRECHAT_USER_EMAIL}}"
|
|
models:
|
|
default: ["deepseek-v4-flash", "gpt-oss-120b"]
|
|
fetch: true
|
|
```
|
|
|
|
## Status
|
|
|
|
**Scaffold** — the three handlers are stubbed with the correct integration
|
|
points. Remaining work before production:
|
|
|
|
- [x] Verify Open Collective webhook event names + payload shape
|
|
→ `order.processed` (every payment), `new member` (first only), `firstPayment` flag
|
|
- [x] Verify Cloudron user-creation API payload (role/group assignment)
|
|
→ `POST /api/v1/users` with `{username, email, displayName, role, active}`
|
|
→ group assignment via `PUT /api/v1/users/:userId/groups` with `{groupIds: [...]}`
|
|
- [x] Add a persistent store (SQLite) for email→key mapping
|
|
→ `key/list` returns token strings (not objects), so we store email→token locally
|
|
- [x] Verify LiteLLM key/generate + key/list payload shapes against live gateway
|
|
→ `key/generate` returns `{key: "sk-..."}`; `key/list` returns `{keys: ["<token>", ...]}`
|
|
- [ ] Add HMAC signature verification for the OC webhook (if OC supports it)
|
|
- [ ] Wire the OIDC addon for admin access
|
|
|
|
## Scheduled tasks (scheduler addon)
|
|
|
|
Three jobs run inside the container via `CloudronManifest.json` → `addons.scheduler`:
|
|
|
|
- **membership_sweep** (nightly 3am) — `sweep.sh` calls the portal's own
|
|
`/admin/sweep` (auth: `X-Admin-Token` = `WEBHOOK_TOKEN`) to reconcile members
|
|
against the live Open Collective list: provisions missed signups, deactivates
|
|
lapsed members, syncs Loomio + Matrix. Idempotent; a missed webhook
|
|
self-heals within 24 hours.
|
|
- **rag_refresh** (nightly 3:30am) — `rag_refresh.py` pulls this repo's
|
|
sibling docs from the Gitea raw API (public, no credentials), uploads changed
|
|
files to Open WebUI (`OWUI_API_KEY`), attaches them to the shared knowledge
|
|
collection (`DOCS_KNOWLEDGE_ID`), reindexes, and re-grants the members group.
|
|
State persists in `/app/data/rag_refresh_state.json`.
|
|
- **metrics_snapshot** (nightly 3:15am) — `metrics_snapshot.py` records one row per
|
|
day into the `metrics_daily` table: members, newsletter subscribers, Matrix
|
|
membership, spend, request counts, and energy/CO₂. A source that cannot be read
|
|
yields NULL for that metric rather than a zero. `--backfill` rebuilds history
|
|
idempotently. Methodology: [Energy and carbon metrics](docs/energy-metrics.md).
|
|
|
|
Verify after any manifest change: the deploy log must show
|
|
`setupAddons: ... scheduler ...` listing all three tasks, and each script must
|
|
succeed when run manually inside the container.
|
|
|
|
### Energy & carbon metrics
|
|
|
|
The daily snapshot also computes energy (Wh) and carbon (gCO₂e) from output tokens,
|
|
using per-model factors whose provenance varies: measured for GreenPT, assumed equal
|
|
for one sibling model, and an explicit order-of-magnitude estimate for providers that
|
|
disclose nothing. Only output tokens count, and a day with no measured tokens is
|
|
recorded as NULL rather than zero.
|
|
|
|
[Energy and carbon metrics](docs/energy-metrics.md) documents the factor table, the
|
|
formulas, the tripwire that catches unit errors, the schedule, and — most importantly —
|
|
which parts are measurements and which are estimates. What members are told is in
|
|
`co-op/docs/energy.md`.
|