Files
member-portal/README.md
T
inference-admin dac8f2c9ea
ci/woodpecker/push/woodpecker Pipeline was canceled
README: the spend-consistency step
2026-11-13 20:41:58 +00:00

383 lines
24 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Member Portal — membership middleware for the Inference Cooperative
Reconciles three systems into one membership lifecycle:
| System | Role | Source of truth for |
|--------|------|---------------------|
| Open Collective | Billing | Who is a *paying* member |
| Cloudron | Identity/SSO | Who can *log in* |
| LiteLLM | Inference | Who can *use models*, and how much |
## Architecture
```
Open Collective (billing)
│ webhook
▼
┌─────────────────────────────┐
│ Member Portal (this app) │
│ - OC webhook handler │
│ - Cloudron user management │
│ - LiteLLM key management │
│ - key-injection proxy │
└─────────────────────────────┘
▲
│ X-User-Email header (LibreChat forwards identity)
│
LibreChat ──→ Member Portal ──→ LiteLLM ──→ backend
```
## The three responsibilities
1. **Webhook handler** (`POST /webhook/opencollective`) — listens for Open
Collective membership events and activates/deactivates members. The same
endpoint carries credit-pack purchases, so a failure here can cost a member
money: it answers **503** (retryable) with an ERROR line naming the exception
when anything unexpected escapes, and **401** when the URL token is wrong —
that status means the webhook URL and `WEBHOOK_TOKEN` disagree, so it is
never swallowed by the error net. Any OC read it makes has an explicit
timeout and one retry (`_oc_graphql_post`); until 2026-11-13
`fetch_all_member_nodes` used httpx's 5-second default and a slow page became
an unhandled 500 on a paid purchase. The durable backstop — reconciling PAID
credit packs against `credit_ledger` on a schedule — is not built yet; see
the app repo's issue *"Backstop: reconcile PAID credit packs against the
ledger"*.
2. **Key injector** (`/v1/*`) — reads the member's email from the
`X-User-Email` header, looks up their LiteLLM key, and forwards the
request to the gateway with that key.
3. **Admin** (`/health`, `/`) — health and status.
## Configuration (environment variables)
| Variable | Purpose |
|----------|---------|
| `CLOUDRON_API` | Cloudron API origin (auto-set by Cloudron) |
| `CLOUDRON_TOKEN` | Cloudron API token (Read+Write) |
| `MEMBERS_GROUP_ID` | Cloudron group ID for members (grants chat access) |
| `LITELLM_BASE` | LiteLLM gateway URL |
| `LITELLM_MASTER_KEY` | LiteLLM master key |
| `PORTAL_KEK` | **Required.** Base64-encoded 32-byte key-encryption key. Member chat keys (`members.key_token`) are stored AES-256-GCM-encrypted at rest (`enc1:` prefix) and decrypted in memory only when served (dashboard, LibreChat key feed, LiteLLM management). Generate with `python3 -c "import base64,os;print(base64.b64encode(os.urandom(32)).decode())"`. If unset, the portal runs but refuses to encrypt new keys, keeps legacy plaintext, and logs CRITICAL. Rotating it requires re-encrypting every row (decrypt with the old key, encrypt with the new). |
| `ADMIN_TOKEN` | The portal's admin token (`X-Admin-Token` header) for the `/admin/*` endpoints the panel calls. Never the webhook URL token — that one cannot unlock `/admin/*` by design. |
| `ALLOWANCE_POLICY` | `report` (default) \| `on`. `report` computes every allowance decision and changes nothing; `on` performs renewals and sends the one reminder. Deactivation is gated separately by `ALLOWANCE_DEACTIVATE`. |
| `ALLOWANCE_DEACTIVATE` | `off` (default) \| `on`. When `on`, a member who was reminded and is still unpaid after the grace window is moved to the inactive group. **Defaults off so renewals/reminders can run before anyone loses access.** |
| `PAYMENT_GRACE_DAYS` | Days after the single reminder before an unpaid member is deactivated (default `7`). |
| `GUEST_BUDGET` | Guest-account credit in USD (default `2.0`). |
| `GUEST_DAYS` | Guest-account time box in days from provisioning (default `60`). The sweep deactivates an expired guest even with credits left. |
| `GUEST_PER_INVITER_DAYS` | Rolling window for the one-guest-per-inviter limit (default `30`). |
| `GUESTS_GROUP_ID` | Cloudron group ID for guest accounts. The chat and dashboard apps' access restrictions list BOTH the members and guests groups, so a guest reaches exactly those two apps. |
| `GUESTS_ENDED_GROUP_ID` | Cloudron group a SELF-CANCELLED guest's user lands in: out of guests (chat access ends) but still admitted by the dashboard app, so the ended state and Become-a-member stay visible. Unset → the shared inactive group (dashboard access ends too). |
## Guest accounts
A member can give a friend access to the co-op with $2 of credits — no Open
Collective membership required. `POST /admin/provision-guest` (X-Admin-Token)
with `{"email", "name", "inviter": "<the guest access member's email>"}` runs the
guest pipeline: Cloudron user in the **guests** group, a LiteLLM team capped at
`GUEST_BUDGET` with **no monthly reset** (the cap is the guest's whole life), the
portal row marked `account_type='guest'` with the $2 held as non-expiring
credit, `guest_expires_at` set to now + `GUEST_DAYS`, the guest welcome email
(setup link, what they can use, Become-a-member, newsletter note), and a
**guests ledger** row (`guests` table: inviter → recipient, expiry, status) for
the abuse audit. Guards: one guest per inviter per `GUEST_PER_INVITER_DAYS`; one
guest per recipient ever; the inviter must be an active member; an existing
portal row under the recipient address is refused. Guests are subscribed to
the newsletter and invited to NOTHING else (no Loomio, no Matrix).
**Member self-serve invitations (provision immediately).** `POST /broker/guests`
(the member-dashboard broker identity — a guest identity fails it) lets a MEMBER
invite a guest from their dashboard: `{"email", "name"}`. The same guards run
BEFORE anything is created (`guest_request_guard`: everything above, plus the
recipient is never a `MANUAL_MEMBERS` address), and a valid invitation
**provisions immediately** — the approval gate was removed on 2026-11-08 as
operator policy; the remaining protections (active member, one guest per
inviter per rolling 30 days, one guest per recipient ever, the $2 cap and the
`GUEST_DAYS` box) were judged sufficient. `provision_guest_member` re-runs
every guard, so a moved-state refusal still refuses. The member sees their
invitation trail — each guest with its status and expiry — via
`GET /broker/guests`.
**Sweep safety.** `reconcile_memberships()` never deactivates a guest for
lacking an OC contribution. A guest lapses only by its own rules, handled by
`reconcile_guest_lifecycles()` inside the sweep: credits spent out (LiteLLM's
cap enforces the limit; the sweep retires the row) or the `GUEST_DAYS` time box.
Deactivation is the same semantics as a member's — guests group → inactive,
budget zeroed, row `active=0`, newsletter unsubscribed — and the account is
KEPT so a returning payer resumes cleanly.
**Self-service cancellation (recipient-initiated).** `POST /broker/cancel-guest`
(dashboard broker identity) ends a guest account at the recipient's own
request: deactivation with the same kept-account semantics, the newsletter
unsubscribed, and — the part a lapse does not do — any UNSPENT credit
returned to the co-op as a negative `credit_ledger` row whose reason names
the cancellation and the original guest reference (`guest-return:<email>` is
the dedupe reference, so a repeat cancel can never double-return). The
guests-ledger row records the full trail: inviter → recipient → 'cancelled
by recipient'. The Cloudron user leaves the guests group for
`GUESTS_ENDED_GROUP_ID` — chat and API access end, while the dashboard app
(which lists that group in its access restriction) still admits them, so
`GET /broker/guest-status` can show the ended state: access ended, data
kept, Become-a-member still one contribution away. A paying member calling
either route gets a 403 — cancelling a membership means ending the Open
Collective contribution, and this endpoint is deliberately guest-only.
**Guest → member conversion (automatic).** When a guest's OC identity appears as
a live BACKER/ADMIN with the account's own email — via the webhook or the
nightly sweep — `provision_member()` routes the call to
`convert_guest_to_member()`: the Cloudron user moves guests → members, the
LiteLLM cap lifts to the member allowance + credits (unspent guest credit is
carried — it already lives in `credit_balance`), the row becomes a normal
member with the standard `MEMBER_BUDGET`, the anchor day (the guest's start
date) is kept and the reset pinned to it, the congratulations email goes out,
and the normal member onboarding runs (newsletter, Loomio + Matrix syncs,
docs-RAG group, held-credit grants). When the contribution arrives under a
DIFFERENT OC email it cannot auto-match: the operator links it with
`POST /admin/guest-convert` `{"email": "<guest account>", "oc_email": "<the OC
email the contribution is under>"}` — the row records `members.oc_email`, and
the sweep and webhook treat that address as an additional matcher from then on.
## Budget model
Governance decision (set in Loomio): a single sliding-scale Membership tier
($5–$20/month flexible) with a FLAT allowance for every member —
`MEMBER_BUDGET` (default `15`), reset on each membership payment and extendable
with non-expiring purchased credit packs. The per-tier budgets the portal once
shipped (free/member/supporter) are retired — there are no per-tier portal
settings; the single env knob is `MEMBER_BUDGET`.
## LibreChat wiring
Point LibreChat's custom endpoint at this portal (not directly at LiteLLM),
and add the identity header:
```yaml
endpoints:
custom:
- name: "Inference Cooperative"
apiKey: "${LITELLM_KEY}"
baseURL: "https://portal.inference.coop/v1"
headers:
X-User-Email: "{{LIBRECHAT_USER_EMAIL}}"
models:
default: ["deepseek-v4-flash", "gpt-oss-120b"]
fetch: true
```
## Status
**Scaffold** — the three handlers are stubbed with the correct integration
points. Remaining work before production:
- [x] Verify Open Collective webhook event names + payload shape
→ `order.processed` (every payment), `new member` (first only), `firstPayment` flag
- [x] Verify Cloudron user-creation API payload (role/group assignment)
→ `POST /api/v1/users` with `{username, email, displayName, role, active}`
→ group assignment via `PUT /api/v1/users/:userId/groups` with `{groupIds: [...]}`
- [x] Add a persistent store (SQLite) for email→key mapping
→ `key/list` returns token strings (not objects), so we store email→token locally
- [x] Verify LiteLLM key/generate + key/list payload shapes against live gateway
→ `key/generate` returns `{key: "sk-..."}`; `key/list` returns `{keys: ["<token>", ...]}`
- [ ] Add HMAC signature verification for the OC webhook (if OC supports it)
- [ ] Wire the OIDC addon for admin access
## Scheduled tasks (scheduler addon)
Four jobs run inside the container via `CloudronManifest.json` → `addons.scheduler`:
- **membership_sweep** (`0 3,11,19 * * *` UTC — three times a day) — `sweep.sh`
calls the portal's own `/admin/sweep` (auth: the `X-Admin-Token` header with
the portal's `ADMIN_TOKEN` — never `WEBHOOK_TOKEN`, which is the webhook's
URL token and returns 403 on `/admin/*`) to reconcile members
against the live Open Collective list: provisions missed signups, deactivates
lapsed members, syncs Loomio + Matrix. Idempotent; a missed webhook
self-heals within a day. The same sweep also runs the payment-driven
allowance reconciliation (below) and the credit-pack reconciliation.
**Credit-pack reconciliation** (`CREDIT_RECONCILE`, default `report`) — credits
are granted on exactly one path, the OC `order.processed` webhook, and that
path can fail AFTER the money has moved: on 2026-11-13 a $50 pack's delivery
500'd on httpx's invisible 5-second default, and a $10 pack had no delivery in
the retained log at all. Neither left a ledger row and nothing noticed. This
pass reads OC's OWN order list (`fetch_collective_orders`, needs the `orders`
scope on `OC_PERSONAL_TOKEN`, granted 2026-11-13) and compares it with our
books over `CREDIT_RECONCILE_DAYS` (default 30 — days, not "whatever our log
still holds": the $10 gap was 11 hours old and the log had already lost the
whole morning). In `report` mode it logs findings and changes NOTHING; grants
happen only when `CREDIT_RECONCILE=on`. Every grant uses
`credit_pack_reference()` — the same per-order key the webhook writes — so a
late delivery and this pass cannot both credit one purchase. A buyer who is not
an active member is reported, never credited: those keep the hold-and-email
path. On demand: `POST /admin/credit-pack-reconcile` (`X-Admin-Token`), a dry
run unless `?apply=1` and the policy is `on`.
Matching caution: our own references are NOT uniform, so a bare
`oc:<slug>:<order.id>` test reports credited purchases as missing (it read 21
of them that way, all false). `credit_pack_accounted()` matches the exact
reference when one exists and otherwise requires a second signal — same member,
same tip-free amount, within days — and counts a `held_credits` row as
accounted for. Each candidate also carries its `nearby_book_rows` (the member's
own ledger/held rows within a week) because a manual grant can be real and carry
no reference at all: the 2026-09-28 launch-day packs were credited as `+20.00`
then corrected by `-10.75`, netting exactly 3 × $5.00 with no reference on any
row. A reviewer sees that, instead of a phantom victim.
**Webhook-failure report** — the portal records every delivery it does not
accept in `webhook_failures` (`record_webhook_failure()`, called from the
webhook wrapper, including the 401 that means the webhook URL and
`WEBHOOK_TOKEN` disagree) and the sweep emails `info@` **once per new batch**
via a marker in `portal_state`, so a wrong token does not mail every sweep while
the rows stay queryable. This replaces the log-scraping watcher that used to run
on the operator's machine: the portal handles every delivery, so the portal is
what must notice and report a failure. (It also removes the dependency on a log
buffer with only hours of memory — when the 2026-11-13 $10 purchase was
investigated, whole hours were missing from it.) Log-scraping jobs cannot be
the primary alert for a signal whose window they cannot see.
**Unmapped-backer check** — a contributor who pays but has no portal row never
gets an account, because `fetch_members()` drops every member node with no
email and an OC Organization never exposes one (the mmoya case, found weeks late
by the member asking). `reconcile_unmapped_backers()` reads the RAW member nodes
— including the ones the sweep must skip — and compares them with `members.db`,
reporting three classes because each needs a different action: an ORG account
(link it with `/admin/org-link`), a VISIBLE email (the sweep should have
provisioned it — check the token), and a HIDDEN email (fetch it out-of-band,
then `/admin/provision`). A node with no account object is attributed through
same-day current orders, and is never reported without a date to attribute it
against. On demand: `POST /admin/unmapped-check`.
**Spend-consistency check** — on 2026-11-05 the portal's all-time-spend aggregate
failed *soft*: its copy of the gateway's LiteLLM DB credential had gone stale, the
query returned nothing, and every surface showed $0 while looking clean — the
portal was the thing that was wrong. `check_spend_consistency()` compares two
genuinely independent reads: the portal's SpendLogs aggregate over its own DB
credential, and LiteLLM's own HTTP API (`/global/spend/keys`) over the master key.
An EMPTY/$0 portal aggregate while LiteLLM reports real spend is a FAILURE named
as the stale-credential case, and an API side with no keys is a failure rather
than agreement. Silent when they agree; a read that cannot be made is `skipped`,
never clean. On demand: `POST /admin/spend-check`.
**Notice policy (all of the above):** email `info@` only when there is a
problem, and only about what is NEW. What a human has been told is a growing SET
of keys in `portal_state` (`order:grant` / `order:hold`, `slug:class`) — a set
that shrinks (a purchase granted, a finding fixed) mails nobody again, a failed
send retries on the next sweep instead of going quiet, a `notify=False` run is a
look and never consumes the alert, and a clean pass clears the marker so a
recurrence is news again.
**Service-invite triggers:** the sweep is the backstop, not the primary path.
Two faster triggers exist, because a member who has just set up their account
should not wait hours for forum + Space access:
1. **First-login activation hook** — when a member first presents themselves
on the broker identity path (the dashboard's first login), the portal
checks — only if their Loomio invite is still outstanding
(`loomio_invited_at` is null) — that their Cloudron account is now
activated (`inviteAccepted`), and if so fires the same idempotent Loomio +
Matrix syncs as a background task. Established members cost one SELECT; the
response never waits on Loomio or Matrix; failures are logged and
non-fatal; unactivated accounts are never invited (the check fails closed).
2. **service_sync** (below) — the scheduled tick that covers members who never
open the dashboard.
- **service_sync** (every 5 minutes) — `service_sync.sh` calls the portal's own
`/admin/service-sync` (same `X-Admin-Token` auth as the sweep). WHY: a new
member is provisioned with an UNACTIVATED Cloudron account, and the syncs
deliberately skip an unactivated account (never invite a half-created one) —
so nobody on our side is watching at the moment the member actually activates.
This job fires the same idempotent Loomio + Matrix syncs the sweep runs, but
ONLY while some member is still awaiting their invites (one indexed SELECT
when nobody is, so it costs nothing in the steady state), and it deliberately
is NOT `/admin/sweep`: it never touches Open Collective, allowance logic or
member mail. An in-process guard drops an overlapping run.
Net effect: a member who has just set up their account waits minutes at most.
- **rag_refresh** (daily 3:30am UTC) — `rag_refresh.py` pulls this repo's
sibling docs from the Gitea raw API (public, no credentials), uploads changed
files to Open WebUI (`OWUI_API_KEY`), attaches them to the shared knowledge
collection (`DOCS_KNOWLEDGE_ID`), reindexes, and re-grants the members group.
State persists in `/app/data/rag_refresh_state.json`.
- **metrics_snapshot** (daily 3:15am UTC) — `metrics_snapshot.py` records one row per
day into the `metrics_daily` table: members, newsletter subscribers, Matrix
membership, spend, request counts, energy/CO₂, and the Open Collective
balance. Each run **completes yesterday's row** with its full-day
spend/requests/tokens/impact aggregates, then records today's row live (its
usage columns carry the hours so far and are completed by the next run's
completion pass). A source that cannot be read
yields NULL for that metric rather than a zero. `spend_usd` is the co-op-wide
total and deliberately spans every team, the co-op's own operator accounts
included; `operator_spend_usd` is the breakout of that portion, served through
`/broker/metrics` so the member dashboard can show the co-op's own share
alongside the total. `oc_balance_usd` is a daily snapshot of the collective's
OC balance (the OC API has no historical series, so balance history accrues
from these snapshots only — 2026-11-08 onward). `--backfill` rebuilds history
idempotently (balance history cannot be backfilled: past balances were never
recorded); `--backfill --since YYYY-MM-DD` repairs recent days' usage columns
only, leaving the membership curve as the nightly snapshots measured it.
Methodology: [Energy and carbon metrics](docs/energy-metrics.md).
Verify after any manifest change: the deploy log must show
`setupAddons: ... scheduler ...` listing all four tasks, and each script must
succeed when run manually inside the container.
### Allowance renewal (payment-driven, join-day anchored)
Each member has a monthly allowance (`MEMBER_BUDGET`, currently $15). The
allowance renews on the day of the month the member *joined*, never on the day
Open Collective happened to charge them — OC aligns every member's charge onto
the collective's common billing day, so anchoring to the payment date let an
early charge move the boundary and skip a cycle entirely. The join day is stored
per member (`members.anchor_day`) and the reset is pinned there.
Whether the allowance renews is decided by asking Open Collective directly, on
the sweep's own schedule (not inferred from webhook delivery): a member with a
live recurring contribution (order status `ACTIVE` or `PROCESSING` — a payment
in flight is not a missed payment) renews; a member with no live recurring
contribution gets **one** reminder email, and is deactivated only after
`PAYMENT_GRACE_DAYS`. One-time payments are not an allowance cycle — they top up
the non-expiring credit balance and are untouched here. There are no annual
contributors (verified 2026-11-04: 104 orders, 87 MONTHLY, 17 ONETIME, zero
annual), so only the monthly case is implemented.
Every decision (renewed / reminded / deactivated, plus the payment date or its
absence) is written to `allowance_log` and answerable from that record. Run
`POST /admin/allowance?mode=report` (X-Admin-Token) for a no-op report that
changes nothing; `mode=send` performs renewals and sends the reminder and
requires `ALLOWANCE_POLICY=on`. Deactivation additionally requires
`ALLOWANCE_DEACTIVATE=on`.
### Allowance shortfall (a paid cycle OC truncated)
Open Collective aligns every recurring charge onto the collective's common
billing day, so a member who joins mid-month is charged again well before a full
month has elapsed — they pay for a full month and receive less than one. The
co-op's remedy: credit the missing **allowance** (the flat `MEMBER_BUDGET`
pro-rated for the days lost, `FULL_PAID_MONTH_DAYS = 30`, unrounded) as
non-expiring credits — the same balance that extends the allowance and is spent
only after it. It is never a share of what they paid.
The set is derived FROM DATA, on every sweep (`POST /admin/sweep` calls
`reconcile_allowance_shortfalls`, as does `POST /admin/shortfall-credits`):
each MONTHLY order's real charge history is read from Open Collective, and any
consecutive pair on the SAME order less than a full month apart is a truncated
cycle. Two subscriptions a member happens to hold are judged separately; a gap
whose dates cannot be determined is skipped and logged. Each credit is keyed to
the specific OC charge that caused it (`credit_ledger.reference` = the charge
`publicId`), so a re-run credits nothing twice — dedupe is on the charge, not
the member — and the ledger `reason` names the cycle so a member's question is
answerable from it.
Member notices ride the same pass, but default OFF
(`ALLOWANCE_SHORTFALL_EMAILS=off`); set it to `on` to send the staged notice,
at most once per charge (`shortfall_notices`). `mode=report` on
`/admin/shortfall-credits` previews the affected set and changes nothing.
### Energy & carbon metrics
The daily snapshot also computes energy (Wh) and carbon (gCO₂e) from output tokens,
using per-model factors whose provenance varies: measured for GreenPT, assumed equal
for one sibling model, and an explicit order-of-magnitude estimate for providers that
disclose nothing. Only output tokens count, and a day with no measured tokens is
recorded as NULL rather than zero.
[Energy and carbon metrics](docs/energy-metrics.md) documents the factor table, the
formulas, the tripwire that catches unit errors, the schedule, and — most importantly —
which parts are measurements and which are estimates. What members are told is in
`co-op/docs/energy.md`.