ci/woodpecker/push/woodpecker Pipeline was canceled
Credits have one path in: the OC order.processed webhook. On 2026-11-13 two paid packs got nothing through it — a $50 whose delivery 500'd on httpx's 5s default, and a $10 with no delivery in the retained log at all — and nothing on a schedule noticed. This adds a pass that reads OC's own order list and compares it with our books over a day-based window (not "whatever the log still holds": the $10 gap was 11h old and the log had lost the whole morning). Report-first: CREDIT_RECONCILE defaults to 'report' (log findings, change nothing) and grants only when set to 'on'. Grants go through add_credits() with credit_pack_reference(), the same per-order key the webhook writes, so a late delivery and this pass are idempotent against each other in both directions. A buyer with no active membership is reported, never credited — those keep the hold-and-email path. credit_pack_accounted() encodes what our own rows actually look like: a v2 id, a hyphenated legacy id, an empty-suffix legacy reference, and held_credits rows. A bare oc:<slug>:<order.id> match read 21 credited purchases as missing during development, so the fallback demands a second signal (member + tip-free amount + date) before calling an order credited. POST /admin/credit-pack-reconcile (X-Admin-Token) runs it on demand, dry unless ?apply=1 and the policy is on. 33 new checks; all 18 suites green.
336 lines
21 KiB
Markdown
336 lines
21 KiB
Markdown
# Member Portal — membership middleware for the Inference Cooperative
|
||
|
||
Reconciles three systems into one membership lifecycle:
|
||
|
||
| System | Role | Source of truth for |
|
||
|--------|------|---------------------|
|
||
| Open Collective | Billing | Who is a *paying* member |
|
||
| Cloudron | Identity/SSO | Who can *log in* |
|
||
| LiteLLM | Inference | Who can *use models*, and how much |
|
||
|
||
## Architecture
|
||
|
||
```
|
||
Open Collective (billing)
|
||
│ webhook
|
||
▼
|
||
┌─────────────────────────────┐
|
||
│ Member Portal (this app) │
|
||
│ - OC webhook handler │
|
||
│ - Cloudron user management │
|
||
│ - LiteLLM key management │
|
||
│ - key-injection proxy │
|
||
└─────────────────────────────┘
|
||
▲
|
||
│ X-User-Email header (LibreChat forwards identity)
|
||
│
|
||
LibreChat ──→ Member Portal ──→ LiteLLM ──→ backend
|
||
```
|
||
|
||
## The three responsibilities
|
||
|
||
1. **Webhook handler** (`POST /webhook/opencollective`) — listens for Open
|
||
Collective membership events and activates/deactivates members. The same
|
||
endpoint carries credit-pack purchases, so a failure here can cost a member
|
||
money: it answers **503** (retryable) with an ERROR line naming the exception
|
||
when anything unexpected escapes, and **401** when the URL token is wrong —
|
||
that status means the webhook URL and `WEBHOOK_TOKEN` disagree, so it is
|
||
never swallowed by the error net. Any OC read it makes has an explicit
|
||
timeout and one retry (`_oc_graphql_post`); until 2026-11-13
|
||
`fetch_all_member_nodes` used httpx's 5-second default and a slow page became
|
||
an unhandled 500 on a paid purchase. The durable backstop — reconciling PAID
|
||
credit packs against `credit_ledger` on a schedule — is not built yet; see
|
||
the app repo's issue *"Backstop: reconcile PAID credit packs against the
|
||
ledger"*.
|
||
2. **Key injector** (`/v1/*`) — reads the member's email from the
|
||
`X-User-Email` header, looks up their LiteLLM key, and forwards the
|
||
request to the gateway with that key.
|
||
3. **Admin** (`/health`, `/`) — health and status.
|
||
|
||
## Configuration (environment variables)
|
||
|
||
| Variable | Purpose |
|
||
|----------|---------|
|
||
| `CLOUDRON_API` | Cloudron API origin (auto-set by Cloudron) |
|
||
| `CLOUDRON_TOKEN` | Cloudron API token (Read+Write) |
|
||
| `MEMBERS_GROUP_ID` | Cloudron group ID for members (grants chat access) |
|
||
| `LITELLM_BASE` | LiteLLM gateway URL |
|
||
| `LITELLM_MASTER_KEY` | LiteLLM master key |
|
||
| `PORTAL_KEK` | **Required.** Base64-encoded 32-byte key-encryption key. Member chat keys (`members.key_token`) are stored AES-256-GCM-encrypted at rest (`enc1:` prefix) and decrypted in memory only when served (dashboard, LibreChat key feed, LiteLLM management). Generate with `python3 -c "import base64,os;print(base64.b64encode(os.urandom(32)).decode())"`. If unset, the portal runs but refuses to encrypt new keys, keeps legacy plaintext, and logs CRITICAL. Rotating it requires re-encrypting every row (decrypt with the old key, encrypt with the new). |
|
||
| `ADMIN_TOKEN` | The portal's admin token (`X-Admin-Token` header) for the `/admin/*` endpoints the panel calls. Never the webhook URL token — that one cannot unlock `/admin/*` by design. |
|
||
| `ALLOWANCE_POLICY` | `report` (default) \| `on`. `report` computes every allowance decision and changes nothing; `on` performs renewals and sends the one reminder. Deactivation is gated separately by `ALLOWANCE_DEACTIVATE`. |
|
||
| `ALLOWANCE_DEACTIVATE` | `off` (default) \| `on`. When `on`, a member who was reminded and is still unpaid after the grace window is moved to the inactive group. **Defaults off so renewals/reminders can run before anyone loses access.** |
|
||
| `PAYMENT_GRACE_DAYS` | Days after the single reminder before an unpaid member is deactivated (default `7`). |
|
||
| `GUEST_BUDGET` | Guest-account credit in USD (default `2.0`). |
|
||
| `GUEST_DAYS` | Guest-account time box in days from provisioning (default `60`). The sweep deactivates an expired guest even with credits left. |
|
||
| `GUEST_PER_INVITER_DAYS` | Rolling window for the one-guest-per-inviter limit (default `30`). |
|
||
| `GUESTS_GROUP_ID` | Cloudron group ID for guest accounts. The chat and dashboard apps' access restrictions list BOTH the members and guests groups, so a guest reaches exactly those two apps. |
|
||
| `GUESTS_ENDED_GROUP_ID` | Cloudron group a SELF-CANCELLED guest's user lands in: out of guests (chat access ends) but still admitted by the dashboard app, so the ended state and Become-a-member stay visible. Unset → the shared inactive group (dashboard access ends too). |
|
||
|
||
## Guest accounts
|
||
|
||
A member can give a friend access to the co-op with $2 of credits — no Open
|
||
Collective membership required. `POST /admin/provision-guest` (X-Admin-Token)
|
||
with `{"email", "name", "inviter": "<the guest access member's email>"}` runs the
|
||
guest pipeline: Cloudron user in the **guests** group, a LiteLLM team capped at
|
||
`GUEST_BUDGET` with **no monthly reset** (the cap is the guest's whole life), the
|
||
portal row marked `account_type='guest'` with the $2 held as non-expiring
|
||
credit, `guest_expires_at` set to now + `GUEST_DAYS`, the guest welcome email
|
||
(setup link, what they can use, Become-a-member, newsletter note), and a
|
||
**guests ledger** row (`guests` table: inviter → recipient, expiry, status) for
|
||
the abuse audit. Guards: one guest per inviter per `GUEST_PER_INVITER_DAYS`; one
|
||
guest per recipient ever; the inviter must be an active member; an existing
|
||
portal row under the recipient address is refused. Guests are subscribed to
|
||
the newsletter and invited to NOTHING else (no Loomio, no Matrix).
|
||
|
||
**Member self-serve invitations (provision immediately).** `POST /broker/guests`
|
||
(the member-dashboard broker identity — a guest identity fails it) lets a MEMBER
|
||
invite a guest from their dashboard: `{"email", "name"}`. The same guards run
|
||
BEFORE anything is created (`guest_request_guard`: everything above, plus the
|
||
recipient is never a `MANUAL_MEMBERS` address), and a valid invitation
|
||
**provisions immediately** — the approval gate was removed on 2026-11-08 as
|
||
operator policy; the remaining protections (active member, one guest per
|
||
inviter per rolling 30 days, one guest per recipient ever, the $2 cap and the
|
||
`GUEST_DAYS` box) were judged sufficient. `provision_guest_member` re-runs
|
||
every guard, so a moved-state refusal still refuses. The member sees their
|
||
invitation trail — each guest with its status and expiry — via
|
||
`GET /broker/guests`.
|
||
|
||
**Sweep safety.** `reconcile_memberships()` never deactivates a guest for
|
||
lacking an OC contribution. A guest lapses only by its own rules, handled by
|
||
`reconcile_guest_lifecycles()` inside the sweep: credits spent out (LiteLLM's
|
||
cap enforces the limit; the sweep retires the row) or the `GUEST_DAYS` time box.
|
||
Deactivation is the same semantics as a member's — guests group → inactive,
|
||
budget zeroed, row `active=0`, newsletter unsubscribed — and the account is
|
||
KEPT so a returning payer resumes cleanly.
|
||
|
||
**Self-service cancellation (recipient-initiated).** `POST /broker/cancel-guest`
|
||
(dashboard broker identity) ends a guest account at the recipient's own
|
||
request: deactivation with the same kept-account semantics, the newsletter
|
||
unsubscribed, and — the part a lapse does not do — any UNSPENT credit
|
||
returned to the co-op as a negative `credit_ledger` row whose reason names
|
||
the cancellation and the original guest reference (`guest-return:<email>` is
|
||
the dedupe reference, so a repeat cancel can never double-return). The
|
||
guests-ledger row records the full trail: inviter → recipient → 'cancelled
|
||
by recipient'. The Cloudron user leaves the guests group for
|
||
`GUESTS_ENDED_GROUP_ID` — chat and API access end, while the dashboard app
|
||
(which lists that group in its access restriction) still admits them, so
|
||
`GET /broker/guest-status` can show the ended state: access ended, data
|
||
kept, Become-a-member still one contribution away. A paying member calling
|
||
either route gets a 403 — cancelling a membership means ending the Open
|
||
Collective contribution, and this endpoint is deliberately guest-only.
|
||
|
||
**Guest → member conversion (automatic).** When a guest's OC identity appears as
|
||
a live BACKER/ADMIN with the account's own email — via the webhook or the
|
||
nightly sweep — `provision_member()` routes the call to
|
||
`convert_guest_to_member()`: the Cloudron user moves guests → members, the
|
||
LiteLLM cap lifts to the member allowance + credits (unspent guest credit is
|
||
carried — it already lives in `credit_balance`), the row becomes a normal
|
||
member with the standard `MEMBER_BUDGET`, the anchor day (the guest's start
|
||
date) is kept and the reset pinned to it, the congratulations email goes out,
|
||
and the normal member onboarding runs (newsletter, Loomio + Matrix syncs,
|
||
docs-RAG group, held-credit grants). When the contribution arrives under a
|
||
DIFFERENT OC email it cannot auto-match: the operator links it with
|
||
`POST /admin/guest-convert` `{"email": "<guest account>", "oc_email": "<the OC
|
||
email the contribution is under>"}` — the row records `members.oc_email`, and
|
||
the sweep and webhook treat that address as an additional matcher from then on.
|
||
|
||
|
||
## Budget model
|
||
|
||
Governance decision (set in Loomio): a single sliding-scale Membership tier
|
||
($5–$20/month flexible) with a FLAT allowance for every member —
|
||
`MEMBER_BUDGET` (default `15`), reset on each membership payment and extendable
|
||
with non-expiring purchased credit packs. The per-tier budgets the portal once
|
||
shipped (free/member/supporter) are retired — there are no per-tier portal
|
||
settings; the single env knob is `MEMBER_BUDGET`.
|
||
|
||
## LibreChat wiring
|
||
|
||
Point LibreChat's custom endpoint at this portal (not directly at LiteLLM),
|
||
and add the identity header:
|
||
|
||
```yaml
|
||
endpoints:
|
||
custom:
|
||
- name: "Inference Cooperative"
|
||
apiKey: "${LITELLM_KEY}"
|
||
baseURL: "https://portal.inference.coop/v1"
|
||
headers:
|
||
X-User-Email: "{{LIBRECHAT_USER_EMAIL}}"
|
||
models:
|
||
default: ["deepseek-v4-flash", "gpt-oss-120b"]
|
||
fetch: true
|
||
```
|
||
|
||
## Status
|
||
|
||
**Scaffold** — the three handlers are stubbed with the correct integration
|
||
points. Remaining work before production:
|
||
|
||
- [x] Verify Open Collective webhook event names + payload shape
|
||
→ `order.processed` (every payment), `new member` (first only), `firstPayment` flag
|
||
- [x] Verify Cloudron user-creation API payload (role/group assignment)
|
||
→ `POST /api/v1/users` with `{username, email, displayName, role, active}`
|
||
→ group assignment via `PUT /api/v1/users/:userId/groups` with `{groupIds: [...]}`
|
||
- [x] Add a persistent store (SQLite) for email→key mapping
|
||
→ `key/list` returns token strings (not objects), so we store email→token locally
|
||
- [x] Verify LiteLLM key/generate + key/list payload shapes against live gateway
|
||
→ `key/generate` returns `{key: "sk-..."}`; `key/list` returns `{keys: ["<token>", ...]}`
|
||
- [ ] Add HMAC signature verification for the OC webhook (if OC supports it)
|
||
- [ ] Wire the OIDC addon for admin access
|
||
|
||
## Scheduled tasks (scheduler addon)
|
||
|
||
Four jobs run inside the container via `CloudronManifest.json` → `addons.scheduler`:
|
||
|
||
- **membership_sweep** (`0 3,11,19 * * *` UTC — three times a day) — `sweep.sh`
|
||
calls the portal's own `/admin/sweep` (auth: the `X-Admin-Token` header with
|
||
the portal's `ADMIN_TOKEN` — never `WEBHOOK_TOKEN`, which is the webhook's
|
||
URL token and returns 403 on `/admin/*`) to reconcile members
|
||
against the live Open Collective list: provisions missed signups, deactivates
|
||
lapsed members, syncs Loomio + Matrix. Idempotent; a missed webhook
|
||
self-heals within a day. The same sweep also runs the payment-driven
|
||
allowance reconciliation (below) and the credit-pack reconciliation.
|
||
|
||
**Credit-pack reconciliation** (`CREDIT_RECONCILE`, default `report`) — credits
|
||
are granted on exactly one path, the OC `order.processed` webhook, and that
|
||
path can fail AFTER the money has moved: on 2026-11-13 a $50 pack's delivery
|
||
500'd on httpx's invisible 5-second default, and a $10 pack had no delivery in
|
||
the retained log at all. Neither left a ledger row and nothing noticed. This
|
||
pass reads OC's OWN order list (`fetch_collective_orders`, needs the `orders`
|
||
scope on `OC_PERSONAL_TOKEN`, granted 2026-11-13) and compares it with our
|
||
books over `CREDIT_RECONCILE_DAYS` (default 30 — days, not "whatever our log
|
||
still holds": the $10 gap was 11 hours old and the log had already lost the
|
||
whole morning). In `report` mode it logs findings and changes NOTHING; grants
|
||
happen only when `CREDIT_RECONCILE=on`. Every grant uses
|
||
`credit_pack_reference()` — the same per-order key the webhook writes — so a
|
||
late delivery and this pass cannot both credit one purchase. A buyer who is not
|
||
an active member is reported, never credited: those keep the hold-and-email
|
||
path. On demand: `POST /admin/credit-pack-reconcile` (`X-Admin-Token`), a dry
|
||
run unless `?apply=1` and the policy is `on`.
|
||
|
||
Matching caution: our own references are NOT uniform, so a bare
|
||
`oc:<slug>:<order.id>` test reports credited purchases as missing (it read 21
|
||
of them that way, all false). `credit_pack_accounted()` matches the exact
|
||
reference when one exists and otherwise requires a second signal — same member,
|
||
same tip-free amount, within days — and counts a `held_credits` row as
|
||
accounted for.
|
||
|
||
**Service-invite triggers:** the sweep is the backstop, not the primary path.
|
||
Two faster triggers exist, because a member who has just set up their account
|
||
should not wait hours for forum + Space access:
|
||
1. **First-login activation hook** — when a member first presents themselves
|
||
on the broker identity path (the dashboard's first login), the portal
|
||
checks — only if their Loomio invite is still outstanding
|
||
(`loomio_invited_at` is null) — that their Cloudron account is now
|
||
activated (`inviteAccepted`), and if so fires the same idempotent Loomio +
|
||
Matrix syncs as a background task. Established members cost one SELECT; the
|
||
response never waits on Loomio or Matrix; failures are logged and
|
||
non-fatal; unactivated accounts are never invited (the check fails closed).
|
||
2. **service_sync** (below) — the scheduled tick that covers members who never
|
||
open the dashboard.
|
||
- **service_sync** (every 5 minutes) — `service_sync.sh` calls the portal's own
|
||
`/admin/service-sync` (same `X-Admin-Token` auth as the sweep). WHY: a new
|
||
member is provisioned with an UNACTIVATED Cloudron account, and the syncs
|
||
deliberately skip an unactivated account (never invite a half-created one) —
|
||
so nobody on our side is watching at the moment the member actually activates.
|
||
This job fires the same idempotent Loomio + Matrix syncs the sweep runs, but
|
||
ONLY while some member is still awaiting their invites (one indexed SELECT
|
||
when nobody is, so it costs nothing in the steady state), and it deliberately
|
||
is NOT `/admin/sweep`: it never touches Open Collective, allowance logic or
|
||
member mail. An in-process guard drops an overlapping run.
|
||
Net effect: a member who has just set up their account waits minutes at most.
|
||
- **rag_refresh** (daily 3:30am UTC) — `rag_refresh.py` pulls this repo's
|
||
sibling docs from the Gitea raw API (public, no credentials), uploads changed
|
||
files to Open WebUI (`OWUI_API_KEY`), attaches them to the shared knowledge
|
||
collection (`DOCS_KNOWLEDGE_ID`), reindexes, and re-grants the members group.
|
||
State persists in `/app/data/rag_refresh_state.json`.
|
||
- **metrics_snapshot** (daily 3:15am UTC) — `metrics_snapshot.py` records one row per
|
||
day into the `metrics_daily` table: members, newsletter subscribers, Matrix
|
||
membership, spend, request counts, energy/CO₂, and the Open Collective
|
||
balance. Each run **completes yesterday's row** with its full-day
|
||
spend/requests/tokens/impact aggregates, then records today's row live (its
|
||
usage columns carry the hours so far and are completed by the next run's
|
||
completion pass). A source that cannot be read
|
||
yields NULL for that metric rather than a zero. `spend_usd` is the co-op-wide
|
||
total and deliberately spans every team, the co-op's own operator accounts
|
||
included; `operator_spend_usd` is the breakout of that portion, served through
|
||
`/broker/metrics` so the member dashboard can show the co-op's own share
|
||
alongside the total. `oc_balance_usd` is a daily snapshot of the collective's
|
||
OC balance (the OC API has no historical series, so balance history accrues
|
||
from these snapshots only — 2026-11-08 onward). `--backfill` rebuilds history
|
||
idempotently (balance history cannot be backfilled: past balances were never
|
||
recorded); `--backfill --since YYYY-MM-DD` repairs recent days' usage columns
|
||
only, leaving the membership curve as the nightly snapshots measured it.
|
||
Methodology: [Energy and carbon metrics](docs/energy-metrics.md).
|
||
|
||
Verify after any manifest change: the deploy log must show
|
||
`setupAddons: ... scheduler ...` listing all four tasks, and each script must
|
||
succeed when run manually inside the container.
|
||
|
||
### Allowance renewal (payment-driven, join-day anchored)
|
||
|
||
Each member has a monthly allowance (`MEMBER_BUDGET`, currently $15). The
|
||
allowance renews on the day of the month the member *joined*, never on the day
|
||
Open Collective happened to charge them — OC aligns every member's charge onto
|
||
the collective's common billing day, so anchoring to the payment date let an
|
||
early charge move the boundary and skip a cycle entirely. The join day is stored
|
||
per member (`members.anchor_day`) and the reset is pinned there.
|
||
|
||
Whether the allowance renews is decided by asking Open Collective directly, on
|
||
the sweep's own schedule (not inferred from webhook delivery): a member with a
|
||
live recurring contribution (order status `ACTIVE` or `PROCESSING` — a payment
|
||
in flight is not a missed payment) renews; a member with no live recurring
|
||
contribution gets **one** reminder email, and is deactivated only after
|
||
`PAYMENT_GRACE_DAYS`. One-time payments are not an allowance cycle — they top up
|
||
the non-expiring credit balance and are untouched here. There are no annual
|
||
contributors (verified 2026-11-04: 104 orders, 87 MONTHLY, 17 ONETIME, zero
|
||
annual), so only the monthly case is implemented.
|
||
|
||
Every decision (renewed / reminded / deactivated, plus the payment date or its
|
||
absence) is written to `allowance_log` and answerable from that record. Run
|
||
`POST /admin/allowance?mode=report` (X-Admin-Token) for a no-op report that
|
||
changes nothing; `mode=send` performs renewals and sends the reminder and
|
||
requires `ALLOWANCE_POLICY=on`. Deactivation additionally requires
|
||
`ALLOWANCE_DEACTIVATE=on`.
|
||
|
||
### Allowance shortfall (a paid cycle OC truncated)
|
||
|
||
Open Collective aligns every recurring charge onto the collective's common
|
||
billing day, so a member who joins mid-month is charged again well before a full
|
||
month has elapsed — they pay for a full month and receive less than one. The
|
||
co-op's remedy: credit the missing **allowance** (the flat `MEMBER_BUDGET`
|
||
pro-rated for the days lost, `FULL_PAID_MONTH_DAYS = 30`, unrounded) as
|
||
non-expiring credits — the same balance that extends the allowance and is spent
|
||
only after it. It is never a share of what they paid.
|
||
|
||
The set is derived FROM DATA, on every sweep (`POST /admin/sweep` calls
|
||
`reconcile_allowance_shortfalls`, as does `POST /admin/shortfall-credits`):
|
||
each MONTHLY order's real charge history is read from Open Collective, and any
|
||
consecutive pair on the SAME order less than a full month apart is a truncated
|
||
cycle. Two subscriptions a member happens to hold are judged separately; a gap
|
||
whose dates cannot be determined is skipped and logged. Each credit is keyed to
|
||
the specific OC charge that caused it (`credit_ledger.reference` = the charge
|
||
`publicId`), so a re-run credits nothing twice — dedupe is on the charge, not
|
||
the member — and the ledger `reason` names the cycle so a member's question is
|
||
answerable from it.
|
||
|
||
Member notices ride the same pass, but default OFF
|
||
(`ALLOWANCE_SHORTFALL_EMAILS=off`); set it to `on` to send the staged notice,
|
||
at most once per charge (`shortfall_notices`). `mode=report` on
|
||
`/admin/shortfall-credits` previews the affected set and changes nothing.
|
||
|
||
### Energy & carbon metrics
|
||
|
||
The daily snapshot also computes energy (Wh) and carbon (gCO₂e) from output tokens,
|
||
using per-model factors whose provenance varies: measured for GreenPT, assumed equal
|
||
for one sibling model, and an explicit order-of-magnitude estimate for providers that
|
||
disclose nothing. Only output tokens count, and a day with no measured tokens is
|
||
recorded as NULL rather than zero.
|
||
|
||
[Energy and carbon metrics](docs/energy-metrics.md) documents the factor table, the
|
||
formulas, the tripwire that catches unit errors, the schedule, and — most importantly —
|
||
which parts are measurements and which are estimates. What members are told is in
|
||
`co-op/docs/energy.md`.
|