Members need to see how much each API key has spent. LiteLLM's per-key
'spend' counter is cycle-only (resets at budget_reset_at), so derive it
from the LiteLLM_SpendLogs table instead: sum spend grouped by api_key,
matching rows to member_api_keys.hash by name and labelling unmatched
rows 'chat' (the portal-held chat key is never in member_api_keys).
broker_get_usage now returns key_spend: [{name, is_chat, spend, tokens,
requests}], API keys by spend desc with the chat row last.
The team spend counter is cycle-only and resets at budget_reset_at, which for the agent teams means the 1st of the month. The panel's agents table read that field directly, so on 1 November the co-op's own bot accounts appeared to have had their spend zeroed out. Members already had an all-time figure for exactly this reason; agent teams did not, because the all-time query filtered to member:% only. Widen it to agent:% and expose all_time_spend on agent rows.
Removes 'nothing else about your account changes' from both the body and the footer; it goes without saying. Button label becomes 'Buy credit packs', and the sentence above it says the same thing so the two do not disagree.
Lets the letter be read in a mail client before the sender is armed. Uses the member's real numbers when the address matches one, writes nothing to budget_notices so the once-per-crossing latch is not consumed, and works while BUDGET_NOTICES is off.
Trigger becomes proportional (90% of the balance used) instead of a $2 floor, so a member on $15 and one on $35 are treated the same. The once-per-crossing latch keys on the same fraction. membership_sweep moves from '0 3 * * *' to '0 3,11,19 * * *' so the notice arrives within hours of crossing rather than up to a day later. BUDGET_NOTICES remains off.
Wording no longer names an allowance or a credit pack: it says balance, so it is true whether the member is short on allowance, on purchased credit, or both. Trigger is an absolute $2 floor on remaining budget rather than a percentage, so it means the same at any budget size. Recipient greeting uses members.slug (the username) instead of the email local part, which produced 'Hi n,'. The once-only latch now clears when the balance rises back above the floor, so it fires once per crossing rather than once per budget cycle. BUDGET_NOTICES stays off.
Warns a member at 80% of their monthly cap and again when it is exhausted, using the welcome email's template shell. Spend and budget come from LiteLLM's Postgres, the same source the dashboard shows. One notice per level per budget cycle. BUDGET_NOTICES=off|dryrun|on gates sending; POST /admin/budget-notices previews at any time and refuses to send unless the flag is on. Rides the existing sweep cron, wrapped so it cannot break reconciliation.
Thread the per-factor source through impact_for() instead of discarding it.
Numeric output (energy_wh, co2e_g) is byte-identical — this is a reporting
change, not a recalculation.
#6 persistence: additive columns tokens_measured, tokens_estimated and
factor_revision (a stable hash of the factor table) on metrics_daily, added via
the existing guarded ALTER TABLE idiom in both app/main.py and
metrics_snapshot.py. Existing rows keep NULL.
#4 API: metrics_series() selects the new columns and derives a per-row
provenance indicator (measured | mixed | estimated | none) from the split; both
/admin/metrics/series and /broker/metrics return it. NULL/unknown splits report
'none', never a confident 'measured'.
docs/energy-metrics.md documents the indicator, the new columns, and the
forward-only-vs-backfill behaviour.
sandbox:librechat-test is a throwaway test account rather than an agent of the co-op,
so it drops out of the panel. Its spend then appears nowhere — accepted knowingly.
Agents also carry a display `name` (the alias minus its "agent:" prefix), so the panel
does not have to strip prefixes itself.
The panel has always excluded operator/utility emails from member counts
(OPERATOR_EMAILS), but the metrics counted every row in the members table, so the
panel said 75 while every metric surface said 76. The one affected row is
info@inference.coop — the reading mailbox, not a person.
- metrics_snapshot.py now applies the same OPERATOR_EMAILS definition (same variable
name, same default) so the two cannot drift apart again. No backfill: history stays
as published and the daily job writes the corrected value going forward.
- admin_overview now also returns an `agents` list, built from the LiteLLM teams it
already fetches (anything not member:*), so bot and sandbox spend is visible rather
than structurally invisible. Kept out of member counts and member aggregates, like
the operator category.
The comment above ADMIN_TOKEN still said "set ADMIN_TOKEN_LEGACY=0 and rotate
both". Stale — and the direct reason a report claimed a rotation was
outstanding: ADMIN_TOKEN was minted new and WEBHOOK_TOKEN rotated on 2026-10-27,
and the legacy path was closed on 2026-10-28.
Comment only, no behaviour change. Rebranched onto the current main: an earlier
attempt had pinned an image built from a tree that predated the key-encryption
work, which would have reverted it.
The build service's context dropped new files added under app/
(keycrypto.py never reached the image while main.py did), causing a
crash loop. Inline the module so the deploy cannot miss it. Behavior
unchanged: AES-256-GCM at rest, KEK from PORTAL_KEK, enc1: format,
legacy plaintext passthrough, fail-closed migration.
Member feedback: keys stored in plaintext in the portal DB are a
key-compromise incident waiting to happen. Members.key_token (the chat
key) is now encrypted with AES-256-GCM using a KEK from the PORTAL_KEK
env var. Plaintext exists only in memory when a key is served (dashboard,
LibreChat key feed, LiteLLM management). Startup migration encrypts
existing rows idempotently; fail-closed when the KEK is absent.
Member-managed API keys were already reveal-once (hash-only at rest).
Implements the approach Nathan settled on in co-op/support#26: per-token
estimates based on what GreenPT publishes, rather than waiting on the upstream
LiteLLM `impact` passthrough.
- New daily columns: out_tokens, energy_wh, co2e_g.
- Tokens come from OUR OWN spend logs (completion_tokens per model per day), so
member-visible impact does not depend on any client persisting an `impact`
object. GreenPT's metric counts OUTPUT tokens only — that is the unit — so
input tokens are excluded deliberately.
- Factors live in one table with a source per row. The greenpt/* rows are
MEASURED: fitted from live calls to their API across prompt sizes
(energy = overhead + marginal x output tokens, R2 >= 0.999; carbon from the
grid intensity their own emissions imply — ug/mWh and g/kWh are the same
ratio). Everything else falls back to a documented order-of-magnitude proxy,
so a day's total is never silently missing models.
- Migration: CREATE IF NOT EXISTS never adds columns, so both the script and
the app ALTER TABLE on connect.
Refining a factor is a data edit in IMPACT_FACTORS, not a code change.
My earlier patch inserted the new handler between the existing
/broker/model-leaderboard decorator and its function, leaving two routes
pointing at model-leaderboard and none at /broker/metrics, which 404'd. The
dashboard's co-op leaderboard was being served the wrong payload.
Caught by the post-deploy check (the 404 on a route whose code was present in
the container), not by the build. Adds a pre-build route-table assertion —
every path must be followed by its intended handler — so this shape of mistake
fails before a build rather than after a deploy.
Members asked to see how the co-op is doing, not just their own usage. The
series was admin-only; now the same query serves both surfaces.
- metrics_series(days) holds the query once; /admin/metrics/series and the new
/broker/metrics both call it (no second copy to drift).
- /broker/metrics is broker-authenticated and returns AGGREGATE, NON-PERSONAL
figures only: co-op size, requests served, spend. Same class as
/broker/model-leaderboard, and the same rule — no per-member figure passes
through the broker.
The co-op had no longitudinal data: every number was an ad-hoc query, so
nothing compounded and no single source owned "how many members do we have,
and when did that change". That cost us a wrong figure in the newsletter ("six
new members" — actually 21; the six were forum joins).
New: metrics_snapshot.py writes one row per day into the portal's SQLite store
(the same members.db), and GET /admin/metrics/series returns them oldest-first
for the panel and dashboard to chart later.
- Cumulative columns (members_active, members_total, members_new,
newsletter_subs, matrix_space_members) are counts AS OF that day;
spend_usd/requests are THAT day's traffic.
- FAIL CLOSED PER METRIC: an unreadable source yields NULL, never a zero — a
zero reads as "nobody" and would bake a lie into a chart. Failures print to
stdout, and a scheduler task's stdout lands in the app log.
- Listmonk's cached subscriber_count is NOT trusted (it reads 142 for a list
holding 71): the count comes from paging the subscribers endpoint.
- Member growth comes from welcome_sent_at, the provisioning timestamp —
the closest thing the portal stores to a join date. Manual/founder members
have no welcome email, so backfilled days are a documented LOWER BOUND;
today's row carries the exact count.
- Runs as an in-app scheduler task (addons.scheduler, 03:15 UTC) rather than
an agent cron: member-facing plumbing should not depend on the agent VM.
- Backfill is idempotent; member growth is reconstructible back to
September and spend/requests back to the gateway's first request.
Charts land later, on the surfaces, once the shape of the series is reviewed.
Two fixes, both filed rather than deferred.
1. member-portal#2 — the sweep deactivates every member whose email is missing
from the Open Collective member list, and the existing fail-safe only covers a
fetch that RAISES. A fetch that SUCCEEDS while returning nothing or far less
than usual is unguarded, and is what a narrowed token scope, an expired
credential with partial access, or a renamed Membership tier produces (the
fetch filters on the tier's name). That path deactivates the membership
instead of failing. fetch_members() now checks the result against the local
active count before any caller can act on it: empty, or at or below half of
local, raises — so the sweep's fail-safe takes over and members stay active.
Skipped when we have no active members of our own (fresh install).
Correction to the issue as filed: listmonk_reconcile reads the local members
table, not this fetch, so the newsletter path was never exposed to it. The
sweep is the only destructive consumer.
2. ops/security#2 — OC's generic webhooks cannot send headers, so the webhook
secret must travel in the URL path, and uvicorn wrote it into the access log
on every delivery. A logging filter on uvicorn.access and on the root handler
replaces the token with <redacted>. The token cannot be kept out of the URL;
it can be kept out of the retention.
Both pieces are unit-tested in isolation before shipping: the guard's decision
function over eight cases, and the filter against a real access-log record, an
unrelated path (no over-redaction), and a missing dependency (must not break
logging).
Member disclosure, three findings (fixes 1-3 of 5):
1. WEBHOOK_TOKEN was both the Open Collective webhook URL token and the
X-Admin-Token for every /admin/* route, so anything that learned the webhook
URL (OC admins, OC logs, this app's access logs) gained portal admin. The URL
token is forced by OC's generic webhooks, which cannot set headers — the reuse
is not. Introduce ADMIN_TOKEN, header-only. ADMIN_TOKEN_LEGACY=1 (default)
still accepts WEBHOOK_TOKEN and logs a warning, so the admin panel and the
scheduler keep working; set ADMIN_TOKEN everywhere, then flip it to 0 and
rotate. sweep.sh and the digest script now send ADMIN_TOKEN.
2. The credit-pack path fell back to the webhook payload's own amount when OC
could not resolve the order, so a forged POST naming its own totalAmount
minted credits for any slug. The payload is now a hint only: an unresolvable
order is refused, logged loudly, and emailed to the operator for manual
reconciliation. Transient OC failures hold instead of granting blind.
3. Lapsing did not revoke access: both deactivation sites (lapse webhook, nightly
sweep) discarded the key token deactivate_member() returns. Member keys carry
no max_budget of their own, so they inherit the team's — zero the team budget
on deactivation, which blocks every key under it. Key deletion is deliberately
NOT used: provisioning reuses the stored sk- token on reactivation, and
get_or_create_team does not update an existing team's budget, so the two
renewal paths now re-sync the budget explicitly. Failures are loud (log +
alert email); deactivation still completes.
info@inference.coop is on the Members list but has no members row, so the
reconcile classified the co-op's own inbox as a stale subscriber and would have
unsubscribed it.
The data model already distinguishes this: OPERATOR_EMAILS (default
info@inference.coop,bot@inference.coop) are infrastructure accounts — the SSO
owner and the bot — which the panel surfaces under "admins" and excludes from
member counts. They live outside the members table by design. The Loomio
reconciliation applies the same rule to the same accounts via LOOMIO_ALWAYS_KEEP
("must always remain in the group").
So operators become the third class here: neither members nor stale. Their
subscription is left exactly as it is (never unsubscribed, and the reconcile
never subscribes, so an operator who unsubscribes stays unsubscribed). The
response reports how many were left, and the log line carries counts only.
The end-to-end probe against the live listmonk caught this: PUT
/api/subscribers/{id} with {"lists": []} returns 400 {"message":"Invalid
email."}, leaving the subscriber on the list. It needs {"email": ..., "lists":
[]}. The failed request was only a log warning, so the reconcile's main job —
taking lapsed members off the Members list — has never worked; only the DELETE
branch did anything, which is the branch this series just removed.
Now both classes unsubscribe successfully and keep their record.
Two findings from the retrospective review of 394cccfb..e24c272e (#34).
listmonk_reconcile() deleted subscribers with no members row outright. That
destroyed the subscriber's consent/unsubscribe history, and a stray test account
is indistinguishable from a real person who subscribed outside the portal. Both
classes now leave the list the same way — PUT {"lists": []} — and the record is
kept as a suppression marker; nothing is purged automatically. The summary log
line reports counts by reason instead of subscriber addresses. Subscriber
pagination also gains a short-page fallback so a missing `total` cannot truncate
the reconcile at page 1.
reconcile_memberships() called fetch_members() unguarded, so the documented
fail-safe — skip reconciliation rather than deactivate everyone missing from a
partial list — was unreachable: since fetch_members() now raises, the sweep
returned a 500 instead of {"status": "skipped"}. It now catches the failure and
returns reason "member_fetch_failed". The two webhook call sites are left
unguarded on purpose: a payment webhook should fail loudly so OC retries it, and
it self-heals.
Refs #34
Testing the previous commit against the live Loomio user list showed it would
also have renamed a member whose handle was chosen by hand (a working name
would have become the sanitized OIDC uid) — the uid is domain-shaped, so the
sanitizer produced a different, wrong target. The function is scoped to
Loomio's own auto-generated handles: the email local part with
non-alphanumerics stripped plus a random suffix. Gate on that shape.
Verified against the live user list (70 users) before building: exactly one
handle changes — the member whose username fix 422'd on every sweep — no
computed target violates Loomio's rule, and the hand-chosen handle is left
alone.
The username fix rejected a dot-containing OAuth uid and then substituted the
email local part, which carries the same dots — so Loomio answered 422
("Usernames must contain lower case letters, numbers, underscores, and internal
hyphens only") on every sweep, for the one member whose Cloudron username has a
dot. Normalize instead: disallowed characters become underscores, and leading
and trailing hyphens/underscores are stripped. Also log the Loomio user id
rather than the member's email.
Root cause of test accounts receiving the newsletter: LISTMONK_API_USER was
set EMPTY in the portal env - env-present beats the code default, so every
listmonk_sync authenticated with an empty user and 403'd silently. Fixed both
the env (set to portal-sync) and the code (empty env falls back to default).
New: listmonk_reconcile() - the Members list must mirror ACTIVE members.
Stale subscribers are removed: members marked inactive are taken off the list
(record kept as suppression marker); addresses that are not members at all
(test/deleted accounts) are deleted outright. Wired as step 4 of the nightly
membership sweep + POST /admin/newsletter/reconcile for on-demand runs.
LiteLLM requires unique key aliases, so re-creating a key with an existing
(email, name) pair 400s ('alias already exists') — breaking key rotation
from the dashboard. The stored LiteLLM hash lets us retire the old key
first; the ON CONFLICT upsert then tracks the new key.
Surfaced by the LibreChat sandbox work: the portal chat-key rotation for
an SSO member failed with exactly this 400.
InferenceBot
inference-bot