The team spend counter is cycle-only and resets at budget_reset_at, which for the agent teams means the 1st of the month. The panel's agents table read that field directly, so on 1 November the co-op's own bot accounts appeared to have had their spend zeroed out. Members already had an all-time figure for exactly this reason; agent teams did not, because the all-time query filtered to member:% only. Widen it to agent:% and expose all_time_spend on agent rows.
Removes 'nothing else about your account changes' from both the body and the footer; it goes without saying. Button label becomes 'Buy credit packs', and the sentence above it says the same thing so the two do not disagree.
Lets the letter be read in a mail client before the sender is armed. Uses the member's real numbers when the address matches one, writes nothing to budget_notices so the once-per-crossing latch is not consumed, and works while BUDGET_NOTICES is off.
Trigger becomes proportional (90% of the balance used) instead of a $2 floor, so a member on $15 and one on $35 are treated the same. The once-per-crossing latch keys on the same fraction. membership_sweep moves from '0 3 * * *' to '0 3,11,19 * * *' so the notice arrives within hours of crossing rather than up to a day later. BUDGET_NOTICES remains off.
Wording no longer names an allowance or a credit pack: it says balance, so it is true whether the member is short on allowance, on purchased credit, or both. Trigger is an absolute $2 floor on remaining budget rather than a percentage, so it means the same at any budget size. Recipient greeting uses members.slug (the username) instead of the email local part, which produced 'Hi n,'. The once-only latch now clears when the balance rises back above the floor, so it fires once per crossing rather than once per budget cycle. BUDGET_NOTICES stays off.
Warns a member at 80% of their monthly cap and again when it is exhausted, using the welcome email's template shell. Spend and budget come from LiteLLM's Postgres, the same source the dashboard shows. One notice per level per budget cycle. BUDGET_NOTICES=off|dryrun|on gates sending; POST /admin/budget-notices previews at any time and refuses to send unless the flag is on. Rides the existing sweep cron, wrapped so it cannot break reconciliation.
Thread the per-factor source through impact_for() instead of discarding it.
Numeric output (energy_wh, co2e_g) is byte-identical — this is a reporting
change, not a recalculation.
#6 persistence: additive columns tokens_measured, tokens_estimated and
factor_revision (a stable hash of the factor table) on metrics_daily, added via
the existing guarded ALTER TABLE idiom in both app/main.py and
metrics_snapshot.py. Existing rows keep NULL.
#4 API: metrics_series() selects the new columns and derives a per-row
provenance indicator (measured | mixed | estimated | none) from the split; both
/admin/metrics/series and /broker/metrics return it. NULL/unknown splits report
'none', never a confident 'measured'.
docs/energy-metrics.md documents the indicator, the new columns, and the
forward-only-vs-backfill behaviour.
sandbox:librechat-test is a throwaway test account rather than an agent of the co-op,
so it drops out of the panel. Its spend then appears nowhere — accepted knowingly.
Agents also carry a display `name` (the alias minus its "agent:" prefix), so the panel
does not have to strip prefixes itself.
The panel has always excluded operator/utility emails from member counts
(OPERATOR_EMAILS), but the metrics counted every row in the members table, so the
panel said 75 while every metric surface said 76. The one affected row is
info@inference.coop — the reading mailbox, not a person.
- metrics_snapshot.py now applies the same OPERATOR_EMAILS definition (same variable
name, same default) so the two cannot drift apart again. No backfill: history stays
as published and the daily job writes the corrected value going forward.
- admin_overview now also returns an `agents` list, built from the LiteLLM teams it
already fetches (anything not member:*), so bot and sandbox spend is visible rather
than structurally invisible. Kept out of member counts and member aggregates, like
the operator category.
The calculation lived only in code comments. Adds docs/energy-metrics.md and wires it
into the README, per the issue's proposed scope. No calculation changes.
Contents: the input (completion_tokens, output tokens only, grouped on model_group)
and why output-only understates total compute; the factor table with its three
provenance tiers (measured / same-family / estimated) and the fact that the tiers are
never exposed; the formulas; NULL-not-zero and the reason (the first backfill claimed
every day used no energy); the implied-grid-intensity tripwire and the 1000x carbon
error it exists to catch; the 03:15 schedule and that --backfill REWRITES history.
States plainly what is measured and what is not: GreenPT is measured, glm-5.3 is
assumed from its flash sibling, and all Tinfoil and Regolo traffic currently falls to
the proxy estimate — with the consequence that the 250 gCO2e/kWh proxy OVERSTATES
Regolo, which runs on renewable energy.
README: "Two jobs" -> "Three", the metrics_snapshot task documented, and a short
Energy & carbon metrics subsection linking the new doc. Also corrects "listing both
tasks" in the post-change verification step.
The comment above ADMIN_TOKEN still said "set ADMIN_TOKEN_LEGACY=0 and rotate
both". Stale — and the direct reason a report claimed a rotation was
outstanding: ADMIN_TOKEN was minted new and WEBHOOK_TOKEN rotated on 2026-10-27,
and the legacy path was closed on 2026-10-28.
Comment only, no behaviour change. Rebranched onto the current main: an earlier
attempt had pinned an image built from a tree that predated the key-encryption
work, which would have reverted it.
The build service's context dropped new files added under app/
(keycrypto.py never reached the image while main.py did), causing a
crash loop. Inline the module so the deploy cannot miss it. Behavior
unchanged: AES-256-GCM at rest, KEK from PORTAL_KEK, enc1: format,
legacy plaintext passthrough, fail-closed migration.
Member feedback: keys stored in plaintext in the portal DB are a
key-compromise incident waiting to happen. Members.key_token (the chat
key) is now encrypted with AES-256-GCM using a KEK from the PORTAL_KEK
env var. Plaintext exists only in memory when a key is served (dashboard,
LibreChat key feed, LiteLLM management). Startup migration encrypts
existing rows idempotently; fail-closed when the KEK is absent.
Member-managed API keys were already reveal-once (hash-only at rest).
ADMIN_TOKEN_LEGACY is "0" and verified: ADMIN_TOKEN returns 200 on /admin/*,
WEBHOOK_TOKEN returns 403. The fallback in sweep.sh would therefore turn
"ADMIN_TOKEN unset" into a confusing mid-run 403 rather than the clear abort
that already exists three lines below it. Same removal in the digest script.
Not a behaviour change in normal operation — ADMIN_TOKEN is set (len 64) and
already preferred — but it removes a foot-gun that could mask a broken
configuration, and leaves the code honest about which credential is live.
The carbon column was 1000x too low: 2.9 gCO2e for 55 days and 14.35 kWh,
where 14.35 kWh at the factor table's own intensities must be ~2.9 kg. Energy
was right, so nothing looked broken unless you did the arithmetic — 938 Wh in a
day cannot be 0.2 g of CO2.
Adds a tripwire in impact_for(): if the implied grid intensity falls outside
5-500 gCO2e/kWh, it prints a warning rather than quietly returning numbers. The
factors themselves span 24-250, so a blend outside that band means a unit or
factor error, not a cleaner grid.
Two bugs from the first deploy, both caught by reading the numbers back:
1. Tokens were summed per raw `model`, which carries the UPSTREAM name
(openai/deepseek-v4.1-flash), while the factor table is keyed on our
provider aliases (greenpt/deepseek-v4.1-flash). Every model therefore
matched nothing. Now grouped on model_group, the same convention
aggregate_model_usage uses, with model as the fallback for old rows.
2. Every day was written as energy_wh = 0.0 / co2e_g = 0.0 — a false claim that
the co-op used no energy, and precisely the fail-open this module is written
to avoid. A day with no measured tokens now yields NULL for all three impact
columns, and impact_for() refuses to compute from an empty total.
The backfill is idempotent, so re-running it overwrote the zero rows.
Implements the approach Nathan settled on in co-op/support#26: per-token
estimates based on what GreenPT publishes, rather than waiting on the upstream
LiteLLM `impact` passthrough.
- New daily columns: out_tokens, energy_wh, co2e_g.
- Tokens come from OUR OWN spend logs (completion_tokens per model per day), so
member-visible impact does not depend on any client persisting an `impact`
object. GreenPT's metric counts OUTPUT tokens only — that is the unit — so
input tokens are excluded deliberately.
- Factors live in one table with a source per row. The greenpt/* rows are
MEASURED: fitted from live calls to their API across prompt sizes
(energy = overhead + marginal x output tokens, R2 >= 0.999; carbon from the
grid intensity their own emissions imply — ug/mWh and g/kWh are the same
ratio). Everything else falls back to a documented order-of-magnitude proxy,
so a day's total is never silently missing models.
- Migration: CREATE IF NOT EXISTS never adds columns, so both the script and
the app ALTER TABLE on connect.
Refining a factor is a data edit in IMPACT_FACTORS, not a code change.
My earlier patch inserted the new handler between the existing
/broker/model-leaderboard decorator and its function, leaving two routes
pointing at model-leaderboard and none at /broker/metrics, which 404'd. The
dashboard's co-op leaderboard was being served the wrong payload.
Caught by the post-deploy check (the 404 on a route whose code was present in
the container), not by the build. Adds a pre-build route-table assertion —
every path must be followed by its intended handler — so this shape of mistake
fails before a build rather than after a deploy.
Members asked to see how the co-op is doing, not just their own usage. The
series was admin-only; now the same query serves both surfaces.
- metrics_series(days) holds the query once; /admin/metrics/series and the new
/broker/metrics both call it (no second copy to drift).
- /broker/metrics is broker-authenticated and returns AGGREGATE, NON-PERSONAL
figures only: co-op size, requests served, spend. Same class as
/broker/model-leaderboard, and the same rule — no per-member figure passes
through the broker.
The backfill's first real curve was wrong: built on welcome_sent_at it showed
34 members on 19 October, when the co-op had ~65. Most members were provisioned
before the portal sent welcome emails, so that field lags by dozens of people —
and a chart built on it would have misrepresented the co-op's growth.
Membership now comes from Listmonk's per-subscription subscription_created_at
on the members list, which the portal's nightly reconcile keeps mirroring
active members. That reproduces the real curve (13 members mid-September → 71
today). Past days are the cumulative joins of today's members — an honest floor,
documented as such, since members who left are absent.
The first live run recorded newsletter_subs = 0 — a wrong zero, the exact
fail-open this module documents against. Cause: Listmonk's subscriber record
carries status enabled/blocked, while the LIST membership carries
subscription_status confirmed/unconfirmed. Filtering on the subscriber's own
status matched nothing and returned a silent 0.
Now counts memberships whose subscription_status is confirmed (verified: 71,
matching the list), and returns NULL when the endpoint yields no rows at all —
"not measured" rather than "nobody is subscribed". The daily job is the
regression test: it must record 71, not 0.
The co-op had no longitudinal data: every number was an ad-hoc query, so
nothing compounded and no single source owned "how many members do we have,
and when did that change". That cost us a wrong figure in the newsletter ("six
new members" — actually 21; the six were forum joins).
New: metrics_snapshot.py writes one row per day into the portal's SQLite store
(the same members.db), and GET /admin/metrics/series returns them oldest-first
for the panel and dashboard to chart later.
- Cumulative columns (members_active, members_total, members_new,
newsletter_subs, matrix_space_members) are counts AS OF that day;
spend_usd/requests are THAT day's traffic.
- FAIL CLOSED PER METRIC: an unreadable source yields NULL, never a zero — a
zero reads as "nobody" and would bake a lie into a chart. Failures print to
stdout, and a scheduler task's stdout lands in the app log.
- Listmonk's cached subscriber_count is NOT trusted (it reads 142 for a list
holding 71): the count comes from paging the subscribers endpoint.
- Member growth comes from welcome_sent_at, the provisioning timestamp —
the closest thing the portal stores to a join date. Manual/founder members
have no welcome email, so backfilled days are a documented LOWER BOUND;
today's row carries the exact count.
- Runs as an in-app scheduler task (addons.scheduler, 03:15 UTC) rather than
an agent cron: member-facing plumbing should not depend on the agent VM.
- Backfill is idempotent; member growth is reconstructible back to
September and spend/requests back to the gateway's first request.
Charts land later, on the surfaces, once the shape of the series is reviewed.
Two fixes, both filed rather than deferred.
1. member-portal#2 — the sweep deactivates every member whose email is missing
from the Open Collective member list, and the existing fail-safe only covers a
fetch that RAISES. A fetch that SUCCEEDS while returning nothing or far less
than usual is unguarded, and is what a narrowed token scope, an expired
credential with partial access, or a renamed Membership tier produces (the
fetch filters on the tier's name). That path deactivates the membership
instead of failing. fetch_members() now checks the result against the local
active count before any caller can act on it: empty, or at or below half of
local, raises — so the sweep's fail-safe takes over and members stay active.
Skipped when we have no active members of our own (fresh install).
Correction to the issue as filed: listmonk_reconcile reads the local members
table, not this fetch, so the newsletter path was never exposed to it. The
sweep is the only destructive consumer.
2. ops/security#2 — OC's generic webhooks cannot send headers, so the webhook
secret must travel in the URL path, and uvicorn wrote it into the access log
on every delivery. A logging filter on uvicorn.access and on the root handler
replaces the token with <redacted>. The token cannot be kept out of the URL;
it can be kept out of the retention.
Both pieces are unit-tested in isolation before shipping: the guard's decision
function over eight cases, and the filter against a real access-log record, an
unrelated path (no over-redaction), and a missing dependency (must not break
logging).
Member disclosure, three findings (fixes 1-3 of 5):
1. WEBHOOK_TOKEN was both the Open Collective webhook URL token and the
X-Admin-Token for every /admin/* route, so anything that learned the webhook
URL (OC admins, OC logs, this app's access logs) gained portal admin. The URL
token is forced by OC's generic webhooks, which cannot set headers — the reuse
is not. Introduce ADMIN_TOKEN, header-only. ADMIN_TOKEN_LEGACY=1 (default)
still accepts WEBHOOK_TOKEN and logs a warning, so the admin panel and the
scheduler keep working; set ADMIN_TOKEN everywhere, then flip it to 0 and
rotate. sweep.sh and the digest script now send ADMIN_TOKEN.
2. The credit-pack path fell back to the webhook payload's own amount when OC
could not resolve the order, so a forged POST naming its own totalAmount
minted credits for any slug. The payload is now a hint only: an unresolvable
order is refused, logged loudly, and emailed to the operator for manual
reconciliation. Transient OC failures hold instead of granting blind.
3. Lapsing did not revoke access: both deactivation sites (lapse webhook, nightly
sweep) discarded the key token deactivate_member() returns. Member keys carry
no max_budget of their own, so they inherit the team's — zero the team budget
on deactivation, which blocks every key under it. Key deletion is deliberately
NOT used: provisioning reuses the stored sk- token on reactivation, and
get_or_create_team does not update an existing team's budget, so the two
renewal paths now re-sync the budget explicitly. Failures are loud (log +
alert email); deactivation still completes.
info@inference.coop is on the Members list but has no members row, so the
reconcile classified the co-op's own inbox as a stale subscriber and would have
unsubscribed it.
The data model already distinguishes this: OPERATOR_EMAILS (default
info@inference.coop,bot@inference.coop) are infrastructure accounts — the SSO
owner and the bot — which the panel surfaces under "admins" and excludes from
member counts. They live outside the members table by design. The Loomio
reconciliation applies the same rule to the same accounts via LOOMIO_ALWAYS_KEEP
("must always remain in the group").
So operators become the third class here: neither members nor stale. Their
subscription is left exactly as it is (never unsubscribed, and the reconcile
never subscribes, so an operator who unsubscribes stays unsubscribed). The
response reports how many were left, and the log line carries counts only.
inference-bot
portal-dev