The panel has always excluded operator/utility emails from member counts
(OPERATOR_EMAILS), but the metrics counted every row in the members table, so the
panel said 75 while every metric surface said 76. The one affected row is
info@inference.coop — the reading mailbox, not a person.
- metrics_snapshot.py now applies the same OPERATOR_EMAILS definition (same variable
name, same default) so the two cannot drift apart again. No backfill: history stays
as published and the daily job writes the corrected value going forward.
- admin_overview now also returns an `agents` list, built from the LiteLLM teams it
already fetches (anything not member:*), so bot and sandbox spend is visible rather
than structurally invisible. Kept out of member counts and member aggregates, like
the operator category.
The calculation lived only in code comments. Adds docs/energy-metrics.md and wires it
into the README, per the issue's proposed scope. No calculation changes.
Contents: the input (completion_tokens, output tokens only, grouped on model_group)
and why output-only understates total compute; the factor table with its three
provenance tiers (measured / same-family / estimated) and the fact that the tiers are
never exposed; the formulas; NULL-not-zero and the reason (the first backfill claimed
every day used no energy); the implied-grid-intensity tripwire and the 1000x carbon
error it exists to catch; the 03:15 schedule and that --backfill REWRITES history.
States plainly what is measured and what is not: GreenPT is measured, glm-5.3 is
assumed from its flash sibling, and all Tinfoil and Regolo traffic currently falls to
the proxy estimate — with the consequence that the 250 gCO2e/kWh proxy OVERSTATES
Regolo, which runs on renewable energy.
README: "Two jobs" -> "Three", the metrics_snapshot task documented, and a short
Energy & carbon metrics subsection linking the new doc. Also corrects "listing both
tasks" in the post-change verification step.
The comment above ADMIN_TOKEN still said "set ADMIN_TOKEN_LEGACY=0 and rotate
both". Stale — and the direct reason a report claimed a rotation was
outstanding: ADMIN_TOKEN was minted new and WEBHOOK_TOKEN rotated on 2026-10-27,
and the legacy path was closed on 2026-10-28.
Comment only, no behaviour change. Rebranched onto the current main: an earlier
attempt had pinned an image built from a tree that predated the key-encryption
work, which would have reverted it.
The build service's context dropped new files added under app/
(keycrypto.py never reached the image while main.py did), causing a
crash loop. Inline the module so the deploy cannot miss it. Behavior
unchanged: AES-256-GCM at rest, KEK from PORTAL_KEK, enc1: format,
legacy plaintext passthrough, fail-closed migration.
Member feedback: keys stored in plaintext in the portal DB are a
key-compromise incident waiting to happen. Members.key_token (the chat
key) is now encrypted with AES-256-GCM using a KEK from the PORTAL_KEK
env var. Plaintext exists only in memory when a key is served (dashboard,
LibreChat key feed, LiteLLM management). Startup migration encrypts
existing rows idempotently; fail-closed when the KEK is absent.
Member-managed API keys were already reveal-once (hash-only at rest).
ADMIN_TOKEN_LEGACY is "0" and verified: ADMIN_TOKEN returns 200 on /admin/*,
WEBHOOK_TOKEN returns 403. The fallback in sweep.sh would therefore turn
"ADMIN_TOKEN unset" into a confusing mid-run 403 rather than the clear abort
that already exists three lines below it. Same removal in the digest script.
Not a behaviour change in normal operation — ADMIN_TOKEN is set (len 64) and
already preferred — but it removes a foot-gun that could mask a broken
configuration, and leaves the code honest about which credential is live.
The carbon column was 1000x too low: 2.9 gCO2e for 55 days and 14.35 kWh,
where 14.35 kWh at the factor table's own intensities must be ~2.9 kg. Energy
was right, so nothing looked broken unless you did the arithmetic — 938 Wh in a
day cannot be 0.2 g of CO2.
Adds a tripwire in impact_for(): if the implied grid intensity falls outside
5-500 gCO2e/kWh, it prints a warning rather than quietly returning numbers. The
factors themselves span 24-250, so a blend outside that band means a unit or
factor error, not a cleaner grid.
Two bugs from the first deploy, both caught by reading the numbers back:
1. Tokens were summed per raw `model`, which carries the UPSTREAM name
(openai/deepseek-v4.1-flash), while the factor table is keyed on our
provider aliases (greenpt/deepseek-v4.1-flash). Every model therefore
matched nothing. Now grouped on model_group, the same convention
aggregate_model_usage uses, with model as the fallback for old rows.
2. Every day was written as energy_wh = 0.0 / co2e_g = 0.0 — a false claim that
the co-op used no energy, and precisely the fail-open this module is written
to avoid. A day with no measured tokens now yields NULL for all three impact
columns, and impact_for() refuses to compute from an empty total.
The backfill is idempotent, so re-running it overwrote the zero rows.
Implements the approach Nathan settled on in co-op/support#26: per-token
estimates based on what GreenPT publishes, rather than waiting on the upstream
LiteLLM `impact` passthrough.
- New daily columns: out_tokens, energy_wh, co2e_g.
- Tokens come from OUR OWN spend logs (completion_tokens per model per day), so
member-visible impact does not depend on any client persisting an `impact`
object. GreenPT's metric counts OUTPUT tokens only — that is the unit — so
input tokens are excluded deliberately.
- Factors live in one table with a source per row. The greenpt/* rows are
MEASURED: fitted from live calls to their API across prompt sizes
(energy = overhead + marginal x output tokens, R2 >= 0.999; carbon from the
grid intensity their own emissions imply — ug/mWh and g/kWh are the same
ratio). Everything else falls back to a documented order-of-magnitude proxy,
so a day's total is never silently missing models.
- Migration: CREATE IF NOT EXISTS never adds columns, so both the script and
the app ALTER TABLE on connect.
Refining a factor is a data edit in IMPACT_FACTORS, not a code change.
My earlier patch inserted the new handler between the existing
/broker/model-leaderboard decorator and its function, leaving two routes
pointing at model-leaderboard and none at /broker/metrics, which 404'd. The
dashboard's co-op leaderboard was being served the wrong payload.
Caught by the post-deploy check (the 404 on a route whose code was present in
the container), not by the build. Adds a pre-build route-table assertion —
every path must be followed by its intended handler — so this shape of mistake
fails before a build rather than after a deploy.
Members asked to see how the co-op is doing, not just their own usage. The
series was admin-only; now the same query serves both surfaces.
- metrics_series(days) holds the query once; /admin/metrics/series and the new
/broker/metrics both call it (no second copy to drift).
- /broker/metrics is broker-authenticated and returns AGGREGATE, NON-PERSONAL
figures only: co-op size, requests served, spend. Same class as
/broker/model-leaderboard, and the same rule — no per-member figure passes
through the broker.
The backfill's first real curve was wrong: built on welcome_sent_at it showed
34 members on 19 October, when the co-op had ~65. Most members were provisioned
before the portal sent welcome emails, so that field lags by dozens of people —
and a chart built on it would have misrepresented the co-op's growth.
Membership now comes from Listmonk's per-subscription subscription_created_at
on the members list, which the portal's nightly reconcile keeps mirroring
active members. That reproduces the real curve (13 members mid-September → 71
today). Past days are the cumulative joins of today's members — an honest floor,
documented as such, since members who left are absent.
The first live run recorded newsletter_subs = 0 — a wrong zero, the exact
fail-open this module documents against. Cause: Listmonk's subscriber record
carries status enabled/blocked, while the LIST membership carries
subscription_status confirmed/unconfirmed. Filtering on the subscriber's own
status matched nothing and returned a silent 0.
Now counts memberships whose subscription_status is confirmed (verified: 71,
matching the list), and returns NULL when the endpoint yields no rows at all —
"not measured" rather than "nobody is subscribed". The daily job is the
regression test: it must record 71, not 0.
The co-op had no longitudinal data: every number was an ad-hoc query, so
nothing compounded and no single source owned "how many members do we have,
and when did that change". That cost us a wrong figure in the newsletter ("six
new members" — actually 21; the six were forum joins).
New: metrics_snapshot.py writes one row per day into the portal's SQLite store
(the same members.db), and GET /admin/metrics/series returns them oldest-first
for the panel and dashboard to chart later.
- Cumulative columns (members_active, members_total, members_new,
newsletter_subs, matrix_space_members) are counts AS OF that day;
spend_usd/requests are THAT day's traffic.
- FAIL CLOSED PER METRIC: an unreadable source yields NULL, never a zero — a
zero reads as "nobody" and would bake a lie into a chart. Failures print to
stdout, and a scheduler task's stdout lands in the app log.
- Listmonk's cached subscriber_count is NOT trusted (it reads 142 for a list
holding 71): the count comes from paging the subscribers endpoint.
- Member growth comes from welcome_sent_at, the provisioning timestamp —
the closest thing the portal stores to a join date. Manual/founder members
have no welcome email, so backfilled days are a documented LOWER BOUND;
today's row carries the exact count.
- Runs as an in-app scheduler task (addons.scheduler, 03:15 UTC) rather than
an agent cron: member-facing plumbing should not depend on the agent VM.
- Backfill is idempotent; member growth is reconstructible back to
September and spend/requests back to the gateway's first request.
Charts land later, on the surfaces, once the shape of the series is reviewed.
Two fixes, both filed rather than deferred.
1. member-portal#2 — the sweep deactivates every member whose email is missing
from the Open Collective member list, and the existing fail-safe only covers a
fetch that RAISES. A fetch that SUCCEEDS while returning nothing or far less
than usual is unguarded, and is what a narrowed token scope, an expired
credential with partial access, or a renamed Membership tier produces (the
fetch filters on the tier's name). That path deactivates the membership
instead of failing. fetch_members() now checks the result against the local
active count before any caller can act on it: empty, or at or below half of
local, raises — so the sweep's fail-safe takes over and members stay active.
Skipped when we have no active members of our own (fresh install).
Correction to the issue as filed: listmonk_reconcile reads the local members
table, not this fetch, so the newsletter path was never exposed to it. The
sweep is the only destructive consumer.
2. ops/security#2 — OC's generic webhooks cannot send headers, so the webhook
secret must travel in the URL path, and uvicorn wrote it into the access log
on every delivery. A logging filter on uvicorn.access and on the root handler
replaces the token with <redacted>. The token cannot be kept out of the URL;
it can be kept out of the retention.
Both pieces are unit-tested in isolation before shipping: the guard's decision
function over eight cases, and the filter against a real access-log record, an
unrelated path (no over-redaction), and a missing dependency (must not break
logging).
Member disclosure, three findings (fixes 1-3 of 5):
1. WEBHOOK_TOKEN was both the Open Collective webhook URL token and the
X-Admin-Token for every /admin/* route, so anything that learned the webhook
URL (OC admins, OC logs, this app's access logs) gained portal admin. The URL
token is forced by OC's generic webhooks, which cannot set headers — the reuse
is not. Introduce ADMIN_TOKEN, header-only. ADMIN_TOKEN_LEGACY=1 (default)
still accepts WEBHOOK_TOKEN and logs a warning, so the admin panel and the
scheduler keep working; set ADMIN_TOKEN everywhere, then flip it to 0 and
rotate. sweep.sh and the digest script now send ADMIN_TOKEN.
2. The credit-pack path fell back to the webhook payload's own amount when OC
could not resolve the order, so a forged POST naming its own totalAmount
minted credits for any slug. The payload is now a hint only: an unresolvable
order is refused, logged loudly, and emailed to the operator for manual
reconciliation. Transient OC failures hold instead of granting blind.
3. Lapsing did not revoke access: both deactivation sites (lapse webhook, nightly
sweep) discarded the key token deactivate_member() returns. Member keys carry
no max_budget of their own, so they inherit the team's — zero the team budget
on deactivation, which blocks every key under it. Key deletion is deliberately
NOT used: provisioning reuses the stored sk- token on reactivation, and
get_or_create_team does not update an existing team's budget, so the two
renewal paths now re-sync the budget explicitly. Failures are loud (log +
alert email); deactivation still completes.
info@inference.coop is on the Members list but has no members row, so the
reconcile classified the co-op's own inbox as a stale subscriber and would have
unsubscribed it.
The data model already distinguishes this: OPERATOR_EMAILS (default
info@inference.coop,bot@inference.coop) are infrastructure accounts — the SSO
owner and the bot — which the panel surfaces under "admins" and excludes from
member counts. They live outside the members table by design. The Loomio
reconciliation applies the same rule to the same accounts via LOOMIO_ALWAYS_KEEP
("must always remain in the group").
So operators become the third class here: neither members nor stale. Their
subscription is left exactly as it is (never unsubscribed, and the reconcile
never subscribes, so an operator who unsubscribes stays unsubscribed). The
response reports how many were left, and the log line carries counts only.
The end-to-end probe against the live listmonk caught this: PUT
/api/subscribers/{id} with {"lists": []} returns 400 {"message":"Invalid
email."}, leaving the subscriber on the list. It needs {"email": ..., "lists":
[]}. The failed request was only a log warning, so the reconcile's main job —
taking lapsed members off the Members list — has never worked; only the DELETE
branch did anything, which is the branch this series just removed.
Now both classes unsubscribe successfully and keep their record.
Two findings from the retrospective review of 394cccfb..e24c272e (#34).
listmonk_reconcile() deleted subscribers with no members row outright. That
destroyed the subscriber's consent/unsubscribe history, and a stray test account
is indistinguishable from a real person who subscribed outside the portal. Both
classes now leave the list the same way — PUT {"lists": []} — and the record is
kept as a suppression marker; nothing is purged automatically. The summary log
line reports counts by reason instead of subscriber addresses. Subscriber
pagination also gains a short-page fallback so a missing `total` cannot truncate
the reconcile at page 1.
reconcile_memberships() called fetch_members() unguarded, so the documented
fail-safe — skip reconciliation rather than deactivate everyone missing from a
partial list — was unreachable: since fetch_members() now raises, the sweep
returned a 500 instead of {"status": "skipped"}. It now catches the failure and
returns reason "member_fetch_failed". The two webhook call sites are left
unguarded on purpose: a payment webhook should fail loudly so OC retries it, and
it self-heals.
Refs #34
The repo didn't record which image is live, so it couldn't answer "what is
deployed here?" — the tag lived only on the Cloudron app. Pin it to the image
deployed 2026-10-26 so the repo mirrors production, as our other repos do.
Consequence, same as librechat-custom: when the CLI runs from this directory this
manifest now wins over `cloudron update --image <tag>`, so a future release pins
the new tag here and commits it as part of the deploy.
Testing the previous commit against the live Loomio user list showed it would
also have renamed a member whose handle was chosen by hand (a working name
would have become the sanitized OIDC uid) — the uid is domain-shaped, so the
sanitizer produced a different, wrong target. The function is scoped to
Loomio's own auto-generated handles: the email local part with
non-alphanumerics stripped plus a random suffix. Gate on that shape.
Verified against the live user list (70 users) before building: exactly one
handle changes — the member whose username fix 422'd on every sweep — no
computed target violates Loomio's rule, and the hand-chosen handle is left
alone.
The username fix rejected a dot-containing OAuth uid and then substituted the
email local part, which carries the same dots — so Loomio answered 422
("Usernames must contain lower case letters, numbers, underscores, and internal
hyphens only") on every sweep, for the one member whose Cloudron username has a
dot. Normalize instead: disallowed characters become underscores, and leading
and trailing hyphens/underscores are stripped. Also log the Loomio user id
rather than the member's email.
Root cause of test accounts receiving the newsletter: LISTMONK_API_USER was
set EMPTY in the portal env - env-present beats the code default, so every
listmonk_sync authenticated with an empty user and 403'd silently. Fixed both
the env (set to portal-sync) and the code (empty env falls back to default).
New: listmonk_reconcile() - the Members list must mirror ACTIVE members.
Stale subscribers are removed: members marked inactive are taken off the list
(record kept as suppression marker); addresses that are not members at all
(test/deleted accounts) are deleted outright. Wired as step 4 of the nightly
membership sweep + POST /admin/newsletter/reconcile for on-demand runs.
inference-bot
inference-bot