Member feedback: keys stored in plaintext in the portal DB are a
key-compromise incident waiting to happen. Members.key_token (the chat
key) is now encrypted with AES-256-GCM using a KEK from the PORTAL_KEK
env var. Plaintext exists only in memory when a key is served (dashboard,
LibreChat key feed, LiteLLM management). Startup migration encrypts
existing rows idempotently; fail-closed when the KEK is absent.
Member-managed API keys were already reveal-once (hash-only at rest).
ADMIN_TOKEN_LEGACY is "0" and verified: ADMIN_TOKEN returns 200 on /admin/*,
WEBHOOK_TOKEN returns 403. The fallback in sweep.sh would therefore turn
"ADMIN_TOKEN unset" into a confusing mid-run 403 rather than the clear abort
that already exists three lines below it. Same removal in the digest script.
Not a behaviour change in normal operation — ADMIN_TOKEN is set (len 64) and
already preferred — but it removes a foot-gun that could mask a broken
configuration, and leaves the code honest about which credential is live.
The carbon column was 1000x too low: 2.9 gCO2e for 55 days and 14.35 kWh,
where 14.35 kWh at the factor table's own intensities must be ~2.9 kg. Energy
was right, so nothing looked broken unless you did the arithmetic — 938 Wh in a
day cannot be 0.2 g of CO2.
Adds a tripwire in impact_for(): if the implied grid intensity falls outside
5-500 gCO2e/kWh, it prints a warning rather than quietly returning numbers. The
factors themselves span 24-250, so a blend outside that band means a unit or
factor error, not a cleaner grid.
Two bugs from the first deploy, both caught by reading the numbers back:
1. Tokens were summed per raw `model`, which carries the UPSTREAM name
(openai/deepseek-v4.1-flash), while the factor table is keyed on our
provider aliases (greenpt/deepseek-v4.1-flash). Every model therefore
matched nothing. Now grouped on model_group, the same convention
aggregate_model_usage uses, with model as the fallback for old rows.
2. Every day was written as energy_wh = 0.0 / co2e_g = 0.0 — a false claim that
the co-op used no energy, and precisely the fail-open this module is written
to avoid. A day with no measured tokens now yields NULL for all three impact
columns, and impact_for() refuses to compute from an empty total.
The backfill is idempotent, so re-running it overwrote the zero rows.
Implements the approach Nathan settled on in co-op/support#26: per-token
estimates based on what GreenPT publishes, rather than waiting on the upstream
LiteLLM `impact` passthrough.
- New daily columns: out_tokens, energy_wh, co2e_g.
- Tokens come from OUR OWN spend logs (completion_tokens per model per day), so
member-visible impact does not depend on any client persisting an `impact`
object. GreenPT's metric counts OUTPUT tokens only — that is the unit — so
input tokens are excluded deliberately.
- Factors live in one table with a source per row. The greenpt/* rows are
MEASURED: fitted from live calls to their API across prompt sizes
(energy = overhead + marginal x output tokens, R2 >= 0.999; carbon from the
grid intensity their own emissions imply — ug/mWh and g/kWh are the same
ratio). Everything else falls back to a documented order-of-magnitude proxy,
so a day's total is never silently missing models.
- Migration: CREATE IF NOT EXISTS never adds columns, so both the script and
the app ALTER TABLE on connect.
Refining a factor is a data edit in IMPACT_FACTORS, not a code change.
My earlier patch inserted the new handler between the existing
/broker/model-leaderboard decorator and its function, leaving two routes
pointing at model-leaderboard and none at /broker/metrics, which 404'd. The
dashboard's co-op leaderboard was being served the wrong payload.
Caught by the post-deploy check (the 404 on a route whose code was present in
the container), not by the build. Adds a pre-build route-table assertion —
every path must be followed by its intended handler — so this shape of mistake
fails before a build rather than after a deploy.
Members asked to see how the co-op is doing, not just their own usage. The
series was admin-only; now the same query serves both surfaces.
- metrics_series(days) holds the query once; /admin/metrics/series and the new
/broker/metrics both call it (no second copy to drift).
- /broker/metrics is broker-authenticated and returns AGGREGATE, NON-PERSONAL
figures only: co-op size, requests served, spend. Same class as
/broker/model-leaderboard, and the same rule — no per-member figure passes
through the broker.
The backfill's first real curve was wrong: built on welcome_sent_at it showed
34 members on 19 October, when the co-op had ~65. Most members were provisioned
before the portal sent welcome emails, so that field lags by dozens of people —
and a chart built on it would have misrepresented the co-op's growth.
Membership now comes from Listmonk's per-subscription subscription_created_at
on the members list, which the portal's nightly reconcile keeps mirroring
active members. That reproduces the real curve (13 members mid-September → 71
today). Past days are the cumulative joins of today's members — an honest floor,
documented as such, since members who left are absent.
The first live run recorded newsletter_subs = 0 — a wrong zero, the exact
fail-open this module documents against. Cause: Listmonk's subscriber record
carries status enabled/blocked, while the LIST membership carries
subscription_status confirmed/unconfirmed. Filtering on the subscriber's own
status matched nothing and returned a silent 0.
Now counts memberships whose subscription_status is confirmed (verified: 71,
matching the list), and returns NULL when the endpoint yields no rows at all —
"not measured" rather than "nobody is subscribed". The daily job is the
regression test: it must record 71, not 0.
The co-op had no longitudinal data: every number was an ad-hoc query, so
nothing compounded and no single source owned "how many members do we have,
and when did that change". That cost us a wrong figure in the newsletter ("six
new members" — actually 21; the six were forum joins).
New: metrics_snapshot.py writes one row per day into the portal's SQLite store
(the same members.db), and GET /admin/metrics/series returns them oldest-first
for the panel and dashboard to chart later.
- Cumulative columns (members_active, members_total, members_new,
newsletter_subs, matrix_space_members) are counts AS OF that day;
spend_usd/requests are THAT day's traffic.
- FAIL CLOSED PER METRIC: an unreadable source yields NULL, never a zero — a
zero reads as "nobody" and would bake a lie into a chart. Failures print to
stdout, and a scheduler task's stdout lands in the app log.
- Listmonk's cached subscriber_count is NOT trusted (it reads 142 for a list
holding 71): the count comes from paging the subscribers endpoint.
- Member growth comes from welcome_sent_at, the provisioning timestamp —
the closest thing the portal stores to a join date. Manual/founder members
have no welcome email, so backfilled days are a documented LOWER BOUND;
today's row carries the exact count.
- Runs as an in-app scheduler task (addons.scheduler, 03:15 UTC) rather than
an agent cron: member-facing plumbing should not depend on the agent VM.
- Backfill is idempotent; member growth is reconstructible back to
September and spend/requests back to the gateway's first request.
Charts land later, on the surfaces, once the shape of the series is reviewed.
Two fixes, both filed rather than deferred.
1. member-portal#2 — the sweep deactivates every member whose email is missing
from the Open Collective member list, and the existing fail-safe only covers a
fetch that RAISES. A fetch that SUCCEEDS while returning nothing or far less
than usual is unguarded, and is what a narrowed token scope, an expired
credential with partial access, or a renamed Membership tier produces (the
fetch filters on the tier's name). That path deactivates the membership
instead of failing. fetch_members() now checks the result against the local
active count before any caller can act on it: empty, or at or below half of
local, raises — so the sweep's fail-safe takes over and members stay active.
Skipped when we have no active members of our own (fresh install).
Correction to the issue as filed: listmonk_reconcile reads the local members
table, not this fetch, so the newsletter path was never exposed to it. The
sweep is the only destructive consumer.
2. ops/security#2 — OC's generic webhooks cannot send headers, so the webhook
secret must travel in the URL path, and uvicorn wrote it into the access log
on every delivery. A logging filter on uvicorn.access and on the root handler
replaces the token with <redacted>. The token cannot be kept out of the URL;
it can be kept out of the retention.
Both pieces are unit-tested in isolation before shipping: the guard's decision
function over eight cases, and the filter against a real access-log record, an
unrelated path (no over-redaction), and a missing dependency (must not break
logging).
Member disclosure, three findings (fixes 1-3 of 5):
1. WEBHOOK_TOKEN was both the Open Collective webhook URL token and the
X-Admin-Token for every /admin/* route, so anything that learned the webhook
URL (OC admins, OC logs, this app's access logs) gained portal admin. The URL
token is forced by OC's generic webhooks, which cannot set headers — the reuse
is not. Introduce ADMIN_TOKEN, header-only. ADMIN_TOKEN_LEGACY=1 (default)
still accepts WEBHOOK_TOKEN and logs a warning, so the admin panel and the
scheduler keep working; set ADMIN_TOKEN everywhere, then flip it to 0 and
rotate. sweep.sh and the digest script now send ADMIN_TOKEN.
2. The credit-pack path fell back to the webhook payload's own amount when OC
could not resolve the order, so a forged POST naming its own totalAmount
minted credits for any slug. The payload is now a hint only: an unresolvable
order is refused, logged loudly, and emailed to the operator for manual
reconciliation. Transient OC failures hold instead of granting blind.
3. Lapsing did not revoke access: both deactivation sites (lapse webhook, nightly
sweep) discarded the key token deactivate_member() returns. Member keys carry
no max_budget of their own, so they inherit the team's — zero the team budget
on deactivation, which blocks every key under it. Key deletion is deliberately
NOT used: provisioning reuses the stored sk- token on reactivation, and
get_or_create_team does not update an existing team's budget, so the two
renewal paths now re-sync the budget explicitly. Failures are loud (log +
alert email); deactivation still completes.
The username fix rejected a dot-containing OAuth uid and then substituted the
email local part, which carries the same dots — so Loomio answered 422
("Usernames must contain lower case letters, numbers, underscores, and internal
hyphens only") on every sweep, for the one member whose Cloudron username has a
dot. Normalize instead: disallowed characters become underscores, and leading
and trailing hyphens/underscores are stripped. Also log the Loomio user id
rather than the member's email.
Root cause of test accounts receiving the newsletter: LISTMONK_API_USER was
set EMPTY in the portal env - env-present beats the code default, so every
listmonk_sync authenticated with an empty user and 403'd silently. Fixed both
the env (set to portal-sync) and the code (empty env falls back to default).
New: listmonk_reconcile() - the Members list must mirror ACTIVE members.
Stale subscribers are removed: members marked inactive are taken off the list
(record kept as suppression marker); addresses that are not members at all
(test/deleted accounts) are deleted outright. Wired as step 4 of the nightly
membership sweep + POST /admin/newsletter/reconcile for on-demand runs.
LiteLLM requires unique key aliases, so re-creating a key with an existing
(email, name) pair 400s ('alias already exists') — breaking key rotation
from the dashboard. The stored LiteLLM hash lets us retire the old key
first; the ON CONFLICT upsert then tracks the new key.
Surfaced by the LibreChat sandbox work: the portal chat-key rotation for
an SSO member failed with exactly this 400.
inference-bot
inference-bot