Members asked to see how the co-op is doing, not just their own usage. The
series was admin-only; now the same query serves both surfaces.
- metrics_series(days) holds the query once; /admin/metrics/series and the new
/broker/metrics both call it (no second copy to drift).
- /broker/metrics is broker-authenticated and returns AGGREGATE, NON-PERSONAL
figures only: co-op size, requests served, spend. Same class as
/broker/model-leaderboard, and the same rule — no per-member figure passes
through the broker.
The co-op had no longitudinal data: every number was an ad-hoc query, so
nothing compounded and no single source owned "how many members do we have,
and when did that change". That cost us a wrong figure in the newsletter ("six
new members" — actually 21; the six were forum joins).
New: metrics_snapshot.py writes one row per day into the portal's SQLite store
(the same members.db), and GET /admin/metrics/series returns them oldest-first
for the panel and dashboard to chart later.
- Cumulative columns (members_active, members_total, members_new,
newsletter_subs, matrix_space_members) are counts AS OF that day;
spend_usd/requests are THAT day's traffic.
- FAIL CLOSED PER METRIC: an unreadable source yields NULL, never a zero — a
zero reads as "nobody" and would bake a lie into a chart. Failures print to
stdout, and a scheduler task's stdout lands in the app log.
- Listmonk's cached subscriber_count is NOT trusted (it reads 142 for a list
holding 71): the count comes from paging the subscribers endpoint.
- Member growth comes from welcome_sent_at, the provisioning timestamp —
the closest thing the portal stores to a join date. Manual/founder members
have no welcome email, so backfilled days are a documented LOWER BOUND;
today's row carries the exact count.
- Runs as an in-app scheduler task (addons.scheduler, 03:15 UTC) rather than
an agent cron: member-facing plumbing should not depend on the agent VM.
- Backfill is idempotent; member growth is reconstructible back to
September and spend/requests back to the gateway's first request.
Charts land later, on the surfaces, once the shape of the series is reviewed.
Two fixes, both filed rather than deferred.
1. member-portal#2 — the sweep deactivates every member whose email is missing
from the Open Collective member list, and the existing fail-safe only covers a
fetch that RAISES. A fetch that SUCCEEDS while returning nothing or far less
than usual is unguarded, and is what a narrowed token scope, an expired
credential with partial access, or a renamed Membership tier produces (the
fetch filters on the tier's name). That path deactivates the membership
instead of failing. fetch_members() now checks the result against the local
active count before any caller can act on it: empty, or at or below half of
local, raises — so the sweep's fail-safe takes over and members stay active.
Skipped when we have no active members of our own (fresh install).
Correction to the issue as filed: listmonk_reconcile reads the local members
table, not this fetch, so the newsletter path was never exposed to it. The
sweep is the only destructive consumer.
2. ops/security#2 — OC's generic webhooks cannot send headers, so the webhook
secret must travel in the URL path, and uvicorn wrote it into the access log
on every delivery. A logging filter on uvicorn.access and on the root handler
replaces the token with <redacted>. The token cannot be kept out of the URL;
it can be kept out of the retention.
Both pieces are unit-tested in isolation before shipping: the guard's decision
function over eight cases, and the filter against a real access-log record, an
unrelated path (no over-redaction), and a missing dependency (must not break
logging).
Member disclosure, three findings (fixes 1-3 of 5):
1. WEBHOOK_TOKEN was both the Open Collective webhook URL token and the
X-Admin-Token for every /admin/* route, so anything that learned the webhook
URL (OC admins, OC logs, this app's access logs) gained portal admin. The URL
token is forced by OC's generic webhooks, which cannot set headers — the reuse
is not. Introduce ADMIN_TOKEN, header-only. ADMIN_TOKEN_LEGACY=1 (default)
still accepts WEBHOOK_TOKEN and logs a warning, so the admin panel and the
scheduler keep working; set ADMIN_TOKEN everywhere, then flip it to 0 and
rotate. sweep.sh and the digest script now send ADMIN_TOKEN.
2. The credit-pack path fell back to the webhook payload's own amount when OC
could not resolve the order, so a forged POST naming its own totalAmount
minted credits for any slug. The payload is now a hint only: an unresolvable
order is refused, logged loudly, and emailed to the operator for manual
reconciliation. Transient OC failures hold instead of granting blind.
3. Lapsing did not revoke access: both deactivation sites (lapse webhook, nightly
sweep) discarded the key token deactivate_member() returns. Member keys carry
no max_budget of their own, so they inherit the team's — zero the team budget
on deactivation, which blocks every key under it. Key deletion is deliberately
NOT used: provisioning reuses the stored sk- token on reactivation, and
get_or_create_team does not update an existing team's budget, so the two
renewal paths now re-sync the budget explicitly. Failures are loud (log +
alert email); deactivation still completes.
info@inference.coop is on the Members list but has no members row, so the
reconcile classified the co-op's own inbox as a stale subscriber and would have
unsubscribed it.
The data model already distinguishes this: OPERATOR_EMAILS (default
info@inference.coop,bot@inference.coop) are infrastructure accounts — the SSO
owner and the bot — which the panel surfaces under "admins" and excludes from
member counts. They live outside the members table by design. The Loomio
reconciliation applies the same rule to the same accounts via LOOMIO_ALWAYS_KEEP
("must always remain in the group").
So operators become the third class here: neither members nor stale. Their
subscription is left exactly as it is (never unsubscribed, and the reconcile
never subscribes, so an operator who unsubscribes stays unsubscribed). The
response reports how many were left, and the log line carries counts only.
The end-to-end probe against the live listmonk caught this: PUT
/api/subscribers/{id} with {"lists": []} returns 400 {"message":"Invalid
email."}, leaving the subscriber on the list. It needs {"email": ..., "lists":
[]}. The failed request was only a log warning, so the reconcile's main job —
taking lapsed members off the Members list — has never worked; only the DELETE
branch did anything, which is the branch this series just removed.
Now both classes unsubscribe successfully and keep their record.
Two findings from the retrospective review of 394cccfb..e24c272e (#34).
listmonk_reconcile() deleted subscribers with no members row outright. That
destroyed the subscriber's consent/unsubscribe history, and a stray test account
is indistinguishable from a real person who subscribed outside the portal. Both
classes now leave the list the same way — PUT {"lists": []} — and the record is
kept as a suppression marker; nothing is purged automatically. The summary log
line reports counts by reason instead of subscriber addresses. Subscriber
pagination also gains a short-page fallback so a missing `total` cannot truncate
the reconcile at page 1.
reconcile_memberships() called fetch_members() unguarded, so the documented
fail-safe — skip reconciliation rather than deactivate everyone missing from a
partial list — was unreachable: since fetch_members() now raises, the sweep
returned a 500 instead of {"status": "skipped"}. It now catches the failure and
returns reason "member_fetch_failed". The two webhook call sites are left
unguarded on purpose: a payment webhook should fail loudly so OC retries it, and
it self-heals.
Refs #34
Testing the previous commit against the live Loomio user list showed it would
also have renamed a member whose handle was chosen by hand (a working name
would have become the sanitized OIDC uid) — the uid is domain-shaped, so the
sanitizer produced a different, wrong target. The function is scoped to
Loomio's own auto-generated handles: the email local part with
non-alphanumerics stripped plus a random suffix. Gate on that shape.
Verified against the live user list (70 users) before building: exactly one
handle changes — the member whose username fix 422'd on every sweep — no
computed target violates Loomio's rule, and the hand-chosen handle is left
alone.
The username fix rejected a dot-containing OAuth uid and then substituted the
email local part, which carries the same dots — so Loomio answered 422
("Usernames must contain lower case letters, numbers, underscores, and internal
hyphens only") on every sweep, for the one member whose Cloudron username has a
dot. Normalize instead: disallowed characters become underscores, and leading
and trailing hyphens/underscores are stripped. Also log the Loomio user id
rather than the member's email.
Root cause of test accounts receiving the newsletter: LISTMONK_API_USER was
set EMPTY in the portal env - env-present beats the code default, so every
listmonk_sync authenticated with an empty user and 403'd silently. Fixed both
the env (set to portal-sync) and the code (empty env falls back to default).
New: listmonk_reconcile() - the Members list must mirror ACTIVE members.
Stale subscribers are removed: members marked inactive are taken off the list
(record kept as suppression marker); addresses that are not members at all
(test/deleted accounts) are deleted outright. Wired as step 4 of the nightly
membership sweep + POST /admin/newsletter/reconcile for on-demand runs.
LiteLLM requires unique key aliases, so re-creating a key with an existing
(email, name) pair 400s ('alias already exists') — breaking key rotation
from the dashboard. The stored LiteLLM hash lets us retire the old key
first; the ON CONFLICT upsert then tracks the new key.
Surfaced by the LibreChat sandbox work: the portal chat-key rotation for
an SSO member failed with exactly this 400.
inference-bot
inference-bot