Commit Graph
164 Commits
Author SHA1 Message Date
inference-bot 00f60f1374 ops: bound allowance_log growth; correct stale README claims (audit L4+L5)
allowance_log: the sweep and every report-mode run used to append a row
per member per run (~500/day measured). The insert is now an upsert per
(member, cycle, decision) — a re-derived decision updates the row's
evidence and timestamp; a decision CHANGE still gets its own row. No
index or migration; the latest-decision readers are unaffected; history
is not rewritten.

README: scheduler runs 3x/day (0 3,11,19 UTC), not nightly-3am; the
sweep's auth is the ADMIN_TOKEN X-Admin-Token header, not WEBHOOK_TOKEN;
the retired free/member/supporter tier budgets replaced by the actual
single MEMBER_BUDGET model; the unread OPENCOLLECTIVE_WEBHOOK_SECRET env
row removed.
2026-11-05 16:26:16 +00:00
inference-bot ce1bc7bee6 security: log every key-feed fetch (audit M2)
/admin/librechat/keys deliberately serves every member's chat key on one
sync token (the LibreChat sandbox seeds from it; a per-member fetch would
require changing the sandbox's identity model — decision recorded in
ops-docs maintenance/librechat-key-feed.md). Every fetch now logs a
count-only WARNING (no key material; the email redactor strips addresses)
with caller + user-agent, and the daily digest watches the rate against
the sandbox's ~10-min poll cadence.
2026-11-05 16:21:51 +00:00
inference-bot d63f702ec3 privacy: redact member emails from portal logs (audit M1)
The _RedactWebhookPath filter now also strips email-shaped strings from
every log record — messages, %-format args (recursing into tuple/list/
dict args), and any exception text already attached to the record — the
same retention discipline the webhook token got in ops/security#2.
_send_email recipients, listmonk failures, Loomio invite lists and
provisioning/credit lines carry member addresses into container logs,
which persist on the host and ride the backups. MXIDs, counts, slugs and
amounts stay readable; the daily digest's own sanitisation is unchanged.
Filter keeps its never-break-logging guarantee (all redaction inside
try/except; non-string args untouched).

Live pre-state measured: 99 email-bearing lines in the current app log.
2026-11-05 16:17:21 +00:00
inference-bot b9561a49d2 correctness: paginate Cloudron user fetches; atomic held-credit grant (audit M5+M6)
M6: a single-page ?per_page=100 fetch silently truncates past 100 users —
the box is at 99. New shared walkers (async + sync) page through
/api/v1/users like the Matrix paths, capped at 50 pages. All six single-
page call sites now walk: duplicate-email check (provisioning 409s),
reinvite loop, Loomio activation filter, admin overview, member-email
resolution, local MXID resolution. Failure behavior per site is preserved
(page-1 failure fails loudly; later-page failure keeps what was read).

M5: provisioning-time held-credit grant moved into grant_held_credits():
per hold, the status flip, balance move and ledger row commit in ONE
transaction (the old code committed the flip before add_credits, so an
exception between them left hold=granted and the member uncredited).
Mirrors admin_held_grant, keyed on held:<hold_id> so re-runs are no-ops;
mixed-case emails normalized. Tests in tests/test_portal_hardening.py.
2026-11-05 16:06:52 +00:00
inference-bot 22c0167978 privacy: de-PII the shortfall exclusion and policy comment (audit H2)
SHORTFALL_REMEDIATED is keyed on the OC charge reference alone (the same
idempotence key the shortfall path uses); the member's identity and the
timeline of the 2026-11-04 manual allowance reset move to the operations
docs and stay in the ledger. The policy comment that named a member's
join/charge dates is generalized. No behavioral change: the exclusion
matches the same charge it always matched.
2026-11-05 15:59:07 +00:00
inference-bot e6ab177a05 credits: make the credit-pack webhook grant idempotent (audit H1)
- reference built per-order from the webhook's order.idV2 (the id the
  amount lookup resolves) instead of the absent data.id that produced
  the legacy empty-suffix 'oc:<slug>:' format; legacy ledger rows are
  neither re-keyed nor rejected (a new-format reference can never match
  one)
- credit_reference_exists() gates add_credits; a re-delivered order
  returns 'already_credited' with the standing balance
- hold path: one hold per purchase via a pre-check plus a UNIQUE index
  on held_credits.oc_reference (index creation logs CRITICAL and
  continues if the live table ever carried duplicates — audited: none)
- get_order_amount sends the webhook-supplied order id as a GraphQL
  variable instead of interpolating it into the query (audit L9)
- regression test: replay credits exactly once; legacy rows never
  block a first grant; unique-index semantics incl. NULLs
2026-11-05 15:54:29 +00:00
inference-bot 6337bcfc24 portal: don't double the oc: prefix in the held-credit grant reason 2026-11-05 03:48:01 +00:00
inference-bot e103e6779b portal: add admin endpoint to grant a held credit pack to a member
Held credit packs (purchases by non-members) could previously only be
refunded or auto-granted when the SAME email/slug later became a member.
A purchase made on another account's behalf therefore had no admin path
and forced a hand-edit of money state.

Add POST /admin/held-credits/{hold_id}/grant with body {email}: applies
the hold to a specified member account in one transaction (balance +
ledger + hold -> granted), idempotent on reference held:<hold_id>.
2026-11-05 03:45:14 +00:00
inference-bot db64dac235 allowances: credit the shortfall when OC's billing day truncates a paid cycle
Open Collective aligns recurring charges onto the collective's common billing
day, so a mid-month joiner is charged again before a full month is up: they pay
for a full month and receive less than one. Credit the missing ALLOWANCE (the
flat MEMBER_BUDGET pro-rated for the days lost, unrounded) as non-expiring
credits.

- derived from the collective's real per-order charge history, on every sweep;
  any consecutive pair on the same order < a full month apart is truncated
- idempotent on the OC charge reference (dedupe on the charge, not the member)
- skips and logs any cycle whose dates cannot be determined
- ledger reason names the cycle; the notice is staged and defaults OFF
- [redacted member reference]'s remediated charge is excluded by ref

Tests: tests/test_allowance_shortfall.py (pure derivation).
2026-11-04 20:01:10 +00:00
inference-bot f7bbec8a0a loomio: enable by default; idempotent add (never re-invite) + targeted remove
The guard direction from 2026-11-04 is corrected: the co-op DOES use Loomio and
wants every new member added. Removing members/notifications is not the goal —
not duplicating them is. The feature is switched back ON and the sync is made
idempotent instead.

Enablement
  * LOOMIO_SYNC_ENABLED now defaults to ENABLED (kill switch: set =0 to pause).
    Restoring LOOMIO_API_KEY and deploying is all that is needed again.

Non-duplicating ADD (the real fix), two layers:
  * (a) live pre-check against Loomio group membership (b3 index, ordered and
    paginated stably). An address already in the group is skipped SILENTLY —
    it is never placed in an invite payload, so no API call about it can fire
    a notification or email.
  * (b) members.loomio_invited_at: an address we have handed over before is
    never sent again, even if it is not (yet) visible in the live roster.
  * FAIL CLOSED if the live roster cannot be read.
  * remove_absent is never sent.

Removal reconciliation preserved (deactivation must still propagate):
  * A member in the group who is no longer active is removed via a TARGETED
    b3 DELETE by membership id — it names only the member being removed, so it
    cannot re-invite or notify anyone staying. Operator allowlist
    (LOOMIO_ALWAYS_KEEP) is never removed. 404 (already gone) counts as
    success, so a re-run neither errors nor churns.
  * The keep-set is computed from the UNFILTERED active list: an active member
    whose Cloudron account is not yet activated is not removed (only their
    invite is deferred).

Tests: tests/test_loomio_guard.py (26/26) — add/dedupe, silent skip, record
guard, fail-closed, remove-once, allowlist-never-removed, targeted removal
(no present member in any payload), double-run no-op for both phases, 404
tolerance, kill switch.
2026-11-04 19:25:45 +00:00
inference-bot 120a24678d loomio: never invite an existing member (off by default + positive exclusion)
Incident 2026-11-04: a live run of sync_loomio_memberships() emailed a Loomio
invite notification to essentially the WHOLE membership (notification 744:
~83 in-app recipients, ~79 emails 'sent' 18:53-18:56 UTC). The sync handed
Loomio the full active-member list every run with no record of who had been
invited, relying on Loomio's add-if-absent semantics — which produced a
whole-group notification.

Guards now make a repeat impossible:
  * LOOMIO_SYNC_ENABLED (off by default) — restoring LOOMIO_API_KEY alone no
    longer re-arms the integration.
  * members.loomio_invited_at — positive exclusion: an address is sent only if
    it is neither in Loomio's current member list NOR ever handed over, and
    each is recorded once. (The new column starts NULL for the ~89 members
    already in Loomio, so the current-member check is what closes that gap.)
  * FAIL CLOSED — no invite if Loomio's member list cannot be read.
  * remove_absent is dropped: no evict-then-re-add (re-invite) path.

Tests: tests/test_loomio_guard.py (10/10).
2026-11-04 19:09:33 +00:00
inference-bot 8c85c912e8 allowances: add a would_deactivate PREVIEW (deactivation stays gated off)
Report mode (and a send run while ALLOWANCE_DEACTIVATE is off) now reports a
member whose reminder is past the grace window as would_deactivate, with the
reminder date, grace length and deadline in the evidence. The real action is
unchanged and still needs BOTH ALLOWANCE_POLICY=on AND ALLOWANCE_DEACTIVATE=on.

The lapse decision is extracted into a pure function (allowance_lapse_decision)
so it is unit-testable against a synthetic reminder time:
tests/test_allowance_lapse.py (would_deactivate past grace; in_grace inside).
2026-11-04 18:46:54 +00:00
inference-bot f1945aaf53 admin overview: expose derived membership state (active / in_grace / awaiting_contribution / inactive)
The admin panel needs to show a member's membership state without inventing
policy of its own. Compute it in the portal (the source of truth) from what the
allowance reconciliation already knows — the payment_reminders send record and
the latest allowance_log decision — and carry it on each /admin/overview row as
state + state_reason. Read-only; no side effects.
2026-11-04 18:30:42 +00:00
inference-bot a8200fdf54 allowances: correct the dormant-yearly note (collective order list is complete; a member's account.orders spans collectives) 2026-11-04 17:31:11 +00:00
inference-bot a984a443e1 allowances: recognise YEARLY contributions so a yearly payer isn't read as unpaid
_payment_verdict treated MONTHLY as the only recurring frequency, so a live
ACTIVE YEARLY contribution — one charge, current for twelve cycles — fell
through to "no recurring (monthly) contribution" and would have been reminded
and (with deactivation on) deactivated for eleven of those months.

Make the verdict frequency-aware: only a MONTHLY or YEARLY order can cover an
allowance cycle; an ACTIVE YEARLY covers while its window (last charge + 1yr,
or nextChargeDate) includes now; PROCESSING stays paid-pending; a ONE-TIME
payment is credit, never an allowance period and never a mask. A failed
monthly charge still takes the remind path.

Dormant on today's data: inference-cooperative has 0 yearly orders (104 orders:
88 MONTHLY, 16 ONETIME), verified per member ACCOUNT because a member's own
yearly order does not appear in the collective-level order list. This changes
no current member's decision — the report is identical before and after.

Adds tests/test_payment_verdict.py (yearly branch + monthly/processing/onetime/
error/cancelled cases), runnable in the app container.
2026-11-04 17:29:47 +00:00
inference-bot ced8c4f082 email: set an explicit Message-ID on outbound mail so a send is traceable 2026-11-04 05:19:43 +00:00
inference-bot 5237e6352b allowances: add mode=test to /admin/allowance to verify a reminder really sends 2026-11-04 05:18:39 +00:00
inference-bot a8bdb70e08 allowances: fix decision-label mapping (would_renew not would_renewed) 2026-11-04 05:17:54 +00:00
inference-bot d878c32d68 allowances: restore fetch_member_orders/_payment_verdict/payment_reminder_email dropped in the slug fix 2026-11-04 05:17:12 +00:00
inference-bot f8b3eaacd6 allowances: resolve OC slug from the member's email, not the stored column
The first report run matched orders on members.slug, which holds a stale
guest-… slug from first provisioning for 18 members — every one of them read
as unpaid while actually being current payers. Resolve the live slug from the
member's email via the OC member list, refresh the stored slug, and treat
members absent from the OC member list as unverifiable (skip, never punish).
2026-11-04 05:16:32 +00:00
inference-bot ccce643b39 allowances: payment-driven renewal anchored to the member's join day
Replaces the payment-date re-anchoring that let Open Collective's common
billing day move a member's renewal and skip a cycle ([redacted member reference]:
[redacted timeline].

- Reconcile against OC orders directly on the sweep's schedule: a live
  recurring contribution (ACTIVE/PROCESSING) renews; otherwise one reminder
  email, then deactivation only after PAYMENT_GRACE_DAYS.
- Anchor each member's reset to their join day (members.anchor_day), not the
  payment date; the webhook now only confirms the cycle.
- Decision log (allowance_log) records renewed/reminded/deactivated with the
  payment evidence.
- Report-only by default (ALLOWANCE_POLICY=report); deactivation gated
  separately by ALLOWANCE_DEACTIVATE=off.
- POST /admin/allowance?mode=report|send.
2026-11-04 05:15:24 +00:00
inference-bot a89c464834 sync: retire identities from all invite rooms, not just Space+General
The retirement half of sync_matrix_space() iterated [MATRIX_SPACE_ID] + the
General room while the invite half used member_invite_rooms() (MATRIX_INVITE_ROOMS,
7 rooms including Skillshare). A retired local identity was therefore left in
Design, Community, Infrastructure, Updates and Skillshare — a privacy problem in
member-only rooms. Iterate member_invite_rooms() in both halves so they cannot
drift again. Retirement logic (kick joined / revoke invite-state) is unchanged.
2026-11-04 01:05:41 +00:00
inference-bot c53112fbaf model usage: Requests counts only successful calls
The Requests column summed every spend-log row (COUNT(*)), so a model's
call count included refused/failed attempts — e.g. Metis showed 26 calls
when only 16 succeeded (the other 10 were 401s on /v1/systemone).

Count only rows whose status is not 'failure', and report the refused
count separately as 'refused' (per model and in totals) so a failure
cannot inflate use without disappearing from view. spend and tokens stay
summed over all rows: a call that failed after consuming tokens still
cost money.

Fixes the panel and dashboard model tables, which both read this
aggregation (panel via /admin/model-usage, dashboard via
/broker/model-leaderboard).
2026-11-03 23:39:01 +00:00
inference-bot 8bcb7ff049 sync: invite members to configured MATRIX_INVITE_ROOMS (Skillshare)
sync_matrix_space() now invites every active member to the rooms in
member_invite_rooms(): MATRIX_INVITE_ROOMS when set, else the co-op Space +
General room; legacy MATRIX_EXTRA_ROOMS is still appended. The set is explicit
configuration, never "every child of the Space", so operator-only rooms are not
pushed at members. Skillshare (!NSPfCdVP7TWTtFoAz1TxVOcqnoDIpMoJDiPb_XS79ys)
was missing from the invite set, so no member who joined after it was created
was ever told about it; including it here makes the nightly sweep self-heal.
2026-11-02 19:51:11 +00:00
inference-bot 5f8bed33e6 metrics: break out the co-op's own (operator) spend in the daily series
metrics_daily.spend_usd is the co-op-wide total and deliberately spans every
team, operators included — the numbers were already right. What was missing
was a way to tell how much of that total is the co-op's OWN usage, so a
member-facing chart could not be misread as member-only activity.

Adds metrics_daily.operator_spend_usd: the per-day portion of spend_usd that
belongs to the accounts in OPERATOR_EMAILS (teams aliased member:<email>, like
anyone else). NULL = not measured (never a confident 0), mirroring spend_usd.

- metrics_snapshot.py: operator_spend_by_day() + write it on snapshot/backfill
- app/main.py: expose it in metrics_series (so /broker/metrics and
  /admin/metrics/series carry it), and add operator_spend /
  operator_all_time_spend to /admin/overview totals as the explicit breakout
  of total_spend / total_all_time_spend.

spend_usd and every existing column are untouched.
2026-11-02 15:42:30 +00:00
InferenceBot 6a7eaf0ff8 broker: per-key spend (chat separate) in /broker/usage
Members need to see how much each API key has spent. LiteLLM's per-key
'spend' counter is cycle-only (resets at budget_reset_at), so derive it
from the LiteLLM_SpendLogs table instead: sum spend grouped by api_key,
matching rows to member_api_keys.hash by name and labelling unmatched
rows 'chat' (the portal-held chat key is never in member_api_keys).

broker_get_usage now returns key_spend: [{name, is_chat, spend, tokens,
requests}], API keys by spend desc with the chat row last.
2026-11-01 18:00:25 +00:00
inference-bot e1ec25f1e4 admin overview: give agent teams an all-time spend, not just the cycle counter
The team spend counter is cycle-only and resets at budget_reset_at, which for the agent teams means the 1st of the month. The panel's agents table read that field directly, so on 1 November the co-op's own bot accounts appeared to have had their spend zeroed out. Members already had an all-time figure for exactly this reason; agent teams did not, because the all-time query filtered to member:% only. Widen it to agent:% and expose all_time_spend on agent rows.
2026-11-01 17:16:33 +00:00
inference-bot 6f6957c779 portal: budget notice copy — drop the reassurance, button says 'Buy credit packs'
Removes 'nothing else about your account changes' from both the body and the footer; it goes without saying. Button label becomes 'Buy credit packs', and the sentence above it says the same thing so the two do not disagree.
2026-10-31 16:57:40 +00:00
inference-bot b998d8d535 portal: budget notices gain mode=test (email one sample, no ledger row)
Lets the letter be read in a mail client before the sender is armed. Uses the member's real numbers when the address matches one, writes nothing to budget_notices so the once-per-crossing latch is not consumed, and works while BUDGET_NOTICES is off.
2026-10-31 16:53:22 +00:00
inference-bot 7d2507a927 portal: 90% threshold for budget notices; sweep 3x a day
Trigger becomes proportional (90% of the balance used) instead of a $2 floor, so a member on $15 and one on $35 are treated the same. The once-per-crossing latch keys on the same fraction. membership_sweep moves from '0 3 * * *' to '0 3,11,19 * * *' so the notice arrives within hours of crossing rather than up to a day later. BUDGET_NOTICES remains off.
2026-10-31 16:49:22 +00:00
inference-bot d5fbb2ed84 portal: budget notice -> generic low-balance wording, $2 floor, username, once per crossing
Wording no longer names an allowance or a credit pack: it says balance, so it is true whether the member is short on allowance, on purchased credit, or both. Trigger is an absolute $2 floor on remaining budget rather than a percentage, so it means the same at any budget size. Recipient greeting uses members.slug (the username) instead of the email local part, which produced 'Hi n,'. The once-only latch now clears when the balance rises back above the floor, so it fires once per crossing rather than once per budget cycle. BUDGET_NOTICES stays off.
2026-10-31 16:35:35 +00:00
inference-bot 87e7c5b30e portal: budget notices by email, off by default
Warns a member at 80% of their monthly cap and again when it is exhausted, using the welcome email's template shell. Spend and budget come from LiteLLM's Postgres, the same source the dashboard shows. One notice per level per budget cycle. BUDGET_NOTICES=off|dryrun|on gates sending; POST /admin/budget-notices previews at any time and refuses to send unless the flag is on. Rides the existing sweep cron, wrapped so it cannot break reconciliation.
2026-10-31 16:25:36 +00:00
portal-dev 6475a51067 metrics: expose + persist measured-vs-estimated provenance (#4, #6)
Thread the per-factor source through impact_for() instead of discarding it.
Numeric output (energy_wh, co2e_g) is byte-identical — this is a reporting
change, not a recalculation.

#6 persistence: additive columns tokens_measured, tokens_estimated and
factor_revision (a stable hash of the factor table) on metrics_daily, added via
the existing guarded ALTER TABLE idiom in both app/main.py and
metrics_snapshot.py. Existing rows keep NULL.

#4 API: metrics_series() selects the new columns and derives a per-row
provenance indicator (measured | mixed | estimated | none) from the split; both
/admin/metrics/series and /broker/metrics return it. NULL/unknown splits report
'none', never a confident 'measured'.

docs/energy-metrics.md documents the indicator, the new columns, and the
forward-only-vs-backfill behaviour.
2026-10-30 19:12:20 +00:00
inference-bot 9f198fcf4c overview: exclude sandbox teams from the agents list, and name agents for display
sandbox:librechat-test is a throwaway test account rather than an agent of the co-op,
so it drops out of the panel. Its spend then appears nowhere — accepted knowingly.

Agents also carry a display `name` (the alias minus its "agent:" prefix), so the panel
does not have to strip prefixes itself.
2026-10-29 21:19:21 +00:00
inference-bot 592cf83656 metrics + overview: stop counting utility accounts as members, and surface agent usage
The panel has always excluded operator/utility emails from member counts
(OPERATOR_EMAILS), but the metrics counted every row in the members table, so the
panel said 75 while every metric surface said 76. The one affected row is
info@inference.coop — the reading mailbox, not a person.

- metrics_snapshot.py now applies the same OPERATOR_EMAILS definition (same variable
  name, same default) so the two cannot drift apart again. No backfill: history stays
  as published and the daily job writes the corrected value going forward.
- admin_overview now also returns an `agents` list, built from the LiteLLM teams it
  already fetches (anything not member:*), so bot and sandbox spend is visible rather
  than structurally invisible. Kept out of member counts and member aggregates, like
  the operator category.
2026-10-29 20:57:10 +00:00
inference-bot 42bdfa25d3 portal: record that the admin-token cutover is complete (comment only)
The comment above ADMIN_TOKEN still said "set ADMIN_TOKEN_LEGACY=0 and rotate
both". Stale — and the direct reason a report claimed a rotation was
outstanding: ADMIN_TOKEN was minted new and WEBHOOK_TOKEN rotated on 2026-10-27,
and the legacy path was closed on 2026-10-28.

Comment only, no behaviour change. Rebranched onto the current main: an earlier
attempt had pinned an image built from a tree that predated the key-encryption
work, which would have reverted it.
2026-10-29 04:19:17 +00:00
inference-bot a5410ac237 Inline key-encryption module into main.py
The build service's context dropped new files added under app/
(keycrypto.py never reached the image while main.py did), causing a
crash loop. Inline the module so the deploy cannot miss it. Behavior
unchanged: AES-256-GCM at rest, KEK from PORTAL_KEK, enc1: format,
legacy plaintext passthrough, fail-closed migration.
2026-10-28 17:11:12 -06:00
inference-bot 3ee385de12 Encrypt member chat keys at rest (AES-256-GCM)
Member feedback: keys stored in plaintext in the portal DB are a
key-compromise incident waiting to happen. Members.key_token (the chat
key) is now encrypted with AES-256-GCM using a KEK from the PORTAL_KEK
env var. Plaintext exists only in memory when a key is served (dashboard,
LibreChat key feed, LiteLLM management). Startup migration encrypts
existing rows idempotently; fail-closed when the KEK is absent.
Member-managed API keys were already reveal-once (hash-only at rest).
2026-10-28 16:48:58 -06:00
inference-bot af610fe25f metrics: energy and carbon per day, from measured GreenPT factors
Implements the approach Nathan settled on in co-op/support#26: per-token
estimates based on what GreenPT publishes, rather than waiting on the upstream
LiteLLM `impact` passthrough.

- New daily columns: out_tokens, energy_wh, co2e_g.
- Tokens come from OUR OWN spend logs (completion_tokens per model per day), so
  member-visible impact does not depend on any client persisting an `impact`
  object. GreenPT's metric counts OUTPUT tokens only — that is the unit — so
  input tokens are excluded deliberately.
- Factors live in one table with a source per row. The greenpt/* rows are
  MEASURED: fitted from live calls to their API across prompt sizes
  (energy = overhead + marginal x output tokens, R2 >= 0.999; carbon from the
  grid intensity their own emissions imply — ug/mWh and g/kWh are the same
  ratio). Everything else falls back to a documented order-of-magnitude proxy,
  so a day's total is never silently missing models.
- Migration: CREATE IF NOT EXISTS never adds columns, so both the script and
  the app ALTER TABLE on connect.

Refining a factor is a data edit in IMPACT_FACTORS, not a code change.
2026-10-28 04:32:05 +00:00
inference-bot 27d30a1af2 portal: fix the broker metrics route (decorator landed on the wrong handler)
My earlier patch inserted the new handler between the existing
/broker/model-leaderboard decorator and its function, leaving two routes
pointing at model-leaderboard and none at /broker/metrics, which 404'd. The
dashboard's co-op leaderboard was being served the wrong payload.

Caught by the post-deploy check (the 404 on a route whose code was present in
the container), not by the build. Adds a pre-build route-table assertion —
every path must be followed by its intended handler — so this shape of mistake
fails before a build rather than after a deploy.
2026-10-28 04:17:03 +00:00
inference-bot 4ffadd223c metrics: share the series with members through the broker
Members asked to see how the co-op is doing, not just their own usage. The
series was admin-only; now the same query serves both surfaces.

- metrics_series(days) holds the query once; /admin/metrics/series and the new
  /broker/metrics both call it (no second copy to drift).
- /broker/metrics is broker-authenticated and returns AGGREGATE, NON-PERSONAL
  figures only: co-op size, requests served, spend. Same class as
  /broker/model-leaderboard, and the same rule — no per-member figure passes
  through the broker.
2026-10-28 04:15:03 +00:00
inference-bot 70ae435867 metrics: daily snapshot of members and usage, backfillable, plus a read route
The co-op had no longitudinal data: every number was an ad-hoc query, so
nothing compounded and no single source owned "how many members do we have,
and when did that change". That cost us a wrong figure in the newsletter ("six
new members" — actually 21; the six were forum joins).

New: metrics_snapshot.py writes one row per day into the portal's SQLite store
(the same members.db), and GET /admin/metrics/series returns them oldest-first
for the panel and dashboard to chart later.

  - Cumulative columns (members_active, members_total, members_new,
    newsletter_subs, matrix_space_members) are counts AS OF that day;
    spend_usd/requests are THAT day's traffic.
  - FAIL CLOSED PER METRIC: an unreadable source yields NULL, never a zero — a
    zero reads as "nobody" and would bake a lie into a chart. Failures print to
    stdout, and a scheduler task's stdout lands in the app log.
  - Listmonk's cached subscriber_count is NOT trusted (it reads 142 for a list
    holding 71): the count comes from paging the subscribers endpoint.
  - Member growth comes from welcome_sent_at, the provisioning timestamp —
    the closest thing the portal stores to a join date. Manual/founder members
    have no welcome email, so backfilled days are a documented LOWER BOUND;
    today's row carries the exact count.
  - Runs as an in-app scheduler task (addons.scheduler, 03:15 UTC) rather than
    an agent cron: member-facing plumbing should not depend on the agent VM.
  - Backfill is idempotent; member growth is reconstructible back to
    September and spend/requests back to the gateway's first request.

Charts land later, on the surfaces, once the shape of the series is reviewed.
2026-10-28 03:52:49 +00:00
inference-bot 452a27a778 portal: guard against an implausible member list; redact the webhook token from logs
Two fixes, both filed rather than deferred.

1. member-portal#2 — the sweep deactivates every member whose email is missing
   from the Open Collective member list, and the existing fail-safe only covers a
   fetch that RAISES. A fetch that SUCCEEDS while returning nothing or far less
   than usual is unguarded, and is what a narrowed token scope, an expired
   credential with partial access, or a renamed Membership tier produces (the
   fetch filters on the tier's name). That path deactivates the membership
   instead of failing. fetch_members() now checks the result against the local
   active count before any caller can act on it: empty, or at or below half of
   local, raises — so the sweep's fail-safe takes over and members stay active.
   Skipped when we have no active members of our own (fresh install).

   Correction to the issue as filed: listmonk_reconcile reads the local members
   table, not this fetch, so the newsletter path was never exposed to it. The
   sweep is the only destructive consumer.

2. ops/security#2 — OC's generic webhooks cannot send headers, so the webhook
   secret must travel in the URL path, and uvicorn wrote it into the access log
   on every delivery. A logging filter on uvicorn.access and on the root handler
   replaces the token with <redacted>. The token cannot be kept out of the URL;
   it can be kept out of the retention.

Both pieces are unit-tested in isolation before shipping: the guard's decision
function over eight cases, and the filter against a real access-log record, an
unrelated path (no over-redaction), and a missing dependency (must not break
logging).
2026-10-27 05:04:05 +00:00
inference-bot 5720a63958 security: separate admin token, verify credit-pack orders, revoke access on lapse
Member disclosure, three findings (fixes 1-3 of 5):

1. WEBHOOK_TOKEN was both the Open Collective webhook URL token and the
   X-Admin-Token for every /admin/* route, so anything that learned the webhook
   URL (OC admins, OC logs, this app's access logs) gained portal admin. The URL
   token is forced by OC's generic webhooks, which cannot set headers — the reuse
   is not. Introduce ADMIN_TOKEN, header-only. ADMIN_TOKEN_LEGACY=1 (default)
   still accepts WEBHOOK_TOKEN and logs a warning, so the admin panel and the
   scheduler keep working; set ADMIN_TOKEN everywhere, then flip it to 0 and
   rotate. sweep.sh and the digest script now send ADMIN_TOKEN.

2. The credit-pack path fell back to the webhook payload's own amount when OC
   could not resolve the order, so a forged POST naming its own totalAmount
   minted credits for any slug. The payload is now a hint only: an unresolvable
   order is refused, logged loudly, and emailed to the operator for manual
   reconciliation. Transient OC failures hold instead of granting blind.

3. Lapsing did not revoke access: both deactivation sites (lapse webhook, nightly
   sweep) discarded the key token deactivate_member() returns. Member keys carry
   no max_budget of their own, so they inherit the team's — zero the team budget
   on deactivation, which blocks every key under it. Key deletion is deliberately
   NOT used: provisioning reuses the stored sk- token on reactivation, and
   get_or_create_team does not update an existing team's budget, so the two
   renewal paths now re-sync the budget explicitly. Failures are loud (log +
   alert email); deactivation still completes.
2026-10-27 03:31:40 +00:00
Ubuntu e55d8a3f1e newsletter: operator accounts are not stale subscribers
info@inference.coop is on the Members list but has no members row, so the
reconcile classified the co-op's own inbox as a stale subscriber and would have
unsubscribed it.

The data model already distinguishes this: OPERATOR_EMAILS (default
info@inference.coop,bot@inference.coop) are infrastructure accounts — the SSO
owner and the bot — which the panel surfaces under "admins" and excludes from
member counts. They live outside the members table by design. The Loomio
reconciliation applies the same rule to the same accounts via LOOMIO_ALWAYS_KEEP
("must always remain in the group").

So operators become the third class here: neither members nor stale. Their
subscription is left exactly as it is (never unsubscribed, and the reconcile
never subscribes, so an operator who unsubscribes stays unsubscribed). The
response reports how many were left, and the log line carries counts only.
2026-10-26 16:54:53 +00:00
Ubuntu 9495daa701 newsletter: the unsubscribe PUT needs the email field — it was a silent 400
The end-to-end probe against the live listmonk caught this: PUT
/api/subscribers/{id} with {"lists": []} returns 400 {"message":"Invalid
email."}, leaving the subscriber on the list. It needs {"email": ..., "lists":
[]}. The failed request was only a log warning, so the reconcile's main job —
taking lapsed members off the Members list — has never worked; only the DELETE
branch did anything, which is the branch this series just removed.

Now both classes unsubscribe successfully and keep their record.
2026-10-26 16:42:10 +00:00
Ubuntu 7bb676eb68 newsletter: unsubscribe stale subscribers instead of deleting them; sweep: apply the documented fail-safe
Two findings from the retrospective review of 394cccfb..e24c272e (#34).

listmonk_reconcile() deleted subscribers with no members row outright. That
destroyed the subscriber's consent/unsubscribe history, and a stray test account
is indistinguishable from a real person who subscribed outside the portal. Both
classes now leave the list the same way — PUT {"lists": []} — and the record is
kept as a suppression marker; nothing is purged automatically. The summary log
line reports counts by reason instead of subscriber addresses. Subscriber
pagination also gains a short-page fallback so a missing `total` cannot truncate
the reconcile at page 1.

reconcile_memberships() called fetch_members() unguarded, so the documented
fail-safe — skip reconciliation rather than deactivate everyone missing from a
partial list — was unreachable: since fetch_members() now raises, the sweep
returned a 500 instead of {"status": "skipped"}. It now catches the failure and
returns reason "member_fetch_failed". The two webhook call sites are left
unguarded on purpose: a payment webhook should fail loudly so OC retries it, and
it self-heals.

Refs #34
2026-10-26 16:38:52 +00:00
Ubuntu 492019088e loomio: only rename auto-generated handles, never one a human chose
Testing the previous commit against the live Loomio user list showed it would
also have renamed a member whose handle was chosen by hand (a working name
would have become the sanitized OIDC uid) — the uid is domain-shaped, so the
sanitizer produced a different, wrong target. The function is scoped to
Loomio's own auto-generated handles: the email local part with
non-alphanumerics stripped plus a random suffix. Gate on that shape.

Verified against the live user list (70 users) before building: exactly one
handle changes — the member whose username fix 422'd on every sweep — no
computed target violates Loomio's rule, and the hand-chosen handle is left
alone.
2026-10-26 16:19:42 +00:00
inference-bot 178a4aa360 loomio: sanitize usernames instead of falling back to another invalid one
The username fix rejected a dot-containing OAuth uid and then substituted the
email local part, which carries the same dots — so Loomio answered 422
("Usernames must contain lower case letters, numbers, underscores, and internal
hyphens only") on every sweep, for the one member whose Cloudron username has a
dot. Normalize instead: disallowed characters become underscores, and leading
and trailing hyphens/underscores are stripped. Also log the Loomio user id
rather than the member's email.
2026-10-26 16:07:23 +00:00
inference-bot e24c272e2e org-link fix: INSERT bound 4 values to 3 placeholders (500 on every link); slug = org slug 2026-10-21 21:16:59 -06:00