Commit Graph
171 Commits
Author SHA1 Message Date
inference-bot 0b0bdaab6f gift emails: invitation at gift time + once-only reaching-limits notice
- gift_welcome_email becomes gift_invitation_email: the ONE email a gift
  sends (the invitation and the guest welcome merged, not two mails at
  once). Subject 'You've been invited to the Inference Cooperative'; names
  the inviter (member slug), the $5 credits, the setup link, the
  Become-a-member button on BECOME_MEMBER_URL. Uses the shared
  PINE_LOGO_SVG instead of an inline copy.
- reaching-limits email (gift_limits_email, 'Your guest access has ended —
  become a member to continue') wired into the sweep's gift deactivation:
  sent once per guest, gated by a gift_lapse_notices sent-state row (same
  pattern as mark_shortfall_notice_sent). Says the access ended, the
  account and data are kept, and membership ($15/mo via Open Collective)
  restores a full monthly allowance + community spaces.
- catch-up pass in reconcile_gift_lifecycles: guests deactivated before the
  email shipped (or a failed send) get exactly one notice on a later sweep;
  converted guests are excluded by account_type.
- sweep summary now reports gift_lapse_notices.
- test_gift_lifecycle: invitation-subject assertion; send_email stub
  returns True (the bool is the gate's contract); new once-only + catch-up
  + healthy-untouched checks. 43/43.
2026-11-08 02:22:59 +00:00
inference-bot c6eb2b44e8 overview: guests are their own list, excluded from member counts
/admin/overview now returns gift (guest) accounts in a separate guests[]
list instead of inside members[], and excludes them from totals.members,
totals.active and the member spend totals. Guests get their own totals
breakout (guests, guests_active, guest_spend, guest_all_time_spend) so
their spend stays visible without ever reading as member activity — the
same reasoning as the operator (admins) and agent categories.

Builds on the gift feature's account_type marker: a guest is any row with
account_type='gift'. The panel reads guests[] and ignores gift rows in
its member count either way.
2026-11-08 02:09:28 +00:00
inference-bot 6d9074eefe gift accounts: $5 guest passes with time box, inviter ledger, auto member conversion
Operator feature: a member gives a friend access with $5 of credits, no OC
membership. Cloudron user lands in the guests group (chat + dashboard only),
LiteLLM team capped at GIFT_BUDGET with NO monthly reset, portal row marked
account_type='gift' with gift_expires_at set, gift welcome email, newsletter
subscription, and a gifts ledger recording inviter -> recipient (one gift per
inviter per 30 days, one per recipient ever).

Sweep safety: reconcile_memberships never deactivates a gift for lacking an
OC contribution; reconcile_gift_lifecycles lapses gifts by their own rules
(credits spent out, or the GIFT_DAYS time box) with member-deactivation
semantics. Guests are excluded from Loomio/Matrix invite lists, the reinvite
path, budget notices, the allowance reconciliation and the metrics member
counts.

Conversion: a gift whose OC identity appears under its own email converts
automatically (webhook + sweep + /admin/provision all funnel through
provision_member): group guests->members, cap lifted to allowance + carried
credits, anchor day kept, congrats email, full member onboarding.
/admin/gift-convert links a membership paid under a different OC email
(members.oc_email, matched by the sweep and webhook).

Fixture test: tests/test_gift_lifecycle.py (37 checks, network replaced by
a raising guard).
2026-11-08 02:05:37 +00:00
inference-bot 2d2d191e11 money-series: drop the caveat field from the payload
The operator removed all caveats from the finance chart; the balance-history
note is gone from the API payload too so it cannot reappear anywhere.
balance_history_start, the series, totals, sources and errors are unchanged.
2026-11-08 01:55:23 +00:00
inference-bot dcce9fcb35 money series: /admin/money-series (revenue/spend/OC balance) + daily OC balance snapshot
- /admin/money-series (X-Admin-Token, 30/min): three series for the admin
  panel's Money chart, hourly in-memory cache with stale-on-error fallback.
  revenue = OC contribution transactions (CREDIT/CONTRIBUTION/non-refund,
  monthly + one-time, org-backed included), cumulative from 2026-09-06;
  spend = one GROUP BY day pass over the full LiteLLM SpendLogs (no team
  filter, same definition as metrics_snapshot.spend_by_day); balance = OC
  current balance + daily snapshots.
- OC exposes only the CURRENT balance — no historical series exists — so
  metrics_snapshot.py now writes oc_balance_usd into metrics_daily daily
  (from 2026-11-08 onward; balance history cannot be backfilled). The
  endpoint seeds an existing row's balance with the live value without
  touching the snapshot's other columns.
- Aggregate totals only: no member emails or names selected or returned.
2026-11-08 01:50:17 +00:00
inference-bot ed66d73327 activation hook: fire Loomio+Matrix syncs at first broker login
New members were skipped by the syncs until their Cloudron account was
activated (deliberately - never invite a half-created account), but nothing
fired the syncs at the moment they DID activate: the 3x/day sweep was the
only pickup, so a newly-activated member could wait hours for forum/space
access (limdingwen waited ~5h on 2026-11-06).

Hook the broker identity path (_verify_broker_secret): after an active
member resolves, if loomio_invited_at is still null, confirm their Cloudron
account is activated (inviteAccepted, fail-CLOSED - an unreadable user list
never fires) and run the same idempotent syncs the sweep runs, as a
background task off the response. Established members pay one indexed
SELECT; concurrent first-logins collapse into one run via an inflight flag;
each service is independently best-effort so a Loomio outage cannot block
the Matrix invite or the response.

Tests: tests/test_first_login_sync.py (18 checks, fixture db + stubbed
network). Also fix test_allowance_shortfall, whose expected exclusion
reference still asserted the pre-a88ca86 placeholder.

Sweep, provisioning and deactivation paths are untouched.
2026-11-06 19:51:12 +00:00
inference-bot a88ca860d4 fix: restore the real OC tx reference in SHORTFALL_REMEDIATED (repo must match the deployed image)
The deployed portal image (20261105-ca3b8db) was built from a working tree
carrying the real transaction id; commit 22c0167 over-redacted it to a
placeholder that matches nothing, so the next build would lose the exclusion
that prevents double-crediting the manually-remediated 2026-11-04 shortfall
charge. Sync HEAD to the deployed bytes (md5-identical to the running
container). The tx id alone is the H2-approved form: it identifies the OC
transaction, carries no member email.
2026-11-05 18:55:15 +00:00
inference-bot 00f60f1374 ops: bound allowance_log growth; correct stale README claims (audit L4+L5)
allowance_log: the sweep and every report-mode run used to append a row
per member per run (~500/day measured). The insert is now an upsert per
(member, cycle, decision) — a re-derived decision updates the row's
evidence and timestamp; a decision CHANGE still gets its own row. No
index or migration; the latest-decision readers are unaffected; history
is not rewritten.

README: scheduler runs 3x/day (0 3,11,19 UTC), not nightly-3am; the
sweep's auth is the ADMIN_TOKEN X-Admin-Token header, not WEBHOOK_TOKEN;
the retired free/member/supporter tier budgets replaced by the actual
single MEMBER_BUDGET model; the unread OPENCOLLECTIVE_WEBHOOK_SECRET env
row removed.
2026-11-05 16:26:16 +00:00
inference-bot ce1bc7bee6 security: log every key-feed fetch (audit M2)
/admin/librechat/keys deliberately serves every member's chat key on one
sync token (the LibreChat sandbox seeds from it; a per-member fetch would
require changing the sandbox's identity model — decision recorded in
ops-docs maintenance/librechat-key-feed.md). Every fetch now logs a
count-only WARNING (no key material; the email redactor strips addresses)
with caller + user-agent, and the daily digest watches the rate against
the sandbox's ~10-min poll cadence.
2026-11-05 16:21:51 +00:00
inference-bot d63f702ec3 privacy: redact member emails from portal logs (audit M1)
The _RedactWebhookPath filter now also strips email-shaped strings from
every log record — messages, %-format args (recursing into tuple/list/
dict args), and any exception text already attached to the record — the
same retention discipline the webhook token got in ops/security#2.
_send_email recipients, listmonk failures, Loomio invite lists and
provisioning/credit lines carry member addresses into container logs,
which persist on the host and ride the backups. MXIDs, counts, slugs and
amounts stay readable; the daily digest's own sanitisation is unchanged.
Filter keeps its never-break-logging guarantee (all redaction inside
try/except; non-string args untouched).

Live pre-state measured: 99 email-bearing lines in the current app log.
2026-11-05 16:17:21 +00:00
inference-bot b9561a49d2 correctness: paginate Cloudron user fetches; atomic held-credit grant (audit M5+M6)
M6: a single-page ?per_page=100 fetch silently truncates past 100 users —
the box is at 99. New shared walkers (async + sync) page through
/api/v1/users like the Matrix paths, capped at 50 pages. All six single-
page call sites now walk: duplicate-email check (provisioning 409s),
reinvite loop, Loomio activation filter, admin overview, member-email
resolution, local MXID resolution. Failure behavior per site is preserved
(page-1 failure fails loudly; later-page failure keeps what was read).

M5: provisioning-time held-credit grant moved into grant_held_credits():
per hold, the status flip, balance move and ledger row commit in ONE
transaction (the old code committed the flip before add_credits, so an
exception between them left hold=granted and the member uncredited).
Mirrors admin_held_grant, keyed on held:<hold_id> so re-runs are no-ops;
mixed-case emails normalized. Tests in tests/test_portal_hardening.py.
2026-11-05 16:06:52 +00:00
inference-bot 22c0167978 privacy: de-PII the shortfall exclusion and policy comment (audit H2)
SHORTFALL_REMEDIATED is keyed on the OC charge reference alone (the same
idempotence key the shortfall path uses); the member's identity and the
timeline of the 2026-11-04 manual allowance reset move to the operations
docs and stay in the ledger. The policy comment that named a member's
join/charge dates is generalized. No behavioral change: the exclusion
matches the same charge it always matched.
2026-11-05 15:59:07 +00:00
inference-bot e6ab177a05 credits: make the credit-pack webhook grant idempotent (audit H1)
- reference built per-order from the webhook's order.idV2 (the id the
  amount lookup resolves) instead of the absent data.id that produced
  the legacy empty-suffix 'oc:<slug>:' format; legacy ledger rows are
  neither re-keyed nor rejected (a new-format reference can never match
  one)
- credit_reference_exists() gates add_credits; a re-delivered order
  returns 'already_credited' with the standing balance
- hold path: one hold per purchase via a pre-check plus a UNIQUE index
  on held_credits.oc_reference (index creation logs CRITICAL and
  continues if the live table ever carried duplicates — audited: none)
- get_order_amount sends the webhook-supplied order id as a GraphQL
  variable instead of interpolating it into the query (audit L9)
- regression test: replay credits exactly once; legacy rows never
  block a first grant; unique-index semantics incl. NULLs
2026-11-05 15:54:29 +00:00
inference-bot 6337bcfc24 portal: don't double the oc: prefix in the held-credit grant reason 2026-11-05 03:48:01 +00:00
inference-bot e103e6779b portal: add admin endpoint to grant a held credit pack to a member
Held credit packs (purchases by non-members) could previously only be
refunded or auto-granted when the SAME email/slug later became a member.
A purchase made on another account's behalf therefore had no admin path
and forced a hand-edit of money state.

Add POST /admin/held-credits/{hold_id}/grant with body {email}: applies
the hold to a specified member account in one transaction (balance +
ledger + hold -> granted), idempotent on reference held:<hold_id>.
2026-11-05 03:45:14 +00:00
inference-bot db64dac235 allowances: credit the shortfall when OC's billing day truncates a paid cycle
Open Collective aligns recurring charges onto the collective's common billing
day, so a mid-month joiner is charged again before a full month is up: they pay
for a full month and receive less than one. Credit the missing ALLOWANCE (the
flat MEMBER_BUDGET pro-rated for the days lost, unrounded) as non-expiring
credits.

- derived from the collective's real per-order charge history, on every sweep;
  any consecutive pair on the same order < a full month apart is truncated
- idempotent on the OC charge reference (dedupe on the charge, not the member)
- skips and logs any cycle whose dates cannot be determined
- ledger reason names the cycle; the notice is staged and defaults OFF
- [redacted member reference]'s remediated charge is excluded by ref

Tests: tests/test_allowance_shortfall.py (pure derivation).
2026-11-04 20:01:10 +00:00
inference-bot f7bbec8a0a loomio: enable by default; idempotent add (never re-invite) + targeted remove
The guard direction from 2026-11-04 is corrected: the co-op DOES use Loomio and
wants every new member added. Removing members/notifications is not the goal —
not duplicating them is. The feature is switched back ON and the sync is made
idempotent instead.

Enablement
  * LOOMIO_SYNC_ENABLED now defaults to ENABLED (kill switch: set =0 to pause).
    Restoring LOOMIO_API_KEY and deploying is all that is needed again.

Non-duplicating ADD (the real fix), two layers:
  * (a) live pre-check against Loomio group membership (b3 index, ordered and
    paginated stably). An address already in the group is skipped SILENTLY —
    it is never placed in an invite payload, so no API call about it can fire
    a notification or email.
  * (b) members.loomio_invited_at: an address we have handed over before is
    never sent again, even if it is not (yet) visible in the live roster.
  * FAIL CLOSED if the live roster cannot be read.
  * remove_absent is never sent.

Removal reconciliation preserved (deactivation must still propagate):
  * A member in the group who is no longer active is removed via a TARGETED
    b3 DELETE by membership id — it names only the member being removed, so it
    cannot re-invite or notify anyone staying. Operator allowlist
    (LOOMIO_ALWAYS_KEEP) is never removed. 404 (already gone) counts as
    success, so a re-run neither errors nor churns.
  * The keep-set is computed from the UNFILTERED active list: an active member
    whose Cloudron account is not yet activated is not removed (only their
    invite is deferred).

Tests: tests/test_loomio_guard.py (26/26) — add/dedupe, silent skip, record
guard, fail-closed, remove-once, allowlist-never-removed, targeted removal
(no present member in any payload), double-run no-op for both phases, 404
tolerance, kill switch.
2026-11-04 19:25:45 +00:00
inference-bot 120a24678d loomio: never invite an existing member (off by default + positive exclusion)
Incident 2026-11-04: a live run of sync_loomio_memberships() emailed a Loomio
invite notification to essentially the WHOLE membership (notification 744:
~83 in-app recipients, ~79 emails 'sent' 18:53-18:56 UTC). The sync handed
Loomio the full active-member list every run with no record of who had been
invited, relying on Loomio's add-if-absent semantics — which produced a
whole-group notification.

Guards now make a repeat impossible:
  * LOOMIO_SYNC_ENABLED (off by default) — restoring LOOMIO_API_KEY alone no
    longer re-arms the integration.
  * members.loomio_invited_at — positive exclusion: an address is sent only if
    it is neither in Loomio's current member list NOR ever handed over, and
    each is recorded once. (The new column starts NULL for the ~89 members
    already in Loomio, so the current-member check is what closes that gap.)
  * FAIL CLOSED — no invite if Loomio's member list cannot be read.
  * remove_absent is dropped: no evict-then-re-add (re-invite) path.

Tests: tests/test_loomio_guard.py (10/10).
2026-11-04 19:09:33 +00:00
inference-bot 8c85c912e8 allowances: add a would_deactivate PREVIEW (deactivation stays gated off)
Report mode (and a send run while ALLOWANCE_DEACTIVATE is off) now reports a
member whose reminder is past the grace window as would_deactivate, with the
reminder date, grace length and deadline in the evidence. The real action is
unchanged and still needs BOTH ALLOWANCE_POLICY=on AND ALLOWANCE_DEACTIVATE=on.

The lapse decision is extracted into a pure function (allowance_lapse_decision)
so it is unit-testable against a synthetic reminder time:
tests/test_allowance_lapse.py (would_deactivate past grace; in_grace inside).
2026-11-04 18:46:54 +00:00
inference-bot f1945aaf53 admin overview: expose derived membership state (active / in_grace / awaiting_contribution / inactive)
The admin panel needs to show a member's membership state without inventing
policy of its own. Compute it in the portal (the source of truth) from what the
allowance reconciliation already knows — the payment_reminders send record and
the latest allowance_log decision — and carry it on each /admin/overview row as
state + state_reason. Read-only; no side effects.
2026-11-04 18:30:42 +00:00
inference-bot a8200fdf54 allowances: correct the dormant-yearly note (collective order list is complete; a member's account.orders spans collectives) 2026-11-04 17:31:11 +00:00
inference-bot a984a443e1 allowances: recognise YEARLY contributions so a yearly payer isn't read as unpaid
_payment_verdict treated MONTHLY as the only recurring frequency, so a live
ACTIVE YEARLY contribution — one charge, current for twelve cycles — fell
through to "no recurring (monthly) contribution" and would have been reminded
and (with deactivation on) deactivated for eleven of those months.

Make the verdict frequency-aware: only a MONTHLY or YEARLY order can cover an
allowance cycle; an ACTIVE YEARLY covers while its window (last charge + 1yr,
or nextChargeDate) includes now; PROCESSING stays paid-pending; a ONE-TIME
payment is credit, never an allowance period and never a mask. A failed
monthly charge still takes the remind path.

Dormant on today's data: inference-cooperative has 0 yearly orders (104 orders:
88 MONTHLY, 16 ONETIME), verified per member ACCOUNT because a member's own
yearly order does not appear in the collective-level order list. This changes
no current member's decision — the report is identical before and after.

Adds tests/test_payment_verdict.py (yearly branch + monthly/processing/onetime/
error/cancelled cases), runnable in the app container.
2026-11-04 17:29:47 +00:00
inference-bot ced8c4f082 email: set an explicit Message-ID on outbound mail so a send is traceable 2026-11-04 05:19:43 +00:00
inference-bot 5237e6352b allowances: add mode=test to /admin/allowance to verify a reminder really sends 2026-11-04 05:18:39 +00:00
inference-bot a8bdb70e08 allowances: fix decision-label mapping (would_renew not would_renewed) 2026-11-04 05:17:54 +00:00
inference-bot d878c32d68 allowances: restore fetch_member_orders/_payment_verdict/payment_reminder_email dropped in the slug fix 2026-11-04 05:17:12 +00:00
inference-bot f8b3eaacd6 allowances: resolve OC slug from the member's email, not the stored column
The first report run matched orders on members.slug, which holds a stale
guest-… slug from first provisioning for 18 members — every one of them read
as unpaid while actually being current payers. Resolve the live slug from the
member's email via the OC member list, refresh the stored slug, and treat
members absent from the OC member list as unverifiable (skip, never punish).
2026-11-04 05:16:32 +00:00
inference-bot ccce643b39 allowances: payment-driven renewal anchored to the member's join day
Replaces the payment-date re-anchoring that let Open Collective's common
billing day move a member's renewal and skip a cycle ([redacted member reference]:
[redacted timeline].

- Reconcile against OC orders directly on the sweep's schedule: a live
  recurring contribution (ACTIVE/PROCESSING) renews; otherwise one reminder
  email, then deactivation only after PAYMENT_GRACE_DAYS.
- Anchor each member's reset to their join day (members.anchor_day), not the
  payment date; the webhook now only confirms the cycle.
- Decision log (allowance_log) records renewed/reminded/deactivated with the
  payment evidence.
- Report-only by default (ALLOWANCE_POLICY=report); deactivation gated
  separately by ALLOWANCE_DEACTIVATE=off.
- POST /admin/allowance?mode=report|send.
2026-11-04 05:15:24 +00:00
inference-bot a89c464834 sync: retire identities from all invite rooms, not just Space+General
The retirement half of sync_matrix_space() iterated [MATRIX_SPACE_ID] + the
General room while the invite half used member_invite_rooms() (MATRIX_INVITE_ROOMS,
7 rooms including Skillshare). A retired local identity was therefore left in
Design, Community, Infrastructure, Updates and Skillshare — a privacy problem in
member-only rooms. Iterate member_invite_rooms() in both halves so they cannot
drift again. Retirement logic (kick joined / revoke invite-state) is unchanged.
2026-11-04 01:05:41 +00:00
inference-bot c53112fbaf model usage: Requests counts only successful calls
The Requests column summed every spend-log row (COUNT(*)), so a model's
call count included refused/failed attempts — e.g. Metis showed 26 calls
when only 16 succeeded (the other 10 were 401s on /v1/systemone).

Count only rows whose status is not 'failure', and report the refused
count separately as 'refused' (per model and in totals) so a failure
cannot inflate use without disappearing from view. spend and tokens stay
summed over all rows: a call that failed after consuming tokens still
cost money.

Fixes the panel and dashboard model tables, which both read this
aggregation (panel via /admin/model-usage, dashboard via
/broker/model-leaderboard).
2026-11-03 23:39:01 +00:00
inference-bot 8bcb7ff049 sync: invite members to configured MATRIX_INVITE_ROOMS (Skillshare)
sync_matrix_space() now invites every active member to the rooms in
member_invite_rooms(): MATRIX_INVITE_ROOMS when set, else the co-op Space +
General room; legacy MATRIX_EXTRA_ROOMS is still appended. The set is explicit
configuration, never "every child of the Space", so operator-only rooms are not
pushed at members. Skillshare (!NSPfCdVP7TWTtFoAz1TxVOcqnoDIpMoJDiPb_XS79ys)
was missing from the invite set, so no member who joined after it was created
was ever told about it; including it here makes the nightly sweep self-heal.
2026-11-02 19:51:11 +00:00
inference-bot 5f8bed33e6 metrics: break out the co-op's own (operator) spend in the daily series
metrics_daily.spend_usd is the co-op-wide total and deliberately spans every
team, operators included — the numbers were already right. What was missing
was a way to tell how much of that total is the co-op's OWN usage, so a
member-facing chart could not be misread as member-only activity.

Adds metrics_daily.operator_spend_usd: the per-day portion of spend_usd that
belongs to the accounts in OPERATOR_EMAILS (teams aliased member:<email>, like
anyone else). NULL = not measured (never a confident 0), mirroring spend_usd.

- metrics_snapshot.py: operator_spend_by_day() + write it on snapshot/backfill
- app/main.py: expose it in metrics_series (so /broker/metrics and
  /admin/metrics/series carry it), and add operator_spend /
  operator_all_time_spend to /admin/overview totals as the explicit breakout
  of total_spend / total_all_time_spend.

spend_usd and every existing column are untouched.
2026-11-02 15:42:30 +00:00
InferenceBot 6a7eaf0ff8 broker: per-key spend (chat separate) in /broker/usage
Members need to see how much each API key has spent. LiteLLM's per-key
'spend' counter is cycle-only (resets at budget_reset_at), so derive it
from the LiteLLM_SpendLogs table instead: sum spend grouped by api_key,
matching rows to member_api_keys.hash by name and labelling unmatched
rows 'chat' (the portal-held chat key is never in member_api_keys).

broker_get_usage now returns key_spend: [{name, is_chat, spend, tokens,
requests}], API keys by spend desc with the chat row last.
2026-11-01 18:00:25 +00:00
inference-bot e1ec25f1e4 admin overview: give agent teams an all-time spend, not just the cycle counter
The team spend counter is cycle-only and resets at budget_reset_at, which for the agent teams means the 1st of the month. The panel's agents table read that field directly, so on 1 November the co-op's own bot accounts appeared to have had their spend zeroed out. Members already had an all-time figure for exactly this reason; agent teams did not, because the all-time query filtered to member:% only. Widen it to agent:% and expose all_time_spend on agent rows.
2026-11-01 17:16:33 +00:00
inference-bot 6f6957c779 portal: budget notice copy — drop the reassurance, button says 'Buy credit packs'
Removes 'nothing else about your account changes' from both the body and the footer; it goes without saying. Button label becomes 'Buy credit packs', and the sentence above it says the same thing so the two do not disagree.
2026-10-31 16:57:40 +00:00
inference-bot b998d8d535 portal: budget notices gain mode=test (email one sample, no ledger row)
Lets the letter be read in a mail client before the sender is armed. Uses the member's real numbers when the address matches one, writes nothing to budget_notices so the once-per-crossing latch is not consumed, and works while BUDGET_NOTICES is off.
2026-10-31 16:53:22 +00:00
inference-bot 7d2507a927 portal: 90% threshold for budget notices; sweep 3x a day
Trigger becomes proportional (90% of the balance used) instead of a $2 floor, so a member on $15 and one on $35 are treated the same. The once-per-crossing latch keys on the same fraction. membership_sweep moves from '0 3 * * *' to '0 3,11,19 * * *' so the notice arrives within hours of crossing rather than up to a day later. BUDGET_NOTICES remains off.
2026-10-31 16:49:22 +00:00
inference-bot d5fbb2ed84 portal: budget notice -> generic low-balance wording, $2 floor, username, once per crossing
Wording no longer names an allowance or a credit pack: it says balance, so it is true whether the member is short on allowance, on purchased credit, or both. Trigger is an absolute $2 floor on remaining budget rather than a percentage, so it means the same at any budget size. Recipient greeting uses members.slug (the username) instead of the email local part, which produced 'Hi n,'. The once-only latch now clears when the balance rises back above the floor, so it fires once per crossing rather than once per budget cycle. BUDGET_NOTICES stays off.
2026-10-31 16:35:35 +00:00
inference-bot 87e7c5b30e portal: budget notices by email, off by default
Warns a member at 80% of their monthly cap and again when it is exhausted, using the welcome email's template shell. Spend and budget come from LiteLLM's Postgres, the same source the dashboard shows. One notice per level per budget cycle. BUDGET_NOTICES=off|dryrun|on gates sending; POST /admin/budget-notices previews at any time and refuses to send unless the flag is on. Rides the existing sweep cron, wrapped so it cannot break reconciliation.
2026-10-31 16:25:36 +00:00
portal-dev 6475a51067 metrics: expose + persist measured-vs-estimated provenance (#4, #6)
Thread the per-factor source through impact_for() instead of discarding it.
Numeric output (energy_wh, co2e_g) is byte-identical — this is a reporting
change, not a recalculation.

#6 persistence: additive columns tokens_measured, tokens_estimated and
factor_revision (a stable hash of the factor table) on metrics_daily, added via
the existing guarded ALTER TABLE idiom in both app/main.py and
metrics_snapshot.py. Existing rows keep NULL.

#4 API: metrics_series() selects the new columns and derives a per-row
provenance indicator (measured | mixed | estimated | none) from the split; both
/admin/metrics/series and /broker/metrics return it. NULL/unknown splits report
'none', never a confident 'measured'.

docs/energy-metrics.md documents the indicator, the new columns, and the
forward-only-vs-backfill behaviour.
2026-10-30 19:12:20 +00:00
inference-bot 9f198fcf4c overview: exclude sandbox teams from the agents list, and name agents for display
sandbox:librechat-test is a throwaway test account rather than an agent of the co-op,
so it drops out of the panel. Its spend then appears nowhere — accepted knowingly.

Agents also carry a display `name` (the alias minus its "agent:" prefix), so the panel
does not have to strip prefixes itself.
2026-10-29 21:19:21 +00:00
inference-bot 592cf83656 metrics + overview: stop counting utility accounts as members, and surface agent usage
The panel has always excluded operator/utility emails from member counts
(OPERATOR_EMAILS), but the metrics counted every row in the members table, so the
panel said 75 while every metric surface said 76. The one affected row is
info@inference.coop — the reading mailbox, not a person.

- metrics_snapshot.py now applies the same OPERATOR_EMAILS definition (same variable
  name, same default) so the two cannot drift apart again. No backfill: history stays
  as published and the daily job writes the corrected value going forward.
- admin_overview now also returns an `agents` list, built from the LiteLLM teams it
  already fetches (anything not member:*), so bot and sandbox spend is visible rather
  than structurally invisible. Kept out of member counts and member aggregates, like
  the operator category.
2026-10-29 20:57:10 +00:00
inference-bot 42bdfa25d3 portal: record that the admin-token cutover is complete (comment only)
The comment above ADMIN_TOKEN still said "set ADMIN_TOKEN_LEGACY=0 and rotate
both". Stale — and the direct reason a report claimed a rotation was
outstanding: ADMIN_TOKEN was minted new and WEBHOOK_TOKEN rotated on 2026-10-27,
and the legacy path was closed on 2026-10-28.

Comment only, no behaviour change. Rebranched onto the current main: an earlier
attempt had pinned an image built from a tree that predated the key-encryption
work, which would have reverted it.
2026-10-29 04:19:17 +00:00
inference-bot a5410ac237 Inline key-encryption module into main.py
The build service's context dropped new files added under app/
(keycrypto.py never reached the image while main.py did), causing a
crash loop. Inline the module so the deploy cannot miss it. Behavior
unchanged: AES-256-GCM at rest, KEK from PORTAL_KEK, enc1: format,
legacy plaintext passthrough, fail-closed migration.
2026-10-28 17:11:12 -06:00
inference-bot 3ee385de12 Encrypt member chat keys at rest (AES-256-GCM)
Member feedback: keys stored in plaintext in the portal DB are a
key-compromise incident waiting to happen. Members.key_token (the chat
key) is now encrypted with AES-256-GCM using a KEK from the PORTAL_KEK
env var. Plaintext exists only in memory when a key is served (dashboard,
LibreChat key feed, LiteLLM management). Startup migration encrypts
existing rows idempotently; fail-closed when the KEK is absent.
Member-managed API keys were already reveal-once (hash-only at rest).
2026-10-28 16:48:58 -06:00
inference-bot af610fe25f metrics: energy and carbon per day, from measured GreenPT factors
Implements the approach Nathan settled on in co-op/support#26: per-token
estimates based on what GreenPT publishes, rather than waiting on the upstream
LiteLLM `impact` passthrough.

- New daily columns: out_tokens, energy_wh, co2e_g.
- Tokens come from OUR OWN spend logs (completion_tokens per model per day), so
  member-visible impact does not depend on any client persisting an `impact`
  object. GreenPT's metric counts OUTPUT tokens only — that is the unit — so
  input tokens are excluded deliberately.
- Factors live in one table with a source per row. The greenpt/* rows are
  MEASURED: fitted from live calls to their API across prompt sizes
  (energy = overhead + marginal x output tokens, R2 >= 0.999; carbon from the
  grid intensity their own emissions imply — ug/mWh and g/kWh are the same
  ratio). Everything else falls back to a documented order-of-magnitude proxy,
  so a day's total is never silently missing models.
- Migration: CREATE IF NOT EXISTS never adds columns, so both the script and
  the app ALTER TABLE on connect.

Refining a factor is a data edit in IMPACT_FACTORS, not a code change.
2026-10-28 04:32:05 +00:00
inference-bot 27d30a1af2 portal: fix the broker metrics route (decorator landed on the wrong handler)
My earlier patch inserted the new handler between the existing
/broker/model-leaderboard decorator and its function, leaving two routes
pointing at model-leaderboard and none at /broker/metrics, which 404'd. The
dashboard's co-op leaderboard was being served the wrong payload.

Caught by the post-deploy check (the 404 on a route whose code was present in
the container), not by the build. Adds a pre-build route-table assertion —
every path must be followed by its intended handler — so this shape of mistake
fails before a build rather than after a deploy.
2026-10-28 04:17:03 +00:00
inference-bot 4ffadd223c metrics: share the series with members through the broker
Members asked to see how the co-op is doing, not just their own usage. The
series was admin-only; now the same query serves both surfaces.

- metrics_series(days) holds the query once; /admin/metrics/series and the new
  /broker/metrics both call it (no second copy to drift).
- /broker/metrics is broker-authenticated and returns AGGREGATE, NON-PERSONAL
  figures only: co-op size, requests served, spend. Same class as
  /broker/model-leaderboard, and the same rule — no per-member figure passes
  through the broker.
2026-10-28 04:15:03 +00:00
inference-bot 70ae435867 metrics: daily snapshot of members and usage, backfillable, plus a read route
The co-op had no longitudinal data: every number was an ad-hoc query, so
nothing compounded and no single source owned "how many members do we have,
and when did that change". That cost us a wrong figure in the newsletter ("six
new members" — actually 21; the six were forum joins).

New: metrics_snapshot.py writes one row per day into the portal's SQLite store
(the same members.db), and GET /admin/metrics/series returns them oldest-first
for the panel and dashboard to chart later.

  - Cumulative columns (members_active, members_total, members_new,
    newsletter_subs, matrix_space_members) are counts AS OF that day;
    spend_usd/requests are THAT day's traffic.
  - FAIL CLOSED PER METRIC: an unreadable source yields NULL, never a zero — a
    zero reads as "nobody" and would bake a lie into a chart. Failures print to
    stdout, and a scheduler task's stdout lands in the app log.
  - Listmonk's cached subscriber_count is NOT trusted (it reads 142 for a list
    holding 71): the count comes from paging the subscribers endpoint.
  - Member growth comes from welcome_sent_at, the provisioning timestamp —
    the closest thing the portal stores to a join date. Manual/founder members
    have no welcome email, so backfilled days are a documented LOWER BOUND;
    today's row carries the exact count.
  - Runs as an in-app scheduler task (addons.scheduler, 03:15 UTC) rather than
    an agent cron: member-facing plumbing should not depend on the agent VM.
  - Backfill is idempotent; member growth is reconstructible back to
    September and spend/requests back to the gateway's first request.

Charts land later, on the surfaces, once the shape of the series is reviewed.
2026-10-28 03:52:49 +00:00
inference-bot 452a27a778 portal: guard against an implausible member list; redact the webhook token from logs
Two fixes, both filed rather than deferred.

1. member-portal#2 — the sweep deactivates every member whose email is missing
   from the Open Collective member list, and the existing fail-safe only covers a
   fetch that RAISES. A fetch that SUCCEEDS while returning nothing or far less
   than usual is unguarded, and is what a narrowed token scope, an expired
   credential with partial access, or a renamed Membership tier produces (the
   fetch filters on the tier's name). That path deactivates the membership
   instead of failing. fetch_members() now checks the result against the local
   active count before any caller can act on it: empty, or at or below half of
   local, raises — so the sweep's fail-safe takes over and members stay active.
   Skipped when we have no active members of our own (fresh install).

   Correction to the issue as filed: listmonk_reconcile reads the local members
   table, not this fetch, so the newsletter path was never exposed to it. The
   sweep is the only destructive consumer.

2. ops/security#2 — OC's generic webhooks cannot send headers, so the webhook
   secret must travel in the URL path, and uvicorn wrote it into the access log
   on every delivery. A logging filter on uvicorn.access and on the root handler
   replaces the token with <redacted>. The token cannot be kept out of the URL;
   it can be kept out of the retention.

Both pieces are unit-tested in isolation before shipping: the guard's decision
function over eight cases, and the filter against a real access-log record, an
unrelated path (no over-redaction), and a missing dependency (must not break
logging).
2026-10-27 05:04:05 +00:00