Commit Graph
100 Commits
Author SHA1 Message Date
inference-bot 73d8c61aa7 manifest: pin to 20261108-2d2d191 2026-11-08 01:55:29 +00:00
inference-bot 2d2d191e11 money-series: drop the caveat field from the payload
The operator removed all caveats from the finance chart; the balance-history
note is gone from the API payload too so it cannot reappear anywhere.
balance_history_start, the series, totals, sources and errors are unchanged.
2026-11-08 01:55:23 +00:00
inference-bot f44c4bb6bd manifest: pin to 20261108-dcce9fc 2026-11-08 01:50:18 +00:00
inference-bot dcce9fcb35 money series: /admin/money-series (revenue/spend/OC balance) + daily OC balance snapshot
- /admin/money-series (X-Admin-Token, 30/min): three series for the admin
  panel's Money chart, hourly in-memory cache with stale-on-error fallback.
  revenue = OC contribution transactions (CREDIT/CONTRIBUTION/non-refund,
  monthly + one-time, org-backed included), cumulative from 2026-09-06;
  spend = one GROUP BY day pass over the full LiteLLM SpendLogs (no team
  filter, same definition as metrics_snapshot.spend_by_day); balance = OC
  current balance + daily snapshots.
- OC exposes only the CURRENT balance — no historical series exists — so
  metrics_snapshot.py now writes oc_balance_usd into metrics_daily daily
  (from 2026-11-08 onward; balance history cannot be backfilled). The
  endpoint seeds an existing row's balance with the live value without
  touching the snapshot's other columns.
- Aggregate totals only: no member emails or names selected or returned.
2026-11-08 01:50:17 +00:00
inference-bot 3cc2fad1dd manifest: pin to 20261106-ed66d73 2026-11-06 19:51:34 +00:00
inference-bot ed66d73327 activation hook: fire Loomio+Matrix syncs at first broker login
New members were skipped by the syncs until their Cloudron account was
activated (deliberately - never invite a half-created account), but nothing
fired the syncs at the moment they DID activate: the 3x/day sweep was the
only pickup, so a newly-activated member could wait hours for forum/space
access (limdingwen waited ~5h on 2026-11-06).

Hook the broker identity path (_verify_broker_secret): after an active
member resolves, if loomio_invited_at is still null, confirm their Cloudron
account is activated (inviteAccepted, fail-CLOSED - an unreadable user list
never fires) and run the same idempotent syncs the sweep runs, as a
background task off the response. Established members pay one indexed
SELECT; concurrent first-logins collapse into one run via an inflight flag;
each service is independently best-effort so a Loomio outage cannot block
the Matrix invite or the response.

Tests: tests/test_first_login_sync.py (18 checks, fixture db + stubbed
network). Also fix test_allowance_shortfall, whose expected exclusion
reference still asserted the pre-a88ca86 placeholder.

Sweep, provisioning and deactivation paths are untouched.
2026-11-06 19:51:12 +00:00
inference-bot a88ca860d4 fix: restore the real OC tx reference in SHORTFALL_REMEDIATED (repo must match the deployed image)
The deployed portal image (20261105-ca3b8db) was built from a working tree
carrying the real transaction id; commit 22c0167 over-redacted it to a
placeholder that matches nothing, so the next build would lose the exclusion
that prevents double-crediting the manually-remediated 2026-11-04 shortfall
charge. Sync HEAD to the deployed bytes (md5-identical to the running
container). The tx id alone is the H2-approved form: it identifies the OC
transaction, carries no member email.
2026-11-05 18:55:15 +00:00
inference-bot 8720d623ea manifest: pin to 20261105-00f60f1 2026-11-05 16:26:16 +00:00
inference-bot 00f60f1374 ops: bound allowance_log growth; correct stale README claims (audit L4+L5)
allowance_log: the sweep and every report-mode run used to append a row
per member per run (~500/day measured). The insert is now an upsert per
(member, cycle, decision) — a re-derived decision updates the row's
evidence and timestamp; a decision CHANGE still gets its own row. No
index or migration; the latest-decision readers are unaffected; history
is not rewritten.

README: scheduler runs 3x/day (0 3,11,19 UTC), not nightly-3am; the
sweep's auth is the ADMIN_TOKEN X-Admin-Token header, not WEBHOOK_TOKEN;
the retired free/member/supporter tier budgets replaced by the actual
single MEMBER_BUDGET model; the unread OPENCOLLECTIVE_WEBHOOK_SECRET env
row removed.
2026-11-05 16:26:16 +00:00
inference-bot 1127b41ba3 manifest: pin to 20261105-ce1bc7b 2026-11-05 16:21:51 +00:00
inference-bot ce1bc7bee6 security: log every key-feed fetch (audit M2)
/admin/librechat/keys deliberately serves every member's chat key on one
sync token (the LibreChat sandbox seeds from it; a per-member fetch would
require changing the sandbox's identity model — decision recorded in
ops-docs maintenance/librechat-key-feed.md). Every fetch now logs a
count-only WARNING (no key material; the email redactor strips addresses)
with caller + user-agent, and the daily digest watches the rate against
the sandbox's ~10-min poll cadence.
2026-11-05 16:21:51 +00:00
inference-bot 8a9222033d manifest: pin to 20261105-d63f702 2026-11-05 16:17:21 +00:00
inference-bot d63f702ec3 privacy: redact member emails from portal logs (audit M1)
The _RedactWebhookPath filter now also strips email-shaped strings from
every log record — messages, %-format args (recursing into tuple/list/
dict args), and any exception text already attached to the record — the
same retention discipline the webhook token got in ops/security#2.
_send_email recipients, listmonk failures, Loomio invite lists and
provisioning/credit lines carry member addresses into container logs,
which persist on the host and ride the backups. MXIDs, counts, slugs and
amounts stay readable; the daily digest's own sanitisation is unchanged.
Filter keeps its never-break-logging guarantee (all redaction inside
try/except; non-string args untouched).

Live pre-state measured: 99 email-bearing lines in the current app log.
2026-11-05 16:17:21 +00:00
inference-bot 2b501b75be manifest: pin to 20261105-b9561a4 2026-11-05 16:06:52 +00:00
inference-bot b9561a49d2 correctness: paginate Cloudron user fetches; atomic held-credit grant (audit M5+M6)
M6: a single-page ?per_page=100 fetch silently truncates past 100 users —
the box is at 99. New shared walkers (async + sync) page through
/api/v1/users like the Matrix paths, capped at 50 pages. All six single-
page call sites now walk: duplicate-email check (provisioning 409s),
reinvite loop, Loomio activation filter, admin overview, member-email
resolution, local MXID resolution. Failure behavior per site is preserved
(page-1 failure fails loudly; later-page failure keeps what was read).

M5: provisioning-time held-credit grant moved into grant_held_credits():
per hold, the status flip, balance move and ledger row commit in ONE
transaction (the old code committed the flip before add_credits, so an
exception between them left hold=granted and the member uncredited).
Mirrors admin_held_grant, keyed on held:<hold_id> so re-runs are no-ops;
mixed-case emails normalized. Tests in tests/test_portal_hardening.py.
2026-11-05 16:06:52 +00:00
inference-bot 0f32c3a868 manifest: pin to 20261105-22c0167 2026-11-05 15:59:07 +00:00
inference-bot 22c0167978 privacy: de-PII the shortfall exclusion and policy comment (audit H2)
SHORTFALL_REMEDIATED is keyed on the OC charge reference alone (the same
idempotence key the shortfall path uses); the member's identity and the
timeline of the 2026-11-04 manual allowance reset move to the operations
docs and stay in the ledger. The policy comment that named a member's
join/charge dates is generalized. No behavioral change: the exclusion
matches the same charge it always matched.
2026-11-05 15:59:07 +00:00
inference-bot f0d67e7efa manifest: pin to 20261105-e6ab177 2026-11-05 15:54:34 +00:00
inference-bot e6ab177a05 credits: make the credit-pack webhook grant idempotent (audit H1)
- reference built per-order from the webhook's order.idV2 (the id the
  amount lookup resolves) instead of the absent data.id that produced
  the legacy empty-suffix 'oc:<slug>:' format; legacy ledger rows are
  neither re-keyed nor rejected (a new-format reference can never match
  one)
- credit_reference_exists() gates add_credits; a re-delivered order
  returns 'already_credited' with the standing balance
- hold path: one hold per purchase via a pre-check plus a UNIQUE index
  on held_credits.oc_reference (index creation logs CRITICAL and
  continues if the live table ever carried duplicates — audited: none)
- get_order_amount sends the webhook-supplied order id as a GraphQL
  variable instead of interpolating it into the query (audit L9)
- regression test: replay credits exactly once; legacy rows never
  block a first grant; unique-index semantics incl. NULLs
2026-11-05 15:54:29 +00:00
inference-bot 48128ab97b manifest: pin to 20261105-6337bcf 2026-11-05 03:48:17 +00:00
inference-bot 6337bcfc24 portal: don't double the oc: prefix in the held-credit grant reason 2026-11-05 03:48:01 +00:00
inference-bot 11a74d955e manifest: pin to 20261105-e103e67 2026-11-05 03:47:21 +00:00
inference-bot e103e6779b portal: add admin endpoint to grant a held credit pack to a member
Held credit packs (purchases by non-members) could previously only be
refunded or auto-granted when the SAME email/slug later became a member.
A purchase made on another account's behalf therefore had no admin path
and forced a hand-edit of money state.

Add POST /admin/held-credits/{hold_id}/grant with body {email}: applies
the hold to a specified member account in one transaction (balance +
ledger + hold -> granted), idempotent on reference held:<hold_id>.
2026-11-05 03:45:14 +00:00
inference-bot ed1e361b14 manifest: pin to 20261104-db64dac 2026-11-04 20:01:28 +00:00
inference-bot db64dac235 allowances: credit the shortfall when OC's billing day truncates a paid cycle
Open Collective aligns recurring charges onto the collective's common billing
day, so a mid-month joiner is charged again before a full month is up: they pay
for a full month and receive less than one. Credit the missing ALLOWANCE (the
flat MEMBER_BUDGET pro-rated for the days lost, unrounded) as non-expiring
credits.

- derived from the collective's real per-order charge history, on every sweep;
  any consecutive pair on the same order < a full month apart is truncated
- idempotent on the OC charge reference (dedupe on the charge, not the member)
- skips and logs any cycle whose dates cannot be determined
- ledger reason names the cycle; the notice is staged and defaults OFF
- [redacted member reference]'s remediated charge is excluded by ref

Tests: tests/test_allowance_shortfall.py (pure derivation).
2026-11-04 20:01:10 +00:00
inference-bot c468b61d3e manifest: pin to 20261104-f7bbec8 2026-11-04 19:26:04 +00:00
inference-bot f7bbec8a0a loomio: enable by default; idempotent add (never re-invite) + targeted remove
The guard direction from 2026-11-04 is corrected: the co-op DOES use Loomio and
wants every new member added. Removing members/notifications is not the goal —
not duplicating them is. The feature is switched back ON and the sync is made
idempotent instead.

Enablement
  * LOOMIO_SYNC_ENABLED now defaults to ENABLED (kill switch: set =0 to pause).
    Restoring LOOMIO_API_KEY and deploying is all that is needed again.

Non-duplicating ADD (the real fix), two layers:
  * (a) live pre-check against Loomio group membership (b3 index, ordered and
    paginated stably). An address already in the group is skipped SILENTLY —
    it is never placed in an invite payload, so no API call about it can fire
    a notification or email.
  * (b) members.loomio_invited_at: an address we have handed over before is
    never sent again, even if it is not (yet) visible in the live roster.
  * FAIL CLOSED if the live roster cannot be read.
  * remove_absent is never sent.

Removal reconciliation preserved (deactivation must still propagate):
  * A member in the group who is no longer active is removed via a TARGETED
    b3 DELETE by membership id — it names only the member being removed, so it
    cannot re-invite or notify anyone staying. Operator allowlist
    (LOOMIO_ALWAYS_KEEP) is never removed. 404 (already gone) counts as
    success, so a re-run neither errors nor churns.
  * The keep-set is computed from the UNFILTERED active list: an active member
    whose Cloudron account is not yet activated is not removed (only their
    invite is deferred).

Tests: tests/test_loomio_guard.py (26/26) — add/dedupe, silent skip, record
guard, fail-closed, remove-once, allowlist-never-removed, targeted removal
(no present member in any payload), double-run no-op for both phases, 404
tolerance, kill switch.
2026-11-04 19:25:45 +00:00
inference-bot 17329c5d6f manifest: pin to 20261104-120a246 2026-11-04 19:09:33 +00:00
inference-bot 120a24678d loomio: never invite an existing member (off by default + positive exclusion)
Incident 2026-11-04: a live run of sync_loomio_memberships() emailed a Loomio
invite notification to essentially the WHOLE membership (notification 744:
~83 in-app recipients, ~79 emails 'sent' 18:53-18:56 UTC). The sync handed
Loomio the full active-member list every run with no record of who had been
invited, relying on Loomio's add-if-absent semantics — which produced a
whole-group notification.

Guards now make a repeat impossible:
  * LOOMIO_SYNC_ENABLED (off by default) — restoring LOOMIO_API_KEY alone no
    longer re-arms the integration.
  * members.loomio_invited_at — positive exclusion: an address is sent only if
    it is neither in Loomio's current member list NOR ever handed over, and
    each is recorded once. (The new column starts NULL for the ~89 members
    already in Loomio, so the current-member check is what closes that gap.)
  * FAIL CLOSED — no invite if Loomio's member list cannot be read.
  * remove_absent is dropped: no evict-then-re-add (re-invite) path.

Tests: tests/test_loomio_guard.py (10/10).
2026-11-04 19:09:33 +00:00
inference-bot 947e4d3cf2 manifest: pin to 20261104-8c85c91 2026-11-04 18:47:18 +00:00
inference-bot 8c85c912e8 allowances: add a would_deactivate PREVIEW (deactivation stays gated off)
Report mode (and a send run while ALLOWANCE_DEACTIVATE is off) now reports a
member whose reminder is past the grace window as would_deactivate, with the
reminder date, grace length and deadline in the evidence. The real action is
unchanged and still needs BOTH ALLOWANCE_POLICY=on AND ALLOWANCE_DEACTIVATE=on.

The lapse decision is extracted into a pure function (allowance_lapse_decision)
so it is unit-testable against a synthetic reminder time:
tests/test_allowance_lapse.py (would_deactivate past grace; in_grace inside).
2026-11-04 18:46:54 +00:00
inference-bot ee7ed77697 manifest: pin to 20261104-f1945aa 2026-11-04 18:31:06 +00:00
inference-bot f1945aaf53 admin overview: expose derived membership state (active / in_grace / awaiting_contribution / inactive)
The admin panel needs to show a member's membership state without inventing
policy of its own. Compute it in the portal (the source of truth) from what the
allowance reconciliation already knows — the payment_reminders send record and
the latest allowance_log decision — and carry it on each /admin/overview row as
state + state_reason. Read-only; no side effects.
2026-11-04 18:30:42 +00:00
inference-bot 97da432bd4 manifest: pin to 20261104-a8200fd 2026-11-04 17:31:33 +00:00
inference-bot a8200fdf54 allowances: correct the dormant-yearly note (collective order list is complete; a member's account.orders spans collectives) 2026-11-04 17:31:11 +00:00
inference-bot 36b7c2f77e manifest: pin to 20261104-a984a44 2026-11-04 17:30:09 +00:00
inference-bot a984a443e1 allowances: recognise YEARLY contributions so a yearly payer isn't read as unpaid
_payment_verdict treated MONTHLY as the only recurring frequency, so a live
ACTIVE YEARLY contribution — one charge, current for twelve cycles — fell
through to "no recurring (monthly) contribution" and would have been reminded
and (with deactivation on) deactivated for eleven of those months.

Make the verdict frequency-aware: only a MONTHLY or YEARLY order can cover an
allowance cycle; an ACTIVE YEARLY covers while its window (last charge + 1yr,
or nextChargeDate) includes now; PROCESSING stays paid-pending; a ONE-TIME
payment is credit, never an allowance period and never a mask. A failed
monthly charge still takes the remind path.

Dormant on today's data: inference-cooperative has 0 yearly orders (104 orders:
88 MONTHLY, 16 ONETIME), verified per member ACCOUNT because a member's own
yearly order does not appear in the collective-level order list. This changes
no current member's decision — the report is identical before and after.

Adds tests/test_payment_verdict.py (yearly branch + monthly/processing/onetime/
error/cancelled cases), runnable in the app container.
2026-11-04 17:29:47 +00:00
inference-bot 86e77a0ad4 manifest: pin to 20261104-ced8c4f 2026-11-04 05:19:58 +00:00
inference-bot ced8c4f082 email: set an explicit Message-ID on outbound mail so a send is traceable 2026-11-04 05:19:43 +00:00
inference-bot 75959e6045 manifest: pin to 20261104-5237e63 2026-11-04 05:18:54 +00:00
inference-bot 5237e6352b allowances: add mode=test to /admin/allowance to verify a reminder really sends 2026-11-04 05:18:39 +00:00
inference-bot f68e4fb652 manifest: pin to 20261104-a8bdb70 2026-11-04 05:18:09 +00:00
inference-bot a8bdb70e08 allowances: fix decision-label mapping (would_renew not would_renewed) 2026-11-04 05:17:54 +00:00
inference-bot 326fec9214 manifest: pin to 20261104-d878c32 2026-11-04 05:17:26 +00:00
inference-bot d878c32d68 allowances: restore fetch_member_orders/_payment_verdict/payment_reminder_email dropped in the slug fix 2026-11-04 05:17:12 +00:00
inference-bot ea4660690d manifest: pin to 20261104-f8b3eaa 2026-11-04 05:16:46 +00:00
inference-bot f8b3eaacd6 allowances: resolve OC slug from the member's email, not the stored column
The first report run matched orders on members.slug, which holds a stale
guest-… slug from first provisioning for 18 members — every one of them read
as unpaid while actually being current payers. Resolve the live slug from the
member's email via the OC member list, refresh the stored slug, and treat
members absent from the OC member list as unverifiable (skip, never punish).
2026-11-04 05:16:32 +00:00
inference-bot 985b55f6d9 manifest: pin to 20261104-ccce643 2026-11-04 05:15:41 +00:00
inference-bot ccce643b39 allowances: payment-driven renewal anchored to the member's join day
Replaces the payment-date re-anchoring that let Open Collective's common
billing day move a member's renewal and skip a cycle ([redacted member reference]:
[redacted timeline].

- Reconcile against OC orders directly on the sweep's schedule: a live
  recurring contribution (ACTIVE/PROCESSING) renews; otherwise one reminder
  email, then deactivation only after PAYMENT_GRACE_DAYS.
- Anchor each member's reset to their join day (members.anchor_day), not the
  payment date; the webhook now only confirms the cycle.
- Decision log (allowance_log) records renewed/reminded/deactivated with the
  payment evidence.
- Report-only by default (ALLOWANCE_POLICY=report); deactivation gated
  separately by ALLOWANCE_DEACTIVATE=off.
- POST /admin/allowance?mode=report|send.
2026-11-04 05:15:24 +00:00
inference-bot 5da1981823 manifest: pin to 20261104-a89c464 2026-11-04 01:05:43 +00:00
inference-bot a89c464834 sync: retire identities from all invite rooms, not just Space+General
The retirement half of sync_matrix_space() iterated [MATRIX_SPACE_ID] + the
General room while the invite half used member_invite_rooms() (MATRIX_INVITE_ROOMS,
7 rooms including Skillshare). A retired local identity was therefore left in
Design, Community, Infrastructure, Updates and Skillshare — a privacy problem in
member-only rooms. Iterate member_invite_rooms() in both halves so they cannot
drift again. Retirement logic (kick joined / revoke invite-state) is unchanged.
2026-11-04 01:05:41 +00:00
inference-bot d5b4a2e03a manifest: pin to 20261103-c53112f
The registry builder is working again (the build pushed cleanly), so
restore the dockerImage pin the portal dropped in f45ca19. Tag carries
the Requests-count fix in app/main.py (commit c53112f).
2026-11-03 23:40:33 +00:00
inference-bot c53112fbaf model usage: Requests counts only successful calls
The Requests column summed every spend-log row (COUNT(*)), so a model's
call count included refused/failed attempts — e.g. Metis showed 26 calls
when only 16 succeeded (the other 10 were 401s on /v1/systemone).

Count only rows whose status is not 'failure', and report the refused
count separately as 'refused' (per model and in totals) so a failure
cannot inflate use without disappearing from view. spend and tokens stay
summed over all rows: a call that failed after consuming tokens still
cost money.

Fixes the panel and dashboard model tables, which both read this
aggregation (panel via /admin/model-usage, dashboard via
/broker/model-leaderboard).
2026-11-03 23:39:01 +00:00
inference-bot 8bcb7ff049 sync: invite members to configured MATRIX_INVITE_ROOMS (Skillshare)
sync_matrix_space() now invites every active member to the rooms in
member_invite_rooms(): MATRIX_INVITE_ROOMS when set, else the co-op Space +
General room; legacy MATRIX_EXTRA_ROOMS is still appended. The set is explicit
configuration, never "every child of the Space", so operator-only rooms are not
pushed at members. Skillshare (!NSPfCdVP7TWTtFoAz1TxVOcqnoDIpMoJDiPb_XS79ys)
was missing from the invite set, so no member who joined after it was created
was ever told about it; including it here makes the nightly sweep self-heal.
2026-11-02 19:51:11 +00:00
inference-bot ea648aab95 README: document operator_spend_usd in the metrics snapshot 2026-11-02 15:51:27 +00:00
inference-bot f45ca199cf manifest: build on-server (drop the registry image pin)
The remote registry builder is currently unusable from this Cloudron host:
its in-build `docker login registry.inference.coop` resolves the registry to
127.0.0.1:443 and is refused, so no new image can be pushed to the registry
(the box's own docker CAN reach the registry — a pull of a missing tag returns
a clean 404 — so this is the builder's container network, not the registry).
The panel and dashboard are already unpinned and build on-server via
`cloudron update` from the repo dir; the portal joins them so its code can ship
the operator-spend breakout the dashboard reads. Restore the pin when the
builder is fixed.
2026-11-02 15:49:11 +00:00
inference-bot 8fc670b240 manifest: pin to 20261102-5f8bed3 2026-11-02 15:42:38 +00:00
inference-bot 5f8bed33e6 metrics: break out the co-op's own (operator) spend in the daily series
metrics_daily.spend_usd is the co-op-wide total and deliberately spans every
team, operators included — the numbers were already right. What was missing
was a way to tell how much of that total is the co-op's OWN usage, so a
member-facing chart could not be misread as member-only activity.

Adds metrics_daily.operator_spend_usd: the per-day portion of spend_usd that
belongs to the accounts in OPERATOR_EMAILS (teams aliased member:<email>, like
anyone else). NULL = not measured (never a confident 0), mirroring spend_usd.

- metrics_snapshot.py: operator_spend_by_day() + write it on snapshot/backfill
- app/main.py: expose it in metrics_series (so /broker/metrics and
  /admin/metrics/series carry it), and add operator_spend /
  operator_all_time_spend to /admin/overview totals as the explicit breakout
  of total_spend / total_all_time_spend.

spend_usd and every existing column are untouched.
2026-11-02 15:42:30 +00:00
InferenceBot ddeefdebe0 manifest: pin to 20261101-6a7eaf0 2026-11-01 18:00:30 +00:00
InferenceBot 6a7eaf0ff8 broker: per-key spend (chat separate) in /broker/usage
Members need to see how much each API key has spent. LiteLLM's per-key
'spend' counter is cycle-only (resets at budget_reset_at), so derive it
from the LiteLLM_SpendLogs table instead: sum spend grouped by api_key,
matching rows to member_api_keys.hash by name and labelling unmatched
rows 'chat' (the portal-held chat key is never in member_api_keys).

broker_get_usage now returns key_spend: [{name, is_chat, spend, tokens,
requests}], API keys by spend desc with the chat row last.
2026-11-01 18:00:25 +00:00
inference-bot b065b7301f manifest: correct the registry path in the pin
I had rewritten co-op/member-portal to code/member-portal while bumping the tag. The repository path in a live pin is the registry location the image is stored and fetched from; changing it points Cloudron at a repo that does not hold this app. Only the tag should move.
2026-11-01 17:17:13 +00:00
inference-bot abdc56e84c manifest: pin to 20261101-e1ec25f
Tag names the commit it was computed from, after committing the agent all-time-spend change.
2026-11-01 17:17:00 +00:00
inference-bot e1ec25f1e4 admin overview: give agent teams an all-time spend, not just the cycle counter
The team spend counter is cycle-only and resets at budget_reset_at, which for the agent teams means the 1st of the month. The panel's agents table read that field directly, so on 1 November the co-op's own bot accounts appeared to have had their spend zeroed out. Members already had an all-time figure for exactly this reason; agent teams did not, because the all-time query filtered to member:% only. Widen it to agent:% and expose all_time_spend on agent rows.
2026-11-01 17:16:33 +00:00
inference-bot 5beffaf7d5 manifest: pin to 20261031-6f6957c 2026-10-31 16:58:03 +00:00
inference-bot 6f6957c779 portal: budget notice copy — drop the reassurance, button says 'Buy credit packs'
Removes 'nothing else about your account changes' from both the body and the footer; it goes without saying. Button label becomes 'Buy credit packs', and the sentence above it says the same thing so the two do not disagree.
2026-10-31 16:57:40 +00:00
inference-bot d2b6f1551f manifest: pin to 20261031-b998d8d 2026-10-31 16:53:50 +00:00
inference-bot b998d8d535 portal: budget notices gain mode=test (email one sample, no ledger row)
Lets the letter be read in a mail client before the sender is armed. Uses the member's real numbers when the address matches one, writes nothing to budget_notices so the once-per-crossing latch is not consumed, and works while BUDGET_NOTICES is off.
2026-10-31 16:53:22 +00:00
inference-bot c21421a89b manifest: pin to 20261031-7d2507a 2026-10-31 16:50:01 +00:00
inference-bot 7d2507a927 portal: 90% threshold for budget notices; sweep 3x a day
Trigger becomes proportional (90% of the balance used) instead of a $2 floor, so a member on $15 and one on $35 are treated the same. The once-per-crossing latch keys on the same fraction. membership_sweep moves from '0 3 * * *' to '0 3,11,19 * * *' so the notice arrives within hours of crossing rather than up to a day later. BUDGET_NOTICES remains off.
2026-10-31 16:49:22 +00:00
inference-bot 26be6156ae manifest: pin to 20261031-d5fbb2e (previous commit dropped the date prefix and 404ed the pull) 2026-10-31 16:44:23 +00:00
inference-bot bb864055e6 manifest: pin to d5fbb2e (computed before the commit again — the code commit is 49e79aa) 2026-10-31 16:35:48 +00:00
inference-bot d5fbb2ed84 portal: budget notice -> generic low-balance wording, $2 floor, username, once per crossing
Wording no longer names an allowance or a credit pack: it says balance, so it is true whether the member is short on allowance, on purchased credit, or both. Trigger is an absolute $2 floor on remaining budget rather than a percentage, so it means the same at any budget size. Recipient greeting uses members.slug (the username) instead of the email local part, which produced 'Hi n,'. The once-only latch now clears when the balance rises back above the floor, so it fires once per crossing rather than once per budget cycle. BUDGET_NOTICES stays off.
2026-10-31 16:35:35 +00:00
inference-bot 49e79aadcc manifest: pin to 20261031-87e7c5b (the earlier pin named the pre-commit hash) 2026-10-31 16:25:51 +00:00
inference-bot 87e7c5b30e portal: budget notices by email, off by default
Warns a member at 80% of their monthly cap and again when it is exhausted, using the welcome email's template shell. Spend and budget come from LiteLLM's Postgres, the same source the dashboard shows. One notice per level per budget cycle. BUDGET_NOTICES=off|dryrun|on gates sending; POST /admin/budget-notices previews at any time and refuses to send unless the flag is on. Rides the existing sweep cron, wrapped so it cannot break reconciliation.
2026-10-31 16:25:36 +00:00
inference-bot 76d59b3fb6 manifest: pin dockerImage to 20261030-6475a51 2026-10-30 19:22:05 +00:00
inference-bot e5c6b6060e manifest: pin dockerImage to 20261029-9f198fc 2026-10-29 21:20:19 +00:00
inference-bot 9f198fcf4c overview: exclude sandbox teams from the agents list, and name agents for display
sandbox:librechat-test is a throwaway test account rather than an agent of the co-op,
so it drops out of the panel. Its spend then appears nowhere — accepted knowingly.

Agents also carry a display `name` (the alias minus its "agent:" prefix), so the panel
does not have to strip prefixes itself.
2026-10-29 21:19:21 +00:00
inference-bot a0040f52e0 manifest: pin dockerImage to 20261029-592cf83 2026-10-29 20:57:38 +00:00
inference-bot 592cf83656 metrics + overview: stop counting utility accounts as members, and surface agent usage
The panel has always excluded operator/utility emails from member counts
(OPERATOR_EMAILS), but the metrics counted every row in the members table, so the
panel said 75 while every metric surface said 76. The one affected row is
info@inference.coop — the reading mailbox, not a person.

- metrics_snapshot.py now applies the same OPERATOR_EMAILS definition (same variable
  name, same default) so the two cannot drift apart again. No backfill: history stays
  as published and the daily job writes the corrected value going forward.
- admin_overview now also returns an `agents` list, built from the LiteLLM teams it
  already fetches (anything not member:*), so bot and sandbox spend is visible rather
  than structurally invisible. Kept out of member counts and member aggregates, like
  the operator category.
2026-10-29 20:57:10 +00:00
inference-bot b11552fa49 docs: write down how the energy/CO2 metrics actually work (#3)
The calculation lived only in code comments. Adds docs/energy-metrics.md and wires it
into the README, per the issue's proposed scope. No calculation changes.

Contents: the input (completion_tokens, output tokens only, grouped on model_group)
and why output-only understates total compute; the factor table with its three
provenance tiers (measured / same-family / estimated) and the fact that the tiers are
never exposed; the formulas; NULL-not-zero and the reason (the first backfill claimed
every day used no energy); the implied-grid-intensity tripwire and the 1000x carbon
error it exists to catch; the 03:15 schedule and that --backfill REWRITES history.

States plainly what is measured and what is not: GreenPT is measured, glm-5.3 is
assumed from its flash sibling, and all Tinfoil and Regolo traffic currently falls to
the proxy estimate — with the consequence that the 250 gCO2e/kWh proxy OVERSTATES
Regolo, which runs on renewable energy.

README: "Two jobs" -> "Three", the metrics_snapshot task documented, and a short
Energy & carbon metrics subsection linking the new doc. Also corrects "listing both
tasks" in the post-change verification step.
2026-10-29 20:15:10 +00:00
inference-bot 42cf43a125 manifest: pin dockerImage to 20261029-42bdfa2 2026-10-29 04:19:33 +00:00
inference-bot 42bdfa25d3 portal: record that the admin-token cutover is complete (comment only)
The comment above ADMIN_TOKEN still said "set ADMIN_TOKEN_LEGACY=0 and rotate
both". Stale — and the direct reason a report claimed a rotation was
outstanding: ADMIN_TOKEN was minted new and WEBHOOK_TOKEN rotated on 2026-10-27,
and the legacy path was closed on 2026-10-28.

Comment only, no behaviour change. Rebranched onto the current main: an earlier
attempt had pinned an image built from a tree that predated the key-encryption
work, which would have reverted it.
2026-10-29 04:19:17 +00:00
inference-bot d5a7a72416 docs: PORTAL_KEK env var (key encryption at rest) 2026-10-28 17:15:07 -06:00
inference-bot ceeaf277e2 manifest: pin dockerImage to 20261028-a5410ac (inlined key crypto) 2026-10-28 17:11:40 -06:00
inference-bot a5410ac237 Inline key-encryption module into main.py
The build service's context dropped new files added under app/
(keycrypto.py never reached the image while main.py did), causing a
crash loop. Inline the module so the deploy cannot miss it. Behavior
unchanged: AES-256-GCM at rest, KEK from PORTAL_KEK, enc1: format,
legacy plaintext passthrough, fail-closed migration.
2026-10-28 17:11:12 -06:00
inference-bot 4a48529cbb manifest: pin dockerImage to 20261028-3ee385d-nc (rebuilt no-cache, keycrypto present) 2026-10-28 17:01:36 -06:00
inference-bot 8ee1c8cea5 manifest: pin dockerImage to 20261028-3ee385d (key encryption at rest) 2026-10-28 16:50:20 -06:00
inference-bot 3ee385de12 Encrypt member chat keys at rest (AES-256-GCM)
Member feedback: keys stored in plaintext in the portal DB are a
key-compromise incident waiting to happen. Members.key_token (the chat
key) is now encrypted with AES-256-GCM using a KEK from the PORTAL_KEK
env var. Plaintext exists only in memory when a key is served (dashboard,
LibreChat key feed, LiteLLM management). Startup migration encrypts
existing rows idempotently; fail-closed when the KEK is absent.
Member-managed API keys were already reveal-once (hash-only at rest).
2026-10-28 16:48:58 -06:00
inference-bot ac3d79a1df manifest: pin dockerImage to 20261028-a0d15e3 (sweep legacy-token cleanup) 2026-10-28 13:54:04 +00:00
inference-bot a0d15e30d8 sweep: drop the legacy WEBHOOK_TOKEN fallback (cutover is complete)
ADMIN_TOKEN_LEGACY is "0" and verified: ADMIN_TOKEN returns 200 on /admin/*,
WEBHOOK_TOKEN returns 403. The fallback in sweep.sh would therefore turn
"ADMIN_TOKEN unset" into a confusing mid-run 403 rather than the clear abort
that already exists three lines below it. Same removal in the digest script.

Not a behaviour change in normal operation — ADMIN_TOKEN is set (len 64) and
already preferred — but it removes a foot-gun that could mask a broken
configuration, and leaves the code honest about which credential is live.
2026-10-28 13:53:25 +00:00
inference-bot e773c5097f manifest: pin dockerImage to 20261028-0ae1404 (carbon unit fix) 2026-10-28 04:35:05 +00:00
inference-bot 0ae1404ab5 metrics: fix carbon unit conversion (Wh -> kWh is /1e3, not /1e6)
The carbon column was 1000x too low: 2.9 gCO2e for 55 days and 14.35 kWh,
where 14.35 kWh at the factor table's own intensities must be ~2.9 kg. Energy
was right, so nothing looked broken unless you did the arithmetic — 938 Wh in a
day cannot be 0.2 g of CO2.

Adds a tripwire in impact_for(): if the implied grid intensity falls outside
5-500 gCO2e/kWh, it prints a warning rather than quietly returning numbers. The
factors themselves span 24-250, so a blend outside that band means a unit or
factor error, not a cleaner grid.
2026-10-28 04:34:50 +00:00
inference-bot 997d047323 manifest: pin dockerImage to 20261028-8e9906d (impact column fixes) 2026-10-28 04:34:06 +00:00
inference-bot 8e9906d844 metrics: fix the impact columns — group by model_group, never write a zero
Two bugs from the first deploy, both caught by reading the numbers back:

1. Tokens were summed per raw `model`, which carries the UPSTREAM name
   (openai/deepseek-v4.1-flash), while the factor table is keyed on our
   provider aliases (greenpt/deepseek-v4.1-flash). Every model therefore
   matched nothing. Now grouped on model_group, the same convention
   aggregate_model_usage uses, with model as the fallback for old rows.

2. Every day was written as energy_wh = 0.0 / co2e_g = 0.0 — a false claim that
   the co-op used no energy, and precisely the fail-open this module is written
   to avoid. A day with no measured tokens now yields NULL for all three impact
   columns, and impact_for() refuses to compute from an empty total.

The backfill is idempotent, so re-running it overwrote the zero rows.
2026-10-28 04:33:50 +00:00
inference-bot 89593cfa74 manifest: pin dockerImage to 20261028-af610fe (energy and carbon metrics) 2026-10-28 04:32:20 +00:00
inference-bot af610fe25f metrics: energy and carbon per day, from measured GreenPT factors
Implements the approach Nathan settled on in co-op/support#26: per-token
estimates based on what GreenPT publishes, rather than waiting on the upstream
LiteLLM `impact` passthrough.

- New daily columns: out_tokens, energy_wh, co2e_g.
- Tokens come from OUR OWN spend logs (completion_tokens per model per day), so
  member-visible impact does not depend on any client persisting an `impact`
  object. GreenPT's metric counts OUTPUT tokens only — that is the unit — so
  input tokens are excluded deliberately.
- Factors live in one table with a source per row. The greenpt/* rows are
  MEASURED: fitted from live calls to their API across prompt sizes
  (energy = overhead + marginal x output tokens, R2 >= 0.999; carbon from the
  grid intensity their own emissions imply — ug/mWh and g/kWh are the same
  ratio). Everything else falls back to a documented order-of-magnitude proxy,
  so a day's total is never silently missing models.
- Migration: CREATE IF NOT EXISTS never adds columns, so both the script and
  the app ALTER TABLE on connect.

Refining a factor is a data edit in IMPACT_FACTORS, not a code change.
2026-10-28 04:32:05 +00:00
inference-bot 5a9b0d2b0d manifest: pin dockerImage to 20261028-27d30a1 (broker metrics route fix) 2026-10-28 04:17:18 +00:00
inference-bot 27d30a1af2 portal: fix the broker metrics route (decorator landed on the wrong handler)
My earlier patch inserted the new handler between the existing
/broker/model-leaderboard decorator and its function, leaving two routes
pointing at model-leaderboard and none at /broker/metrics, which 404'd. The
dashboard's co-op leaderboard was being served the wrong payload.

Caught by the post-deploy check (the 404 on a route whose code was present in
the container), not by the build. Adds a pre-build route-table assertion —
every path must be followed by its intended handler — so this shape of mistake
fails before a build rather than after a deploy.
2026-10-28 04:17:03 +00:00
inference-bot acd01afc99 manifest: pin dockerImage to 20261028-4ffadd2 (broker metrics route) 2026-10-28 04:15:19 +00:00
inference-bot 4ffadd223c metrics: share the series with members through the broker
Members asked to see how the co-op is doing, not just their own usage. The
series was admin-only; now the same query serves both surfaces.

- metrics_series(days) holds the query once; /admin/metrics/series and the new
  /broker/metrics both call it (no second copy to drift).
- /broker/metrics is broker-authenticated and returns AGGREGATE, NON-PERSONAL
  figures only: co-op size, requests served, spend. Same class as
  /broker/model-leaderboard, and the same rule — no per-member figure passes
  through the broker.
2026-10-28 04:15:03 +00:00