Commit Graph
15 Commits
Author SHA1 Message Date
inference-bot cad5897e9a metrics: self-heal missing OC balance days from the OC ledger
The 2026-11-12 hole was NOT a missed sample: the 03:15 run recorded
oc_balance_usd=1357.39, and the evening repair backfill's full-row
upsert of the then-current day — whose dict carries no balance key —
nulled it on conflict. Nothing ever revisited a day's balance, so the
hole was permanent.

Two fixes:
- upsert() can no longer erase a balance it did not measure: the
  conflict clause COALESCEs, so a NULL (a failed OC read) keeps the
  previously recorded value.
- the nightly snapshot heals any recent day (within a week, after the
  series' first recorded balance) whose balance is missing: the value is
  reconstructed from Open Collective's transaction ledger
  (balance(cutoff) = current - net flow after the 03:15 sample moment,
  ledger window clamped to the balance read), matching the daily sample
  semantics exactly. It is written balance-column-only, so membership
  and usage series are untouched. Days before 2026-11-08 stay NULL by
  design. Fail-soft: an OC failure prints and retries next night.

The reconstruction was verified to the cent against OC's own ledger:
full-ledger signed sum == stats.balance (1470.44 on 2026-11-13), and
1470.44 - 113.05 (net flow 2026-11-12 03:15 -> 2026-11-13 03:15:58)
== 1357.39, exactly the sample the 03:15 run had recorded.
2026-11-13 04:04:43 +00:00
inference-bot cf3dbb7489 metrics: complete YESTERDAY's row at 03:15 — every day was frozen at its first ~3 hours
ci/woodpecker/push/woodpecker Pipeline was successful
snapshot() fetched the full spend window (since = yesterday) but kept only
spend.get(TODAY): the day's first hours, written at 03:15 and never revisited.
Every metrics_daily row after the last backfill under-counted its day by the
hours after 03:15 (Nov 7 recorded $2.19/691 requests against a true
$29.06/3384). The completed day's aggregates were computed and thrown away.

Now each run completes the day that just ENDED — full-day spend, requests,
tokens and impact, written to the usage columns only (upsert_usage: membership
columns are point-in-time measurements and are never rewritten) — and then
records today's live partial as before.

backfill() gains --since YYYY-MM-DD: a targeted usage-only repair that cannot
touch the published membership curve.

Regression test: tests/test_metrics_completion.py (day completion, membership
preservation, since-mode repair boundaries, full-backfill parity).
2026-11-12 18:40:31 +00:00
inference-bot 6be8e2d632 Guest cap $5 -> $2: GUEST_BUDGET default, emails, docs, tests 2026-11-08 20:12:47 +00:00
inference-bot 27f9cc06b2 Rename gift to guest throughout: routes, ledger table, account_type values, env names, emails, tests
Pure rename, no behaviour change, matching the renamed panel and dashboard
builds. Includes one-time idempotent migrations for existing databases
(gifts->guests table, gift_expires_at->guest_expires_at column,
account_type 'gift'->'guest' values, gift_lapse_notices->guest_lapse_notices)
and four transitional alias routes (admin/provision-gift, admin/gift-convert,
broker/gift-status, broker/cancel-gift) so pre-rename dashboard/panel builds
keep working until they are redeployed.
2026-11-08 02:41:05 +00:00
inference-bot 6d9074eefe gift accounts: $5 guest passes with time box, inviter ledger, auto member conversion
Operator feature: a member gives a friend access with $5 of credits, no OC
membership. Cloudron user lands in the guests group (chat + dashboard only),
LiteLLM team capped at GIFT_BUDGET with NO monthly reset, portal row marked
account_type='gift' with gift_expires_at set, gift welcome email, newsletter
subscription, and a gifts ledger recording inviter -> recipient (one gift per
inviter per 30 days, one per recipient ever).

Sweep safety: reconcile_memberships never deactivates a gift for lacking an
OC contribution; reconcile_gift_lifecycles lapses gifts by their own rules
(credits spent out, or the GIFT_DAYS time box) with member-deactivation
semantics. Guests are excluded from Loomio/Matrix invite lists, the reinvite
path, budget notices, the allowance reconciliation and the metrics member
counts.

Conversion: a gift whose OC identity appears under its own email converts
automatically (webhook + sweep + /admin/provision all funnel through
provision_member): group guests->members, cap lifted to allowance + carried
credits, anchor day kept, congrats email, full member onboarding.
/admin/gift-convert links a membership paid under a different OC email
(members.oc_email, matched by the sweep and webhook).

Fixture test: tests/test_gift_lifecycle.py (37 checks, network replaced by
a raising guard).
2026-11-08 02:05:37 +00:00
inference-bot dcce9fcb35 money series: /admin/money-series (revenue/spend/OC balance) + daily OC balance snapshot
- /admin/money-series (X-Admin-Token, 30/min): three series for the admin
  panel's Money chart, hourly in-memory cache with stale-on-error fallback.
  revenue = OC contribution transactions (CREDIT/CONTRIBUTION/non-refund,
  monthly + one-time, org-backed included), cumulative from 2026-09-06;
  spend = one GROUP BY day pass over the full LiteLLM SpendLogs (no team
  filter, same definition as metrics_snapshot.spend_by_day); balance = OC
  current balance + daily snapshots.
- OC exposes only the CURRENT balance — no historical series exists — so
  metrics_snapshot.py now writes oc_balance_usd into metrics_daily daily
  (from 2026-11-08 onward; balance history cannot be backfilled). The
  endpoint seeds an existing row's balance with the live value without
  touching the snapshot's other columns.
- Aggregate totals only: no member emails or names selected or returned.
2026-11-08 01:50:17 +00:00
inference-bot 5f8bed33e6 metrics: break out the co-op's own (operator) spend in the daily series
metrics_daily.spend_usd is the co-op-wide total and deliberately spans every
team, operators included — the numbers were already right. What was missing
was a way to tell how much of that total is the co-op's OWN usage, so a
member-facing chart could not be misread as member-only activity.

Adds metrics_daily.operator_spend_usd: the per-day portion of spend_usd that
belongs to the accounts in OPERATOR_EMAILS (teams aliased member:<email>, like
anyone else). NULL = not measured (never a confident 0), mirroring spend_usd.

- metrics_snapshot.py: operator_spend_by_day() + write it on snapshot/backfill
- app/main.py: expose it in metrics_series (so /broker/metrics and
  /admin/metrics/series carry it), and add operator_spend /
  operator_all_time_spend to /admin/overview totals as the explicit breakout
  of total_spend / total_all_time_spend.

spend_usd and every existing column are untouched.
2026-11-02 15:42:30 +00:00
portal-dev 6475a51067 metrics: expose + persist measured-vs-estimated provenance (#4, #6)
Thread the per-factor source through impact_for() instead of discarding it.
Numeric output (energy_wh, co2e_g) is byte-identical — this is a reporting
change, not a recalculation.

#6 persistence: additive columns tokens_measured, tokens_estimated and
factor_revision (a stable hash of the factor table) on metrics_daily, added via
the existing guarded ALTER TABLE idiom in both app/main.py and
metrics_snapshot.py. Existing rows keep NULL.

#4 API: metrics_series() selects the new columns and derives a per-row
provenance indicator (measured | mixed | estimated | none) from the split; both
/admin/metrics/series and /broker/metrics return it. NULL/unknown splits report
'none', never a confident 'measured'.

docs/energy-metrics.md documents the indicator, the new columns, and the
forward-only-vs-backfill behaviour.
2026-10-30 19:12:20 +00:00
inference-bot 592cf83656 metrics + overview: stop counting utility accounts as members, and surface agent usage
The panel has always excluded operator/utility emails from member counts
(OPERATOR_EMAILS), but the metrics counted every row in the members table, so the
panel said 75 while every metric surface said 76. The one affected row is
info@inference.coop — the reading mailbox, not a person.

- metrics_snapshot.py now applies the same OPERATOR_EMAILS definition (same variable
  name, same default) so the two cannot drift apart again. No backfill: history stays
  as published and the daily job writes the corrected value going forward.
- admin_overview now also returns an `agents` list, built from the LiteLLM teams it
  already fetches (anything not member:*), so bot and sandbox spend is visible rather
  than structurally invisible. Kept out of member counts and member aggregates, like
  the operator category.
2026-10-29 20:57:10 +00:00
inference-bot 0ae1404ab5 metrics: fix carbon unit conversion (Wh -> kWh is /1e3, not /1e6)
The carbon column was 1000x too low: 2.9 gCO2e for 55 days and 14.35 kWh,
where 14.35 kWh at the factor table's own intensities must be ~2.9 kg. Energy
was right, so nothing looked broken unless you did the arithmetic — 938 Wh in a
day cannot be 0.2 g of CO2.

Adds a tripwire in impact_for(): if the implied grid intensity falls outside
5-500 gCO2e/kWh, it prints a warning rather than quietly returning numbers. The
factors themselves span 24-250, so a blend outside that band means a unit or
factor error, not a cleaner grid.
2026-10-28 04:34:50 +00:00
inference-bot 8e9906d844 metrics: fix the impact columns — group by model_group, never write a zero
Two bugs from the first deploy, both caught by reading the numbers back:

1. Tokens were summed per raw `model`, which carries the UPSTREAM name
   (openai/deepseek-v4.1-flash), while the factor table is keyed on our
   provider aliases (greenpt/deepseek-v4.1-flash). Every model therefore
   matched nothing. Now grouped on model_group, the same convention
   aggregate_model_usage uses, with model as the fallback for old rows.

2. Every day was written as energy_wh = 0.0 / co2e_g = 0.0 — a false claim that
   the co-op used no energy, and precisely the fail-open this module is written
   to avoid. A day with no measured tokens now yields NULL for all three impact
   columns, and impact_for() refuses to compute from an empty total.

The backfill is idempotent, so re-running it overwrote the zero rows.
2026-10-28 04:33:50 +00:00
inference-bot af610fe25f metrics: energy and carbon per day, from measured GreenPT factors
Implements the approach Nathan settled on in co-op/support#26: per-token
estimates based on what GreenPT publishes, rather than waiting on the upstream
LiteLLM `impact` passthrough.

- New daily columns: out_tokens, energy_wh, co2e_g.
- Tokens come from OUR OWN spend logs (completion_tokens per model per day), so
  member-visible impact does not depend on any client persisting an `impact`
  object. GreenPT's metric counts OUTPUT tokens only — that is the unit — so
  input tokens are excluded deliberately.
- Factors live in one table with a source per row. The greenpt/* rows are
  MEASURED: fitted from live calls to their API across prompt sizes
  (energy = overhead + marginal x output tokens, R2 >= 0.999; carbon from the
  grid intensity their own emissions imply — ug/mWh and g/kWh are the same
  ratio). Everything else falls back to a documented order-of-magnitude proxy,
  so a day's total is never silently missing models.
- Migration: CREATE IF NOT EXISTS never adds columns, so both the script and
  the app ALTER TABLE on connect.

Refining a factor is a data edit in IMPACT_FACTORS, not a code change.
2026-10-28 04:32:05 +00:00
inference-bot 1cf78c4e56 metrics: membership history from list-join dates, not welcome_sent_at
The backfill's first real curve was wrong: built on welcome_sent_at it showed
34 members on 19 October, when the co-op had ~65. Most members were provisioned
before the portal sent welcome emails, so that field lags by dozens of people —
and a chart built on it would have misrepresented the co-op's growth.

Membership now comes from Listmonk's per-subscription subscription_created_at
on the members list, which the portal's nightly reconcile keeps mirroring
active members. That reproduces the real curve (13 members mid-September → 71
today). Past days are the cumulative joins of today's members — an honest floor,
documented as such, since members who left are absent.
2026-10-28 03:56:06 +00:00
inference-bot 27c32fc75f metrics: count LIST membership status, not the subscriber's own status
The first live run recorded newsletter_subs = 0 — a wrong zero, the exact
fail-open this module documents against. Cause: Listmonk's subscriber record
carries status enabled/blocked, while the LIST membership carries
subscription_status confirmed/unconfirmed. Filtering on the subscriber's own
status matched nothing and returned a silent 0.

Now counts memberships whose subscription_status is confirmed (verified: 71,
matching the list), and returns NULL when the endpoint yields no rows at all —
"not measured" rather than "nobody is subscribed". The daily job is the
regression test: it must record 71, not 0.
2026-10-28 03:54:42 +00:00
inference-bot 70ae435867 metrics: daily snapshot of members and usage, backfillable, plus a read route
The co-op had no longitudinal data: every number was an ad-hoc query, so
nothing compounded and no single source owned "how many members do we have,
and when did that change". That cost us a wrong figure in the newsletter ("six
new members" — actually 21; the six were forum joins).

New: metrics_snapshot.py writes one row per day into the portal's SQLite store
(the same members.db), and GET /admin/metrics/series returns them oldest-first
for the panel and dashboard to chart later.

  - Cumulative columns (members_active, members_total, members_new,
    newsletter_subs, matrix_space_members) are counts AS OF that day;
    spend_usd/requests are THAT day's traffic.
  - FAIL CLOSED PER METRIC: an unreadable source yields NULL, never a zero — a
    zero reads as "nobody" and would bake a lie into a chart. Failures print to
    stdout, and a scheduler task's stdout lands in the app log.
  - Listmonk's cached subscriber_count is NOT trusted (it reads 142 for a list
    holding 71): the count comes from paging the subscribers endpoint.
  - Member growth comes from welcome_sent_at, the provisioning timestamp —
    the closest thing the portal stores to a join date. Manual/founder members
    have no welcome email, so backfilled days are a documented LOWER BOUND;
    today's row carries the exact count.
  - Runs as an in-app scheduler task (addons.scheduler, 03:15 UTC) rather than
    an agent cron: member-facing plumbing should not depend on the agent VM.
  - Backfill is idempotent; member growth is reconstructible back to
    September and spend/requests back to the gateway's first request.

Charts land later, on the surfaces, once the shape of the series is reviewed.
2026-10-28 03:52:49 +00:00