metrics: expose + persist measured-vs-estimated provenance (#4, #6)

Thread the per-factor source through impact_for() instead of discarding it.
Numeric output (energy_wh, co2e_g) is byte-identical — this is a reporting
change, not a recalculation.

#6 persistence: additive columns tokens_measured, tokens_estimated and
factor_revision (a stable hash of the factor table) on metrics_daily, added via
the existing guarded ALTER TABLE idiom in both app/main.py and
metrics_snapshot.py. Existing rows keep NULL.

#4 API: metrics_series() selects the new columns and derives a per-row
provenance indicator (measured | mixed | estimated | none) from the split; both
/admin/metrics/series and /broker/metrics return it. NULL/unknown splits report
'none', never a confident 'measured'.

docs/energy-metrics.md documents the indicator, the new columns, and the
forward-only-vs-backfill behaviour.
This commit is contained in:
portal-dev committed 2026-10-30 19:12:20 +00:00
1 parent e5c6b6060e
commit 6475a51067
3 files changed
+236 -25

No files matched your search

+58 -9
View File
@@ -49,9 +49,55 @@ Factors live in `metrics_snapshot.py` (`IMPACT_FACTORS`, `IMPACT_PROXY`). Matchi
not disclose impact, **deliberately**, so that a day's total is never quietly
missing models. It is not a number we know.
**No tier is exposed anywhere.** The endpoints (`/admin/metrics/series`,
`/broker/metrics`) return bare numbers, so a consumer cannot tell measured from
estimated. Exposing provenance is tracked as a separate issue.
**No tier is exposed on its own**, but every row now carries a *provenance* read
derived from the measured/estimated token split — see "Provenance in the API"
below. A consumer can tell how much of a day's figure rested on a measurement.
## Provenance in the API
Every row returned by `/admin/metrics/series` and `/broker/metrics` carries the
measured-vs-estimated split, so a consumer can tell how much of a day's figure
rested on a real measurement. Four fields:
| field | meaning |
|---|---|
| `tokens_measured` | output tokens that day attributed to a factor whose `source` is **MEASURED** — a direct provider measurement. |
| `tokens_estimated` | the rest: **SAME_FAMILY** (borrowed from a sibling) and the flat **proxy**. A borrowed or proxied factor is an assumption, so it is **not** counted as measured. |
| `factor_revision` | a short, stable hash of the factor table (`IMPACT_FACTORS` + `IMPACT_PROXY`) in effect when the row was computed. |
| `provenance` | the derived read, below. |
`tokens_measured + tokens_estimated = out_tokens` for any day with tokens. A day
with no output tokens has NULL in both (fail-closed, same rule as energy/carbon).
`provenance` is one of:
- **`measured`** — every token rested on a measured factor.
- **`estimated`** — every token rested on a non-measured factor (borrowed or proxy).
- **`mixed`** — both are present; the day's total is a blend.
- **`none`** — no provenance. This is **either** a day with no output tokens **or**
a row written before these columns existed (older rows keep NULL). It is
deliberately **not** `measured`: an unknown split must never read as a confident
measurement. This is the whole point of the indicator — failure reads as unknown.
`factor_revision` is a deterministic hash, **not a timestamp**: two runs against the
same table produce the same string. It changes if any factor value or source label
changes, or if the matching order changes (order is meaningful — longest prefix
first). It is "what the numbers were computed with", not "what was known then".
### Forward-only, then backfill
The columns are added by an **additive** migration — a guarded
`ALTER TABLE metrics_daily ADD COLUMN …`, the same idiom used when the impact
columns first arrived. On an existing database this is safe: it never errors,
existing rows survive untouched, and they simply keep NULL in the new columns.
Nothing is guessed for history on migration; those rows read as `provenance: "none"`.
To give past rows a real split, run `metrics_snapshot.py --backfill` (or re-run the
affected days). Backfill rewrites each day from the current LiteLLM token split and
the **current** factor table, so it also updates `factor_revision` to today's
revision — a backfilled historical row records the revision used to recompute it,
not the one that was current at the time. Honest for a chart label; not a historical
audit trail.
## Formulas
@@ -67,8 +113,8 @@ intended — which is why the tripwire below exists.
## NULL, never zero
`impact_for()` returns `None, None` for a day with no measured tokens, and the row is
written with NULL energy and carbon. It never writes `0.0`:
`impact_for()` returns an all-NULL impact block for a day with no measured tokens,
and the row is written with NULL energy and carbon. It never writes `0.0`:
> a zero reads as "the co-op used no energy", which is a claim we cannot support, and
> it would survive into a chart as a fact.
@@ -107,7 +153,8 @@ than by us.
- It writes **one row per day** into the `metrics_daily` table, in the same SQLite
database as the members table (`members.db`). Columns: `day`, `members_active`,
`members_total`, `members_new`, `newsletter_subs`, `matrix_space_members`,
`spend_usd`, `requests`, `out_tokens`, `energy_wh`, `co2e_g`, `recorded_at`.
`spend_usd`, `requests`, `out_tokens`, `energy_wh`, `co2e_g`, `tokens_measured`,
`tokens_estimated`, `factor_revision`, `recorded_at`.
- **`metrics_snapshot.py --backfill` rebuilds history** from the sources and is
idempotent — it rewrites a day rather than duplicating it. It is the tool to run
after changing a factor, and the reason a published chart can move: history is
@@ -122,9 +169,11 @@ Stated plainly, because a mixed total is easy to over-read:
- **Estimated:** everything else. Currently that means **all Tinfoil and Regolo
traffic**, which is a real share of co-op usage.
So a day's figure is a blend, and **the split is not recorded per day** — you cannot
tell from `metrics_daily` how much of a total rested on the proxy. Persisting that
split is a separate follow-up.
So a day's figure is a blend, and **the split is now recorded per day** —
`tokens_measured` and `tokens_estimated` say how much of the total rested on a
measurement, and the API derives a `provenance` read from them. Before those
columns existed the split was not recorded; such rows carry NULL and read as
`provenance: "none"`. See "Provenance in the API".
One consequence worth knowing: the proxy assumes a 250 gCO2e/kWh grid, while Regolo
runs on 100% renewable energy, so the proxy **overstates** Regolo's carbon. Per-provider