Thread the per-factor source through impact_for() instead of discarding it. Numeric output (energy_wh, co2e_g) is byte-identical — this is a reporting change, not a recalculation. #6 persistence: additive columns tokens_measured, tokens_estimated and factor_revision (a stable hash of the factor table) on metrics_daily, added via the existing guarded ALTER TABLE idiom in both app/main.py and metrics_snapshot.py. Existing rows keep NULL. #4 API: metrics_series() selects the new columns and derives a per-row provenance indicator (measured | mixed | estimated | none) from the split; both /admin/metrics/series and /broker/metrics return it. NULL/unknown splits report 'none', never a confident 'measured'. docs/energy-metrics.md documents the indicator, the new columns, and the forward-only-vs-backfill behaviour.
This commit is contained in:
1 parent
e5c6b6060e
commit
6475a51067
3 files changed
+236
-25
No files matched your search
+58
-9
@@ -49,9 +49,55 @@ Factors live in `metrics_snapshot.py` (`IMPACT_FACTORS`, `IMPACT_PROXY`). Matchi
|
||||
not disclose impact, **deliberately**, so that a day's total is never quietly
|
||||
missing models. It is not a number we know.
|
||||
|
||||
**No tier is exposed anywhere.** The endpoints (`/admin/metrics/series`,
|
||||
`/broker/metrics`) return bare numbers, so a consumer cannot tell measured from
|
||||
estimated. Exposing provenance is tracked as a separate issue.
|
||||
**No tier is exposed on its own**, but every row now carries a *provenance* read
|
||||
derived from the measured/estimated token split — see "Provenance in the API"
|
||||
below. A consumer can tell how much of a day's figure rested on a measurement.
|
||||
|
||||
## Provenance in the API
|
||||
|
||||
Every row returned by `/admin/metrics/series` and `/broker/metrics` carries the
|
||||
measured-vs-estimated split, so a consumer can tell how much of a day's figure
|
||||
rested on a real measurement. Four fields:
|
||||
|
||||
| field | meaning |
|
||||
|---|---|
|
||||
| `tokens_measured` | output tokens that day attributed to a factor whose `source` is **MEASURED** — a direct provider measurement. |
|
||||
| `tokens_estimated` | the rest: **SAME_FAMILY** (borrowed from a sibling) and the flat **proxy**. A borrowed or proxied factor is an assumption, so it is **not** counted as measured. |
|
||||
| `factor_revision` | a short, stable hash of the factor table (`IMPACT_FACTORS` + `IMPACT_PROXY`) in effect when the row was computed. |
|
||||
| `provenance` | the derived read, below. |
|
||||
|
||||
`tokens_measured + tokens_estimated = out_tokens` for any day with tokens. A day
|
||||
with no output tokens has NULL in both (fail-closed, same rule as energy/carbon).
|
||||
|
||||
`provenance` is one of:
|
||||
|
||||
- **`measured`** — every token rested on a measured factor.
|
||||
- **`estimated`** — every token rested on a non-measured factor (borrowed or proxy).
|
||||
- **`mixed`** — both are present; the day's total is a blend.
|
||||
- **`none`** — no provenance. This is **either** a day with no output tokens **or**
|
||||
a row written before these columns existed (older rows keep NULL). It is
|
||||
deliberately **not** `measured`: an unknown split must never read as a confident
|
||||
measurement. This is the whole point of the indicator — failure reads as unknown.
|
||||
|
||||
`factor_revision` is a deterministic hash, **not a timestamp**: two runs against the
|
||||
same table produce the same string. It changes if any factor value or source label
|
||||
changes, or if the matching order changes (order is meaningful — longest prefix
|
||||
first). It is "what the numbers were computed with", not "what was known then".
|
||||
|
||||
### Forward-only, then backfill
|
||||
|
||||
The columns are added by an **additive** migration — a guarded
|
||||
`ALTER TABLE metrics_daily ADD COLUMN …`, the same idiom used when the impact
|
||||
columns first arrived. On an existing database this is safe: it never errors,
|
||||
existing rows survive untouched, and they simply keep NULL in the new columns.
|
||||
Nothing is guessed for history on migration; those rows read as `provenance: "none"`.
|
||||
|
||||
To give past rows a real split, run `metrics_snapshot.py --backfill` (or re-run the
|
||||
affected days). Backfill rewrites each day from the current LiteLLM token split and
|
||||
the **current** factor table, so it also updates `factor_revision` to today's
|
||||
revision — a backfilled historical row records the revision used to recompute it,
|
||||
not the one that was current at the time. Honest for a chart label; not a historical
|
||||
audit trail.
|
||||
|
||||
## Formulas
|
||||
|
||||
@@ -67,8 +113,8 @@ intended — which is why the tripwire below exists.
|
||||
|
||||
## NULL, never zero
|
||||
|
||||
`impact_for()` returns `None, None` for a day with no measured tokens, and the row is
|
||||
written with NULL energy and carbon. It never writes `0.0`:
|
||||
`impact_for()` returns an all-NULL impact block for a day with no measured tokens,
|
||||
and the row is written with NULL energy and carbon. It never writes `0.0`:
|
||||
|
||||
> a zero reads as "the co-op used no energy", which is a claim we cannot support, and
|
||||
> it would survive into a chart as a fact.
|
||||
@@ -107,7 +153,8 @@ than by us.
|
||||
- It writes **one row per day** into the `metrics_daily` table, in the same SQLite
|
||||
database as the members table (`members.db`). Columns: `day`, `members_active`,
|
||||
`members_total`, `members_new`, `newsletter_subs`, `matrix_space_members`,
|
||||
`spend_usd`, `requests`, `out_tokens`, `energy_wh`, `co2e_g`, `recorded_at`.
|
||||
`spend_usd`, `requests`, `out_tokens`, `energy_wh`, `co2e_g`, `tokens_measured`,
|
||||
`tokens_estimated`, `factor_revision`, `recorded_at`.
|
||||
- **`metrics_snapshot.py --backfill` rebuilds history** from the sources and is
|
||||
idempotent — it rewrites a day rather than duplicating it. It is the tool to run
|
||||
after changing a factor, and the reason a published chart can move: history is
|
||||
@@ -122,9 +169,11 @@ Stated plainly, because a mixed total is easy to over-read:
|
||||
- **Estimated:** everything else. Currently that means **all Tinfoil and Regolo
|
||||
traffic**, which is a real share of co-op usage.
|
||||
|
||||
So a day's figure is a blend, and **the split is not recorded per day** — you cannot
|
||||
tell from `metrics_daily` how much of a total rested on the proxy. Persisting that
|
||||
split is a separate follow-up.
|
||||
So a day's figure is a blend, and **the split is now recorded per day** —
|
||||
`tokens_measured` and `tokens_estimated` say how much of the total rested on a
|
||||
measurement, and the API derives a `provenance` read from them. Before those
|
||||
columns existed the split was not recorded; such rows carry NULL and read as
|
||||
`provenance: "none"`. See "Provenance in the API".
|
||||
|
||||
One consequence worth knowing: the proxy assumes a 250 gCO2e/kWh grid, while Regolo
|
||||
runs on 100% renewable energy, so the proxy **overstates** Regolo's carbon. Per-provider
|
||||
|
||||
Reference in new issue
Block a user