metrics: expose + persist measured-vs-estimated provenance (#4, #6)

Thread the per-factor source through impact_for() instead of discarding it.
Numeric output (energy_wh, co2e_g) is byte-identical — this is a reporting
change, not a recalculation.

#6 persistence: additive columns tokens_measured, tokens_estimated and
factor_revision (a stable hash of the factor table) on metrics_daily, added via
the existing guarded ALTER TABLE idiom in both app/main.py and
metrics_snapshot.py. Existing rows keep NULL.

#4 API: metrics_series() selects the new columns and derives a per-row
provenance indicator (measured | mixed | estimated | none) from the split; both
/admin/metrics/series and /broker/metrics return it. NULL/unknown splits report
'none', never a confident 'measured'.

docs/energy-metrics.md documents the indicator, the new columns, and the
forward-only-vs-backfill behaviour.
This commit is contained in:
portal-dev committed 2026-10-30 19:12:20 +00:00
1 parent e5c6b6060e
commit 6475a51067
3 files changed
+236 -25

No files matched your search

+70 -4
View File
@@ -698,12 +698,24 @@ def get_db() -> sqlite3.Connection:
"out_tokens INTEGER, "
"energy_wh REAL, "
"co2e_g REAL, "
"tokens_measured INTEGER, "
"tokens_estimated INTEGER, "
"factor_revision TEXT, "
"recorded_at TEXT"
")"
)
# CREATE IF NOT EXISTS never adds columns to an existing table, so migrate:
# impact columns arrived after the table did (see co-op/support#26).
for _col, _decl in (("out_tokens", "INTEGER"), ("energy_wh", "REAL"), ("co2e_g", "REAL")):
# impact columns arrived after the table did (see co-op/support#26), and the
# measured-vs-estimated provenance columns after those. All additive: existing
# rows keep NULL, and a NULL split reports as unknown — never as "measured".
for _col, _decl in (
("out_tokens", "INTEGER"),
("energy_wh", "REAL"),
("co2e_g", "REAL"),
("tokens_measured", "INTEGER"),
("tokens_estimated", "INTEGER"),
("factor_revision", "TEXT"),
):
try:
conn.execute(f"ALTER TABLE metrics_daily ADD COLUMN {_col} {_decl}")
except sqlite3.OperationalError:
@@ -3165,6 +3177,31 @@ async def admin_overview(request: Request):
}
def _metrics_provenance(tokens_measured: int | None, tokens_estimated: int | None) -> str:
"""Classify a day's measured-vs-estimated split for the API.
"measured" — every token rested on a directly measured factor.
"estimated" — every token rested on a non-measured factor (borrowed
SAME_FAMILY or the flat proxy).
"mixed" — the day's total is a blend of both.
"none" — no provenance recorded: either the day has no output tokens
or the row predates these columns (both split columns NULL).
Deliberately NOT "measured" — an unknown split must never read
as a confident measurement.
"""
if tokens_measured is None and tokens_estimated is None:
return "none"
m = tokens_measured or 0
e = tokens_estimated or 0
if m and e:
return "mixed"
if m:
return "measured"
if e:
return "estimated"
return "none"
def metrics_series(days: int = 90) -> list[dict]:
"""Daily metrics rows, oldest first.
@@ -3176,13 +3213,21 @@ def metrics_series(days: int = 90) -> list[dict]:
energy_wh and co2e_g are computed in metrics_snapshot.py from the day's
output tokens against a factor table: measured for GreenPT models (their API
returns an `impact` object), estimated for the rest. See co-op/support#26.
Each row also carries its provenance: `tokens_measured`/`tokens_estimated`
(the output-token split across measured vs non-measured factors),
`factor_revision` (a short stable hash of the factor table that produced the
row), and a derived `provenance` indicator — measured | mixed | estimated |
none. Rows written before these columns exist have NULL splits and report
`provenance: "none"`, never a confident "measured". See docs/energy-metrics.md.
"""
limit = max(1, min(int(days or 90), 730))
conn = get_db()
try:
rows = conn.execute(
"SELECT day, members_active, members_total, members_new, newsletter_subs, "
"matrix_space_members, spend_usd, requests, out_tokens, energy_wh, co2e_g, recorded_at "
"matrix_space_members, spend_usd, requests, out_tokens, energy_wh, co2e_g, "
"tokens_measured, tokens_estimated, factor_revision, recorded_at "
"FROM metrics_daily ORDER BY day DESC LIMIT ?",
(limit,),
).fetchall()
@@ -3200,9 +3245,19 @@ def metrics_series(days: int = 90) -> list[dict]:
"out_tokens",
"energy_wh",
"co2e_g",
"tokens_measured",
"tokens_estimated",
"factor_revision",
"recorded_at",
]
return [dict(zip(cols, row)) for row in reversed(rows)]
series = []
for row in reversed(rows):
rec = dict(zip(cols, row))
rec["provenance"] = _metrics_provenance(
rec["tokens_measured"], rec["tokens_estimated"]
)
series.append(rec)
return series
@app.get("/admin/metrics/series")
@@ -3219,6 +3274,11 @@ async def admin_metrics_series(request: Request, days: int = 90):
counts as of that day; spend_usd/requests are that day's traffic only.
Backfilled days carry a LOWER BOUND for members_total (members provisioned
without a welcome email are absent from the history).
Each row carries its energy provenance — `provenance` (measured | mixed |
estimated | none) plus `tokens_measured`, `tokens_estimated` and
`factor_revision`. A row written before those columns existed reports
`provenance: "none"`, not "measured". See docs/energy-metrics.md.
"""
_verify_admin_token(request)
series = metrics_series(days)
@@ -3607,6 +3667,12 @@ async def broker_metrics(request: Request, days: int = 90):
it is AGGREGATE and non-personal — co-op size, requests served, spend — the
same class of social-proof data as /broker/model-leaderboard. No per-member
figure passes through here, and members asked to see how the co-op is doing.
Rows carry the energy provenance the admin series does: `provenance`
(measured | mixed | estimated | none) plus `tokens_measured`,
`tokens_estimated` and `factor_revision` — so a member-facing chart can say
how much of a day's figure rested on a measurement. NULL splits (older rows)
report `provenance: "none"`. See docs/energy-metrics.md.
"""
_verify_broker_secret(request)
series = metrics_series(days)