Expose provenance for energy/CO2 metrics through the API #4

New Issue

2026-10-30T17:09:09Z

ed commented

2026-10-30 17:09:09 +00:00

Context

metrics_snapshot.py already classifies every factor by provenance
(metrics_snapshot.py:95-97):

MEASURED — fitted from GreenPT's disclosed impact values
SAME_FAMILY — borrowed from a sibling model
ESTIMATED — the IMPACT_PROXY order-of-magnitude fallback

That label is stored per factor (metrics_snapshot.py:101-106) but never used.
It is dropped before anything is written or served:

impact_for() returns only (wh, co2) (metrics_snapshot.py:317-358), so the
per-factor source is discarded during aggregation.
metrics_daily has no provenance column (app/main.py:689-710).
metrics_series() returns a fixed column list with no provenance
(app/main.py:3146-3183).
Both /admin/metrics/series (app/main.py:3186) and /broker/metrics
(app/main.py:3579) therefore return energy_wh / co2e_g as bare numbers.

So a dashboard cannot tell how much of a given day's figure is measured versus a
placeholder, even though the code knows.

Problem

A day mixes models with different provenance. Summing them into one unlabelled
number makes an estimate look as authoritative as a measurement. As non-GreenPT
traffic grows, the series is increasingly proxy-derived with no visible signal.

Proposed scope

Define what provenance means at day granularity (a day mixes models).
Suggested: a per-day estimated_tokens_share (fraction of output tokens that
fell through to IMPACT_PROXY), and/or a status enum such as
measured / partial / estimated.
Persist it (see the related "persist measured-vs-proxy split" issue) and add it
to metrics_series() output and both endpoints.
Document the field so consumers know to render/shade it.

Acceptance criteria

Each metrics row carries a provenance indicator distinguishing measured
from estimated.
metrics_series() and both endpoints expose it.
The metrics documentation describes how to interpret it.

References

metrics_snapshot.py:95-106, :317-358
app/main.py:3146-3183, :3186, :3579

## Context `metrics_snapshot.py` already classifies every factor by provenance (`metrics_snapshot.py:95-97`): - `MEASURED` — fitted from GreenPT's disclosed `impact` values - `SAME_FAMILY` — borrowed from a sibling model - `ESTIMATED` — the `IMPACT_PROXY` order-of-magnitude fallback That label is stored per factor (`metrics_snapshot.py:101-106`) but never used. It is dropped before anything is written or served: - `impact_for()` returns only `(wh, co2)` (`metrics_snapshot.py:317-358`), so the per-factor `source` is discarded during aggregation. - `metrics_daily` has no provenance column (`app/main.py:689-710`). - `metrics_series()` returns a fixed column list with no provenance (`app/main.py:3146-3183`). - Both `/admin/metrics/series` (`app/main.py:3186`) and `/broker/metrics` (`app/main.py:3579`) therefore return `energy_wh` / `co2e_g` as bare numbers. So a dashboard cannot tell how much of a given day's figure is measured versus a placeholder, even though the code knows. ## Problem A day mixes models with different provenance. Summing them into one unlabelled number makes an estimate look as authoritative as a measurement. As non-GreenPT traffic grows, the series is increasingly proxy-derived with no visible signal. ## Proposed scope - Define what provenance means at **day granularity** (a day mixes models). Suggested: a per-day `estimated_tokens_share` (fraction of output tokens that fell through to `IMPACT_PROXY`), and/or a status enum such as `measured` / `partial` / `estimated`. - Persist it (see the related "persist measured-vs-proxy split" issue) and add it to `metrics_series()` output and both endpoints. - Document the field so consumers know to render/shade it. ## Acceptance criteria - [ ] Each metrics row carries a provenance indicator distinguishing measured from estimated. - [ ] `metrics_series()` and both endpoints expose it. - [ ] The metrics documentation describes how to interpret it. ## References - `metrics_snapshot.py:95-106`, `:317-358` - `app/main.py:3146-3183`, `:3186`, `:3579`

ed commented

2026-10-30 17:14:31 +00:00

Status after b11552f (the docs commit for #3): deliverables here are unchanged.

The shipped doc docs/energy-metrics.md now records that provenance is not exposed ("No tier is exposed anywhere") and defers it to this issue — so the gap is documented, not fixed.
No acceptance criterion is met: no per-row indicator, and nothing added to metrics_series() or either endpoint.

Still open. One note for the implementation: co-op/docs/energy.md states GreenPT's energy is itself derived from inference time, not a per-call meter reading — worth reflecting in whatever tier labels get exposed.

Status after `b11552f` (the docs commit for #3): deliverables here are unchanged. - The shipped doc `docs/energy-metrics.md` now records that provenance is **not** exposed ("No tier is exposed anywhere") and defers it to this issue — so the gap is documented, not fixed. - No acceptance criterion is met: no per-row indicator, and nothing added to `metrics_series()` or either endpoint. Still open. One note for the implementation: `co-op/docs/energy.md` states GreenPT's energy is itself derived from inference *time*, not a per-call meter reading — worth reflecting in whatever tier labels get exposed.

inference-bot referenced this issue from a commit

2026-10-30 19:22:08 +00:00

metrics: expose + persist measured-vs-estimated provenance (#4, #6)

inference-bot commented

2026-10-30 19:24:33 +00:00

Done — shipped in 6475a51 (image 20261030-6475a51, deployed and verified live).

Acceptance criteria

Each metrics row carries a provenance indicator — metrics_series() derives provenance (measured | mixed | estimated | none) from the measured/estimated token split.
metrics_series() and both endpoints expose it — verified live on both. /admin/metrics/series returns it directly; /broker/metrics carries it with the columns documented in the route docstring.
The docs describe how to interpret it — new "Provenance in the API" section in docs/energy-metrics.md.

Live rows, right now: 2026-10-30 reads provenance: "mixed", tokens_measured: 1,428,402, tokens_estimated: 585,247, factor_revision: 2f6d7e3b (the split sums to out_tokens). 2026-10-28 and 10-29 read provenance: "none" — they predate the columns, so they report unknown rather than claiming measured. That was the deliberate choice: a day whose provenance is unknown must not read as a confident label.

One judgement call worth your review: SAME_FAMILY counts as not measured — a borrowed value is an assumption, not a measurement. If you'd rather it counted as measured-with-caveat, that's a one-line change and I'd want your reasoning first.

Done — shipped in `6475a51` (image `20261030-6475a51`, deployed and verified live). **Acceptance criteria** - [x] Each metrics row carries a provenance indicator — `metrics_series()` derives `provenance` (`measured` | `mixed` | `estimated` | `none`) from the measured/estimated token split. - [x] `metrics_series()` and both endpoints expose it — verified live on both. `/admin/metrics/series` returns it directly; `/broker/metrics` carries it with the columns documented in the route docstring. - [x] The docs describe how to interpret it — new "Provenance in the API" section in `docs/energy-metrics.md`. **Live rows, right now:** `2026-10-30` reads `provenance: "mixed"`, `tokens_measured: 1,428,402`, `tokens_estimated: 585,247`, `factor_revision: 2f6d7e3b` (the split sums to `out_tokens`). `2026-10-28` and `10-29` read `provenance: "none"` — they predate the columns, so they report unknown rather than claiming `measured`. That was the deliberate choice: a day whose provenance is unknown must not read as a confident label. **One judgement call worth your review:** `SAME_FAMILY` counts as *not measured* — a borrowed value is an assumption, not a measurement. If you'd rather it counted as measured-with-caveat, that's a one-line change and I'd want your reasoning first.

inference-bot closed this issue

2026-10-30 19:24:34 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: code/member-portal#4