Persist the measured-vs-proxy split for daily energy metrics #6

New Issue

2026-10-30T17:09:10Z

ed commented

2026-10-30 17:09:10 +00:00

Context

A day's energy/CO2 total is computed by summing across all models and then
storing only the aggregate:

impact_for() loops over every model and returns a single (wh, co2) tuple
(metrics_snapshot.py:317-358).
metrics_daily stores out_tokens, energy_wh, co2e_g — one row per day,
no breakdown (app/main.py:689-710).
tokens_by_day_model() is recomputed from LiteLLM on every run and never
persisted (metrics_snapshot.py:266-314).

So the database keeps the derived total but discards the inputs to it.

Problem

From the stored row alone we cannot tell:

how many tokens were measured (GreenPT factors) vs estimated
(IMPACT_PROXY);
which model or provider contributed what;
which factor revision produced the number.

Consequences:

The estimate share of a day is invisible, so a chart can be mostly placeholder
without saying so.
If the factor table changes and --backfill is run, energy_wh / co2e_g are
overwritten with no record that the underlying factor changed
(metrics_snapshot.py:434-489), and there is no way to recompute the old value.

Proposed scope

Add columns to metrics_daily to record the split, e.g.
out_tokens_measured and out_tokens_estimated (or a per-provider breakdown),
plus a factor_revision identifier.
Populate them in impact_for() / snapshot() / backfill(), mirroring the
existing out_tokens handling.
Use the established migration pattern: ALTER TABLE ... ADD COLUMN guarded by
try/except (app/main.py:706-710, metrics_snapshot.py:66-70).
Decide and document whether factor changes are forward-only or retroactive via
--backfill, and store the revision so either choice is auditable.

Acceptance criteria

Each row records the measured vs estimated token split (or per-provider
breakdown) and the factor revision used.
The migration is additive and safe on existing databases.
The metrics documentation explains the new columns and the
forward-only-vs-backfill behaviour.

References

metrics_snapshot.py:266-314, :317-358, :434-489
app/main.py:689-710

## Context A day's energy/CO2 total is computed by summing across all models and then storing only the aggregate: - `impact_for()` loops over every model and returns a single `(wh, co2)` tuple (`metrics_snapshot.py:317-358`). - `metrics_daily` stores `out_tokens`, `energy_wh`, `co2e_g` — one row per day, no breakdown (`app/main.py:689-710`). - `tokens_by_day_model()` is recomputed from LiteLLM on every run and never persisted (`metrics_snapshot.py:266-314`). So the database keeps the derived total but discards the inputs to it. ## Problem From the stored row alone we cannot tell: - how many tokens were measured (GreenPT factors) vs estimated (`IMPACT_PROXY`); - which model or provider contributed what; - which factor revision produced the number. Consequences: - The estimate share of a day is invisible, so a chart can be mostly placeholder without saying so. - If the factor table changes and `--backfill` is run, `energy_wh` / `co2e_g` are overwritten with no record that the underlying factor changed (`metrics_snapshot.py:434-489`), and there is no way to recompute the old value. ## Proposed scope - Add columns to `metrics_daily` to record the split, e.g. `out_tokens_measured` and `out_tokens_estimated` (or a per-provider breakdown), plus a `factor_revision` identifier. - Populate them in `impact_for()` / `snapshot()` / `backfill()`, mirroring the existing `out_tokens` handling. - Use the established migration pattern: `ALTER TABLE ... ADD COLUMN` guarded by `try/except` (`app/main.py:706-710`, `metrics_snapshot.py:66-70`). - Decide and document whether factor changes are forward-only or retroactive via `--backfill`, and store the revision so either choice is auditable. ## Acceptance criteria - [ ] Each row records the measured vs estimated token split (or per-provider breakdown) and the factor revision used. - [ ] The migration is additive and safe on existing databases. - [ ] The metrics documentation explains the new columns and the forward-only-vs-backfill behaviour. ## References - `metrics_snapshot.py:266-314`, `:317-358`, `:434-489` - `app/main.py:689-710`

ed commented

2026-10-30 17:14:32 +00:00

Status after b11552f (the docs commit for #3): partly covered by documentation, substance still open.

Covered: the dev doc explains the backfill behaviour ("history is rewritten, not appended to") and states plainly that the split is not recorded per day — the behaviour half of the third criterion.
Open: metrics_daily still has no measured/estimated split columns and no factor revision; the migration list is unchanged. The doc describes the forward-only-vs-retroactive choice but does not decide it.

Suggest keeping open — the new columns are the point of this issue.

Status after `b11552f` (the docs commit for #3): partly covered by documentation, substance still open. - Covered: the dev doc explains the backfill behaviour ("history is **rewritten**, not appended to") and states plainly that the split is not recorded per day — the behaviour half of the third criterion. - Open: `metrics_daily` still has no measured/estimated split columns and no factor revision; the migration list is unchanged. The doc describes the forward-only-vs-retroactive choice but does not decide it. Suggest keeping open — the new columns are the point of this issue.

inference-bot referenced this issue from a commit

2026-10-30 19:22:08 +00:00

metrics: expose + persist measured-vs-estimated provenance (#4, #6)

inference-bot commented

2026-10-30 19:24:35 +00:00

Done — shipped in 6475a51 (image 20261030-6475a51, deployed and verified live).

Acceptance criteria

Each row records the measured vs estimated token split and the factor revision — tokens_measured, tokens_estimated, factor_revision (a stable short hash of the factor table, order-sensitive, no timestamps).
Additive and safe on existing databases — guarded ALTER TABLE ... ADD COLUMN in both metrics_snapshot.py and app/main.py; the live database migrated in place and all pre-existing columns and rows survived.
Docs explain the new columns and the forward-only-vs-backfill behaviour — "Forward-only, then backfill" section documents that history stays NULL until --backfill runs, and that a backfill rewrites factor_revision to today's.

The check I'd want if I were you: the arithmetic did not move. I captured the values the old code had written, deployed, re-ran the snapshot, and compared:

day          field          before            after   same
2026-10-28   out_tokens     533066            533066  True
2026-10-28   energy_wh      218.102129        218.102129  True
2026-10-28   co2e_g         43.324056         43.324056  True
2026-10-29   out_tokens     1864676           1864676  True
2026-10-29   energy_wh      793.543927        793.543927  True
2026-10-29   co2e_g         167.074746        167.074746  True

Byte-identical on closed days, which is the real proof — this is a reporting change, not a recalculation. 2026-10-30 did change (312,025 → 2,013,649 tokens), and that is the day still accumulating between the two runs, not a different computation.

Still open from your set: #5, the per-provider factors. Deliberately untouched — that one needs a stated method per provider rather than a code edit, and I'd rather not invent a Regolar factor.

Done — shipped in `6475a51` (image `20261030-6475a51`, deployed and verified live). **Acceptance criteria** - [x] Each row records the measured vs estimated token split and the factor revision — `tokens_measured`, `tokens_estimated`, `factor_revision` (a stable short hash of the factor table, order-sensitive, no timestamps). - [x] Additive and safe on existing databases — guarded `ALTER TABLE ... ADD COLUMN` in both `metrics_snapshot.py` and `app/main.py`; the live database migrated in place and all pre-existing columns and rows survived. - [x] Docs explain the new columns and the forward-only-vs-backfill behaviour — "Forward-only, then backfill" section documents that history stays NULL until `--backfill` runs, and that a backfill rewrites `factor_revision` to today's. **The check I'd want if I were you: the arithmetic did not move.** I captured the values the old code had written, deployed, re-ran the snapshot, and compared: ``` day field before after same 2026-10-28 out_tokens 533066 533066 True 2026-10-28 energy_wh 218.102129 218.102129 True 2026-10-28 co2e_g 43.324056 43.324056 True 2026-10-29 out_tokens 1864676 1864676 True 2026-10-29 energy_wh 793.543927 793.543927 True 2026-10-29 co2e_g 167.074746 167.074746 True ``` Byte-identical on closed days, which is the real proof — this is a reporting change, not a recalculation. `2026-10-30` *did* change (312,025 → 2,013,649 tokens), and that is the day still accumulating between the two runs, not a different computation. Still open from your set: #5, the per-provider factors. Deliberately untouched — that one needs a stated method per provider rather than a code edit, and I'd rather not invent a Regolar factor.

inference-bot closed this issue

2026-10-30 19:24:36 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: code/member-portal#6