Document how the energy/CO2 metrics pipeline works #3

New Issue

2026-10-29T20:11:37Z

ed commented

2026-10-29 20:11:37 +00:00

Context

The portal computes daily energy_wh and co2e_g in metrics_snapshot.py and serves them via /admin/metrics/series and /broker/metrics. The calculation is explained inside the code (comments and docstrings), but not anywhere a reader would normally look first. This issue tracks writing that down. It is not about changing the calculation.

What exists today

metrics_snapshot.py:80-97 — the factor table and its provenance comments (the only real explanation).
app/main.py:3146-3157 — a docstring on metrics_series.
Nothing else.

What's missing

README's Scheduled tasks section says "Two jobs"; metrics_snapshot (CloudronManifest.json:25-28) is not mentioned, and nothing in the README describes energy at all.
No markdown file in the repo mentions energy / CO2 / impact (verified).
The provenance labels (MEASURED / SAME_FAMILY / ESTIMATED) are never exposed — the endpoints return bare numbers, so a consumer cannot tell measured from estimated.
No methodology note, no factor changelog, and no reference to how the factors were fitted (no fitting script or data is checked in).
The tracking issue referenced in code (co-op/support#26) is closed.

Proposed scope

README — document the third scheduled task and add a short "Energy & carbon metrics" subsection linking to a methodology doc.
New doc (e.g. docs/energy-metrics.md, or a section in the docs repo) that explains:
- the input — output tokens (completion_tokens) from LiteLLM's LiteLLM_SpendLogs, keyed on model_group;
- the factor table and its three provenance tiers (measured / same-family / estimated);
- the formulas — Wh per 1,000 output tokens; gCO2e = kWh x grid intensity;
- the proxy fallback used for non-GreenPT providers and what "estimated" concretely means;
- NULL-not-zero semantics and the implied-grid-intensity tripwire;
- the daily schedule and the fact that --backfill rewrites history.
State clearly which parts are measured vs estimated, so downstream charts can abel them.

Non-blocking follow-ups (separate issues)

Expose provenance (source label / factor revision) through the metrics endpoints.
Consider per-provider factors, especially Regolo (renewables).
Persist a measured-vs-proxy split per day so the estimate share is auditable.

Acceptance criteria

README mentions the metrics_snapshot task.
A methodology doc explains collection, calculation, provenance tiers, the fallback, and NULL semantics.
The doc states plainly that non-GreenPT values are estimates.

References

metrics_snapshot.py
app/main.py:3146
CloudronManifest.json:25-28
co-op/support#26 (closed)

## Context The portal computes daily `energy_wh` and `co2e_g` in `metrics_snapshot.py` and serves them via `/admin/metrics/series` and `/broker/metrics`. The calculation is explained inside the code (comments and docstrings), but not anywhere a reader would normally look first. This issue tracks writing that down. It is not about changing the calculation. ## What exists today - `metrics_snapshot.py:80-97` — the factor table and its provenance comments (the only real explanation). - `app/main.py:3146-3157` — a docstring on `metrics_series`. - Nothing else. ## What's missing - README's *Scheduled tasks* section says "Two jobs"; `metrics_snapshot` (`CloudronManifest.json:25-28`) is not mentioned, and nothing in the README describes energy at all. - No markdown file in the repo mentions energy / CO2 / impact (verified). - The provenance labels (`MEASURED` / `SAME_FAMILY` / `ESTIMATED`) are never exposed — the endpoints return bare numbers, so a consumer cannot tell measured from estimated. - No methodology note, no factor changelog, and no reference to how the factors were fitted (no fitting script or data is checked in). - The tracking issue referenced in code (`co-op/support#26`) is closed. ## Proposed scope 1. **README** — document the third scheduled task and add a short "Energy & carbon metrics" subsection linking to a methodology doc. 2. **New doc** (e.g. `docs/energy-metrics.md`, or a section in the docs repo) that explains: - the input — output tokens (`completion_tokens`) from LiteLLM's `LiteLLM_SpendLogs`, keyed on `model_group`; - the factor table and its three provenance tiers (measured / same-family / estimated); - the formulas — Wh per 1,000 output tokens; `gCO2e = kWh x grid intensity`; - the proxy fallback used for non-GreenPT providers and what "estimated" concretely means; - NULL-not-zero semantics and the implied-grid-intensity tripwire; - the daily schedule and the fact that `--backfill` rewrites history. 3. State clearly which parts are measured vs estimated, so downstream charts can abel them. ## Non-blocking follow-ups (separate issues) - Expose provenance (source label / factor revision) through the metrics endpoints. - Consider per-provider factors, especially Regolo (renewables). - Persist a measured-vs-proxy split per day so the estimate share is auditable. ## Acceptance criteria - [ ] README mentions the `metrics_snapshot` task. - [ ] A methodology doc explains collection, calculation, provenance tiers, the fallback, and NULL semantics. - [ ] The doc states plainly that non-GreenPT values are estimates. ## References - `metrics_snapshot.py` - `app/main.py:3146` - `CloudronManifest.json:25-28` - `co-op/support#26` (closed)

inference-bot referenced this issue from a commit

2026-10-29 20:15:14 +00:00

docs: write down how the energy/CO2 metrics actually work (#3)

inference-bot commented

2026-10-29 20:15:27 +00:00

Done — thank you for the write-up. It was a fair complaint: the only real explanation was in code comments, and the README actively undercounted the tasks.

Shipped in b11552f (docs only; no calculation touched):

docs/energy-metrics.md — the methodology doc, covering everything in your proposed scope: the input (completion_tokens, output tokens only, grouped on model_group), the factor table with its three provenance tiers, the formulas, NULL-not-zero, the tripwire, the 03:15 schedule, and that --backfill rewrites history rather than appending.
README — "Two jobs" → "Three", metrics_snapshot documented, and an "Energy & carbon metrics" subsection linking the doc. Also fixed "listing both tasks" in the post-change verification step, which would have been the next wrong sentence.

On your third scope point, which I read as the important one: the doc states plainly what is measured and what is not — GreenPT is measured, glm-5.3 is assumed from its flash sibling, and all Tinfoil and Regolo traffic currently falls to the proxy estimate. Two consequences are written down rather than left for a reader to infer:

a day's figure is a blend, and the split is not recorded per day, so you cannot tell from metrics_daily how much of a total rested on the proxy;
the proxy assumes a 250 gCO2e/kWh grid while Regolo runs on 100% renewable energy, so the proxy overstates Regolo's carbon — Regolo and Tinfoil figures should be read as upper bounds, not measurements.

Two things I did not do, because you filed them as separate follow-ups and I agree they are separate:

Exposing provenance through the endpoints. The doc records that the tiers are never exposed; changing that is an API change, not documentation.
Per-provider factors, and persisting a measured-vs-proxy split per day. Both would change the numbers or the schema.

One correction to the issue's own text, for the record: it lists the tracking issue co-op/support#26 as closed, which is right, but the reference is not dead — it is the thread where these factors were agreed, so the doc points at it rather than removing it.

Two other things the doc fixes while it was open: it records why the tripwire exists (carbon was once understated exactly 1000×, while the energy column stayed plausible — so nothing looked wrong), and it names the model_group-vs-model mistake explicitly, because that one silently matches no factor and produces zeros rather than an error.

I have left the issue open for you to check — close it if the doc answers what you needed, and reopen if anything reads wrong. The member-facing counterpart is co-op/docs/energy.md; the two should not disagree, and the doc says so.

Done — thank you for the write-up. It was a fair complaint: the only real explanation was in code comments, and the README actively undercounted the tasks. **Shipped in `b11552f`** (docs only; no calculation touched): - **`docs/energy-metrics.md`** — the methodology doc, covering everything in your proposed scope: the input (`completion_tokens`, output tokens only, grouped on `model_group`), the factor table with its three provenance tiers, the formulas, NULL-not-zero, the tripwire, the 03:15 schedule, and that `--backfill` **rewrites** history rather than appending. - **README** — "Two jobs" → "Three", `metrics_snapshot` documented, and an "Energy & carbon metrics" subsection linking the doc. Also fixed "listing both tasks" in the post-change verification step, which would have been the next wrong sentence. **On your third scope point, which I read as the important one:** the doc states plainly what is measured and what is not — GreenPT is measured, `glm-5.3` is assumed from its flash sibling, and **all Tinfoil and Regolo traffic currently falls to the proxy estimate**. Two consequences are written down rather than left for a reader to infer: - a day's figure is a blend, and **the split is not recorded per day**, so you cannot tell from `metrics_daily` how much of a total rested on the proxy; - the proxy assumes a 250 gCO2e/kWh grid while Regolo runs on 100% renewable energy, so **the proxy overstates Regolo's carbon** — Regolo and Tinfoil figures should be read as upper bounds, not measurements. **Two things I did not do**, because you filed them as separate follow-ups and I agree they are separate: 1. **Exposing provenance through the endpoints.** The doc records that the tiers are never exposed; changing that is an API change, not documentation. 2. **Per-provider factors, and persisting a measured-vs-proxy split per day.** Both would change the numbers or the schema. **One correction to the issue's own text, for the record:** it lists the tracking issue `co-op/support#26` as closed, which is right, but the reference is not dead — it is the thread where these factors were agreed, so the doc points at it rather than removing it. Two other things the doc fixes while it was open: it records *why* the tripwire exists (carbon was once understated exactly 1000×, while the energy column stayed plausible — so nothing looked wrong), and it names the `model_group`-vs-`model` mistake explicitly, because that one silently matches no factor and produces zeros rather than an error. I have left the issue open for you to check — close it if the doc answers what you needed, and reopen if anything reads wrong. The member-facing counterpart is `co-op/docs/energy.md`; the two should not disagree, and the doc says so.

ed commented

2026-10-29 22:00:19 +00:00

Thanks for providing this documentation!

ed closed this issue

2026-10-29 22:00:19 +00:00

ed referenced this issue

2026-10-30 17:14:31 +00:00

Expose provenance for energy/CO2 metrics through the API #4

ed referenced this issue

2026-10-30 17:14:31 +00:00

Add per-provider impact factors (start with Regolo) #5

ed referenced this issue