docs: write down how the energy/CO2 metrics actually work (#3)
The calculation lived only in code comments. Adds docs/energy-metrics.md and wires it into the README, per the issue's proposed scope. No calculation changes. Contents: the input (completion_tokens, output tokens only, grouped on model_group) and why output-only understates total compute; the factor table with its three provenance tiers (measured / same-family / estimated) and the fact that the tiers are never exposed; the formulas; NULL-not-zero and the reason (the first backfill claimed every day used no energy); the implied-grid-intensity tripwire and the 1000x carbon error it exists to catch; the 03:15 schedule and that --backfill REWRITES history. States plainly what is measured and what is not: GreenPT is measured, glm-5.3 is assumed from its flash sibling, and all Tinfoil and Regolo traffic currently falls to the proxy estimate — with the consequence that the 250 gCO2e/kWh proxy OVERSTATES Regolo, which runs on renewable energy. README: "Two jobs" -> "Three", the metrics_snapshot task documented, and a short Energy & carbon metrics subsection linking the new doc. Also corrects "listing both tasks" in the post-change verification step.
This commit is contained in:
1 parent
42cf43a125
commit
b11552fa49
2 files changed
+159
-2
No files matched your search
@@ -0,0 +1,139 @@
|
||||
# Energy and carbon metrics
|
||||
|
||||
How the portal's daily `energy_wh` and `co2e_g` figures are produced, what is
|
||||
measured versus estimated, and how to change them. Written because the calculation
|
||||
previously lived only in code comments (see `code/member-portal#3`).
|
||||
|
||||
For what members are told, see `co-op/docs/energy.md`. This document is the
|
||||
developer-facing counterpart and is deliberately blunter about the limits.
|
||||
|
||||
## Where the numbers come from
|
||||
|
||||
**Input: output tokens only, per day, per model.**
|
||||
|
||||
- Source: LiteLLM's spend logs (`LiteLLM_SpendLogs`), column `completion_tokens`.
|
||||
- Grouped on **`model_group`** — our alias, e.g. `greenpt/deepseek-v4.1-flash` —
|
||||
**not** `model`, which carries the upstream name (`openai/deepseek-v4.1-flash`).
|
||||
Grouping on the wrong column matches no factor and yields zeros for every day. That
|
||||
mistake shipped once during development and is the reason this is called out.
|
||||
|
||||
**Only output tokens count**, matching the unit GreenPT publishes (mWh per 100 output
|
||||
tokens). This is a deliberate choice to line up with the provider's own metric, not an
|
||||
oversight — and it means these figures **understate total compute**, because input
|
||||
tokens consume energy too. Treat the numbers as comparable and directionally true
|
||||
rather than as a total energy accounting.
|
||||
|
||||
## The factor table
|
||||
|
||||
Factors live in `metrics_snapshot.py` (`IMPACT_FACTORS`, `IMPACT_PROXY`). Matching is
|
||||
`str.startswith()`, so **the longest prefix must come first** — order matters.
|
||||
|
||||
| model prefix | Wh / 1k output tokens | gCO2e / kWh | provenance |
|
||||
|---|---|---|---|
|
||||
| `greenpt/deepseek-v4.1-flash` | 0.3641 | 193.0 | MEASURED |
|
||||
| `greenpt/glm-5.3-flash` | 1.0782 | 193.0 | MEASURED |
|
||||
| `greenpt/glm-5.3` | 1.0782 | 193.0 | SAME_FAMILY |
|
||||
| `greenpt/green-r` | 0.5535 | 23.6 | MEASURED |
|
||||
| *anything else* | 0.5000 | 250.0 | ESTIMATED |
|
||||
|
||||
### The three provenance tiers
|
||||
|
||||
- **MEASURED** — fitted from live calls to GreenPT's API, which returns an `impact`
|
||||
object on every request (energy in watt-milliseconds, emissions in µgCO2e). Energy
|
||||
was regressed on output tokens across prompt sizes (R² ≥ 0.999); grid intensity is
|
||||
what their own emissions figures imply. Fitted 2026-10-28.
|
||||
- **SAME_FAMILY** — measured for `glm-5.3-flash` and assumed to hold for `glm-5.3`.
|
||||
An assumption of similarity, not a measurement.
|
||||
- **ESTIMATED** — the proxy row: an order-of-magnitude figure near the middle of the
|
||||
measured range, with an EU grid average. It is applied to every provider that does
|
||||
not disclose impact, **deliberately**, so that a day's total is never quietly
|
||||
missing models. It is not a number we know.
|
||||
|
||||
**No tier is exposed anywhere.** The endpoints (`/admin/metrics/series`,
|
||||
`/broker/metrics`) return bare numbers, so a consumer cannot tell measured from
|
||||
estimated. Exposing provenance is tracked as a separate issue.
|
||||
|
||||
## Formulas
|
||||
|
||||
```
|
||||
energy_wh = Σ_models ( out_tokens / 1000 × wh_per_1k_out )
|
||||
|
||||
co2e_g = Σ_models ( out_tokens / 1000 × wh_per_1k_out / 1000 × gco2e_per_kwh )
|
||||
└────────────── Wh ──────────────┘ └─ Wh → kWh ─┘
|
||||
```
|
||||
|
||||
Carbon was once understated by exactly 1000× here — a `/1e6` where `/1e3` was
|
||||
intended — which is why the tripwire below exists.
|
||||
|
||||
## NULL, never zero
|
||||
|
||||
`impact_for()` returns `None, None` for a day with no measured tokens, and the row is
|
||||
written with NULL energy and carbon. It never writes `0.0`:
|
||||
|
||||
> a zero reads as "the co-op used no energy", which is a claim we cannot support, and
|
||||
> it would survive into a chart as a fact.
|
||||
|
||||
The first backfill shipped exactly that bug — it summed the wrong column, every model
|
||||
totalled zero, and every day in history claimed to have used no energy.
|
||||
|
||||
A NULL day renders as a **bare gap in the charts, with nothing explaining it.** That is
|
||||
deliberate: the surfaces carry no caveat text. Accepted knowingly.
|
||||
|
||||
The same fail-closed rule applies to every metric in the snapshot, not only energy:
|
||||
an unreadable source (Listmonk, Matrix, LiteLLM) yields NULL for that metric and
|
||||
prints to the log, rather than a number that looks real.
|
||||
|
||||
## The tripwire
|
||||
|
||||
After summing, `impact_for()` computes the grid intensity its own mix implies and
|
||||
compares it against a plausible band:
|
||||
|
||||
```
|
||||
implied = co2e_g / energy_wh × 1000 # gCO2e per kWh
|
||||
if not (5.0 <= implied <= 500.0):
|
||||
warn "suspect a unit error before trusting these numbers"
|
||||
```
|
||||
|
||||
It warns; it never alters the numbers or suppresses the row. Its purpose is narrow and
|
||||
earned: the 1000× carbon error kept the energy column plausible, so nothing looked
|
||||
wrong, and a silent error of that kind would otherwise be found by a member rather
|
||||
than by us.
|
||||
|
||||
## Schedule and backfill
|
||||
|
||||
- **`metrics_snapshot` runs daily at 03:15**, from Cloudron's scheduler addon
|
||||
(`CloudronManifest.json` → `addons.scheduler`), via `/app/code/metrics_snapshot.py`.
|
||||
It runs after `membership_sweep` (03:00) and `rag_refresh` (03:30).
|
||||
- It writes **one row per day** into the `metrics_daily` table, in the same SQLite
|
||||
database as the members table (`members.db`). Columns: `day`, `members_active`,
|
||||
`members_total`, `members_new`, `newsletter_subs`, `matrix_space_members`,
|
||||
`spend_usd`, `requests`, `out_tokens`, `energy_wh`, `co2e_g`, `recorded_at`.
|
||||
- **`metrics_snapshot.py --backfill` rebuilds history** from the sources and is
|
||||
idempotent — it rewrites a day rather than duplicating it. It is the tool to run
|
||||
after changing a factor, and the reason a published chart can move: history is
|
||||
**rewritten**, not appended to.
|
||||
|
||||
## What is measured, and what is not
|
||||
|
||||
Stated plainly, because a mixed total is easy to over-read:
|
||||
|
||||
- **Measured:** GreenPT models — the providers whose API reports impact.
|
||||
- **Assumed equal to a measurement:** `greenpt/glm-5.3`, from its flash sibling.
|
||||
- **Estimated:** everything else. Currently that means **all Tinfoil and Regolo
|
||||
traffic**, which is a real share of co-op usage.
|
||||
|
||||
So a day's figure is a blend, and **the split is not recorded per day** — you cannot
|
||||
tell from `metrics_daily` how much of a total rested on the proxy. Persisting that
|
||||
split is a separate follow-up.
|
||||
|
||||
One consequence worth knowing: the proxy assumes a 250 gCO2e/kWh grid, while Regolo
|
||||
runs on 100% renewable energy, so the proxy **overstates** Regolo's carbon. Per-provider
|
||||
factors are a follow-up; until then, Regolo and Tinfoil figures should be read as upper
|
||||
bounds rather than measurements.
|
||||
|
||||
## Changing the factors
|
||||
|
||||
Edit the table in `metrics_snapshot.py` and redeploy — refining a factor is a data
|
||||
edit, not a code change. Then run `--backfill` if the change should apply to history.
|
||||
The energy factors are also recorded in `co-op/docs/energy.md` for members, and the two
|
||||
should not disagree.
|
||||
Reference in new issue
Block a user