inference-bot
b11552fa49
The calculation lived only in code comments. Adds docs/energy-metrics.md and wires it into the README, per the issue's proposed scope. No calculation changes. Contents: the input (completion_tokens, output tokens only, grouped on model_group) and why output-only understates total compute; the factor table with its three provenance tiers (measured / same-family / estimated) and the fact that the tiers are never exposed; the formulas; NULL-not-zero and the reason (the first backfill claimed every day used no energy); the implied-grid-intensity tripwire and the 1000x carbon error it exists to catch; the 03:15 schedule and that --backfill REWRITES history. States plainly what is measured and what is not: GreenPT is measured, glm-5.3 is assumed from its flash sibling, and all Tinfoil and Regolo traffic currently falls to the proxy estimate — with the consequence that the 250 gCO2e/kWh proxy OVERSTATES Regolo, which runs on renewable energy. README: "Two jobs" -> "Three", the metrics_snapshot task documented, and a short Energy & carbon metrics subsection linking the new doc. Also corrects "listing both tasks" in the post-change verification step.
140 lines
6.6 KiB
Markdown
140 lines
6.6 KiB
Markdown
# Energy and carbon metrics
|
||
|
||
How the portal's daily `energy_wh` and `co2e_g` figures are produced, what is
|
||
measured versus estimated, and how to change them. Written because the calculation
|
||
previously lived only in code comments (see `code/member-portal#3`).
|
||
|
||
For what members are told, see `co-op/docs/energy.md`. This document is the
|
||
developer-facing counterpart and is deliberately blunter about the limits.
|
||
|
||
## Where the numbers come from
|
||
|
||
**Input: output tokens only, per day, per model.**
|
||
|
||
- Source: LiteLLM's spend logs (`LiteLLM_SpendLogs`), column `completion_tokens`.
|
||
- Grouped on **`model_group`** — our alias, e.g. `greenpt/deepseek-v4.1-flash` —
|
||
**not** `model`, which carries the upstream name (`openai/deepseek-v4.1-flash`).
|
||
Grouping on the wrong column matches no factor and yields zeros for every day. That
|
||
mistake shipped once during development and is the reason this is called out.
|
||
|
||
**Only output tokens count**, matching the unit GreenPT publishes (mWh per 100 output
|
||
tokens). This is a deliberate choice to line up with the provider's own metric, not an
|
||
oversight — and it means these figures **understate total compute**, because input
|
||
tokens consume energy too. Treat the numbers as comparable and directionally true
|
||
rather than as a total energy accounting.
|
||
|
||
## The factor table
|
||
|
||
Factors live in `metrics_snapshot.py` (`IMPACT_FACTORS`, `IMPACT_PROXY`). Matching is
|
||
`str.startswith()`, so **the longest prefix must come first** — order matters.
|
||
|
||
| model prefix | Wh / 1k output tokens | gCO2e / kWh | provenance |
|
||
|---|---|---|---|
|
||
| `greenpt/deepseek-v4.1-flash` | 0.3641 | 193.0 | MEASURED |
|
||
| `greenpt/glm-5.3-flash` | 1.0782 | 193.0 | MEASURED |
|
||
| `greenpt/glm-5.3` | 1.0782 | 193.0 | SAME_FAMILY |
|
||
| `greenpt/green-r` | 0.5535 | 23.6 | MEASURED |
|
||
| *anything else* | 0.5000 | 250.0 | ESTIMATED |
|
||
|
||
### The three provenance tiers
|
||
|
||
- **MEASURED** — fitted from live calls to GreenPT's API, which returns an `impact`
|
||
object on every request (energy in watt-milliseconds, emissions in µgCO2e). Energy
|
||
was regressed on output tokens across prompt sizes (R² ≥ 0.999); grid intensity is
|
||
what their own emissions figures imply. Fitted 2026-10-28.
|
||
- **SAME_FAMILY** — measured for `glm-5.3-flash` and assumed to hold for `glm-5.3`.
|
||
An assumption of similarity, not a measurement.
|
||
- **ESTIMATED** — the proxy row: an order-of-magnitude figure near the middle of the
|
||
measured range, with an EU grid average. It is applied to every provider that does
|
||
not disclose impact, **deliberately**, so that a day's total is never quietly
|
||
missing models. It is not a number we know.
|
||
|
||
**No tier is exposed anywhere.** The endpoints (`/admin/metrics/series`,
|
||
`/broker/metrics`) return bare numbers, so a consumer cannot tell measured from
|
||
estimated. Exposing provenance is tracked as a separate issue.
|
||
|
||
## Formulas
|
||
|
||
```
|
||
energy_wh = Σ_models ( out_tokens / 1000 × wh_per_1k_out )
|
||
|
||
co2e_g = Σ_models ( out_tokens / 1000 × wh_per_1k_out / 1000 × gco2e_per_kwh )
|
||
└────────────── Wh ──────────────┘ └─ Wh → kWh ─┘
|
||
```
|
||
|
||
Carbon was once understated by exactly 1000× here — a `/1e6` where `/1e3` was
|
||
intended — which is why the tripwire below exists.
|
||
|
||
## NULL, never zero
|
||
|
||
`impact_for()` returns `None, None` for a day with no measured tokens, and the row is
|
||
written with NULL energy and carbon. It never writes `0.0`:
|
||
|
||
> a zero reads as "the co-op used no energy", which is a claim we cannot support, and
|
||
> it would survive into a chart as a fact.
|
||
|
||
The first backfill shipped exactly that bug — it summed the wrong column, every model
|
||
totalled zero, and every day in history claimed to have used no energy.
|
||
|
||
A NULL day renders as a **bare gap in the charts, with nothing explaining it.** That is
|
||
deliberate: the surfaces carry no caveat text. Accepted knowingly.
|
||
|
||
The same fail-closed rule applies to every metric in the snapshot, not only energy:
|
||
an unreadable source (Listmonk, Matrix, LiteLLM) yields NULL for that metric and
|
||
prints to the log, rather than a number that looks real.
|
||
|
||
## The tripwire
|
||
|
||
After summing, `impact_for()` computes the grid intensity its own mix implies and
|
||
compares it against a plausible band:
|
||
|
||
```
|
||
implied = co2e_g / energy_wh × 1000 # gCO2e per kWh
|
||
if not (5.0 <= implied <= 500.0):
|
||
warn "suspect a unit error before trusting these numbers"
|
||
```
|
||
|
||
It warns; it never alters the numbers or suppresses the row. Its purpose is narrow and
|
||
earned: the 1000× carbon error kept the energy column plausible, so nothing looked
|
||
wrong, and a silent error of that kind would otherwise be found by a member rather
|
||
than by us.
|
||
|
||
## Schedule and backfill
|
||
|
||
- **`metrics_snapshot` runs daily at 03:15**, from Cloudron's scheduler addon
|
||
(`CloudronManifest.json` → `addons.scheduler`), via `/app/code/metrics_snapshot.py`.
|
||
It runs after `membership_sweep` (03:00) and `rag_refresh` (03:30).
|
||
- It writes **one row per day** into the `metrics_daily` table, in the same SQLite
|
||
database as the members table (`members.db`). Columns: `day`, `members_active`,
|
||
`members_total`, `members_new`, `newsletter_subs`, `matrix_space_members`,
|
||
`spend_usd`, `requests`, `out_tokens`, `energy_wh`, `co2e_g`, `recorded_at`.
|
||
- **`metrics_snapshot.py --backfill` rebuilds history** from the sources and is
|
||
idempotent — it rewrites a day rather than duplicating it. It is the tool to run
|
||
after changing a factor, and the reason a published chart can move: history is
|
||
**rewritten**, not appended to.
|
||
|
||
## What is measured, and what is not
|
||
|
||
Stated plainly, because a mixed total is easy to over-read:
|
||
|
||
- **Measured:** GreenPT models — the providers whose API reports impact.
|
||
- **Assumed equal to a measurement:** `greenpt/glm-5.3`, from its flash sibling.
|
||
- **Estimated:** everything else. Currently that means **all Tinfoil and Regolo
|
||
traffic**, which is a real share of co-op usage.
|
||
|
||
So a day's figure is a blend, and **the split is not recorded per day** — you cannot
|
||
tell from `metrics_daily` how much of a total rested on the proxy. Persisting that
|
||
split is a separate follow-up.
|
||
|
||
One consequence worth knowing: the proxy assumes a 250 gCO2e/kWh grid, while Regolo
|
||
runs on 100% renewable energy, so the proxy **overstates** Regolo's carbon. Per-provider
|
||
factors are a follow-up; until then, Regolo and Tinfoil figures should be read as upper
|
||
bounds rather than measurements.
|
||
|
||
## Changing the factors
|
||
|
||
Edit the table in `metrics_snapshot.py` and redeploy — refining a factor is a data
|
||
edit, not a code change. Then run `--backfill` if the change should apply to history.
|
||
The energy factors are also recorded in `co-op/docs/energy.md` for members, and the two
|
||
should not disagree.
|