Files
member-portal/docs/energy-metrics.md
T
inference-bot cf3dbb7489
ci/woodpecker/push/woodpecker Pipeline was successful
metrics: complete YESTERDAY's row at 03:15 — every day was frozen at its first ~3 hours
snapshot() fetched the full spend window (since = yesterday) but kept only
spend.get(TODAY): the day's first hours, written at 03:15 and never revisited.
Every metrics_daily row after the last backfill under-counted its day by the
hours after 03:15 (Nov 7 recorded $2.19/691 requests against a true
$29.06/3384). The completed day's aggregates were computed and thrown away.

Now each run completes the day that just ENDED — full-day spend, requests,
tokens and impact, written to the usage columns only (upsert_usage: membership
columns are point-in-time measurements and are never rewritten) — and then
records today's live partial as before.

backfill() gains --since YYYY-MM-DD: a targeted usage-only repair that cannot
touch the published membership curve.

Regression test: tests/test_metrics_completion.py (day completion, membership
preservation, since-mode repair boundaries, full-backfill parity).
2026-11-12 18:40:31 +00:00

198 lines
10 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Energy and carbon metrics
How the portal's daily `energy_wh` and `co2e_g` figures are produced, what is
measured versus estimated, and how to change them. Written because the calculation
previously lived only in code comments (see `code/member-portal#3`).
For what members are told, see `co-op/docs/energy.md`. This document is the
developer-facing counterpart and is deliberately blunter about the limits.
## Where the numbers come from
**Input: output tokens only, per day, per model.**
- Source: LiteLLM's spend logs (`LiteLLM_SpendLogs`), column `completion_tokens`.
- Grouped on **`model_group`** — our alias, e.g. `greenpt/deepseek-v4.1-flash` —
**not** `model`, which carries the upstream name (`openai/deepseek-v4.1-flash`).
Grouping on the wrong column matches no factor and yields zeros for every day. That
mistake shipped once during development and is the reason this is called out.
**Only output tokens count**, matching the unit GreenPT publishes (mWh per 100 output
tokens). This is a deliberate choice to line up with the provider's own metric, not an
oversight — and it means these figures **understate total compute**, because input
tokens consume energy too. Treat the numbers as comparable and directionally true
rather than as a total energy accounting.
## The factor table
Factors live in `metrics_snapshot.py` (`IMPACT_FACTORS`, `IMPACT_PROXY`). Matching is
`str.startswith()`, so **the longest prefix must come first** — order matters.
| model prefix | Wh / 1k output tokens | gCO2e / kWh | provenance |
|---|---|---|---|
| `greenpt/deepseek-v4.1-flash` | 0.3641 | 193.0 | MEASURED |
| `greenpt/glm-5.3-flash` | 1.0782 | 193.0 | MEASURED |
| `greenpt/glm-5.3` | 1.0782 | 193.0 | SAME_FAMILY |
| `greenpt/green-r` | 0.5535 | 23.6 | MEASURED |
| *anything else* | 0.5000 | 250.0 | ESTIMATED |
### The three provenance tiers
- **MEASURED** — fitted from live calls to GreenPT's API, which returns an `impact`
object on every request (energy in watt-milliseconds, emissions in µgCO2e). Energy
was regressed on output tokens across prompt sizes (R² ≥ 0.999); grid intensity is
what their own emissions figures imply. Fitted 2026-10-28.
- **SAME_FAMILY** — measured for `glm-5.3-flash` and assumed to hold for `glm-5.3`.
An assumption of similarity, not a measurement.
- **ESTIMATED** — the proxy row: an order-of-magnitude figure near the middle of the
measured range, with an EU grid average. It is applied to every provider that does
not disclose impact, **deliberately**, so that a day's total is never quietly
missing models. It is not a number we know.
**No tier is exposed on its own**, but every row now carries a *provenance* read
derived from the measured/estimated token split — see "Provenance in the API"
below. A consumer can tell how much of a day's figure rested on a measurement.
## Provenance in the API
Every row returned by `/admin/metrics/series` and `/broker/metrics` carries the
measured-vs-estimated split, so a consumer can tell how much of a day's figure
rested on a real measurement. Four fields:
| field | meaning |
|---|---|
| `tokens_measured` | output tokens that day attributed to a factor whose `source` is **MEASURED** — a direct provider measurement. |
| `tokens_estimated` | the rest: **SAME_FAMILY** (borrowed from a sibling) and the flat **proxy**. A borrowed or proxied factor is an assumption, so it is **not** counted as measured. |
| `factor_revision` | a short, stable hash of the factor table (`IMPACT_FACTORS` + `IMPACT_PROXY`) in effect when the row was computed. |
| `provenance` | the derived read, below. |
`tokens_measured + tokens_estimated = out_tokens` for any day with tokens. A day
with no output tokens has NULL in both (fail-closed, same rule as energy/carbon).
`provenance` is one of:
- **`measured`** — every token rested on a measured factor.
- **`estimated`** — every token rested on a non-measured factor (borrowed or proxy).
- **`mixed`** — both are present; the day's total is a blend.
- **`none`** — no provenance. This is **either** a day with no output tokens **or**
a row written before these columns existed (older rows keep NULL). It is
deliberately **not** `measured`: an unknown split must never read as a confident
measurement. This is the whole point of the indicator — failure reads as unknown.
`factor_revision` is a deterministic hash, **not a timestamp**: two runs against the
same table produce the same string. It changes if any factor value or source label
changes, or if the matching order changes (order is meaningful — longest prefix
first). It is "what the numbers were computed with", not "what was known then".
### Forward-only, then backfill
The columns are added by an **additive** migration — a guarded
`ALTER TABLE metrics_daily ADD COLUMN …`, the same idiom used when the impact
columns first arrived. On an existing database this is safe: it never errors,
existing rows survive untouched, and they simply keep NULL in the new columns.
Nothing is guessed for history on migration; those rows read as `provenance: "none"`.
To give past rows a real split, run `metrics_snapshot.py --backfill` (or re-run the
affected days). Backfill rewrites each day from the current LiteLLM token split and
the **current** factor table, so it also updates `factor_revision` to today's
revision — a backfilled historical row records the revision used to recompute it,
not the one that was current at the time. Honest for a chart label; not a historical
audit trail.
## Formulas
```
energy_wh = Σ_models ( out_tokens / 1000 × wh_per_1k_out )
co2e_g = Σ_models ( out_tokens / 1000 × wh_per_1k_out / 1000 × gco2e_per_kwh )
└────────────── Wh ──────────────┘ └─ Wh → kWh ─┘
```
Carbon was once understated by exactly 1000× here — a `/1e6` where `/1e3` was
intended — which is why the tripwire below exists.
## NULL, never zero
`impact_for()` returns an all-NULL impact block for a day with no measured tokens,
and the row is written with NULL energy and carbon. It never writes `0.0`:
> a zero reads as "the co-op used no energy", which is a claim we cannot support, and
> it would survive into a chart as a fact.
The first backfill shipped exactly that bug — it summed the wrong column, every model
totalled zero, and every day in history claimed to have used no energy.
A NULL day renders as a **bare gap in the charts, with nothing explaining it.** That is
deliberate: the surfaces carry no caveat text. Accepted knowingly.
The same fail-closed rule applies to every metric in the snapshot, not only energy:
an unreadable source (Listmonk, Matrix, LiteLLM) yields NULL for that metric and
prints to the log, rather than a number that looks real.
## The tripwire
After summing, `impact_for()` computes the grid intensity its own mix implies and
compares it against a plausible band:
```
implied = co2e_g / energy_wh × 1000 # gCO2e per kWh
if not (5.0 <= implied <= 500.0):
warn "suspect a unit error before trusting these numbers"
```
It warns; it never alters the numbers or suppresses the row. Its purpose is narrow and
earned: the 1000× carbon error kept the energy column plausible, so nothing looked
wrong, and a silent error of that kind would otherwise be found by a member rather
than by us.
## Schedule and backfill
- **`metrics_snapshot` runs daily at 03:15**, from Cloudron's scheduler addon
(`CloudronManifest.json` → `addons.scheduler`), via `/app/code/metrics_snapshot.py`.
It runs after `membership_sweep` (03:00) and `rag_refresh` (03:30).
- It writes **one row per day** into the `metrics_daily` table, in the same SQLite
database as the members table (`members.db`). Columns: `day`, `members_active`,
`members_total`, `members_new`, `newsletter_subs`, `matrix_space_members`,
`spend_usd`, `requests`, `out_tokens`, `energy_wh`, `co2e_g`, `tokens_measured`,
`tokens_estimated`, `factor_revision`, `recorded_at`.
- **Each 03:15 run completes YESTERDAY, then records today live.** The day that
just ended is rewritten with its FULL spend/requests/tokens/impact aggregates
(usage columns only — the membership columns are point-in-time measurements
that are not reconstructible afterwards). Today's row carries the first hours
of live data and is completed by the next run. (Until 2026-11-12 the run
wrote only today's partial and never revisited it, which froze every day at
its first ~3 hours — the November under-count.)
- **`metrics_snapshot.py --backfill` rebuilds history** from the sources and is
idempotent — it rewrites a day rather than duplicating it. It is the tool to run
after changing a factor, and the reason a published chart can move: history is
**rewritten**, not appended to. `--backfill --since YYYY-MM-DD` limits the
rewrite to days from that date and touches ONLY the usage/impact columns, so
a targeted repair of recent days cannot alter the published membership curve.
## What is measured, and what is not
Stated plainly, because a mixed total is easy to over-read:
- **Measured:** GreenPT models — the providers whose API reports impact.
- **Assumed equal to a measurement:** `greenpt/glm-5.3`, from its flash sibling.
- **Estimated:** everything else. Currently that means **all Tinfoil and Regolo
traffic**, which is a real share of co-op usage.
So a day's figure is a blend, and **the split is now recorded per day** —
`tokens_measured` and `tokens_estimated` say how much of the total rested on a
measurement, and the API derives a `provenance` read from them. Before those
columns existed the split was not recorded; such rows carry NULL and read as
`provenance: "none"`. See "Provenance in the API".
One consequence worth knowing: the proxy assumes a 250 gCO2e/kWh grid, while Regolo
runs on 100% renewable energy, so the proxy **overstates** Regolo's carbon. Per-provider
factors are a follow-up; until then, Regolo and Tinfoil figures should be read as upper
bounds rather than measurements.
## Changing the factors
Edit the table in `metrics_snapshot.py` and redeploy — refining a factor is a data
edit, not a code change. Then run `--backfill` if the change should apply to history.
The energy factors are also recorded in `co-op/docs/energy.md` for members, and the two
should not disagree.