docs: write down how the energy/CO2 metrics actually work (#3)

The calculation lived only in code comments. Adds docs/energy-metrics.md and wires it
into the README, per the issue's proposed scope. No calculation changes.

Contents: the input (completion_tokens, output tokens only, grouped on model_group)
and why output-only understates total compute; the factor table with its three
provenance tiers (measured / same-family / estimated) and the fact that the tiers are
never exposed; the formulas; NULL-not-zero and the reason (the first backfill claimed
every day used no energy); the implied-grid-intensity tripwire and the 1000x carbon
error it exists to catch; the 03:15 schedule and that --backfill REWRITES history.

States plainly what is measured and what is not: GreenPT is measured, glm-5.3 is
assumed from its flash sibling, and all Tinfoil and Regolo traffic currently falls to
the proxy estimate — with the consequence that the 250 gCO2e/kWh proxy OVERSTATES
Regolo, which runs on renewable energy.

README: "Two jobs" -> "Three", the metrics_snapshot task documented, and a short
Energy & carbon metrics subsection linking the new doc. Also corrects "listing both
tasks" in the post-change verification step.
This commit is contained in:
inference-bot committed 2026-10-29 20:15:10 +00:00
1 parent 42cf43a125
commit b11552fa49
2 files changed
+159 -2

No files matched your search

+20 -2
View File
@@ -93,7 +93,7 @@ points. Remaining work before production:
## Scheduled tasks (scheduler addon) ## Scheduled tasks (scheduler addon)
Two jobs run inside the container via `CloudronManifest.json` → `addons.scheduler`: Three jobs run inside the container via `CloudronManifest.json` → `addons.scheduler`:
- **membership_sweep** (nightly 3am) — `sweep.sh` calls the portal's own - **membership_sweep** (nightly 3am) — `sweep.sh` calls the portal's own
`/admin/sweep` (auth: `X-Admin-Token` = `WEBHOOK_TOKEN`) to reconcile members `/admin/sweep` (auth: `X-Admin-Token` = `WEBHOOK_TOKEN`) to reconcile members
@@ -105,7 +105,25 @@ Two jobs run inside the container via `CloudronManifest.json` → `addons.schedu
files to Open WebUI (`OWUI_API_KEY`), attaches them to the shared knowledge files to Open WebUI (`OWUI_API_KEY`), attaches them to the shared knowledge
collection (`DOCS_KNOWLEDGE_ID`), reindexes, and re-grants the members group. collection (`DOCS_KNOWLEDGE_ID`), reindexes, and re-grants the members group.
State persists in `/app/data/rag_refresh_state.json`. State persists in `/app/data/rag_refresh_state.json`.
- **metrics_snapshot** (nightly 3:15am) — `metrics_snapshot.py` records one row per
day into the `metrics_daily` table: members, newsletter subscribers, Matrix
membership, spend, request counts, and energy/CO₂. A source that cannot be read
yields NULL for that metric rather than a zero. `--backfill` rebuilds history
idempotently. Methodology: [Energy and carbon metrics](docs/energy-metrics.md).
Verify after any manifest change: the deploy log must show Verify after any manifest change: the deploy log must show
`setupAddons: ... scheduler ...` listing both tasks, and each script must `setupAddons: ... scheduler ...` listing all three tasks, and each script must
succeed when run manually inside the container. succeed when run manually inside the container.
### Energy & carbon metrics
The daily snapshot also computes energy (Wh) and carbon (gCO₂e) from output tokens,
using per-model factors whose provenance varies: measured for GreenPT, assumed equal
for one sibling model, and an explicit order-of-magnitude estimate for providers that
disclose nothing. Only output tokens count, and a day with no measured tokens is
recorded as NULL rather than zero.
[Energy and carbon metrics](docs/energy-metrics.md) documents the factor table, the
formulas, the tripwire that catches unit errors, the schedule, and — most importantly —
which parts are measurements and which are estimates. What members are told is in
`co-op/docs/energy.md`.
+139
View File
@@ -0,0 +1,139 @@
# Energy and carbon metrics
How the portal's daily `energy_wh` and `co2e_g` figures are produced, what is
measured versus estimated, and how to change them. Written because the calculation
previously lived only in code comments (see `code/member-portal#3`).
For what members are told, see `co-op/docs/energy.md`. This document is the
developer-facing counterpart and is deliberately blunter about the limits.
## Where the numbers come from
**Input: output tokens only, per day, per model.**
- Source: LiteLLM's spend logs (`LiteLLM_SpendLogs`), column `completion_tokens`.
- Grouped on **`model_group`** — our alias, e.g. `greenpt/deepseek-v4.1-flash` —
**not** `model`, which carries the upstream name (`openai/deepseek-v4.1-flash`).
Grouping on the wrong column matches no factor and yields zeros for every day. That
mistake shipped once during development and is the reason this is called out.
**Only output tokens count**, matching the unit GreenPT publishes (mWh per 100 output
tokens). This is a deliberate choice to line up with the provider's own metric, not an
oversight — and it means these figures **understate total compute**, because input
tokens consume energy too. Treat the numbers as comparable and directionally true
rather than as a total energy accounting.
## The factor table
Factors live in `metrics_snapshot.py` (`IMPACT_FACTORS`, `IMPACT_PROXY`). Matching is
`str.startswith()`, so **the longest prefix must come first** — order matters.
| model prefix | Wh / 1k output tokens | gCO2e / kWh | provenance |
|---|---|---|---|
| `greenpt/deepseek-v4.1-flash` | 0.3641 | 193.0 | MEASURED |
| `greenpt/glm-5.3-flash` | 1.0782 | 193.0 | MEASURED |
| `greenpt/glm-5.3` | 1.0782 | 193.0 | SAME_FAMILY |
| `greenpt/green-r` | 0.5535 | 23.6 | MEASURED |
| *anything else* | 0.5000 | 250.0 | ESTIMATED |
### The three provenance tiers
- **MEASURED** — fitted from live calls to GreenPT's API, which returns an `impact`
object on every request (energy in watt-milliseconds, emissions in µgCO2e). Energy
was regressed on output tokens across prompt sizes (R² ≥ 0.999); grid intensity is
what their own emissions figures imply. Fitted 2026-10-28.
- **SAME_FAMILY** — measured for `glm-5.3-flash` and assumed to hold for `glm-5.3`.
An assumption of similarity, not a measurement.
- **ESTIMATED** — the proxy row: an order-of-magnitude figure near the middle of the
measured range, with an EU grid average. It is applied to every provider that does
not disclose impact, **deliberately**, so that a day's total is never quietly
missing models. It is not a number we know.
**No tier is exposed anywhere.** The endpoints (`/admin/metrics/series`,
`/broker/metrics`) return bare numbers, so a consumer cannot tell measured from
estimated. Exposing provenance is tracked as a separate issue.
## Formulas
```
energy_wh = Σ_models ( out_tokens / 1000 × wh_per_1k_out )
co2e_g = Σ_models ( out_tokens / 1000 × wh_per_1k_out / 1000 × gco2e_per_kwh )
└────────────── Wh ──────────────┘ └─ Wh → kWh ─┘
```
Carbon was once understated by exactly 1000× here — a `/1e6` where `/1e3` was
intended — which is why the tripwire below exists.
## NULL, never zero
`impact_for()` returns `None, None` for a day with no measured tokens, and the row is
written with NULL energy and carbon. It never writes `0.0`:
> a zero reads as "the co-op used no energy", which is a claim we cannot support, and
> it would survive into a chart as a fact.
The first backfill shipped exactly that bug — it summed the wrong column, every model
totalled zero, and every day in history claimed to have used no energy.
A NULL day renders as a **bare gap in the charts, with nothing explaining it.** That is
deliberate: the surfaces carry no caveat text. Accepted knowingly.
The same fail-closed rule applies to every metric in the snapshot, not only energy:
an unreadable source (Listmonk, Matrix, LiteLLM) yields NULL for that metric and
prints to the log, rather than a number that looks real.
## The tripwire
After summing, `impact_for()` computes the grid intensity its own mix implies and
compares it against a plausible band:
```
implied = co2e_g / energy_wh × 1000 # gCO2e per kWh
if not (5.0 <= implied <= 500.0):
warn "suspect a unit error before trusting these numbers"
```
It warns; it never alters the numbers or suppresses the row. Its purpose is narrow and
earned: the 1000× carbon error kept the energy column plausible, so nothing looked
wrong, and a silent error of that kind would otherwise be found by a member rather
than by us.
## Schedule and backfill
- **`metrics_snapshot` runs daily at 03:15**, from Cloudron's scheduler addon
(`CloudronManifest.json` → `addons.scheduler`), via `/app/code/metrics_snapshot.py`.
It runs after `membership_sweep` (03:00) and `rag_refresh` (03:30).
- It writes **one row per day** into the `metrics_daily` table, in the same SQLite
database as the members table (`members.db`). Columns: `day`, `members_active`,
`members_total`, `members_new`, `newsletter_subs`, `matrix_space_members`,
`spend_usd`, `requests`, `out_tokens`, `energy_wh`, `co2e_g`, `recorded_at`.
- **`metrics_snapshot.py --backfill` rebuilds history** from the sources and is
idempotent — it rewrites a day rather than duplicating it. It is the tool to run
after changing a factor, and the reason a published chart can move: history is
**rewritten**, not appended to.
## What is measured, and what is not
Stated plainly, because a mixed total is easy to over-read:
- **Measured:** GreenPT models — the providers whose API reports impact.
- **Assumed equal to a measurement:** `greenpt/glm-5.3`, from its flash sibling.
- **Estimated:** everything else. Currently that means **all Tinfoil and Regolo
traffic**, which is a real share of co-op usage.
So a day's figure is a blend, and **the split is not recorded per day** — you cannot
tell from `metrics_daily` how much of a total rested on the proxy. Persisting that
split is a separate follow-up.
One consequence worth knowing: the proxy assumes a 250 gCO2e/kWh grid, while Regolo
runs on 100% renewable energy, so the proxy **overstates** Regolo's carbon. Per-provider
factors are a follow-up; until then, Regolo and Tinfoil figures should be read as upper
bounds rather than measurements.
## Changing the factors
Edit the table in `metrics_snapshot.py` and redeploy — refining a factor is a data
edit, not a code change. Then run `--backfill` if the change should apply to history.
The energy factors are also recorded in `co-op/docs/energy.md` for members, and the two
should not disagree.