From b11552fa495f395808b08e42dbcc3b860286edd0 Mon Sep 17 00:00:00 2001 From: Inferencebot Date: Thu, 29 Oct 2026 20:15:10 +0000 Subject: [PATCH] docs: write down how the energy/CO2 metrics actually work (#3) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The calculation lived only in code comments. Adds docs/energy-metrics.md and wires it into the README, per the issue's proposed scope. No calculation changes. Contents: the input (completion_tokens, output tokens only, grouped on model_group) and why output-only understates total compute; the factor table with its three provenance tiers (measured / same-family / estimated) and the fact that the tiers are never exposed; the formulas; NULL-not-zero and the reason (the first backfill claimed every day used no energy); the implied-grid-intensity tripwire and the 1000x carbon error it exists to catch; the 03:15 schedule and that --backfill REWRITES history. States plainly what is measured and what is not: GreenPT is measured, glm-5.3 is assumed from its flash sibling, and all Tinfoil and Regolo traffic currently falls to the proxy estimate — with the consequence that the 250 gCO2e/kWh proxy OVERSTATES Regolo, which runs on renewable energy. README: "Two jobs" -> "Three", the metrics_snapshot task documented, and a short Energy & carbon metrics subsection linking the new doc. Also corrects "listing both tasks" in the post-change verification step. --- README.md | 22 ++++++- docs/energy-metrics.md | 139 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 159 insertions(+), 2 deletions(-) create mode 100644 docs/energy-metrics.md diff --git a/README.md b/README.md index c5cd7f9..ecddf72 100644 --- a/README.md +++ b/README.md @@ -93,7 +93,7 @@ points. Remaining work before production: ## Scheduled tasks (scheduler addon) -Two jobs run inside the container via `CloudronManifest.json` → `addons.scheduler`: +Three jobs run inside the container via `CloudronManifest.json` → `addons.scheduler`: - **membership_sweep** (nightly 3am) — `sweep.sh` calls the portal's own `/admin/sweep` (auth: `X-Admin-Token` = `WEBHOOK_TOKEN`) to reconcile members @@ -105,7 +105,25 @@ Two jobs run inside the container via `CloudronManifest.json` → `addons.schedu files to Open WebUI (`OWUI_API_KEY`), attaches them to the shared knowledge collection (`DOCS_KNOWLEDGE_ID`), reindexes, and re-grants the members group. State persists in `/app/data/rag_refresh_state.json`. +- **metrics_snapshot** (nightly 3:15am) — `metrics_snapshot.py` records one row per + day into the `metrics_daily` table: members, newsletter subscribers, Matrix + membership, spend, request counts, and energy/CO₂. A source that cannot be read + yields NULL for that metric rather than a zero. `--backfill` rebuilds history + idempotently. Methodology: [Energy and carbon metrics](docs/energy-metrics.md). Verify after any manifest change: the deploy log must show -`setupAddons: ... scheduler ...` listing both tasks, and each script must +`setupAddons: ... scheduler ...` listing all three tasks, and each script must succeed when run manually inside the container. + +### Energy & carbon metrics + +The daily snapshot also computes energy (Wh) and carbon (gCO₂e) from output tokens, +using per-model factors whose provenance varies: measured for GreenPT, assumed equal +for one sibling model, and an explicit order-of-magnitude estimate for providers that +disclose nothing. Only output tokens count, and a day with no measured tokens is +recorded as NULL rather than zero. + +[Energy and carbon metrics](docs/energy-metrics.md) documents the factor table, the +formulas, the tripwire that catches unit errors, the schedule, and — most importantly — +which parts are measurements and which are estimates. What members are told is in +`co-op/docs/energy.md`. diff --git a/docs/energy-metrics.md b/docs/energy-metrics.md new file mode 100644 index 0000000..8a8cc6b --- /dev/null +++ b/docs/energy-metrics.md @@ -0,0 +1,139 @@ +# Energy and carbon metrics + +How the portal's daily `energy_wh` and `co2e_g` figures are produced, what is +measured versus estimated, and how to change them. Written because the calculation +previously lived only in code comments (see `code/member-portal#3`). + +For what members are told, see `co-op/docs/energy.md`. This document is the +developer-facing counterpart and is deliberately blunter about the limits. + +## Where the numbers come from + +**Input: output tokens only, per day, per model.** + +- Source: LiteLLM's spend logs (`LiteLLM_SpendLogs`), column `completion_tokens`. +- Grouped on **`model_group`** — our alias, e.g. `greenpt/deepseek-v4.1-flash` — + **not** `model`, which carries the upstream name (`openai/deepseek-v4.1-flash`). + Grouping on the wrong column matches no factor and yields zeros for every day. That + mistake shipped once during development and is the reason this is called out. + +**Only output tokens count**, matching the unit GreenPT publishes (mWh per 100 output +tokens). This is a deliberate choice to line up with the provider's own metric, not an +oversight — and it means these figures **understate total compute**, because input +tokens consume energy too. Treat the numbers as comparable and directionally true +rather than as a total energy accounting. + +## The factor table + +Factors live in `metrics_snapshot.py` (`IMPACT_FACTORS`, `IMPACT_PROXY`). Matching is +`str.startswith()`, so **the longest prefix must come first** — order matters. + +| model prefix | Wh / 1k output tokens | gCO2e / kWh | provenance | +|---|---|---|---| +| `greenpt/deepseek-v4.1-flash` | 0.3641 | 193.0 | MEASURED | +| `greenpt/glm-5.3-flash` | 1.0782 | 193.0 | MEASURED | +| `greenpt/glm-5.3` | 1.0782 | 193.0 | SAME_FAMILY | +| `greenpt/green-r` | 0.5535 | 23.6 | MEASURED | +| *anything else* | 0.5000 | 250.0 | ESTIMATED | + +### The three provenance tiers + +- **MEASURED** — fitted from live calls to GreenPT's API, which returns an `impact` + object on every request (energy in watt-milliseconds, emissions in µgCO2e). Energy + was regressed on output tokens across prompt sizes (R² ≥ 0.999); grid intensity is + what their own emissions figures imply. Fitted 2026-10-28. +- **SAME_FAMILY** — measured for `glm-5.3-flash` and assumed to hold for `glm-5.3`. + An assumption of similarity, not a measurement. +- **ESTIMATED** — the proxy row: an order-of-magnitude figure near the middle of the + measured range, with an EU grid average. It is applied to every provider that does + not disclose impact, **deliberately**, so that a day's total is never quietly + missing models. It is not a number we know. + +**No tier is exposed anywhere.** The endpoints (`/admin/metrics/series`, +`/broker/metrics`) return bare numbers, so a consumer cannot tell measured from +estimated. Exposing provenance is tracked as a separate issue. + +## Formulas + +``` +energy_wh = Σ_models ( out_tokens / 1000 × wh_per_1k_out ) + +co2e_g = Σ_models ( out_tokens / 1000 × wh_per_1k_out / 1000 × gco2e_per_kwh ) + └────────────── Wh ──────────────┘ └─ Wh → kWh ─┘ +``` + +Carbon was once understated by exactly 1000× here — a `/1e6` where `/1e3` was +intended — which is why the tripwire below exists. + +## NULL, never zero + +`impact_for()` returns `None, None` for a day with no measured tokens, and the row is +written with NULL energy and carbon. It never writes `0.0`: + +> a zero reads as "the co-op used no energy", which is a claim we cannot support, and +> it would survive into a chart as a fact. + +The first backfill shipped exactly that bug — it summed the wrong column, every model +totalled zero, and every day in history claimed to have used no energy. + +A NULL day renders as a **bare gap in the charts, with nothing explaining it.** That is +deliberate: the surfaces carry no caveat text. Accepted knowingly. + +The same fail-closed rule applies to every metric in the snapshot, not only energy: +an unreadable source (Listmonk, Matrix, LiteLLM) yields NULL for that metric and +prints to the log, rather than a number that looks real. + +## The tripwire + +After summing, `impact_for()` computes the grid intensity its own mix implies and +compares it against a plausible band: + +``` +implied = co2e_g / energy_wh × 1000 # gCO2e per kWh +if not (5.0 <= implied <= 500.0): + warn "suspect a unit error before trusting these numbers" +``` + +It warns; it never alters the numbers or suppresses the row. Its purpose is narrow and +earned: the 1000× carbon error kept the energy column plausible, so nothing looked +wrong, and a silent error of that kind would otherwise be found by a member rather +than by us. + +## Schedule and backfill + +- **`metrics_snapshot` runs daily at 03:15**, from Cloudron's scheduler addon + (`CloudronManifest.json` → `addons.scheduler`), via `/app/code/metrics_snapshot.py`. + It runs after `membership_sweep` (03:00) and `rag_refresh` (03:30). +- It writes **one row per day** into the `metrics_daily` table, in the same SQLite + database as the members table (`members.db`). Columns: `day`, `members_active`, + `members_total`, `members_new`, `newsletter_subs`, `matrix_space_members`, + `spend_usd`, `requests`, `out_tokens`, `energy_wh`, `co2e_g`, `recorded_at`. +- **`metrics_snapshot.py --backfill` rebuilds history** from the sources and is + idempotent — it rewrites a day rather than duplicating it. It is the tool to run + after changing a factor, and the reason a published chart can move: history is + **rewritten**, not appended to. + +## What is measured, and what is not + +Stated plainly, because a mixed total is easy to over-read: + +- **Measured:** GreenPT models — the providers whose API reports impact. +- **Assumed equal to a measurement:** `greenpt/glm-5.3`, from its flash sibling. +- **Estimated:** everything else. Currently that means **all Tinfoil and Regolo + traffic**, which is a real share of co-op usage. + +So a day's figure is a blend, and **the split is not recorded per day** — you cannot +tell from `metrics_daily` how much of a total rested on the proxy. Persisting that +split is a separate follow-up. + +One consequence worth knowing: the proxy assumes a 250 gCO2e/kWh grid, while Regolo +runs on 100% renewable energy, so the proxy **overstates** Regolo's carbon. Per-provider +factors are a follow-up; until then, Regolo and Tinfoil figures should be read as upper +bounds rather than measurements. + +## Changing the factors + +Edit the table in `metrics_snapshot.py` and redeploy — refining a factor is a data +edit, not a code change. Then run `--backfill` if the change should apply to history. +The energy factors are also recorded in `co-op/docs/energy.md` for members, and the two +should not disagree.