member-portal/docs/energy-metrics.md

# Energy and carbon metrics

How the portal's daily `energy_wh` and `co2e_g` figures are produced, what is
measured versus estimated, and how to change them. Written because the calculation
previously lived only in code comments (see `code/member-portal#3`).

For what members are told, see `co-op/docs/energy.md`. This document is the
developer-facing counterpart and is deliberately blunter about the limits.

## Where the numbers come from

**Input: output tokens only, per day, per model.**

- Source: LiteLLM's spend logs (`LiteLLM_SpendLogs`), column `completion_tokens`.
- Grouped on **`model_group`** — our alias, e.g. `greenpt/deepseek-v4.1-flash` —
  **not** `model`, which carries the upstream name (`openai/deepseek-v4.1-flash`).
  Grouping on the wrong column matches no factor and yields zeros for every day. That
  mistake shipped once during development and is the reason this is called out.

**Only output tokens count**, matching the unit GreenPT publishes (mWh per 100 output
tokens). This is a deliberate choice to line up with the provider's own metric, not an
oversight — and it means these figures **understate total compute**, because input
tokens consume energy too. Treat the numbers as comparable and directionally true
rather than as a total energy accounting.

## The factor table

Factors live in `metrics_snapshot.py` (`IMPACT_FACTORS`, `IMPACT_PROXY`). Matching is
`str.startswith()`, so **the longest prefix must come first** — order matters.

| model prefix | Wh / 1k output tokens | gCO2e / kWh | provenance |
|---|---|---|---|
| `greenpt/deepseek-v4.1-flash` | 0.3641 | 193.0 | MEASURED |
| `greenpt/glm-5.3-flash` | 1.0782 | 193.0 | MEASURED |
| `greenpt/glm-5.3` | 1.0782 | 193.0 | SAME_FAMILY |
| `greenpt/green-r` | 0.5535 | 23.6 | MEASURED |
| *anything else* | 0.5000 | 250.0 | ESTIMATED |

### The three provenance tiers

- **MEASURED** — fitted from live calls to GreenPT's API, which returns an `impact`
  object on every request (energy in watt-milliseconds, emissions in µgCO2e). Energy
  was regressed on output tokens across prompt sizes (R² ≥ 0.999); grid intensity is
  what their own emissions figures imply. Fitted 2026-10-28.
- **SAME_FAMILY** — measured for `glm-5.3-flash` and assumed to hold for `glm-5.3`.
  An assumption of similarity, not a measurement.
- **ESTIMATED** — the proxy row: an order-of-magnitude figure near the middle of the
  measured range, with an EU grid average. It is applied to every provider that does
  not disclose impact, **deliberately**, so that a day's total is never quietly
  missing models. It is not a number we know.

**No tier is exposed on its own**, but every row now carries a *provenance* read
derived from the measured/estimated token split — see "Provenance in the API"
below. A consumer can tell how much of a day's figure rested on a measurement.

## Provenance in the API

Every row returned by `/admin/metrics/series` and `/broker/metrics` carries the
measured-vs-estimated split, so a consumer can tell how much of a day's figure
rested on a real measurement. Four fields:

| field | meaning |
|---|---|
| `tokens_measured` | output tokens that day attributed to a factor whose `source` is **MEASURED** — a direct provider measurement. |
| `tokens_estimated` | the rest: **SAME_FAMILY** (borrowed from a sibling) and the flat **proxy**. A borrowed or proxied factor is an assumption, so it is **not** counted as measured. |
| `factor_revision` | a short, stable hash of the factor table (`IMPACT_FACTORS` + `IMPACT_PROXY`) in effect when the row was computed. |
| `provenance` | the derived read, below. |

`tokens_measured + tokens_estimated = out_tokens` for any day with tokens. A day
with no output tokens has NULL in both (fail-closed, same rule as energy/carbon).

`provenance` is one of:

- **`measured`** — every token rested on a measured factor.
- **`estimated`** — every token rested on a non-measured factor (borrowed or proxy).
- **`mixed`** — both are present; the day's total is a blend.
- **`none`** — no provenance. This is **either** a day with no output tokens **or**
  a row written before these columns existed (older rows keep NULL). It is
  deliberately **not** `measured`: an unknown split must never read as a confident
  measurement. This is the whole point of the indicator — failure reads as unknown.

`factor_revision` is a deterministic hash, **not a timestamp**: two runs against the
same table produce the same string. It changes if any factor value or source label
changes, or if the matching order changes (order is meaningful — longest prefix
first). It is "what the numbers were computed with", not "what was known then".

### Forward-only, then backfill

The columns are added by an **additive** migration — a guarded
`ALTER TABLE metrics_daily ADD COLUMN …`, the same idiom used when the impact
columns first arrived. On an existing database this is safe: it never errors,
existing rows survive untouched, and they simply keep NULL in the new columns.
Nothing is guessed for history on migration; those rows read as `provenance: "none"`.

To give past rows a real split, run `metrics_snapshot.py --backfill` (or re-run the
affected days). Backfill rewrites each day from the current LiteLLM token split and
the **current** factor table, so it also updates `factor_revision` to today's
revision — a backfilled historical row records the revision used to recompute it,
not the one that was current at the time. Honest for a chart label; not a historical
audit trail.

## Formulas

```
energy_wh = Σ_models ( out_tokens / 1000 × wh_per_1k_out )

co2e_g    = Σ_models ( out_tokens / 1000 × wh_per_1k_out / 1000 × gco2e_per_kwh )
                      └────────────── Wh ──────────────┘   └─ Wh → kWh ─┘
```

Carbon was once understated by exactly 1000× here — a `/1e6` where `/1e3` was
intended — which is why the tripwire below exists.

## NULL, never zero

`impact_for()` returns an all-NULL impact block for a day with no measured tokens,
and the row is written with NULL energy and carbon. It never writes `0.0`:

> a zero reads as "the co-op used no energy", which is a claim we cannot support, and
> it would survive into a chart as a fact.

The first backfill shipped exactly that bug — it summed the wrong column, every model
totalled zero, and every day in history claimed to have used no energy.

A NULL day renders as a **bare gap in the charts, with nothing explaining it.** That is
deliberate: the surfaces carry no caveat text. Accepted knowingly.

The same fail-closed rule applies to every metric in the snapshot, not only energy:
an unreadable source (Listmonk, Matrix, LiteLLM) yields NULL for that metric and
prints to the log, rather than a number that looks real.

## The tripwire

After summing, `impact_for()` computes the grid intensity its own mix implies and
compares it against a plausible band:

```
implied = co2e_g / energy_wh × 1000      # gCO2e per kWh
if not (5.0 <= implied <= 500.0):
    warn "suspect a unit error before trusting these numbers"
```

It warns; it never alters the numbers or suppresses the row. Its purpose is narrow and
earned: the 1000× carbon error kept the energy column plausible, so nothing looked
wrong, and a silent error of that kind would otherwise be found by a member rather
than by us.

## Schedule and backfill

- **`metrics_snapshot` runs daily at 03:15**, from Cloudron's scheduler addon
  (`CloudronManifest.json` → `addons.scheduler`), via `/app/code/metrics_snapshot.py`.
  It runs after `membership_sweep` (03:00) and `rag_refresh` (03:30).
- It writes **one row per day** into the `metrics_daily` table, in the same SQLite
  database as the members table (`members.db`). Columns: `day`, `members_active`,
  `members_total`, `members_new`, `newsletter_subs`, `matrix_space_members`,
  `spend_usd`, `requests`, `out_tokens`, `energy_wh`, `co2e_g`, `tokens_measured`,
  `tokens_estimated`, `factor_revision`, `recorded_at`.
- **`metrics_snapshot.py --backfill` rebuilds history** from the sources and is
  idempotent — it rewrites a day rather than duplicating it. It is the tool to run
  after changing a factor, and the reason a published chart can move: history is
  **rewritten**, not appended to.

## What is measured, and what is not

Stated plainly, because a mixed total is easy to over-read:

- **Measured:** GreenPT models — the providers whose API reports impact.
- **Assumed equal to a measurement:** `greenpt/glm-5.3`, from its flash sibling.
- **Estimated:** everything else. Currently that means **all Tinfoil and Regolo
  traffic**, which is a real share of co-op usage.

So a day's figure is a blend, and **the split is now recorded per day** —
`tokens_measured` and `tokens_estimated` say how much of the total rested on a
measurement, and the API derives a `provenance` read from them. Before those
columns existed the split was not recorded; such rows carry NULL and read as
`provenance: "none"`. See "Provenance in the API".

One consequence worth knowing: the proxy assumes a 250 gCO2e/kWh grid, while Regolo
runs on 100% renewable energy, so the proxy **overstates** Regolo's carbon. Per-provider
factors are a follow-up; until then, Regolo and Tinfoil figures should be read as upper
bounds rather than measurements.

## Changing the factors

Edit the table in `metrics_snapshot.py` and redeploy — refining a factor is a data
edit, not a code change. Then run `--backfill` if the change should apply to history.
The energy factors are also recorded in `co-op/docs/energy.md` for members, and the two
should not disagree.