Metis decisions route registers no spend against the calling key #43

Closed
opened 2026-11-03 18:47:27 +00:00 by inference-bot · 1 comment
Owner

A call to the Metis decisions route succeeds but registers no spend against the key that made it.

Evidence (2026-11-03): minted a throwaway member-style key, confirmed its spend was 0.0, made one POST /v1/systemone call (200 OK, usage.input_tokens: 140, output_tokens: 0), waited, and re-read the key: spend still 0.0. GET /spend/logs showed rows for chat models (deepseek-v4.1-flash, gpt-oss-120b) with no row for the route.

Cause: the endpoint is a LiteLLM pass_through_endpoints entry with cost_per_request: 0.0001, but that value is not landing on the caller's spend. LiteLLM cannot price this route from the response body, so a flat per-request cost was the intended fallback.

Consequences:

  1. Decisions do not count toward a member's monthly allowance, which is how the co-op budgets. The route is effectively free to members until this is fixed.
  2. The published docs asserted the opposite until it was corrected today (see the docs commit); the model list and API access pages now state that metering is not yet active.

Two things to fix, in order:

  • Make the route register spend at all, so the allowance applies.
  • Then make it accurate: Metis returns exact usage.input_tokens (output_tokens is always 0) and costs $0.12 per 1M input tokens, so a patch in the house style of the existing patch_*_cost.py files in code/litellm can compute the true figure per call instead of a flat approximation.

Found while documenting the route for members.

A call to the Metis decisions route succeeds but registers **no spend** against the key that made it. **Evidence (2026-11-03):** minted a throwaway member-style key, confirmed its spend was `0.0`, made one `POST /v1/systemone` call (200 OK, `usage.input_tokens: 140`, `output_tokens: 0`), waited, and re-read the key: spend still `0.0`. `GET /spend/logs` showed rows for chat models (deepseek-v4.1-flash, gpt-oss-120b) with no row for the route. **Cause:** the endpoint is a LiteLLM `pass_through_endpoints` entry with `cost_per_request: 0.0001`, but that value is not landing on the caller's spend. LiteLLM cannot price this route from the response body, so a flat per-request cost was the intended fallback. **Consequences:** 1. Decisions do not count toward a member's monthly allowance, which is how the co-op budgets. The route is effectively free to members until this is fixed. 2. The published docs asserted the opposite until it was corrected today (see the docs commit); the model list and API access pages now state that metering is not yet active. **Two things to fix, in order:** - Make the route register spend at all, so the allowance applies. - Then make it *accurate*: Metis returns exact `usage.input_tokens` (`output_tokens` is always 0) and costs $0.12 per 1M input tokens, so a patch in the house style of the existing `patch_*_cost.py` files in `code/litellm` can compute the true figure per call instead of a flat approximation. Found while documenting the route for members.
Author
Owner

Fixed, and the original diagnosis was wrong. Recorded here because the wrong version would otherwise be the one the next reader trusts.

Root cause — not what the issue said. The generic pass-through path does dispatch spend logging and does attribute the row to the calling key. The earlier "registers no spend" reading was a false negative from the wrong surface: a short-window GET /spend/logs read too soon after the call, filtered by a model name that never matches (the row's model is the request body's model, i.e. metis, not the path systemone). Measured again in isolation: a fresh member key whose only call was POST /v1/systemone went from spend 0.0 to exactly 0.0001, and the ledger holds one row for it — call_type=pass_through_endpoint, model=metis, spend=0.0001, with a key hash.

So the real defect was accuracy, not dispatch: the flat cost_per_request: 0.0001 was landing, but the route is billed per token ($0.12 per 1M input tokens) and GreenPT returns the exact usage.input_tokens, so the flat estimate was wrong by whatever factor the call differed.

Fix. A new build-time patch in code/litellm, patch_passthrough_token_cost.py, in the house style of the existing patch_*.py files. Any route listed in the new general_settings.coop_passthrough_cost_per_token map (USD per input token, keyed by path) is billed as usage.input_tokens × rate, read from the response body at logging time; cost_per_request stays as the fallback for a listed route whose body carries no token count, and every unlisted route is unchanged. Auth is untouched. This is general to generic pass-through routes, not a /v1/systemone special case — the next non-OpenAI-shaped provider route we add is metered simply by listing it in the map.

Deployed image local/ai.coop.litellm:1.0.0-1793732484679; commit da02f8e on code/litellm main.

Verification (probe key, deployed build). One POST /v1/systemone returned usage.input_tokens: 131.

  • expected cost: 131 × 0.12 / 1,000,000 = 0.00001572
  • observed key-spend delta: 0.00001572 (spend 0.0 → 0.00001572)
  • ledger row present: call_type=pass_through_endpoint, model=metis, spend=0.00001572, attributed to the key.
  • unauthorised access still refused: a bad key to /v1/systemone → 401; a normal chat completion with the same probe key → 200. Probe key deleted.

Docs. The "not yet metered" warnings are replaced with the accurate position — decisions count toward the same allowance, at $0.12 per 1M input tokens from the exact token count. co-op/docs commit cdb62fc (models.md, api-access.md).

With metering live and accurate, this is closed.

**Fixed, and the original diagnosis was wrong.** Recorded here because the wrong version would otherwise be the one the next reader trusts. **Root cause — not what the issue said.** The generic pass-through path *does* dispatch spend logging and *does* attribute the row to the calling key. The earlier "registers no spend" reading was a false negative from the wrong surface: a short-window `GET /spend/logs` read too soon after the call, filtered by a model name that never matches (the row's `model` is the request body's `model`, i.e. `metis`, not the path `systemone`). Measured again in isolation: a fresh member key whose only call was `POST /v1/systemone` went from spend `0.0` to exactly `0.0001`, and the ledger holds one row for it — `call_type=pass_through_endpoint`, `model=metis`, `spend=0.0001`, with a key hash. So the real defect was **accuracy, not dispatch**: the flat `cost_per_request: 0.0001` was landing, but the route is billed per token ($0.12 per 1M input tokens) and GreenPT returns the exact `usage.input_tokens`, so the flat estimate was wrong by whatever factor the call differed. **Fix.** A new build-time patch in `code/litellm`, `patch_passthrough_token_cost.py`, in the house style of the existing `patch_*.py` files. Any route listed in the new `general_settings.coop_passthrough_cost_per_token` map (USD per input token, keyed by path) is billed as `usage.input_tokens × rate`, read from the response body at logging time; `cost_per_request` stays as the fallback for a listed route whose body carries no token count, and every unlisted route is unchanged. Auth is untouched. This is general to generic pass-through routes, not a `/v1/systemone` special case — the next non-OpenAI-shaped provider route we add is metered simply by listing it in the map. Deployed image `local/ai.coop.litellm:1.0.0-1793732484679`; commit `da02f8e` on `code/litellm` main. **Verification (probe key, deployed build).** One `POST /v1/systemone` returned `usage.input_tokens: 131`. - expected cost: 131 × 0.12 / 1,000,000 = **0.00001572** - observed key-spend delta: **0.00001572** (spend `0.0` → `0.00001572`) - ledger row present: `call_type=pass_through_endpoint`, `model=metis`, `spend=0.00001572`, attributed to the key. - unauthorised access still refused: a bad key to `/v1/systemone` → **401**; a normal chat completion with the same probe key → **200**. Probe key deleted. **Docs.** The "not yet metered" warnings are replaced with the accurate position — decisions count toward the same allowance, at $0.12 per 1M input tokens from the exact token count. `co-op/docs` commit `cdb62fc` (models.md, api-access.md). With metering live and accurate, this is closed.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: co-op/support#43