From d6cfe78c775defc9d230f51b5a441fe57f7387a8 Mon Sep 17 00:00:00 2001 From: inference-bot Date: Tue, 3 Nov 2026 19:13:09 +0000 Subject: [PATCH] gateway: record token counts on Metis pass-through spend rows The pass-through row for /v1/systemone carried the correct spend but prompt_tokens/completion_tokens/total_tokens were all 0: the generic pass-through success handler hands the spend writer a StandardPassThroughResponseObject with no usage, and the generic branch builds no standard_logging_object usage, so anything summing tokens for the route under-reported. patch_passthrough_token_cost.py now also sets combined_usage_object (a litellm.Usage carrying prompt_tokens=input_tokens, completion_tokens=0, total_tokens=input_tokens) on the logging object -- the same override the streaming handler uses for recovered partial usage, which the spend writer reads before the response object. Spend is unchanged. Defect 2 (a second zero-spend row with call_type=/v1/systemone) is NOT duplication: it is upstream LiteLLM's failure row for a rejected (missing/invalid key) request, written by ProxyLogging._handle_logging_proxy_only_error on the pristine image. One authenticated call produces exactly one priced row. Documented in the README; no code change. --- README.md | 4 ++- patch_passthrough_token_cost.py | 43 ++++++++++++++++++++++++++++++++- 2 files changed, 45 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 3e7f081..6664eb3 100644 --- a/README.md +++ b/README.md @@ -167,7 +167,9 @@ expect a 15–20 min maintenance window and **never panic-restart mid-migration* - Check the [LiteLLM changelog](https://github.com/BerriAI/litellm/releases) for breaking changes - The `start.sh` and `config.yaml` may need adjustment if LiteLLM changes its env-var or config schema - **Build-time patches ride on LiteLLM internals — verify each one after an upgrade.** `patch_budget_message.py`, `patch_image_cost.py` (keeps image-gen pricing on the shared cost-map key) and `patch_audio_request_cost.py` (adds the declared per-request charge on the audio paths: Tinfoil bills `voxtral-tts` $0.002 and `whisper-large-v3-turbo` $0.01 per request, which the cost map cannot express). Each prints `[coop-patch]` in the build log. They are non-fatal by design: a missing anchor warns and the build still succeeds, so the behaviour can vanish silently on a version bump. Related: code/litellm#5. -- **`patch_passthrough_token_cost.py`** prices a generic pass-through route from the token count it reports in its own response body. A `pass_through_endpoints` target whose host matches none of vertex/anthropic/openai/gemini resolves to `EndpointType.GENERIC`: it is logged, but with no model on the route LiteLLM only knows the flat `cost_per_request`, which is a budget estimate, not an invoice. The patch bills any route listed in `general_settings.coop_passthrough_cost_per_token` (USD per input token, keyed by path) as `usage.input_tokens × rate` — Metis (`/v1/systemone`) at $0.12 per 1M input tokens. Same rules: non-fatal, `[coop-patch]` in the build log, verify after every upgrade. `cost_per_request` stays as the fallback for a listed route whose response carries no token count. The block prints `[coop-passthrough-cost] input_tokens=… rate=… -> response_cost=…` on each priced call, so the log is the field check that it fired. +- **`patch_passthrough_token_cost.py`** prices a generic pass-through route from the token count it reports in its own response body. A `pass_through_endpoints` target whose host matches none of vertex/anthropic/openai/gemini resolves to `EndpointType.GENERIC`: it is logged, but with no model on the route LiteLLM only knows the flat `cost_per_request`, which is a budget estimate, not an invoice. The patch bills any route listed in `general_settings.coop_passthrough_cost_per_token` (USD per input token, keyed by path) as `usage.input_tokens × rate` — Metis (`/v1/systemone`) at $0.12 per 1M input tokens — and records the same count on the row (see below). Same rules: non-fatal, `[coop-patch]` in the build log, verify after every upgrade. `cost_per_request` stays as the fallback for a listed route whose response carries no token count. The block prints `[coop-passthrough-cost] input_tokens=… rate=… -> response_cost=…` on each priced call, so the log is the field check that it fired. +- **Pass-through spend rows also carry the token counts (Metis defect fix).** Pricing the row from the body was not enough: the row recorded the correct `spend` but `prompt_tokens`/`completion_tokens`/`total_tokens` stayed `0`, because the pass-through success handler hands the spend writer a `StandardPassThroughResponseObject` whose `usage` is empty and the generic branch builds no `standard_logging_object` usage. Anything summing tokens for the route (panel, dashboard, energy reporting) then under-reported. The same patch now sets `combined_usage_object` (a `litellm.Usage`) on the logging object — the override the streaming handler already uses for recovered partial usage — which the spend writer reads before the response object (`spend_tracking_utils.get_logging_payload`, `StandardLoggingPayloadSetup.get_usage_as_dict`). The row now records `prompt_tokens = usage.input_tokens`, `completion_tokens = 0`, `total_tokens = usage.input_tokens`, with spend unchanged. +- **A zero-spend row with `call_type = /v1/` is NOT a duplicate of the priced row — it is the 401 refusal.** Every *rejected* pass-through request (missing/invalid key) is logged by upstream LiteLLM as a failure row whose `call_type` is the route path and whose `spend` is `0`, via `ProxyLogging._handle_logging_proxy_only_error` (`litellm/proxy/utils.py`, `original_function=route`). That is upstream behaviour on the pristine image, not something our patches add (neither `patch_passthrough_key_access.py` nor `patch_passthrough_token_cost.py` writes a row). A single *authenticated* call produces exactly **one** row (`call_type=pass_through_endpoint`, the real spend). Measured: one 200 → one row; one unauthenticated 401 → one failure row. Count requests for the route by `call_type = pass_through_endpoint`, not by summing all rows carrying the model name. - The CORS env var is `LITELLM_CORS_ORIGINS` (upstream-native since 1.84.0); the old `LITELLM_CORS_ALLOWED_ORIGINS` + sed patch are dead — don't re-add them - Test on a staging instance if the version jump is large diff --git a/patch_passthrough_token_cost.py b/patch_passthrough_token_cost.py index 87c456b..24cb0dc 100644 --- a/patch_passthrough_token_cost.py +++ b/patch_passthrough_token_cost.py @@ -39,6 +39,22 @@ payload — the ``general_settings`` dict is read straight from the proxy module the same way ``patch_passthrough_key_access.py`` reads ``coop_passthrough_open_to_keys``. +TOKEN COUNTS. Pricing the row from the body is not enough: the row still +recorded ``prompt_tokens=0``, ``completion_tokens=0``, ``total_tokens=0``, +because the pass-through success handler hands the spend writer a +``StandardPassThroughResponseObject`` whose text carries the body but whose +``usage`` is empty, and the generic pass-through branch builds no +``standard_logging_object`` usage either. Any per-token reporting (panel, +dashboard, energy) that sums tokens for this route then under-reports. So the +patch also carries the count onto the logging object as +``combined_usage_object`` — the same ``litellm.Usage`` override the streaming +handler uses to recover partial usage. The spend writer reads +``combined_usage_object`` (``spend_tracking_utils.get_logging_payload`` and +``StandardLoggingPayloadSetup.get_usage_as_dict``) before it falls back to the +response object, so the row records ``prompt_tokens=input_tokens``, +``completion_tokens=0``, ``total_tokens=input_tokens`` while spend stays exactly +``input_tokens × rate``. + The exact figure wins; ``cost_per_request`` stays as the fallback for a listed route whose body carries no usable input-token count, and for every route not listed. An upstream that reports its own totals via the x-litellm-* headers still @@ -83,7 +99,8 @@ NEW = ( " # token count rather than cost_per_request's flat estimate. GreenPT's\n" " # Metis decision route (/v1/systemone) is the first such route: it\n" " # returns usage.input_tokens and always 0 output tokens, billed at\n" - " # $0.12 per 1M input tokens. Inert unless a route is listed; a listed\n" + " # $0.12 per 1M input tokens and recorded on the spend row with the\n" + " # same count. Inert unless a route is listed; a listed\n" " # route whose body carries no token count falls through to\n" " # cost_per_request below, and every unlisted route is unchanged.\n" " try:\n" @@ -130,6 +147,30 @@ NEW = ( " _coop_cost = float(_coop_tokens) * float(_coop_rate)\n" " kwargs[\"response_cost\"] = _coop_cost\n" " logging_obj.model_call_details[\"response_cost\"] = _coop_cost\n" + " # [coop-patch] Record the true token counts on the row too.\n" + " # The spend writer reads usage from combined_usage_object\n" + " # (the override the streaming handler already uses for\n" + " # recovered partial usage); without it the generic\n" + " # pass-through response object carries no usage and the\n" + " # row's prompt/completion/total tokens stay 0.\n" + " _coop_usage = None\n" + " try:\n" + " from litellm.types.utils import Usage as _CoopUsage\n" + "\n" + " _coop_usage = _CoopUsage(\n" + " prompt_tokens=int(_coop_tokens),\n" + " completion_tokens=0,\n" + " total_tokens=int(_coop_tokens),\n" + " )\n" + " except Exception as _coop_usage_exc:\n" + " print(\n" + " \"[coop-passthrough-cost] usage object failed: %s: %s\"\n" + " % (type(_coop_usage_exc).__name__, _coop_usage_exc),\n" + " flush=True,\n" + " )\n" + " if _coop_usage is not None:\n" + " kwargs[\"combined_usage_object\"] = _coop_usage\n" + " logging_obj.model_call_details[\"combined_usage_object\"] = _coop_usage\n" " print(\n" " \"[coop-passthrough-cost] input_tokens=%s rate=%s -> response_cost=%s\"\n" " % (_coop_tokens, _coop_rate, _coop_cost),\n"