gateway: record token counts on Metis pass-through spend rows
The pass-through row for /v1/systemone carried the correct spend but prompt_tokens/completion_tokens/total_tokens were all 0: the generic pass-through success handler hands the spend writer a StandardPassThroughResponseObject with no usage, and the generic branch builds no standard_logging_object usage, so anything summing tokens for the route under-reported. patch_passthrough_token_cost.py now also sets combined_usage_object (a litellm.Usage carrying prompt_tokens=input_tokens, completion_tokens=0, total_tokens=input_tokens) on the logging object -- the same override the streaming handler uses for recovered partial usage, which the spend writer reads before the response object. Spend is unchanged. Defect 2 (a second zero-spend row with call_type=/v1/systemone) is NOT duplication: it is upstream LiteLLM's failure row for a rejected (missing/invalid key) request, written by ProxyLogging._handle_logging_proxy_only_error on the pristine image. One authenticated call produces exactly one priced row. Documented in the README; no code change.
This commit is contained in:
1 parent
da02f8eb32
commit
d6cfe78c77
2 files changed
+45
-2
No files matched your search
@@ -39,6 +39,22 @@ payload — the ``general_settings`` dict is read straight from the proxy module
|
||||
the same way ``patch_passthrough_key_access.py`` reads
|
||||
``coop_passthrough_open_to_keys``.
|
||||
|
||||
TOKEN COUNTS. Pricing the row from the body is not enough: the row still
|
||||
recorded ``prompt_tokens=0``, ``completion_tokens=0``, ``total_tokens=0``,
|
||||
because the pass-through success handler hands the spend writer a
|
||||
``StandardPassThroughResponseObject`` whose text carries the body but whose
|
||||
``usage`` is empty, and the generic pass-through branch builds no
|
||||
``standard_logging_object`` usage either. Any per-token reporting (panel,
|
||||
dashboard, energy) that sums tokens for this route then under-reports. So the
|
||||
patch also carries the count onto the logging object as
|
||||
``combined_usage_object`` — the same ``litellm.Usage`` override the streaming
|
||||
handler uses to recover partial usage. The spend writer reads
|
||||
``combined_usage_object`` (``spend_tracking_utils.get_logging_payload`` and
|
||||
``StandardLoggingPayloadSetup.get_usage_as_dict``) before it falls back to the
|
||||
response object, so the row records ``prompt_tokens=input_tokens``,
|
||||
``completion_tokens=0``, ``total_tokens=input_tokens`` while spend stays exactly
|
||||
``input_tokens × rate``.
|
||||
|
||||
The exact figure wins; ``cost_per_request`` stays as the fallback for a listed
|
||||
route whose body carries no usable input-token count, and for every route not
|
||||
listed. An upstream that reports its own totals via the x-litellm-* headers still
|
||||
@@ -83,7 +99,8 @@ NEW = (
|
||||
" # token count rather than cost_per_request's flat estimate. GreenPT's\n"
|
||||
" # Metis decision route (/v1/systemone) is the first such route: it\n"
|
||||
" # returns usage.input_tokens and always 0 output tokens, billed at\n"
|
||||
" # $0.12 per 1M input tokens. Inert unless a route is listed; a listed\n"
|
||||
" # $0.12 per 1M input tokens and recorded on the spend row with the\n"
|
||||
" # same count. Inert unless a route is listed; a listed\n"
|
||||
" # route whose body carries no token count falls through to\n"
|
||||
" # cost_per_request below, and every unlisted route is unchanged.\n"
|
||||
" try:\n"
|
||||
@@ -130,6 +147,30 @@ NEW = (
|
||||
" _coop_cost = float(_coop_tokens) * float(_coop_rate)\n"
|
||||
" kwargs[\"response_cost\"] = _coop_cost\n"
|
||||
" logging_obj.model_call_details[\"response_cost\"] = _coop_cost\n"
|
||||
" # [coop-patch] Record the true token counts on the row too.\n"
|
||||
" # The spend writer reads usage from combined_usage_object\n"
|
||||
" # (the override the streaming handler already uses for\n"
|
||||
" # recovered partial usage); without it the generic\n"
|
||||
" # pass-through response object carries no usage and the\n"
|
||||
" # row's prompt/completion/total tokens stay 0.\n"
|
||||
" _coop_usage = None\n"
|
||||
" try:\n"
|
||||
" from litellm.types.utils import Usage as _CoopUsage\n"
|
||||
"\n"
|
||||
" _coop_usage = _CoopUsage(\n"
|
||||
" prompt_tokens=int(_coop_tokens),\n"
|
||||
" completion_tokens=0,\n"
|
||||
" total_tokens=int(_coop_tokens),\n"
|
||||
" )\n"
|
||||
" except Exception as _coop_usage_exc:\n"
|
||||
" print(\n"
|
||||
" \"[coop-passthrough-cost] usage object failed: %s: %s\"\n"
|
||||
" % (type(_coop_usage_exc).__name__, _coop_usage_exc),\n"
|
||||
" flush=True,\n"
|
||||
" )\n"
|
||||
" if _coop_usage is not None:\n"
|
||||
" kwargs[\"combined_usage_object\"] = _coop_usage\n"
|
||||
" logging_obj.model_call_details[\"combined_usage_object\"] = _coop_usage\n"
|
||||
" print(\n"
|
||||
" \"[coop-passthrough-cost] input_tokens=%s rate=%s -> response_cost=%s\"\n"
|
||||
" % (_coop_tokens, _coop_rate, _coop_cost),\n"
|
||||
|
||||
Reference in new issue
Block a user