gateway: price the Metis pass-through route from its reported tokens
The /v1/systemone pass-through is an EndpointType.GENERIC route: LiteLLM logs it, but with no model on the route the only cost it can apply is the flat cost_per_request (a budget estimate, not an invoice). So decisions were billed at a fixed 0.0001 instead of the real $0.12 per 1M input tokens GreenPT reports back in usage.input_tokens. Price the route from its own response body via a new build-time patch, patch_passthrough_token_cost.py, in the house style of the existing patch_*.py files: any route listed in general_settings.coop_passthrough_cost_per_token (USD per input token, keyed by path) is billed as usage.input_tokens x rate. cost_per_request stays as the fallback for a listed route whose response carries no token count, and every unlisted route is unchanged. Auth is untouched. config.yaml: list /v1/systemone at 0.00000012 ($0.12 per 1M input tokens); update the metering note.
This commit is contained in:
1 parent
557999a16f
commit
da02f8eb32
4 files changed
+221
-6
No files matched your search
+21
-6
@@ -245,12 +245,16 @@ general_settings:
|
||||
# GREENPT_API_KEY the green chat models already use.
|
||||
# Price: $0.12 per 1M *input* tokens; output is free.
|
||||
#
|
||||
# METERING IS APPROXIMATE. LiteLLM cannot price a pass-through route from the
|
||||
# response body, so cost_per_request (USD per request) is a flat estimate used
|
||||
# for budget accounting — 0.0001 ≈ a typical ~800-input-token call. Exact
|
||||
# per-token billing (reading usage.input_tokens from the Metis response) would
|
||||
# need a patch in the house style of the existing patch_*.py files; until then
|
||||
# treat the spend figure for this route as an approximation, not an invoice.
|
||||
# METERING: exact, from the token count GreenPT reports back. Metis is not a
|
||||
# model_list entry (LiteLLM's chat pipeline has nothing to parse), so it is
|
||||
# priced through the coop_passthrough_cost_per_token map below, keyed by this
|
||||
# route's path: a call's cost is usage.input_tokens × the rate. patch_
|
||||
# passthrough_token_cost.py (built in via the Dockerfile) reads
|
||||
# usage.input_tokens from the Metis response body at logging time and writes
|
||||
# the exact figure to member spend. cost_per_request on the route is now only
|
||||
# a fallback: it applies if a response ever arrives without a usable token
|
||||
# count, so a call is never silently free. Keep the two in sync.
|
||||
#
|
||||
# Routes here (matched by prefix) may be called by ANY authenticated member
|
||||
# key. LiteLLM's own gate for auth=true pass-through routes
|
||||
# (`allowed_passthrough_routes`) is an Enterprise-only field, so on this
|
||||
@@ -261,6 +265,16 @@ general_settings:
|
||||
# endpoint still requires a valid key).
|
||||
coop_passthrough_open_to_keys:
|
||||
- "/v1/systemone"
|
||||
# Exact per-input-token pricing (USD per token) for pass-through routes that
|
||||
# report their own usage in the RESPONSE BODY — the shape the x-litellm-*
|
||||
# response-header contract does not cover. Keyed by the gateway route path
|
||||
# (an upstream target URL also matches). Any route listed here is billed from
|
||||
# usage.input_tokens × the rate instead of its flat cost_per_request; a route
|
||||
# not listed is unchanged. Consumed at build time by
|
||||
# patch_passthrough_token_cost.py.
|
||||
coop_passthrough_cost_per_token:
|
||||
# Metis (GreenPT): $0.12 per 1M input tokens, output always free.
|
||||
"/v1/systemone": 0.00000012
|
||||
pass_through_endpoints:
|
||||
- path: "/v1/systemone"
|
||||
target: "https://api.greenpt.ai/v1/systemone"
|
||||
@@ -270,6 +284,7 @@ general_settings:
|
||||
auth: true
|
||||
methods: ["POST"]
|
||||
timeout: 60
|
||||
# Fallback only — the exact token cost above is what bills in practice.
|
||||
cost_per_request: 0.0001
|
||||
|
||||
litellm_settings:
|
||||
|
||||
Reference in new issue
Block a user