gateway: bill Tinfoil's per-request charges on the audio paths
Tinfoil bills voxtral-tts $0.002 and whisper-large-v3-turbo $0.01 per REQUEST, with $0 per token for both. Reading only the two token prices in its model list made TTS look free and left STT billed at an older catalog's $0.05/1M tokens, so neither matched what the co-op actually pays. LiteLLM cannot express a per-request charge on the audio paths: per-request pricing exists only on the search/rerank/MCP paths via input_cost_per_query, and input_cost_per_request is read only by the proxy's cost-tracking settings, never by the model cost calculator. So patch_audio_request_cost.py, a build-time patch in the same shape as patch_image_cost.py, makes response_cost_calculator honour a declared input_cost_per_request. That function is the single funnel for every call type — the proxy dispatches audio as `aspeech` and `transcription` and reaches it through Logging._response_cost_calculator — so one insertion covers both. The charge resolves from the per-deployment custom_pricing the logging layer builds from our model_info, then the shared cost map, then call metadata, and the patch is inert for any model that declares nothing. config.yaml declares the real charges: $0.002/request for voxtral-tts, and $0.01/request for whisper with its token price corrected to $0. Verified before committing: the patch applies to the installed 1.103.2 anchors and compiles; the helper's resolution order passes unit checks (all branches, plus inert and garbage cases); the resulting config validates as YAML with all 12 entries intact. End-to-end spend is proven after deploy by a real TTS/STT call. Refs code/litellm#5
This commit is contained in:
1 parent
f75851d6cf
commit
425d738625
4 files changed
+204
-2
No files matched your search
@@ -35,6 +35,14 @@ RUN python3 /app/code/patch_budget_message.py
|
||||
COPY patch_image_cost.py /app/code/patch_image_cost.py
|
||||
RUN python3 /app/code/patch_image_cost.py
|
||||
|
||||
# Audio per-request pricing: Tinfoil bills voxtral-tts and whisper per request,
|
||||
# which LiteLLM cannot express on the audio paths, so those two were billed at
|
||||
# $0. The patch makes the cost calculator honour a declared
|
||||
# input_cost_per_request. Check for the "[coop-patch] applied" line in the build
|
||||
# log after upgrading LiteLLM: the script is non-fatal and warns instead.
|
||||
COPY patch_audio_request_cost.py /app/code/patch_audio_request_cost.py
|
||||
RUN python3 /app/code/patch_audio_request_cost.py
|
||||
|
||||
# Audio keep-alive: scheduled pings keep the Tinfoil TEE audio enclave
|
||||
# warm so voice mode skips the ~24s cold-start (see audio_keepalive.py header).
|
||||
COPY audio_keepalive.py /app/code/audio_keepalive.py
|
||||
|
||||
Reference in new issue
Block a user