gateway: bill Tinfoil's per-request charges on the audio paths
Tinfoil bills voxtral-tts $0.002 and whisper-large-v3-turbo $0.01 per REQUEST, with $0 per token for both. Reading only the two token prices in its model list made TTS look free and left STT billed at an older catalog's $0.05/1M tokens, so neither matched what the co-op actually pays. LiteLLM cannot express a per-request charge on the audio paths: per-request pricing exists only on the search/rerank/MCP paths via input_cost_per_query, and input_cost_per_request is read only by the proxy's cost-tracking settings, never by the model cost calculator. So patch_audio_request_cost.py, a build-time patch in the same shape as patch_image_cost.py, makes response_cost_calculator honour a declared input_cost_per_request. That function is the single funnel for every call type — the proxy dispatches audio as `aspeech` and `transcription` and reaches it through Logging._response_cost_calculator — so one insertion covers both. The charge resolves from the per-deployment custom_pricing the logging layer builds from our model_info, then the shared cost map, then call metadata, and the patch is inert for any model that declares nothing. config.yaml declares the real charges: $0.002/request for voxtral-tts, and $0.01/request for whisper with its token price corrected to $0. Verified before committing: the patch applies to the installed 1.103.2 anchors and compiles; the helper's resolution order passes unit checks (all branches, plus inert and garbage cases); the resulting config validates as YAML with all 12 entries intact. End-to-end spend is proven after deploy by a real TTS/STT call. Refs code/litellm#5
This commit is contained in:
1 parent
f75851d6cf
commit
425d738625
4 files changed
+204
-2
No files matched your search
+14
-2
@@ -148,7 +148,14 @@ model_list:
|
||||
# models: the member portal's /v1/models filters them out of the chat model
|
||||
# list (see AUDIO_MODEL_PREFIXES there) so members don't see them in the
|
||||
# picker, but they remain fully callable via /v1/audio/* for API users.
|
||||
# whisper-large-v3-turbo: $0.05/1M input tokens. voxtral-tts: listed $0/$0.
|
||||
# whisper-large-v3-turbo: $0/1M tokens + $0.01 per request.
|
||||
# voxtral-tts: $0/1M tokens + $0.002 per request.
|
||||
# Tinfoil bills these per REQUEST (its model list carries a `requestPrice`);
|
||||
# reading only the two token prices made TTS look free and left STT billed off
|
||||
# an older catalog. LiteLLM cannot price per request on the audio paths, so
|
||||
# patch_audio_request_cost.py (built in via the Dockerfile) makes the cost
|
||||
# calculator honour `input_cost_per_request` below. Re-check that build log
|
||||
# line after every LiteLLM upgrade.
|
||||
# NOTE: LiteLLM routes audio by the openai/ provider + the endpoint called
|
||||
# (/v1/audio/transcriptions or /v1/audio/speech) — there is no
|
||||
# "audio_transcription/" provider prefix.
|
||||
@@ -159,7 +166,10 @@ model_list:
|
||||
api_key: os.environ/TINFOIL_API_KEY
|
||||
model_info:
|
||||
mode: audio_transcription
|
||||
input_cost_per_token: 0.00000005
|
||||
# Tinfoil: $0 per token, $0.01 per request. The per-request charge is the
|
||||
# whole cost, and the patch is what makes it billable.
|
||||
input_cost_per_token: 0
|
||||
input_cost_per_request: 0.01
|
||||
|
||||
- model_name: tinfoil/voxtral-tts
|
||||
litellm_params:
|
||||
@@ -168,6 +178,8 @@ model_list:
|
||||
api_key: os.environ/TINFOIL_API_KEY
|
||||
model_info:
|
||||
mode: audio_speech
|
||||
# Tinfoil: $0 per token, $0.002 per request — see the audio section note.
|
||||
input_cost_per_request: 0.002
|
||||
|
||||
# --- Embedding model (Tinfoil, in-TEE) ---
|
||||
# nomic-embed-text — 768-dim text embeddings for RAG/semantic search.
|
||||
|
||||
Reference in new issue
Block a user