gateway: add tinfoil/voxtral-small-24b, Tinfoil's token-priced STT model
Tinfoil's catalog prices exactly one audio model per token: voxtral-small-24b ($0.20 in / $0.60 out per 1M, $0 per request). Wiring it gives members an STT route without the whisper $0.01/request floor (co-op/support#48). Two billing shapes on one model entry: - chat route (input_audio): returns real token usage -> bills exactly per token. - /v1/audio/transcriptions: Tinfoil returns NO token counts (duration only), so per-token billing cannot fire; Tinfoil also tokenizes audio in ~30s chunks (~375-380 tokens each regardless of clip length), making the chunk-equivalent ~$0.00008/request. Declared input_cost_per_request with that measured value, added by patch_audio_request_cost.py. patch_audio_request_cost.py: scope the per-request charge to the audio call types (transcription/atranscription/speech/aspeech) so this model's chat calls bill per token and are not double-billed; extend the diagnostic name-gate to 'voxtral'. Measured on the live catalog + sidecar 2026-11-06 (see issue #48 follow-up).
This commit is contained in:
1 parent
54925cae97
commit
67df968a04
2 files changed
+61
-9
No files matched your search
+27
@@ -11,6 +11,8 @@
|
||||
# tinfoil/gpt-oss-120b: $0.15 in / $0.60 out per 1M
|
||||
# tinfoil/glm-5.3: $1.80 in / $5.75 out / $0.45 cached per 1M
|
||||
# greenpt/qwen3-embedding-8b: $0.25 in / $0 out per 1M (embeddings, 32K ctx)
|
||||
# tinfoil/voxtral-small-24b: $0.20 in / $0.60 out per 1M (token-priced audio;
|
||||
# +$0.00008/request on /v1/audio/transcriptions)
|
||||
# Without these, LiteLLM can't price these custom models, so spend shows $0.00
|
||||
# and the team budget cap is not enforced.
|
||||
#
|
||||
@@ -151,6 +153,8 @@ model_list:
|
||||
# picker, but they remain fully callable via /v1/audio/* for API users.
|
||||
# whisper-large-v3-turbo: $0/1M tokens + $0.01 per request.
|
||||
# voxtral-tts: $0/1M tokens + $0.002 per request.
|
||||
# voxtral-small-24b: $0.20/1M in + $0.60/1M out tokens, $0 per request —
|
||||
# the only TOKEN-PRICED audio model in their catalog (its block below).
|
||||
# Tinfoil bills these per REQUEST (its model list carries a `requestPrice`);
|
||||
# reading only the two token prices made TTS look free and left STT billed off
|
||||
# an older catalog. LiteLLM cannot price per request on the audio paths, so
|
||||
@@ -172,6 +176,29 @@ model_list:
|
||||
input_cost_per_token: 0
|
||||
input_cost_per_request: 0.01
|
||||
|
||||
# voxtral-small-24b — Tinfoil's TOKEN-PRICED STT model: the only audio entry
|
||||
# in their catalog with real token prices ($0.20 in / $0.60 out per 1M, $0
|
||||
# per request). The chat route (audio sent as input_audio) returns real
|
||||
# token usage and bills exactly at those rates. The /v1/audio/transcriptions
|
||||
# route returns NO token counts (duration only), so per-token billing cannot
|
||||
# fire there; Tinfoil tokenizes audio in ~30s chunks (~375-380 tokens each —
|
||||
# measured: a 1-second clip and a 6-second clip both cost ~380 prompt
|
||||
# tokens), so for the short dictation requests this route serves the
|
||||
# chunk-equivalent is ~$0.00008/request — declared as input_cost_per_request
|
||||
# below, added by patch_audio_request_cost.py, which is scoped to the audio
|
||||
# call types so chat calls to this model are NOT double-billed. Audio longer
|
||||
# than ~30s should go through the chat route, which meters exactly.
|
||||
- model_name: tinfoil/voxtral-small-24b
|
||||
litellm_params:
|
||||
model: openai/voxtral-small-24b
|
||||
api_base: http://127.0.0.1:3301/v1
|
||||
api_key: os.environ/TINFOIL_API_KEY
|
||||
model_info:
|
||||
mode: audio_transcription
|
||||
input_cost_per_token: 0.0000002
|
||||
output_cost_per_token: 0.0000006
|
||||
input_cost_per_request: 0.00008
|
||||
|
||||
- model_name: tinfoil/voxtral-tts
|
||||
litellm_params:
|
||||
model: openai/voxtral-tts
|
||||
|
||||
@@ -27,7 +27,13 @@ from the per-deployment ``custom_pricing`` the logging layer builds out of our
|
||||
call metadata.
|
||||
|
||||
Scope: nothing changes unless a model declares ``input_cost_per_request``, so
|
||||
this is inert for every other model on the gateway.
|
||||
this is inert for every other model on the gateway. Since 2026-11-06 the charge
|
||||
is added only on the AUDIO call types (transcription/atranscription/speech/
|
||||
aspeech): a model that also carries per-token pricing (``voxtral-small-24b``)
|
||||
bills its chat calls exactly per token, and a per-request charge on top of
|
||||
that would double-bill them. The audio endpoints are the paths that cannot
|
||||
price per token (the provider returns no token counts there), so the flat
|
||||
charge belongs to them alone.
|
||||
|
||||
Upstream: no issue filed; LiteLLM has no per-request audio pricing as of 1.103.2.
|
||||
"""
|
||||
@@ -58,7 +64,9 @@ HELPER = '''def _coop_per_request_cost(
|
||||
"""
|
||||
def _is_audio(name: Any) -> bool:
|
||||
text = str(name or "").lower()
|
||||
return "tts" in text or "whisper" in text
|
||||
return (
|
||||
"tts" in text or "whisper" in text or "voxtral" in text
|
||||
)
|
||||
|
||||
cost_map = {}
|
||||
try:
|
||||
@@ -140,17 +148,34 @@ PATCHED_RETURN = """ region_name=region_name,
|
||||
# per-request charge for models LiteLLM can only price per unit
|
||||
# (Tinfoil's voxtral-tts, whisper-large-v3-turbo). Inert when the model
|
||||
# declares nothing; the early "provider reported the cost" return above
|
||||
# is unaffected.
|
||||
# is unaffected. Scoped to the AUDIO call types (2026-11-06): a model
|
||||
# that ALSO carries per-token pricing (voxtral-small-24b) bills its
|
||||
# chat calls exactly per token and must not take the per-request
|
||||
# charge on top of that.
|
||||
try:
|
||||
_coop_pr = _coop_per_request_cost(
|
||||
model=model,
|
||||
litellm_model_name=litellm_model_name,
|
||||
custom_pricing=custom_pricing,
|
||||
litellm_logging_obj=litellm_logging_obj,
|
||||
_coop_audio_call = call_type in (
|
||||
"transcription",
|
||||
"atranscription",
|
||||
"speech",
|
||||
"aspeech",
|
||||
)
|
||||
_coop_pr = (
|
||||
_coop_per_request_cost(
|
||||
model=model,
|
||||
litellm_model_name=litellm_model_name,
|
||||
custom_pricing=custom_pricing,
|
||||
litellm_logging_obj=litellm_logging_obj,
|
||||
)
|
||||
if _coop_audio_call
|
||||
else 0.0
|
||||
)
|
||||
response_cost = (response_cost or 0.0) + _coop_pr
|
||||
_coop_name = str(model or "") + str(litellm_model_name or "")
|
||||
if "tts" in _coop_name.lower() or "whisper" in _coop_name.lower():
|
||||
if _coop_audio_call and (
|
||||
"tts" in _coop_name.lower()
|
||||
or "whisper" in _coop_name.lower()
|
||||
or "voxtral" in _coop_name.lower()
|
||||
):
|
||||
print(
|
||||
f"[coop-audio] add per_request={_coop_pr!r} -> response_cost={response_cost!r}",
|
||||
flush=True,
|
||||
|
||||
Reference in new issue
Block a user