gateway: add tinfoil/voxtral-small-24b, Tinfoil's token-priced STT model

Tinfoil's catalog prices exactly one audio model per token: voxtral-small-24b
($0.20 in / $0.60 out per 1M, $0 per request). Wiring it gives members an
STT route without the whisper $0.01/request floor (co-op/support#48).

Two billing shapes on one model entry:
- chat route (input_audio): returns real token usage -> bills exactly per token.
- /v1/audio/transcriptions: Tinfoil returns NO token counts (duration only),
  so per-token billing cannot fire; Tinfoil also tokenizes audio in ~30s
  chunks (~375-380 tokens each regardless of clip length), making the
  chunk-equivalent ~$0.00008/request. Declared input_cost_per_request with
  that measured value, added by patch_audio_request_cost.py.

patch_audio_request_cost.py: scope the per-request charge to the audio call
types (transcription/atranscription/speech/aspeech) so this model's chat
calls bill per token and are not double-billed; extend the diagnostic
name-gate to 'voxtral'.

Measured on the live catalog + sidecar 2026-11-06 (see issue #48 follow-up).
This commit is contained in:
inference-bot committed 2026-11-06 22:33:17 +00:00
1 parent 54925cae97
commit 67df968a04
2 files changed
+61 -9

No files matched your search

+34 -9
View File
@@ -27,7 +27,13 @@ from the per-deployment ``custom_pricing`` the logging layer builds out of our
call metadata.
Scope: nothing changes unless a model declares ``input_cost_per_request``, so
this is inert for every other model on the gateway.
this is inert for every other model on the gateway. Since 2026-11-06 the charge
is added only on the AUDIO call types (transcription/atranscription/speech/
aspeech): a model that also carries per-token pricing (``voxtral-small-24b``)
bills its chat calls exactly per token, and a per-request charge on top of
that would double-bill them. The audio endpoints are the paths that cannot
price per token (the provider returns no token counts there), so the flat
charge belongs to them alone.
Upstream: no issue filed; LiteLLM has no per-request audio pricing as of 1.103.2.
"""
@@ -58,7 +64,9 @@ HELPER = '''def _coop_per_request_cost(
"""
def _is_audio(name: Any) -> bool:
text = str(name or "").lower()
return "tts" in text or "whisper" in text
return (
"tts" in text or "whisper" in text or "voxtral" in text
)
cost_map = {}
try:
@@ -140,17 +148,34 @@ PATCHED_RETURN = """ region_name=region_name,
# per-request charge for models LiteLLM can only price per unit
# (Tinfoil's voxtral-tts, whisper-large-v3-turbo). Inert when the model
# declares nothing; the early "provider reported the cost" return above
# is unaffected.
# is unaffected. Scoped to the AUDIO call types (2026-11-06): a model
# that ALSO carries per-token pricing (voxtral-small-24b) bills its
# chat calls exactly per token and must not take the per-request
# charge on top of that.
try:
_coop_pr = _coop_per_request_cost(
model=model,
litellm_model_name=litellm_model_name,
custom_pricing=custom_pricing,
litellm_logging_obj=litellm_logging_obj,
_coop_audio_call = call_type in (
"transcription",
"atranscription",
"speech",
"aspeech",
)
_coop_pr = (
_coop_per_request_cost(
model=model,
litellm_model_name=litellm_model_name,
custom_pricing=custom_pricing,
litellm_logging_obj=litellm_logging_obj,
)
if _coop_audio_call
else 0.0
)
response_cost = (response_cost or 0.0) + _coop_pr
_coop_name = str(model or "") + str(litellm_model_name or "")
if "tts" in _coop_name.lower() or "whisper" in _coop_name.lower():
if _coop_audio_call and (
"tts" in _coop_name.lower()
or "whisper" in _coop_name.lower()
or "voxtral" in _coop_name.lower()
):
print(
f"[coop-audio] add per_request={_coop_pr!r} -> response_cost={response_cost!r}",
flush=True,