Meter Tinfoil's per-request prices (voxtral-tts, whisper) — LiteLLM has no per-request cost field on the audio path #5

New Issue

2026-10-26T20:22:18Z

inference-bot commented

2026-10-26 20:22:18 +00:00

Verified 2026-10-26 by reading the Tinfoil proxy's model list inside the gateway (http://127.0.0.1:3301/v1/models). Every Tinfoil model carries a pricing object with inputTokenPricePer1M, outputTokenPricePer1M and requestPrice.

What we actually pay

Model	Tinfoil publishes	We declare	Effect
`tinfoil/voxtral-tts`	$0/token + $0.002/request	nothing	speech is free to members; the co-op pays
`tinfoil/whisper-large-v3-turbo`	$0/token + $0.01/request	$0.05/1M input tokens	under-billed on typical dictation

Also published but not served by us: doc-upload $0.05/request, websearch $0.05/request, pii-filter $0.005/request, qwen3-tts $0.01/request.

The config comment for the audio block says voxtral-tts is "listed $0/$0 (verify on invoice — may be beta-free)": it read the two token prices and missed requestPrice. The Tinfoil list is authoritative and machine-readable, which is also what makes the monitor below possible.

The constraint

LiteLLM applies per-request pricing on the search, rerank and MCP-tool paths via input_cost_per_query (litellm/types/utils.py: "per-request pricing: rerank, search, and Bedrock Marengo embed"; consumed in litellm/search/cost_calculator.py as number_of_queries * cost_per_query). The audio paths (audio_speech, audio_transcription) are priced by characters, seconds or tokens. input_cost_per_request exists in the proxies' cost-tracking settings but is not read by the model cost calculator. So a per-request TTS/STT price cannot be expressed through the cost map as things stand.

Options

Calibrate an equivalent — publish input_cost_per_character (TTS) / input_cost_per_token (STT) derived from measured characters or tokens per request. Config-only, no code. Aggregate-accurate, but wrong per request: a burst of very short requests under-bills, which is an abuse hole precisely where requests are cheap and numerous.
Build-time patch — bill the audio paths a per-request cost, the way this repo already patches image-gen pricing (79f0447: "keeping image-gen pricing on the shared cost-map key … so image spend is tracked"). Exact, and consistent with the existing pattern; the cost is carrying another patch across LiteLLM upgrades.
Leave it — accept the subsidy. Not recommended while voice mode is the only unmetered capability on the gateway.

Recommendation: option 2, with the patch listed in the upgrade checklist for this app so it cannot be silently lost on a version bump.

The monitoring half is done: cron job 8f2429d0c4bf diffs our declared prices against Tinfoil's published ones and reports any change, so a price move shows up instead of silently drifting.
code/litellm#4 is closed: it billed $16/1M characters, a market rate rather than our cost.
code/litellm#3 (Bo) is the same class of gap on the discovery fields rather than the cost fields.

Verified 2026-10-26 by reading the Tinfoil proxy's model list inside the gateway (`http://127.0.0.1:3301/v1/models`). Every Tinfoil model carries a `pricing` object with `inputTokenPricePer1M`, `outputTokenPricePer1M` **and `requestPrice`**. ## What we actually pay | Model | Tinfoil publishes | We declare | Effect | |---|---|---|---| | `tinfoil/voxtral-tts` | $0/token + **$0.002/request** | nothing | speech is free to members; the co-op pays | | `tinfoil/whisper-large-v3-turbo` | $0/token + **$0.01/request** | $0.05/1M input tokens | under-billed on typical dictation | Also published but not served by us: `doc-upload` $0.05/request, `websearch` $0.05/request, `pii-filter` $0.005/request, `qwen3-tts` $0.01/request. The config comment for the audio block says voxtral-tts is "listed $0/$0 (verify on invoice — may be beta-free)": it read the two token prices and missed `requestPrice`. The Tinfoil list is authoritative and machine-readable, which is also what makes the monitor below possible. ## The constraint LiteLLM applies per-request pricing on the search, rerank and MCP-tool paths via `input_cost_per_query` (`litellm/types/utils.py`: *"per-request pricing: rerank, search, and Bedrock Marengo embed"*; consumed in `litellm/search/cost_calculator.py` as `number_of_queries * cost_per_query`). The audio paths (`audio_speech`, `audio_transcription`) are priced by characters, seconds or tokens. `input_cost_per_request` exists in the proxies' cost-tracking settings but is **not** read by the model cost calculator. So a per-request TTS/STT price cannot be expressed through the cost map as things stand. ## Options 1. **Calibrate an equivalent** — publish `input_cost_per_character` (TTS) / `input_cost_per_token` (STT) derived from measured characters or tokens per request. Config-only, no code. Aggregate-accurate, but wrong per request: a burst of very short requests under-bills, which is an abuse hole precisely where requests are cheap and numerous. 2. **Build-time patch** — bill the audio paths a per-request cost, the way this repo already patches image-gen pricing (`79f0447`: *"keeping image-gen pricing on the shared cost-map key … so image spend is tracked"*). Exact, and consistent with the existing pattern; the cost is carrying another patch across LiteLLM upgrades. 3. **Leave it** — accept the subsidy. Not recommended while voice mode is the only unmetered capability on the gateway. **Recommendation: option 2**, with the patch listed in the upgrade checklist for this app so it cannot be silently lost on a version bump. ## Related - The monitoring half is done: cron job `8f2429d0c4bf` diffs our declared prices against Tinfoil's published ones and reports any change, so a price move shows up instead of silently drifting. - `code/litellm#4` is closed: it billed $16/1M characters, a market rate rather than our cost. - `code/litellm#3` (Bo) is the same class of gap on the discovery fields rather than the cost fields.

inference-bot referenced this issue from a commit

2026-10-26 20:45:20 +00:00

gateway: bill Tinfoil's per-request charges on the audio paths

inference-bot commented

2026-10-26 21:39:39 +00:00

Status: transcription fixed and verified; speech still unmetered.

Landed and deployed

patch_audio_request_cost.py — the model cost calculator now honours a declared input_cost_per_request. LiteLLM reads that key only in the proxy's cost-tracking settings (proxy/_types.py); the calculator ignores it, which is why "$0.002 per request" was not expressible at all.
patch_audio_router_pricing.py — puts audio deployments' pricing on the shared cost-map key, which the router strips per-deployment pricing from. Without it the cost lookup raised, the spend write was skipped, and no spend row appeared for either audio call type. Same mechanism as the image-generation case patch_image_cost.py already fixes.
config.yaml — declares Tinfoil's actual charges: whisper input_cost_per_token: 0 + input_cost_per_request: 0.01; voxtral-tts input_cost_per_character: 0 + input_cost_per_request: 0.002.
A process fix on the side: /app/data/config.yaml is a persistent volume, so the image's COPY config.yaml is shadowed forever and start.sh only wrote a default when the file was missing — meaning config changes in this repo never reached a running gateway (the audit's byte-identical md5 check was a manual sync). start.sh now syncs the image's copy over the volume at boot.

Verified by measurement

Throwaway member-scope key (deleted afterwards), real calls:

/v1/audio/transcriptions on tinfoil/whisper-large-v3-turbo → +$0.010000 exactly. Whisper's token price is $0, so that is entirely the per-request charge.
That path is done: config → shared cost map → calculator → budget.

Open: `/v1/audio/speech` bills $0.000000

Instrumentation in the patched calculator prints a diagnostic line for audio models. It shows the helper runs and resolves for transcription, and never runs for aspeech — so the speech cost never reaches the model cost calculator at all. Two candidate mechanisms, not yet distinguished:

the cost calculation raises earlier in the funnel and LiteLLM swallows it into response_cost_failure_debug_information (debug level only), or
the speech handler never invokes the cost path for a binary/streaming audio response.

Cheapest next step: instrument the entry of response_cost_calculator — whether it is called at all for a speech request separates those two in one deploy. Note _prompt_for_cost_calculation() in litellm_logging.py does special-case speech ("the raw input string is only priced directly for text-to-speech, which bills per character"), so the intent upstream is that speech is costed.

Secondary gap

Audio charges update the key's spend, so budget enforcement works — but they produce no /spend/logs row, and that endpoint is what the member dashboard reads. Audio usage may therefore be billed yet invisible in the panel.

Exposure while open

voxtral-tts is $0.002 per request and the audio keep-alive job is currently disabled, so the unmetered exposure is small; re-enabling it would be ~$0.29/day at a ten-minute interval.

**Status: transcription fixed and verified; speech still unmetered.** ## Landed and deployed - **`patch_audio_request_cost.py`** — the model cost calculator now honours a declared `input_cost_per_request`. LiteLLM reads that key only in the proxy's cost-tracking settings (`proxy/_types.py`); the calculator ignores it, which is why "$0.002 per request" was not expressible at all. - **`patch_audio_router_pricing.py`** — puts audio deployments' pricing on the **shared** cost-map key, which the router strips per-deployment pricing from. Without it the cost lookup raised, the spend write was skipped, and **no spend row appeared for either audio call type**. Same mechanism as the image-generation case `patch_image_cost.py` already fixes. - **`config.yaml`** — declares Tinfoil's actual charges: whisper `input_cost_per_token: 0` + `input_cost_per_request: 0.01`; voxtral-tts `input_cost_per_character: 0` + `input_cost_per_request: 0.002`. - **A process fix on the side:** `/app/data/config.yaml` is a persistent volume, so the image's `COPY config.yaml` is shadowed forever and `start.sh` only wrote a default when the file was missing — meaning config changes in this repo **never reached a running gateway** (the audit's byte-identical md5 check was a manual sync). `start.sh` now syncs the image's copy over the volume at boot. ## Verified by measurement Throwaway member-scope key (deleted afterwards), real calls: - `/v1/audio/transcriptions` on `tinfoil/whisper-large-v3-turbo` → **+$0.010000 exactly**. Whisper's token price is $0, so that is entirely the per-request charge. - That path is done: config → shared cost map → calculator → budget. ## Open: `/v1/audio/speech` bills $0.000000 Instrumentation in the patched calculator prints a diagnostic line for audio models. It shows the helper **runs and resolves for `transcription`, and never runs for `aspeech`** — so the speech cost never reaches the model cost calculator at all. Two candidate mechanisms, not yet distinguished: 1. the cost calculation raises earlier in the funnel and LiteLLM swallows it into `response_cost_failure_debug_information` (debug level only), or 2. the speech handler never invokes the cost path for a binary/streaming audio response. **Cheapest next step:** instrument the *entry* of `response_cost_calculator` — whether it is called at all for a speech request separates those two in one deploy. Note `_prompt_for_cost_calculation()` in `litellm_logging.py` *does* special-case speech ("the raw input string is only priced directly for text-to-speech, which bills per character"), so the intent upstream is that speech is costed. ## Secondary gap Audio charges update the **key's** spend, so budget enforcement works — but they produce **no `/spend/logs` row**, and that endpoint is what the member dashboard reads. Audio usage may therefore be billed yet invisible in the panel. ## Exposure while open `voxtral-tts` is $0.002 per request and the audio keep-alive job is currently disabled, so the unmetered exposure is small; re-enabling it would be ~$0.29/day at a ten-minute interval.

inference-bot referenced this issue

2026-10-27 05:01:29 +00:00

gateway: bill Tinfoil's per-request charges on the audio paths (voxtral-tts, whisper) #6

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: code/litellm#5

Meter Tinfoil's per-request prices (voxtral-tts, whisper) — LiteLLM has no per-request cost field on the audio path #5