A member with API-spend history for tinfoil/nomic-embed-text reported the
model 'missing'. It was never removed: it has been admin-team scoped
(model_info.team_id) since it was added 2026-10-07 for the co-op's own
docs-RAG, and team-scoped models are hidden from every /v1/models listing
by design — the member's dashboard row was their own refused call. Tinfoil
still serves nomic-embed-text upstream; nothing to restore.
What WAS missing is a member-callable embedding model. GreenPT serves
qwen3-embedding-8b (id verbatim, type EMBEDDINGS): $0.25/1M input from
GreenPT's own /v1/pricing, no output charge, 32K context. Added as a
regular model_list entry (no team_id) so any member key can call it via
/v1/embeddings; the portal filters mode:embedding out of the chat picker.
max_input_tokens declared from the provider's contextWindowTokens.
Additive change only: every pre-existing model_info byte-identical,
fallbacks and pass-through untouched; boot-proved on a spare port in the
container before deploy.
The /v1/systemone pass-through is an EndpointType.GENERIC route: LiteLLM
logs it, but with no model on the route the only cost it can apply is the
flat cost_per_request (a budget estimate, not an invoice). So decisions
were billed at a fixed 0.0001 instead of the real $0.12 per 1M input
tokens GreenPT reports back in usage.input_tokens.
Price the route from its own response body via a new build-time patch,
patch_passthrough_token_cost.py, in the house style of the existing
patch_*.py files: any route listed in
general_settings.coop_passthrough_cost_per_token (USD per input token,
keyed by path) is billed as usage.input_tokens x rate. cost_per_request
stays as the fallback for a listed route whose response carries no token
count, and every unlisted route is unchanged. Auth is untouched.
config.yaml: list /v1/systemone at 0.00000012 ($0.12 per 1M input
tokens); update the metering note.
LiteLLM gates auth=true pass-through routes behind allowed_passthrough_routes,
a LiteLLM Enterprise field: on this community-edition proxy the management API
refuses to set it, so no member key can ever be granted access, and auth:false
would instead expose the route (and our GreenPT key) to anonymous callers.
Add patch_passthrough_key_access.py (house style: build-time, non-fatal,
idempotent, compile-checked) so routes the operator names in
general_settings.coop_passthrough_open_to_keys are callable by ANY
authenticated key. Authentication itself is unchanged: auth=true still installs
Depends(user_api_key_auth), so unauthenticated requests are still rejected.
Validated locally against LiteLLM 1.103.2: a plain generated key gets 200 with
real Metis answers; an unauthenticated POST gets 401.
Metis is a decision endpoint (state + typed questions -> probabilities),
not a chat completion: it emits no text, so it cannot be a model_list entry.
Expose it as a pass-through endpoint under general_settings, behind the
member's normal co-op key (auth: true), with GreenPT's key injected from the
app env (Bearer os.environ/GREENPT_API_KEY). cost_per_request is an
APPROXIMATION for budget accounting (~800 input tokens at $0.12/1M);
exact billing would need a patch_*.py in the house style.
apertus-70b input 0.40->1.00 EUR (+152%), output 2.10->3.40; qwen3.5-122b input 1.00->0.60 (-40%), output 4.20->3.40. EUR->USD at the existing 1.16 peg. Also converts regolo/qwen-image's per-pixel cost out of the EUR-as-USD state it was in (1.5e-8 -> 1.74e-8).
Speech still billed nothing after the router patch put its pricing on the shared
cost map. Cause, read out of cost_per_token: for a speech model
select_cost_metric_for_model picks "cost_per_character", and that branch raises
ValueError("cost for tts call is None") when no character price is declared. The
raise aborts the whole cost calculation, so the spend write is skipped and the
per-request patch never runs — the entry being unpriced per character made the
per-request charge unreachable.
Declaring input_cost_per_character: 0 lets the path complete with a zero
per-unit cost, after which the patch adds the real $0.002 per request.
Measured, not assumed: transcription bills $0.010000 exactly (whisper's token
price is $0, so that is entirely the per-request charge), while speech sat at
$0.000000 across three deploys.
Tinfoil bills voxtral-tts $0.002 and whisper-large-v3-turbo $0.01 per REQUEST,
with $0 per token for both. Reading only the two token prices in its model list
made TTS look free and left STT billed at an older catalog's $0.05/1M tokens, so
neither matched what the co-op actually pays.
LiteLLM cannot express a per-request charge on the audio paths: per-request
pricing exists only on the search/rerank/MCP paths via input_cost_per_query, and
input_cost_per_request is read only by the proxy's cost-tracking settings, never
by the model cost calculator.
So patch_audio_request_cost.py, a build-time patch in the same shape as
patch_image_cost.py, makes response_cost_calculator honour a declared
input_cost_per_request. That function is the single funnel for every call type —
the proxy dispatches audio as `aspeech` and `transcription` and reaches it
through Logging._response_cost_calculator — so one insertion covers both. The
charge resolves from the per-deployment custom_pricing the logging layer builds
from our model_info, then the shared cost map, then call metadata, and the patch
is inert for any model that declares nothing.
config.yaml declares the real charges: $0.002/request for voxtral-tts, and
$0.01/request for whisper with its token price corrected to $0.
Verified before committing: the patch applies to the installed 1.103.2 anchors
and compiles; the helper's resolution order passes unit checks (all branches,
plus inert and garbage cases); the resulting config validates as YAML with all
12 entries intact. End-to-end spend is proven after deploy by a real TTS/STT call.
Refs code/litellm#5
num_retries=2 (retries transient connection errors), retry_policy with
TimeoutErrorRetries=0 (a timed-out generation is already billed by Tinfoil, so
re-hitting it re-bills). Combined with timeout=180, this hedges connection
errors without the silent re-billing that caused the Tinfoil spend gap.
DeepSeek V4.1 Flash is a reasoning model; 30s timeout caused LiteLLM to
abandon in-flight generations (falling back to GPT-OSS) while Tinfoil still
completed and billed them. num_retries re-hit Tinfoil up to 3x. Result: Tinfoil
billed ~2x the tokens LiteLLM logged. Generous timeout + zero retries stops
the silent re-billing.
inference-bot
Ubuntu