The /v1/systemone pass-through is an EndpointType.GENERIC route: LiteLLM
logs it, but with no model on the route the only cost it can apply is the
flat cost_per_request (a budget estimate, not an invoice). So decisions
were billed at a fixed 0.0001 instead of the real $0.12 per 1M input
tokens GreenPT reports back in usage.input_tokens.
Price the route from its own response body via a new build-time patch,
patch_passthrough_token_cost.py, in the house style of the existing
patch_*.py files: any route listed in
general_settings.coop_passthrough_cost_per_token (USD per input token,
keyed by path) is billed as usage.input_tokens x rate. cost_per_request
stays as the fallback for a listed route whose response carries no token
count, and every unlisted route is unchanged. Auth is untouched.
config.yaml: list /v1/systemone at 0.00000012 ($0.12 per 1M input
tokens); update the metering note.
LiteLLM gates auth=true pass-through routes behind allowed_passthrough_routes,
a LiteLLM Enterprise field: on this community-edition proxy the management API
refuses to set it, so no member key can ever be granted access, and auth:false
would instead expose the route (and our GreenPT key) to anonymous callers.
Add patch_passthrough_key_access.py (house style: build-time, non-fatal,
idempotent, compile-checked) so routes the operator names in
general_settings.coop_passthrough_open_to_keys are callable by ANY
authenticated key. Authentication itself is unchanged: auth=true still installs
Depends(user_api_key_auth), so unauthenticated requests are still rejected.
Validated locally against LiteLLM 1.103.2: a plain generated key gets 200 with
real Metis answers; an unauthenticated POST gets 401.
start.sh hardcoded and exported LITELLM_CORS_ORIGINS, so the Cloudron app env that is meant to control it was silently overridden — the change I made earlier tonight looked applied and was inert. It now defaults only when unset, so the app env wins, and the co-op decision to allow all origins (co-op/support#38) actually takes effect. Comment rewritten for the current version and the reasoning. Also adds patch_expose_cost_headers.py (wired into the Dockerfile) so x-litellm-response-cost* can be read by cross-origin clients (co-op/support#39); upstream closed the request as stale, so it is handled at the app layer like the other patches.
LiteLLM copies every upstream response header onto its own response prefixed
'llm_provider-'. For our providers that hands callers operational detail about the
co-op's own account — verified live: llm_provider-x-credits-remaining carried the
remaining provider balance in every member's response.
Build-time patch makes _get_llm_provider_headers() return {}. The four OpenAI-style
x-ratelimit-* headers a provider may send are still forwarded: those describe the
caller's own request, not the co-op's account.
The first audio deploy proved the config and the cost-calculator patch were both
live — /model/info served input_cost_per_request, and the build log confirmed
'[coop-patch] applied' — yet a real /v1/audio/speech and
/v1/audio/transcriptions pair still billed exactly $0.000000, with no spend row
anywhere. Diagnosis: the audio cost paths resolve pricing from the SHARED
cost-map key, and the router strips per-deployment pricing from that key, so the
lookup raises and the spend write is skipped. Same mechanism as the
image-generation case patch_image_cost.py already fixes.
The cost-calculator patch cannot cover this on its own: it adds a per-request
charge on top of a cost the normal path computes, and that path never got there.
This patch re-adds the pricing for audio_speech and audio_transcription
deployments, explicitly allowing input_cost_per_request through — LiteLLM does
not list it among its CustomPricing fields, which is precisely why the
cost-calculator patch has to read it.
Deploying the audio-pricing change proved the gap: the image built and the app
restarted, but /model/info still served the old prices — /app/data is a
persistent volume, so the Dockerfile's COPY config.yaml /app/data/config.yaml is
shadowed by the volume's file from the first install, and start.sh only wrote a
default when the file was missing. Config changes in this repo therefore never
reached a running gateway; the audit's byte-identical md5 check was a manual
sync.
Fix: ship a second copy at /app/code/config.yaml and have start.sh sync it over
the volume at boot whenever they differ. The repo stays the source of truth, the
legacy first-boot default remains, and the README documents the flow.
Tinfoil bills voxtral-tts $0.002 and whisper-large-v3-turbo $0.01 per REQUEST,
with $0 per token for both. Reading only the two token prices in its model list
made TTS look free and left STT billed at an older catalog's $0.05/1M tokens, so
neither matched what the co-op actually pays.
LiteLLM cannot express a per-request charge on the audio paths: per-request
pricing exists only on the search/rerank/MCP paths via input_cost_per_query, and
input_cost_per_request is read only by the proxy's cost-tracking settings, never
by the model cost calculator.
So patch_audio_request_cost.py, a build-time patch in the same shape as
patch_image_cost.py, makes response_cost_calculator honour a declared
input_cost_per_request. That function is the single funnel for every call type —
the proxy dispatches audio as `aspeech` and `transcription` and reaches it
through Logging._response_cost_calculator — so one insertion covers both. The
charge resolves from the per-deployment custom_pricing the logging layer builds
from our model_info, then the shared cost map, then call metadata, and the patch
is inert for any model that declares nothing.
config.yaml declares the real charges: $0.002/request for voxtral-tts, and
$0.01/request for whisper with its token price corrected to $0.
Verified before committing: the patch applies to the installed 1.103.2 anchors
and compiles; the helper's resolution order passes unit checks (all branches,
plus inert and garbage cases); the resulting config validates as YAML with all
12 entries intact. End-to-end spend is proven after deploy by a real TTS/STT call.
Refs code/litellm#5
- Dockerfile: pin ghcr.io/berriai/litellm:v1.84.0 (was floating
main-v1.74.0-stable, which silently moved to a newer digest).
v1.84.0 is the fix line for CVE-2026-35029 (auth bypass on
/config/update, fixed 1.83.0) and CVE-2026-59822 (MCP session
auth bypass, fixed 1.84.0, CISA KEV).
- Drop the v1.74.0-era sed CORS patch: upstream v1.84.0 moved the
file and now reads LITELLM_CORS_ORIGINS natively.
- start.sh: export LITELLM_CORS_ORIGINS (the native v1.84.0 var)
instead of LITELLM_CORS_ALLOWED_ORIGINS (the old sed-patch var
that v1.84.0 ignores) — keeps CORS locked to chat.inference.coop.
- CloudronManifest: upstreamVersion 1.74.0 -> 1.84.0.
Live gateway already runs v1.84.0 (image digest
sha256:dc532d896ba8..., built 2026-10-02 04:50 UTC); this commit
makes the repo mirror the deployed packaging per the no-drift rule.
LITELLM_CORS_ORIGINS also set as a Cloudron env var on the app so
the running container honours it without a rebuild.
inference-bot
inference-bot