Commit Graph
26 Commits
Author SHA1 Message Date
inference-admin 2483715ef3 litellm: register audio per-request prices at interpreter startup (#5)
Build-time edits of the cost-map JSON never reached the running gateway, so the
audio prices now go in through sitecustomize — imported before LiteLLM prices
anything. Idempotent, non-fatal.
2026-11-17 08:44:41 +00:00
inference-admin b81177c3ad litellm: put Tinfoil's per-request audio prices in the shared cost map (#5)
The per-request helper reads litellm.model_cost, which is built from
model_prices_and_context_window.json — not from model_info. Router-side patching
therefore never priced voxtral-tts speech, and every call logged $0. This build
step writes the prices where the calculator looks, and fails the build if the file
is missing instead of warning.
2026-11-17 08:32:56 +00:00
inference-bot e21c24ea48 gateway: fail-closed config validation before boot — unknown top-level key or unresolvable provider prefix refuses the start (ops-docs canonical copy shipped byte-identical) 2026-11-11 03:55:18 +00:00
inference-bot 64d7e5afc5 gateway: single TTS warm path — drop the 10-min scheduler task, in-container loop every 15 min
The scheduler task ran audio_keepalive.py in its own container every 10 min
in parallel with the in-container loop (8 min): two warm paths billing the
same ping twice, and the scheduler container's access to 127.0.0.1:4000 and
/app/data was never proven. Keep the proven in-container path only, at a
15-minute interval (sleep 900).
2026-11-09 14:33:52 +00:00
inference-bot c43d45c455 upgrade: pin LiteLLM v1.108.3 (re-validated on real 1.108.3 sources)
- Dockerfile FROM v1.103.2 -> v1.108.3; manifest upstreamVersion follows.
- Carries main's newer build patches (patch_transcription_response_format.py,
  patch_audio_default_mp3.py, audio_keepalive.py) that the v1.108.0 branch
  predates, plus the 1.108.x CORS anchor fallback from that branch.
- Re-validated all ten patch scripts against the real v1.108.3 wheel sources:
  every anchor matched and the patched files compile. patch_expose_cost_headers
  applies via its 1.108.x anchor; patch_audio_request_cost needs
  LITELLM_COST_CALCULATOR when run outside a full install (noted in README).
- Stop tracking __pycache__.
2026-11-09 04:22:30 +00:00
inference-bot 5d38403892 gateway: fix TTS latency + restore audio warm path
- patch_audio_default_mp3.py: default /v1/audio/speech to response_format=mp3
  when the caller omits it. LibreChat sends no response_format, and Tinfoil's
  voxtral-tts then serves WAV: measured ~8-17s warm (145-300KB) vs ~2.6-3.1s
  for mp3 (28-39KB, provider streams it). LibreChat labels the reply
  audio/mpeg regardless, so WAV bytes rode under an mp3 label. Explicit
  response_format from any caller still wins.
- start.sh: scheduler tasks arrive as ARGUMENTS to this ENTRYPOINT script
  (docker run <image> <command>); they were booting a second tinfoil-proxy +
  LiteLLM and dying on the bind FATAL every run, so audio_keepalive never
  executed. Detect the arg and exec the task directly.
- start.sh: own the warm path in the app container regardless of the
  scheduler container: ping tinfoil/voxtral-tts every 480s in the background
  (silent on success; failures append to /app/data/keepalive.log).
2026-11-07 20:09:54 +00:00
inference-bot 943701581c gateway: make /v1/audio/transcriptions work bare for voxtral-small-24b
LiteLLM's openai transcription transformation force-injects
response_format=verbose_json when the caller omits it (to obtain a duration
for per-second pricing). Tinfoil rejects verbose_json for
voxtral-small-24b with a 400, so every bare transcription call failed —
including the file+model shape members and Open WebUI dictation send.

patch_transcription_response_format.py (new build-time patch, non-fatal):
skip the injection for deployments whose model_info declares
coop_no_verbose_json, with an explicit model-name gate as the reliable
mechanism. Its billing (per-token via chat, per-request via the audio
patch) does not need the duration. whisper-large-v3-turbo is untouched:
Tinfoil serves verbose_json for it and dictation depends on the injection.
2026-11-06 22:40:53 +00:00
inference-bot da02f8eb32 gateway: price the Metis pass-through route from its reported tokens
The /v1/systemone pass-through is an EndpointType.GENERIC route: LiteLLM
logs it, but with no model on the route the only cost it can apply is the
flat cost_per_request (a budget estimate, not an invoice). So decisions
were billed at a fixed 0.0001 instead of the real $0.12 per 1M input
tokens GreenPT reports back in usage.input_tokens.

Price the route from its own response body via a new build-time patch,
patch_passthrough_token_cost.py, in the house style of the existing
patch_*.py files: any route listed in
general_settings.coop_passthrough_cost_per_token (USD per input token,
keyed by path) is billed as usage.input_tokens x rate. cost_per_request
stays as the fallback for a listed route whose response carries no token
count, and every unlisted route is unchanged. Auth is untouched.

config.yaml: list /v1/systemone at 0.00000012 ($0.12 per 1M input
tokens); update the metering note.
2026-11-03 18:58:07 +00:00
inference-bot 557999a16f gateway: let member keys reach the Metis pass-through route
LiteLLM gates auth=true pass-through routes behind allowed_passthrough_routes,
a LiteLLM Enterprise field: on this community-edition proxy the management API
refuses to set it, so no member key can ever be granted access, and auth:false
would instead expose the route (and our GreenPT key) to anonymous callers.

Add patch_passthrough_key_access.py (house style: build-time, non-fatal,
idempotent, compile-checked) so routes the operator names in
general_settings.coop_passthrough_open_to_keys are callable by ANY
authenticated key. Authentication itself is unchanged: auth=true still installs
Depends(user_api_key_auth), so unauthenticated requests are still rejected.

Validated locally against LiteLLM 1.103.2: a plain generated key gets 200 with
real Metis answers; an unauthenticated POST gets 401.
2026-11-03 18:30:22 +00:00
inference-bot 197abce9e6 gateway: honour LITELLM_CORS_ORIGINS from app env; expose cost headers (#38, #39)
start.sh hardcoded and exported LITELLM_CORS_ORIGINS, so the Cloudron app env that is meant to control it was silently overridden — the change I made earlier tonight looked applied and was inert. It now defaults only when unset, so the app env wins, and the co-op decision to allow all origins (co-op/support#38) actually takes effect. Comment rewritten for the current version and the reasoning. Also adds patch_expose_cost_headers.py (wired into the Dockerfile) so x-litellm-response-cost* can be read by cross-origin clients (co-op/support#39); upstream closed the request as stale, so it is handled at the app layer like the other patches.
2026-10-31 17:20:01 +00:00
inference-bot 8ddd023786 gateway: stop forwarding provider response headers to callers
LiteLLM copies every upstream response header onto its own response prefixed
'llm_provider-'. For our providers that hands callers operational detail about the
co-op's own account — verified live: llm_provider-x-credits-remaining carried the
remaining provider balance in every member's response.

Build-time patch makes _get_llm_provider_headers() return {}. The four OpenAI-style
x-ratelimit-* headers a provider may send are still forwarded: those describe the
caller's own request, not the co-op's account.
2026-10-27 04:00:57 +00:00
Ubuntu c03abaa330 gateway: put audio pricing on the shared cost-map key (no SpendLogs row at all)
The first audio deploy proved the config and the cost-calculator patch were both
live — /model/info served input_cost_per_request, and the build log confirmed
'[coop-patch] applied' — yet a real /v1/audio/speech and
/v1/audio/transcriptions pair still billed exactly $0.000000, with no spend row
anywhere. Diagnosis: the audio cost paths resolve pricing from the SHARED
cost-map key, and the router strips per-deployment pricing from that key, so the
lookup raises and the spend write is skipped. Same mechanism as the
image-generation case patch_image_cost.py already fixes.

The cost-calculator patch cannot cover this on its own: it adds a per-request
charge on top of a cost the normal path computes, and that path never got there.

This patch re-adds the pricing for audio_speech and audio_transcription
deployments, explicitly allowing input_cost_per_request through — LiteLLM does
not list it among its CustomPricing fields, which is precisely why the
cost-calculator patch has to read it.
2026-10-26 21:24:42 +00:00
Ubuntu facf01d578 gateway: deploy config.yaml changes (the volume was shadowing the image)
Deploying the audio-pricing change proved the gap: the image built and the app
restarted, but /model/info still served the old prices — /app/data is a
persistent volume, so the Dockerfile's COPY config.yaml /app/data/config.yaml is
shadowed by the volume's file from the first install, and start.sh only wrote a
default when the file was missing. Config changes in this repo therefore never
reached a running gateway; the audit's byte-identical md5 check was a manual
sync.

Fix: ship a second copy at /app/code/config.yaml and have start.sh sync it over
the volume at boot whenever they differ. The repo stays the source of truth, the
legacy first-boot default remains, and the README documents the flow.
2026-10-26 21:17:41 +00:00
Ubuntu 425d738625 gateway: bill Tinfoil's per-request charges on the audio paths
Tinfoil bills voxtral-tts $0.002 and whisper-large-v3-turbo $0.01 per REQUEST,
with $0 per token for both. Reading only the two token prices in its model list
made TTS look free and left STT billed at an older catalog's $0.05/1M tokens, so
neither matched what the co-op actually pays.

LiteLLM cannot express a per-request charge on the audio paths: per-request
pricing exists only on the search/rerank/MCP paths via input_cost_per_query, and
input_cost_per_request is read only by the proxy's cost-tracking settings, never
by the model cost calculator.

So patch_audio_request_cost.py, a build-time patch in the same shape as
patch_image_cost.py, makes response_cost_calculator honour a declared
input_cost_per_request. That function is the single funnel for every call type —
the proxy dispatches audio as `aspeech` and `transcription` and reaches it
through Logging._response_cost_calculator — so one insertion covers both. The
charge resolves from the per-deployment custom_pricing the logging layer builds
from our model_info, then the shared cost map, then call metadata, and the patch
is inert for any model that declares nothing.

config.yaml declares the real charges: $0.002/request for voxtral-tts, and
$0.01/request for whisper with its token price corrected to $0.

Verified before committing: the patch applies to the installed 1.103.2 anchors
and compiles; the helper's resolution order passes unit checks (all branches,
plus inert and garbage cases); the resulting config validates as YAML with all
12 entries intact. End-to-end spend is proven after deploy by a real TTS/STT call.

Refs code/litellm#5
2026-10-26 20:45:06 +00:00
Inferencebot 79f0447089 gateway: build-time patch keeping image-gen pricing on the shared cost-map key (upstream #10023/#12471) so image spend is tracked 2026-10-19 09:26:45 -06:00
Inferencebot acceb43995 Move audio keep-alive into the gateway container (Cloudron scheduler addon, every 10m) 2026-10-12 19:18:46 -06:00
Inferencebot 5389d9acfc credit-pack hint: patch budget-exceeded message at BUILD time (runtime rootfs is read-only; boot-time patch crashed startup) 2026-10-06 14:24:55 -06:00
Inferencebot a2864ce34a Upgrade to LiteLLM v1.103.2 (schema pre-migrated live; CVE fix line + streaming-usage bugfix window) 2026-10-02 15:25:27 -06:00
inference-bot fd34e33508 Sync packaging to live v1.84.0 deployment (CVE-2026-35029/59822 fix line)
- Dockerfile: pin ghcr.io/berriai/litellm:v1.84.0 (was floating
  main-v1.74.0-stable, which silently moved to a newer digest).
  v1.84.0 is the fix line for CVE-2026-35029 (auth bypass on
  /config/update, fixed 1.83.0) and CVE-2026-59822 (MCP session
  auth bypass, fixed 1.84.0, CISA KEV).
- Drop the v1.74.0-era sed CORS patch: upstream v1.84.0 moved the
  file and now reads LITELLM_CORS_ORIGINS natively.
- start.sh: export LITELLM_CORS_ORIGINS (the native v1.84.0 var)
  instead of LITELLM_CORS_ALLOWED_ORIGINS (the old sed-patch var
  that v1.84.0 ignores) — keeps CORS locked to chat.inference.coop.
- CloudronManifest: upstreamVersion 1.74.0 -> 1.84.0.

Live gateway already runs v1.84.0 (image digest
sha256:dc532d896ba8..., built 2026-10-02 04:50 UTC); this commit
makes the repo mirror the deployed packaging per the no-drift rule.
LITELLM_CORS_ORIGINS also set as a Cloudron env var on the app so
the running container honours it without a rebuild.
2026-10-01 23:32:43 -06:00
inference-bot 9cbbc33bc7 Fix CORS patch to target site-packages (the copy the running CLI imports) 2026-09-14 18:01:13 -06:00
inference-bot 0a797ba50e Patch hardcoded origins=[*] to honour LITELLM_CORS_ALLOWED_ORIGINS (v1.74.0 ignores the env var) 2026-09-14 17:55:08 -06:00
inference-bot 8a54e75d4e Download tinfoil-proxy at runtime (build sandbox has no GitHub access) 2026-09-09 13:23:53 -06:00
inference-bot 26b0edc6a8 Hardcode tinfoil-proxy version in Dockerfile 2026-09-09 13:22:21 -06:00
inference-bot e05fec0ca8 Switch to Tinfoil (TEE): add tinfoil-proxy sidecar for EHBP encryption, point LiteLLM at local proxy 2026-09-09 13:21:01 -06:00
inference-bot 41068a6264 Fix Dockerfile for Cloudron: run as root (base image default), override ENTRYPOINT, fix .dockerignore 2026-09-04 10:29:15 -06:00
inference-bot b9b4620e52 Initial commit: LiteLLM Cloudron app package for Inference Cooperative 2026-09-03 21:55:54 -06:00