A member with API-spend history for tinfoil/nomic-embed-text reported the
model 'missing'. It was never removed: it has been admin-team scoped
(model_info.team_id) since it was added 2026-10-07 for the co-op's own
docs-RAG, and team-scoped models are hidden from every /v1/models listing
by design — the member's dashboard row was their own refused call. Tinfoil
still serves nomic-embed-text upstream; nothing to restore.
What WAS missing is a member-callable embedding model. GreenPT serves
qwen3-embedding-8b (id verbatim, type EMBEDDINGS): $0.25/1M input from
GreenPT's own /v1/pricing, no output charge, 32K context. Added as a
regular model_list entry (no team_id) so any member key can call it via
/v1/embeddings; the portal filters mode:embedding out of the chat picker.
max_input_tokens declared from the provider's contextWindowTokens.
Additive change only: every pre-existing model_info byte-identical,
fallbacks and pass-through untouched; boot-proved on a spare port in the
container before deploy.
Wire-verified on the pinned v1.103.2 (2026-11-05): NO_DOCS/NO_REDOC/
NO_OPENAPI do disable /docs, /redoc and /openapi.json (live 404s), but
DISABLE_ADMIN_UI only disables the UI's SSO login-redirect routes in this
version — the static /ui/ bundle is mounted unconditionally and still
serves (login stays master-key-gated). README 'Admin access' and the
start.sh comment now say exactly that, with the build-patch path if the
bundle itself should ever be removed. There is no env/flag for it on
this version, so the README is corrected instead (M4).
CORS reality recorded: LITELLM_CORS_ORIGINS is set at the app level to *
by co-op decision (co-op/support#38); start.sh's chat-origin fallback
applies only if the app env omits it. PublicAI (removed 2026-10-16,
replaced by Regolo) dropped from the provider lists; chat is LibreChat,
not Open WebUI. The embedded first-boot fallback config cannot be served
(overwritten by the same boot's image sync) — noted inline.
Manifest contactEmail: botbot@hermes -> info@inference.coop (L6).
The pass-through row for /v1/systemone carried the correct spend but
prompt_tokens/completion_tokens/total_tokens were all 0: the generic
pass-through success handler hands the spend writer a
StandardPassThroughResponseObject with no usage, and the generic branch
builds no standard_logging_object usage, so anything summing tokens for
the route under-reported.
patch_passthrough_token_cost.py now also sets combined_usage_object (a
litellm.Usage carrying prompt_tokens=input_tokens, completion_tokens=0,
total_tokens=input_tokens) on the logging object -- the same override the
streaming handler uses for recovered partial usage, which the spend
writer reads before the response object. Spend is unchanged.
Defect 2 (a second zero-spend row with call_type=/v1/systemone) is NOT
duplication: it is upstream LiteLLM's failure row for a rejected
(missing/invalid key) request, written by
ProxyLogging._handle_logging_proxy_only_error on the pristine image. One
authenticated call produces exactly one priced row. Documented in the
README; no code change.
The /v1/systemone pass-through is an EndpointType.GENERIC route: LiteLLM
logs it, but with no model on the route the only cost it can apply is the
flat cost_per_request (a budget estimate, not an invoice). So decisions
were billed at a fixed 0.0001 instead of the real $0.12 per 1M input
tokens GreenPT reports back in usage.input_tokens.
Price the route from its own response body via a new build-time patch,
patch_passthrough_token_cost.py, in the house style of the existing
patch_*.py files: any route listed in
general_settings.coop_passthrough_cost_per_token (USD per input token,
keyed by path) is billed as usage.input_tokens x rate. cost_per_request
stays as the fallback for a listed route whose response carries no token
count, and every unlisted route is unchanged. Auth is untouched.
config.yaml: list /v1/systemone at 0.00000012 ($0.12 per 1M input
tokens); update the metering note.
LiteLLM gates auth=true pass-through routes behind allowed_passthrough_routes,
a LiteLLM Enterprise field: on this community-edition proxy the management API
refuses to set it, so no member key can ever be granted access, and auth:false
would instead expose the route (and our GreenPT key) to anonymous callers.
Add patch_passthrough_key_access.py (house style: build-time, non-fatal,
idempotent, compile-checked) so routes the operator names in
general_settings.coop_passthrough_open_to_keys are callable by ANY
authenticated key. Authentication itself is unchanged: auth=true still installs
Depends(user_api_key_auth), so unauthenticated requests are still rejected.
Validated locally against LiteLLM 1.103.2: a plain generated key gets 200 with
real Metis answers; an unauthenticated POST gets 401.
Metis is a decision endpoint (state + typed questions -> probabilities),
not a chat completion: it emits no text, so it cannot be a model_list entry.
Expose it as a pass-through endpoint under general_settings, behind the
member's normal co-op key (auth: true), with GreenPT's key injected from the
app env (Bearer os.environ/GREENPT_API_KEY). cost_per_request is an
APPROXIMATION for budget accounting (~800 input tokens at $0.12/1M);
exact billing would need a patch_*.py in the house style.
apertus-70b input 0.40->1.00 EUR (+152%), output 2.10->3.40; qwen3.5-122b input 1.00->0.60 (-40%), output 4.20->3.40. EUR->USD at the existing 1.16 peg. Also converts regolo/qwen-image's per-pixel cost out of the EUR-as-USD state it was in (1.5e-8 -> 1.74e-8).
Adds x-litellm-model-name, -call-id, -key-spend, -response-duration-ms, -overhead-duration-ms, -callback-duration-ms, -attempted-retries and -attempted-fallbacks to the CORS expose list, on the GM's go-ahead. Renames COST_HEADERS to EXPOSED_HEADERS since the list is no longer only costs. x-litellm-key-spend discloses the caller's own key spend, which is their own data; noted in the thread so it can be dropped if anyone objects.
start.sh hardcoded and exported LITELLM_CORS_ORIGINS, so the Cloudron app env that is meant to control it was silently overridden — the change I made earlier tonight looked applied and was inert. It now defaults only when unset, so the app env wins, and the co-op decision to allow all origins (co-op/support#38) actually takes effect. Comment rewritten for the current version and the reasoning. Also adds patch_expose_cost_headers.py (wired into the Dockerfile) so x-litellm-response-cost* can be read by cross-origin clients (co-op/support#39); upstream closed the request as stale, so it is handled at the app layer like the other patches.
LiteLLM copies every upstream response header onto its own response prefixed
'llm_provider-'. For our providers that hands callers operational detail about the
co-op's own account — verified live: llm_provider-x-credits-remaining carried the
remaining provider balance in every member's response.
Build-time patch makes _get_llm_provider_headers() return {}. The four OpenAI-style
x-ratelimit-* headers a provider may send are still forwarded: those describe the
caller's own request, not the co-op's account.
Speech billed $0.000000 across three deploys while transcription billed
$0.010000 exactly, so the patch works for one audio call type and not the other.
Three hypotheses have now been retired (hidden-params early return; missing
character price; lookup miss) — the remaining question is WHERE the speech cost
is lost, and guessing is more expensive than measuring.
The helper now prints, for TTS/STT models only, whether it ran at all, what it
could see (custom_pricing keys, shared cost-map hits) and what it resolved, into
the app log. If those lines never appear for a speech call, the request never
reaches the model cost calculator and the loss is upstream of this patch.
Also fixes a real defect in the previous revision: the addition was wrapped in a
bare except, so a TypeError from a None base cost would have been swallowed
silently — which is exactly the sort of silence that made this take three
deploys. It now coerces a None base to 0.0 and reports any failure.
Speech still billed nothing after the router patch put its pricing on the shared
cost map. Cause, read out of cost_per_token: for a speech model
select_cost_metric_for_model picks "cost_per_character", and that branch raises
ValueError("cost for tts call is None") when no character price is declared. The
raise aborts the whole cost calculation, so the spend write is skipped and the
per-request patch never runs — the entry being unpriced per character made the
per-request charge unreachable.
Declaring input_cost_per_character: 0 lets the path complete with a zero
per-unit cost, after which the patch adds the real $0.002 per request.
Measured, not assumed: transcription bills $0.010000 exactly (whisper's token
price is $0, so that is entirely the per-request charge), while speech sat at
$0.000000 across three deploys.
The first audio deploy proved the config and the cost-calculator patch were both
live — /model/info served input_cost_per_request, and the build log confirmed
'[coop-patch] applied' — yet a real /v1/audio/speech and
/v1/audio/transcriptions pair still billed exactly $0.000000, with no spend row
anywhere. Diagnosis: the audio cost paths resolve pricing from the SHARED
cost-map key, and the router strips per-deployment pricing from that key, so the
lookup raises and the spend write is skipped. Same mechanism as the
image-generation case patch_image_cost.py already fixes.
The cost-calculator patch cannot cover this on its own: it adds a per-request
charge on top of a cost the normal path computes, and that path never got there.
This patch re-adds the pricing for audio_speech and audio_transcription
deployments, explicitly allowing input_cost_per_request through — LiteLLM does
not list it among its CustomPricing fields, which is precisely why the
cost-calculator patch has to read it.
Deploying the audio-pricing change proved the gap: the image built and the app
restarted, but /model/info still served the old prices — /app/data is a
persistent volume, so the Dockerfile's COPY config.yaml /app/data/config.yaml is
shadowed by the volume's file from the first install, and start.sh only wrote a
default when the file was missing. Config changes in this repo therefore never
reached a running gateway; the audit's byte-identical md5 check was a manual
sync.
Fix: ship a second copy at /app/code/config.yaml and have start.sh sync it over
the volume at boot whenever they differ. The repo stays the source of truth, the
legacy first-boot default remains, and the README documents the flow.
Tinfoil bills voxtral-tts $0.002 and whisper-large-v3-turbo $0.01 per REQUEST,
with $0 per token for both. Reading only the two token prices in its model list
made TTS look free and left STT billed at an older catalog's $0.05/1M tokens, so
neither matched what the co-op actually pays.
LiteLLM cannot express a per-request charge on the audio paths: per-request
pricing exists only on the search/rerank/MCP paths via input_cost_per_query, and
input_cost_per_request is read only by the proxy's cost-tracking settings, never
by the model cost calculator.
So patch_audio_request_cost.py, a build-time patch in the same shape as
patch_image_cost.py, makes response_cost_calculator honour a declared
input_cost_per_request. That function is the single funnel for every call type —
the proxy dispatches audio as `aspeech` and `transcription` and reaches it
through Logging._response_cost_calculator — so one insertion covers both. The
charge resolves from the per-deployment custom_pricing the logging layer builds
from our model_info, then the shared cost map, then call metadata, and the patch
is inert for any model that declares nothing.
config.yaml declares the real charges: $0.002/request for voxtral-tts, and
$0.01/request for whisper with its token price corrected to $0.
Verified before committing: the patch applies to the installed 1.103.2 anchors
and compiles; the helper's resolution order passes unit checks (all branches,
plus inert and garbage cases); the resulting config validates as YAML with all
12 entries intact. End-to-end spend is proven after deploy by a real TTS/STT call.
Refs code/litellm#5
The 3301 'address already in use' on process-level restarts left an unmanaged
old proxy serving (pre-update code/flags) while the new one died. Also: if the
proxy never binds, exit 1 instead of exec-ing LiteLLM against a dead 3301 —
that masqueraded as a provider outage. Cache continuity: with
TINFOIL_USER_CACHE_SECRET now pinned (env, 2026-10-22), restarts keep Tinfoil's
prompt-cache warm either way.
The 2026-10-02 outage had three causes the runbook now prevents:
- floating tag drift (README now mandates exact-tag pinning)
- 256MB default cgroup cap OOM-killing the prisma migration engine
(manifest now sets memoryLimit: 2147483648)
- no _prisma_migrations ledger -> P3005 crash-loop on boot
(start.sh now exports LITELLM_MIGRATION_DIR=/app/data/migrations)
README 'Updating LiteLLM' section rewritten from the old (wrong)
floating-tag procedure into a runbook with rules + verify checklist.
- Dockerfile: pin ghcr.io/berriai/litellm:v1.84.0 (was floating
main-v1.74.0-stable, which silently moved to a newer digest).
v1.84.0 is the fix line for CVE-2026-35029 (auth bypass on
/config/update, fixed 1.83.0) and CVE-2026-59822 (MCP session
auth bypass, fixed 1.84.0, CISA KEV).
- Drop the v1.74.0-era sed CORS patch: upstream v1.84.0 moved the
file and now reads LITELLM_CORS_ORIGINS natively.
- start.sh: export LITELLM_CORS_ORIGINS (the native v1.84.0 var)
instead of LITELLM_CORS_ALLOWED_ORIGINS (the old sed-patch var
that v1.84.0 ignores) — keeps CORS locked to chat.inference.coop.
- CloudronManifest: upstreamVersion 1.74.0 -> 1.84.0.
Live gateway already runs v1.84.0 (image digest
sha256:dc532d896ba8..., built 2026-10-02 04:50 UTC); this commit
makes the repo mirror the deployed packaging per the no-drift rule.
LITELLM_CORS_ORIGINS also set as a Cloudron env var on the app so
the running container honours it without a rebuild.
num_retries=2 (retries transient connection errors), retry_policy with
TimeoutErrorRetries=0 (a timed-out generation is already billed by Tinfoil, so
re-hitting it re-bills). Combined with timeout=180, this hedges connection
errors without the silent re-billing that caused the Tinfoil spend gap.
DeepSeek V4.1 Flash is a reasoning model; 30s timeout caused LiteLLM to
abandon in-flight generations (falling back to GPT-OSS) while Tinfoil still
completed and billed them. num_retries re-hit Tinfoil up to 3x. Result: Tinfoil
billed ~2x the tokens LiteLLM logged. Generous timeout + zero retries stops
the silent re-billing.
inference-bot
inference-bot