code/litellm
2
0 Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
197abce9e6434696b77736d5bbe79263f526f1d0
litellm/config.yaml
T
Ubuntu a6098741d0 gateway: declare a zero character price for voxtral-tts so speech can bill 
Speech still billed nothing after the router patch put its pricing on the shared
cost map. Cause, read out of cost_per_token: for a speech model
select_cost_metric_for_model picks "cost_per_character", and that branch raises
ValueError("cost for tts call is None") when no character price is declared. The
raise aborts the whole cost calculation, so the spend write is skipped and the
per-request patch never runs — the entry being unpriced per character made the
per-request charge unreachable.

Declaring input_cost_per_character: 0 lets the path complete with a zero
per-unit cost, after which the patch adds the real $0.002 per request.

Measured, not assumed: transcription bills $0.010000 exactly (whisper's token
price is $0, so that is entirely the per-request charge), while speech sat at
$0.000000 across three deploys.
2026-10-26 21:29:59 +00:00

241 lines
10 KiB
YAML
Raw Blame History

# LiteLLM config for inference.coop
# PRODUCTION config: Tinfoil (TEE-protected inference)
# LiteLLM talks plaintext OpenAI to a local Tinfoil proxy (127.0.0.1:3301),
# which verifies the enclave attestation and encrypts request/response bodies
# with EHBP (HPKE) before forwarding to the Tinfoil enclave. Provider key set
# via environment variable TINFOIL_API_KEY.
#
# Pricing (model_info) is per-token, derived from provider per-1M rates:
# greenpt/deepseek-v4.1-flash: $0.22 in / $1.10 out / $0.011 cached per 1M
# tinfoil/deepseek-v4-1-flash: $0.65 in / $1.45 out / $0.13 cached per 1M
# tinfoil/gpt-oss-120b: $0.15 in / $0.60 out per 1M
# tinfoil/glm-5.3: $1.80 in / $5.75 out / $0.45 cached per 1M
# Without these, LiteLLM can't price these custom models, so spend shows $0.00
# and the team budget cap is not enforced.
#
# SYNC: the per-model pricing here must mirror the user-facing model list in
# co-op/docs/models.md (and vice-versa). When a rate changes, update BOTH this
# file and models.md — this config is the authoritative source of the numbers;
# models.md is the member-facing mirror.
model_list:
# DeepSeek V4.1 Flash on GreenPT — DEFAULT model. Same model family as Tinfoil's
# deepseek-v4-1-flash, but on renewable energy and NOT marked Experimental by the
# provider (Tinfoil's deepseek + glm-flash carry an "Experimental: may be deprecated
# with minimal notice" flag). Policy-based privacy (GreenPT) vs TEE (Tinfoil).
# €0.22 in / €1.10 out / €0.011 cached per 1M.
- model_name: greenpt/deepseek-v4.1-flash
litellm_params:
model: openai/deepseek-v4.1-flash
api_base: https://api.greenpt.ai/v1
api_key: os.environ/GREENPT_API_KEY
model_info:
input_cost_per_token: 0.00000022
output_cost_per_token: 0.00000110
cache_read_input_token_cost: 0.000000011
# DeepSeek V4.1 Flash on Tinfoil (TEE) — private counterpart to the default.
- model_name: tinfoil/deepseek-v4-1-flash
litellm_params:
model: openai/deepseek-v4-1-flash
api_base: http://127.0.0.1:3301/v1
api_key: os.environ/TINFOIL_API_KEY
model_info:
input_cost_per_token: 0.00000065
output_cost_per_token: 0.00000145
cache_read_input_token_cost: 0.00000013
# GPT-OSS 120B — lightweight fallback
- model_name: tinfoil/gpt-oss-120b
litellm_params:
model: openai/gpt-oss-120b
api_base: http://127.0.0.1:3301/v1
api_key: os.environ/TINFOIL_API_KEY
model_info:
input_cost_per_token: 0.00000015
output_cost_per_token: 0.00000060
# GLM-5.3 (full flagship) on Tinfoil (TEE) — replaced tinfoil/glm-5-3-flash
# (Tinfoil deprecated the flash variant 2026-10-09). $1.80 in / $5.75 out /
# $0.45 cached per 1M. Text-only (no vision), reasoning + tool calling.
- model_name: tinfoil/glm-5.3
litellm_params:
model: openai/glm-5-3
api_base: http://127.0.0.1:3301/v1
api_key: os.environ/TINFOIL_API_KEY
model_info:
input_cost_per_token: 0.0000018
output_cost_per_token: 0.00000575
cache_read_input_token_cost: 0.00000045
# --- GreenPT (renewable energy) ---
# OpenAI-compatible, EU-hosted on 100% renewable energy. No TEE proxy needed —
# LiteLLM talks to GreenPT directly.
# (Removed 2026-10: green-l/green-r — GreenPT now marks them "not recommended".)
# GLM-5.3 Flash on GreenPT (note the dotted slug — distinct from Tinfoil's
# glm-5-3-flash). $0.11 in / $0.44 out / $0.022 cached per 1M.
- model_name: greenpt/glm-5.3-flash
litellm_params:
model: openai/glm-5.3-flash
api_base: https://api.greenpt.ai/v1
api_key: os.environ/GREENPT_API_KEY
model_info:
input_cost_per_token: 0.00000011
output_cost_per_token: 0.00000044
cache_read_input_token_cost: 0.000000022
# GLM-5.3 (full flagship) on GreenPT — strongest agentic model in the green
# tier. $1.10 in / $4.40 out per 1M, 1M context.
- model_name: greenpt/glm-5.3
litellm_params:
model: openai/glm-5.3
api_base: https://api.greenpt.ai/v1
api_key: os.environ/GREENPT_API_KEY
model_info:
input_cost_per_token: 0.0000011
output_cost_per_token: 0.0000044
cache_read_input_token_cost: 0.000000275
# --- Audio models (Tinfoil, inside the TEE) ---
# NOT chat models: these power /v1/audio/* routes (STT for dictation, TTS for
# voice mode). They stay VISIBLE in the API model list so developer members
# can call them, but the portal filters them out of the chat-facing list
# (Open WebUI) by their `mode` field, so they never appear as pickable chat
# options. Pricing per Tinfoil catalog: whisper $0.05/1M in; voxtral-tts
# listed $0/$0 (verify on invoice — may be beta-free).
# --- Regolo (EU, zero data retention, 100% renewables) ---
# Seeweb S.r.l. (Italy). Added 2026-10-16 following member suggestion (#22).
# apertus-70b: replaced PublicAI (removed 2026-10-16) as our Apertus provider —
# same open-weights model, ~45% cheaper input, EU ZDR + renewables.
# qwen3.5-122b: large vision+reasoning MoE (262K ctx). REQUIRES
# chat_template_kwargs: {enable_thinking: false} — otherwise the reasoning
# phase consumes the entire token budget and content comes back empty.
# Qwen-Image: image generation via /v1/images/generations, billed per pixel.
- model_name: regolo/apertus-70b
litellm_params:
model: openai/apertus-70b
api_base: https://api.regolo.ai/v1
api_key: os.environ/REGOLO_API_KEY
model_info:
mode: chat
input_cost_per_token: 4.6e-7
output_cost_per_token: 2.44e-6
- model_name: regolo/qwen3.5-122b
litellm_params:
model: openai/qwen3.5-122b
api_base: https://api.regolo.ai/v1
api_key: os.environ/REGOLO_API_KEY
chat_template_kwargs: {"enable_thinking": false}
model_info:
mode: chat
input_cost_per_token: 1.16e-6
output_cost_per_token: 4.87e-6
- model_name: regolo/qwen-image
litellm_params:
model: openai/Qwen-Image
api_base: https://api.regolo.ai/v1
api_key: os.environ/REGOLO_API_KEY
model_info:
mode: image_generation
input_cost_per_pixel: 1.5e-8
# --- Audio models (Tinfoil, in-TEE) ---
# Used for chat dictation (STT) and voice mode (TTS). These are NOT chat
# models: the member portal's /v1/models filters them out of the chat model
# list (see AUDIO_MODEL_PREFIXES there) so members don't see them in the
# picker, but they remain fully callable via /v1/audio/* for API users.
# whisper-large-v3-turbo: $0/1M tokens + $0.01 per request.
# voxtral-tts: $0/1M tokens + $0.002 per request.
# Tinfoil bills these per REQUEST (its model list carries a `requestPrice`);
# reading only the two token prices made TTS look free and left STT billed off
# an older catalog. LiteLLM cannot price per request on the audio paths, so
# patch_audio_request_cost.py (built in via the Dockerfile) makes the cost
# calculator honour `input_cost_per_request` below. Re-check that build log
# line after every LiteLLM upgrade.
# NOTE: LiteLLM routes audio by the openai/ provider + the endpoint called
# (/v1/audio/transcriptions or /v1/audio/speech) — there is no
# "audio_transcription/" provider prefix.
- model_name: tinfoil/whisper-large-v3-turbo
litellm_params:
model: openai/whisper-large-v3-turbo
api_base: http://127.0.0.1:3301/v1
api_key: os.environ/TINFOIL_API_KEY
model_info:
mode: audio_transcription
# Tinfoil: $0 per token, $0.01 per request. The per-request charge is the
# whole cost, and the patch is what makes it billable.
input_cost_per_token: 0
input_cost_per_request: 0.01
- model_name: tinfoil/voxtral-tts
litellm_params:
model: openai/voxtral-tts
api_base: http://127.0.0.1:3301/v1
api_key: os.environ/TINFOIL_API_KEY
model_info:
mode: audio_speech
# Tinfoil: $0 per token, $0.002 per request — see the audio section note.
#
# input_cost_per_character: 0 is load-bearing, not decoration. The TTS cost
# branch (cost_per_token, cost_metric == "cost_per_character") raises
# ValueError("cost for tts call is None") when there is no character price,
# and that raise aborts the whole cost calculation — so the spend write is
# skipped and speech bills nothing, whatever input_cost_per_request says.
# Declaring the zero lets the path complete; the patch then adds the
# per-request charge.
input_cost_per_character: 0
input_cost_per_request: 0.002
# --- Embedding model (Tinfoil, in-TEE) ---
# nomic-embed-text — 768-dim text embeddings for RAG/semantic search.
# NOT a chat model: it powers /v1/embeddings only. Scoped to the admin
# account's team (model_info.team_id) so member keys cannot call it — it is
# an operator resource for the co-op's own docs-RAG, not a member-facing
# model. The portal also filters it out of the chat-facing /v1/models list by
# its `mode: embedding`. Pricing per Tinfoil catalog: $0.05/1M input, $0 out.
- model_name: tinfoil/nomic-embed-text
litellm_params:
model: openai/nomic-embed-text
api_base: http://127.0.0.1:3301/v1
api_key: os.environ/TINFOIL_API_KEY
model_info:
mode: embedding
team_id: cdfdf422-633f-4432-8893-cffee82a8325
input_cost_per_token: 0.00000005
output_cost_per_token: 0.0
# Fallback: if DeepSeek is down, use GPT-OSS
# NOTE: DeepSeek V4.1 Flash is a reasoning model — generations routinely run
# longer than 30s. A 30s timeout caused LiteLLM to abandon in-flight DeepSeek
# requests (falling back to GPT-OSS), while Tinfoil still completed and billed
# the abandoned generation. Result: Tinfoil billed ~2x the tokens LiteLLM logged.
# Fix: a generous timeout (180s) so generations complete inside LiteLLM and get
# logged; and a retry_policy that retries transient connection errors (base
# num_retries) but NEVER retries timeouts (TimeoutErrorRetries: 0) — a timed-out
# generation is already billed by Tinfoil, so re-hitting it re-bills.
router_settings:
num_retries: 2
timeout: 180
retry_policy:
TimeoutErrorRetries: 0
fallbacks:
- greenpt/deepseek-v4.1-flash:
- tinfoil/deepseek-v4-1-flash
- tinfoil/deepseek-v4-1-flash:
- tinfoil/gpt-oss-120b
general_settings:
master_key: os.environ/LITELLM_MASTER_KEY
database_url: os.environ/DATABASE_URL
store_model_in_db: true
litellm_settings:
salt_key: os.environ/LITELLM_SALT_KEY
drop_params: true
num_threads: 4
request_timeout: 180
Reference in new issue View Git Blame Copy Permalink