LiteLLM Gateway — Cloudron App

A Cloudron app package for LiteLLM, the open-source AI gateway, configured for the Inference Cooperative.

LiteLLM is the single gateway between members and the inference backends. It handles per-member API keys, usage metering, rate limiting (via budget caps), and model routing — all behind one OpenAI-compatible endpoint.

Current base image: LiteLLM v1.103.2, pinned to the exact tag (fix line for CVE-2026-35029 / CVE-2026-59822, CISA KEV-listed; also carries the streaming-usage fix for prompt_tokens_details). See Updating LiteLLM.

What this provides

• LiteLLM proxy with an OpenAI-compatible API on port 4000
• Per-member virtual API keys with budget caps and rate limits
• Usage / spend tracking per key, per model
• Model routing to multiple providers, chosen by member values (private / green / public)
• Cloudron PostgreSQL for keys, teams, and spend logs
• Cloudron Redis for rate limiting and caching
• Automatic SSL, backups, and sandboxing (Cloudron-managed)
• Master key + salt key auto-generated on first start, persisted in /app/data
• Hardened for the public internet: admin UI and API docs disabled at the proxy; CORS locked to the chat origin

Architecture

Members → Open WebUI (chat) / own tools (API) → member portal (key injection)
        → LiteLLM gateway (this app)
            ├→ Tinfoil (TEE enclaves — architectural privacy)
            ├→ GreenPT  (100% renewable energy)
            └→ PublicAI (publicly developed, sovereign models)

LiteLLM is the control plane (keys, metering, routing). It does not run models itself — it forwards to providers, each chosen by members for a different value. A tinfoil-proxy sidecar provides EHBP end-to-end encryption to Tinfoil's enclaves.

Files

• CloudronManifest.json — Cloudron app manifest (addons, ports, memory limit, metadata)
• Dockerfile — wraps LiteLLM's official image for Cloudron's environment (pinned to an exact tag)
• start.sh — startup script: wires Cloudron's Postgres/Redis, downloads the tinfoil-proxy sidecar, exports LITELLM_MIGRATION_DIR
• config.yaml — LiteLLM config: model catalog, provider routing, retry policy, custom per-token pricing
• logo.png — app icon

Models

The catalog is defined in config.yaml and documented for members in co-op/docs models — that page is the single source of truth; this repo's config.yaml mirrors it. Models follow the provider/model-name convention so members can choose not just which model but which provider values it embodies (privacy, renewable energy, public development).

The catalog changes by member decision — expect it to evolve.

---

Building and installing

This package uses Cloudron's on-server build (no local Docker, no registry). The Cloudron CLI uploads the source and Cloudron builds the image on the server.

Prerequisites

• The Cloudron CLI installed on a machine with access to the Cloudron server
• A Cloudron instance with the inference.coop domain

Install

# 1. Install the Cloudron CLI (once)
npm install -g cloudron-cli

# 2. Log in to the Cloudron
cloudron login my.inference.coop

# 3. From this directory, install (builds on the server)
cloudron install --location gateway

--location gateway installs the app at gateway.inference.coop. Omit --location to be prompted.

Configure

After install, set the provider API keys in Cloudron → app → Settings → Environment Variables:

• TINFOIL_API_KEY — Tinfoil (https://tinfoil.sh/)
• GREENPT_API_KEY — GreenPT (https://greenpt.ai/)
• PUBLICAI_API_KEY — PublicAI (https://publicai.co/)

The following are auto-generated and should not be set manually:

• LITELLM_MASTER_KEY — generated on first start, stored in /app/data/.master_key
• LITELLM_SALT_KEY — generated on first start, stored in /app/data/.salt_key
• DATABASE_URL — from Cloudron's PostgreSQL addon
• REDIS_HOST, REDIS_PORT, REDIS_PASSWORD — from Cloudron's Redis addon

Config deployment

config.yaml ships with the image and start.sh syncs it over the persistent /app/data/config.yaml at boot whenever they differ. The volume copy would otherwise shadow the image's file forever — which is exactly how a config change can be merged, built and "deployed" while the gateway keeps serving the old configuration. Change the repo and redeploy; never edit the volume copy directly.

Admin access

The LiteLLM admin UI and API docs are disabled on the public gateway (hardened by default; see start.sh). Administrative operations run through the member portal, which holds the master key server-side. If direct LiteLLM admin access is ever needed, re-enable the UI in start.sh, redeploy, and use the master key from /app/data/.master_key — then disable it again.

---

Updating LiteLLM

This is the routine maintenance path. LiteLLM releases frequently. Upgrading is a code change (pin a new version) plus a DB migration that runs on first boot — expect a 15–20 min maintenance window and never panic-restart mid-migration (each restart resets the clock).

Rules (learned from the 2026-10-02 outage)

• Pin the exact version tag, never a floating tag. Registries silently move floating tags (main-v1.74.0-stable, latest) to newer digests, so a routine rebuild changes the deployed version with no pin change and triggers a full schema migration against an unprepared DB. Always FROM ghcr.io/berriai/litellm:vX.Y.Z.
• Give the app a real memory limit. Cloudron's default when memoryLimit: 0 is a 256 MB cgroup cap; the proxy fits in 256 MB but Prisma's schema engine does NOT, and gets SIGKILL'd (OOM, exit 137) mid-migration. This package sets memoryLimit: 2147483648 (2 GB) in CloudronManifest.json — don't lower it.
• LITELLM_MIGRATION_DIR=/app/data/migrations is exported in start.sh. The database was created by prisma db push, so it has no _prisma_migrations ledger; without a writable migration dir the boot migration hits P3005 ("schema is not empty") and crash-loops. Keep it pointed at the persistent volume.
• Verify with a member-scope key, not only the master key. Master-key requests bypass the member auth path, so they can pass even on a half-migrated schema while every member key 401s. See the checklist below.

Steps

1. Pick a target version from the LiteLLM changelog. Stay at or above v1.84.0 (the fix line for CVE-2026-35029 and CVE-2026-59822, the latter on CISA's KEV list).

2. Bump the pin in Dockerfile:

   FROM ghcr.io/berriai/litellm:v1.84.0
   #                              ^^^^^^ exact tag, never main-*-stable
   

3. Bump the manifest in CloudronManifest.json:

   "upstreamVersion": "1.84.0"
   

4. Redeploy:

   cloudron update --app <app-id>
   

   First boot after the version change runs prisma migrate deploy (or baselines + db push if the ledger is missing) before binding port 4000. Expect 15–20 min of ECONNREFUSED healthcheck errors — this is normal.

Verify after upgrading

1. GET https://gateway.inference.coop/health/liveliness → 200
2. Generate a temp key with the master key → if /key/generate 500s, the schema is incomplete; fix the migration before anything else
3. Use the temp key (not master) for GET /v1/models → full catalog
4. Use the temp key for a small POST /v1/chat/completions → content + usage
5. OPTIONS preflight from a foreign origin → 400; from chat.inference.coop → allowed
6. Cloudron app health = healthy, then delete the temp key

Before updating

• Check the LiteLLM changelog for breaking changes
• The start.sh and config.yaml may need adjustment if LiteLLM changes its env-var or config schema
• Build-time patches ride on LiteLLM internals — verify each one after an upgrade. patch_budget_message.py, patch_image_cost.py (keeps image-gen pricing on the shared cost-map key) and patch_audio_request_cost.py (adds the declared per-request charge on the audio paths: Tinfoil bills voxtral-tts $0.002 and whisper-large-v3-turbo $0.01 per request, which the cost map cannot express). Each prints [coop-patch] in the build log. They are non-fatal by design: a missing anchor warns and the build still succeeds, so the behaviour can vanish silently on a version bump. Related: code/litellm#5.
• patch_passthrough_token_cost.py prices a generic pass-through route from the token count it reports in its own response body. A pass_through_endpoints target whose host matches none of vertex/anthropic/openai/gemini resolves to EndpointType.GENERIC: it is logged, but with no model on the route LiteLLM only knows the flat cost_per_request, which is a budget estimate, not an invoice. The patch bills any route listed in general_settings.coop_passthrough_cost_per_token (USD per input token, keyed by path) as usage.input_tokens × rate — Metis (/v1/systemone) at $0.12 per 1M input tokens — and records the same count on the row (see below). Same rules: non-fatal, [coop-patch] in the build log, verify after every upgrade. cost_per_request stays as the fallback for a listed route whose response carries no token count. The block prints [coop-passthrough-cost] input_tokens=… rate=… -> response_cost=… on each priced call, so the log is the field check that it fired.
• Pass-through spend rows also carry the token counts (Metis defect fix). Pricing the row from the body was not enough: the row recorded the correct spend but prompt_tokens/completion_tokens/total_tokens stayed 0, because the pass-through success handler hands the spend writer a StandardPassThroughResponseObject whose usage is empty and the generic branch builds no standard_logging_object usage. Anything summing tokens for the route (panel, dashboard, energy reporting) then under-reported. The same patch now sets combined_usage_object (a litellm.Usage) on the logging object — the override the streaming handler already uses for recovered partial usage — which the spend writer reads before the response object (spend_tracking_utils.get_logging_payload, StandardLoggingPayloadSetup.get_usage_as_dict). The row now records prompt_tokens = usage.input_tokens, completion_tokens = 0, total_tokens = usage.input_tokens, with spend unchanged.
• A zero-spend row with call_type = /v1/<path> is NOT a duplicate of the priced row — it is the 401 refusal. Every rejected pass-through request (missing/invalid key) is logged by upstream LiteLLM as a failure row whose call_type is the route path and whose spend is 0, via ProxyLogging._handle_logging_proxy_only_error (litellm/proxy/utils.py, original_function=route). That is upstream behaviour on the pristine image, not something our patches add (neither patch_passthrough_key_access.py nor patch_passthrough_token_cost.py writes a row). A single authenticated call produces exactly one row (call_type=pass_through_endpoint, the real spend). Measured: one 200 → one row; one unauthenticated 401 → one failure row. Count requests for the route by call_type = pass_through_endpoint, not by summing all rows carrying the model name.
• The CORS env var is LITELLM_CORS_ORIGINS (upstream-native since 1.84.0); the old LITELLM_CORS_ALLOWED_ORIGINS + sed patch are dead — don't re-add them
• Test on a staging instance if the version jump is large

---

Environment variables

Variable	Required	Purpose
TINFOIL_API_KEY	Yes	Tinfoil provider (TEE inference)
GREENPT_API_KEY	Yes	GreenPT provider (renewable energy)
PUBLICAI_API_KEY	Yes	PublicAI provider (public models)
LITELLM_CORS_ORIGINS	Set by start.sh	CORS allowlist (chat origin only)
LITELLM_MIGRATION_DIR	Set by start.sh	Persistent Prisma migrations ledger

Connecting chat clients

Open WebUI does not talk to this gateway directly — it routes through the member portal, which authenticates the member and injects their personal LiteLLM key per request. Members building their own tools create API keys on the member dashboard and use:

• API Base URL: https://gateway.inference.coop/v1 (or the portal's proxy endpoint for chat-integrated tools)
• API Key: their own virtual key from the dashboard

Related repos

• code/member-portal — membership middleware: provisioning, billing webhooks, key injection
• code/member-dashboard — member-facing usage and API-key management
• code/admin-panel — admin overview
• co-op/website — the public landing page
• co-op/docs — documentation, including the member-facing model list
• co-op/design-assets — logos, fonts, brand materials

Scheduled tasks (scheduler addon)

The gateway pings the Tinfoil audio models every 10 minutes (audio_keepalive.py, declared in CloudronManifest.json → addons.scheduler) to keep the TEE audio enclave warm — voice mode then skips the 24s cold-start after idle. TTS-only: an STT ping would cost $0.01/request ($52/mo at 6/hr) versus $0.002 for TTS.

The script reads the master key from /app/data/.master_key (written by start.sh); the LITELLM_MASTER_KEY env var is only exported into the litellm process, not scheduler tasks. Verify after deploys: cloudron exec --app <id> -- python3 /app/code/audio_keepalive.py should exit 0 silently, and the deploy log must show setupAddons: ... scheduler ... audio_keepalive.