LiteLLM Gateway — Cloudron App

A Cloudron app package for LiteLLM, the open-source AI gateway, configured for the Inference Cooperative.

LiteLLM is the single gateway between members and the inference backends. It handles per-member API keys, usage metering, rate limiting (via budget caps), and model routing — all behind one OpenAI-compatible endpoint.

Current base image: LiteLLM v1.103.2, pinned to the exact tag (fix line for CVE-2026-35029 / CVE-2026-59822, CISA KEV-listed; also carries the streaming-usage fix for prompt_tokens_details). See Updating LiteLLM.

What this provides

• LiteLLM proxy with an OpenAI-compatible API on port 4000
• Per-member virtual API keys with budget caps and rate limits
• Usage / spend tracking per key, per model
• Model routing to multiple providers, chosen by member values (private / green / public)
• Cloudron PostgreSQL for keys, teams, and spend logs
• Cloudron Redis for rate limiting and caching
• Automatic SSL, backups, and sandboxing (Cloudron-managed)
• Master key + salt key auto-generated on first start, persisted in /app/data
• Hardened for the public internet: LiteLLM's built-in API docs surfaces are disabled (NO_DOCS/NO_REDOC/NO_OPENAPI → live 404s). The static /ui/ admin-UI bundle is served on this version (see Admin access); logging into it still requires the master key. CORS is deliberately open (LITELLM_CORS_ORIGINS=*, co-op/support#38) — auth is the per-member bearer key, not origin checking.

Architecture

Members → LibreChat (chat) / own tools (API) → member portal (key injection)
        → LiteLLM gateway (this app)
            ├→ Tinfoil (TEE enclaves — architectural privacy)
            ├→ GreenPT  (100% renewable energy)
            └→ Regolo   (EU-hosted, renewable + zero data retention)

LiteLLM is the control plane (keys, metering, routing). It does not run models itself — it forwards to providers, each chosen by members for a different value. A tinfoil-proxy sidecar provides EHBP end-to-end encryption to Tinfoil's enclaves.

Files

• CloudronManifest.json — Cloudron app manifest (addons, ports, memory limit, metadata)
• Dockerfile — wraps LiteLLM's official image for Cloudron's environment (pinned to an exact tag)
• start.sh — startup script: wires Cloudron's Postgres/Redis, downloads the tinfoil-proxy sidecar, exports LITELLM_MIGRATION_DIR
• config.yaml — LiteLLM config: model catalog, provider routing, retry policy, custom per-token pricing
• logo.png — app icon

Models

The catalog is defined in config.yaml and documented for members in co-op/docs models — that page is the single source of truth; this repo's config.yaml mirrors it. Models follow the provider/model-name convention so members can choose not just which model but which provider values it embodies (privacy, renewable energy, public development).

The catalog changes by member decision — expect it to evolve.

---

Building and installing

This package uses Cloudron's on-server build (no local Docker, no registry). The Cloudron CLI uploads the source and Cloudron builds the image on the server.

Prerequisites

• The Cloudron CLI installed on a machine with access to the Cloudron server
• A Cloudron instance with the inference.coop domain

Install

# 1. Install the Cloudron CLI (once)
npm install -g cloudron-cli

# 2. Log in to the Cloudron
cloudron login my.inference.coop

# 3. From this directory, install (builds on the server)
cloudron install --location gateway

--location gateway installs the app at gateway.inference.coop. Omit --location to be prompted.

Configure

After install, set the provider API keys in Cloudron → app → Settings → Environment Variables:

• TINFOIL_API_KEY — Tinfoil (https://tinfoil.sh/)
• GREENPT_API_KEY — GreenPT (https://greenpt.ai/)
• REGOLO_API_KEY — Regolo (regolo.ai; replaced PublicAI, removed 2026-10-16)

The following are auto-generated and should not be set manually:

• LITELLM_MASTER_KEY — generated on first start, stored in /app/data/.master_key
• LITELLM_SALT_KEY — generated on first start, stored in /app/data/.salt_key
• DATABASE_URL — from Cloudron's PostgreSQL addon
• REDIS_HOST, REDIS_PORT, REDIS_PASSWORD — from Cloudron's Redis addon

Config deployment

config.yaml ships with the image and start.sh syncs it over the persistent /app/data/config.yaml at boot whenever they differ. The volume copy would otherwise shadow the image's file forever — which is exactly how a config change can be merged, built and "deployed" while the gateway keeps serving the old configuration. Change the repo and redeploy; never edit the volume copy directly.

Admin access

The state of the admin surfaces on the pinned v1.103.2, wire-verified 2026-11-05:

• The built-in API docs are disabled (NO_DOCS/NO_REDOC/NO_OPENAPI in start.sh) — /docs, /redoc and /openapi.json return 404.
• The static admin UI bundle is still served at /ui/ — upstream's DISABLE_ADMIN_UI env (which we set) only disables the UI's SSO login redirect routes in this version; nothing on v1.103.2 stops the unconditional app.mount("/ui", …). So GET /ui/ serves the login page. Logging in still requires the master key (auto-generated here, never published), and administrative operations normally run through the member portal, which holds the master key server-side.
• If the login surface itself should disappear, it needs a build-time patch (strip the /ui mount from proxy_server in the Dockerfile, the same non-fatal patch pattern the other 8 build patches use) — there is no env/flag for it on this version. Until then treat /ui/ as a master-key-only endpoint, like any other control-plane route.

If direct LiteLLM admin access is ever needed, log in at /ui/ with the master key from /app/data/.master_key and lock it away again afterwards.

---

Updating LiteLLM

This is the routine maintenance path. LiteLLM releases frequently. Upgrading is a code change (pin a new version) plus a DB migration that runs on first boot — expect a 15–20 min maintenance window and never panic-restart mid-migration (each restart resets the clock).

Rules (learned from the 2026-10-02 outage)

• Pin the exact version tag, never a floating tag. Registries silently move floating tags (main-v1.74.0-stable, latest) to newer digests, so a routine rebuild changes the deployed version with no pin change and triggers a full schema migration against an unprepared DB. Always FROM ghcr.io/berriai/litellm:vX.Y.Z.
• Give the app a real memory limit. Cloudron's default when memoryLimit: 0 is a 256 MB cgroup cap; the proxy fits in 256 MB but Prisma's schema engine does NOT, and gets SIGKILL'd (OOM, exit 137) mid-migration. This package sets memoryLimit: 2147483648 (2 GB) in CloudronManifest.json — don't lower it.
• LITELLM_MIGRATION_DIR=/app/data/migrations is exported in start.sh. The database was created by prisma db push, so it has no _prisma_migrations ledger; without a writable migration dir the boot migration hits P3005 ("schema is not empty") and crash-loops. Keep it pointed at the persistent volume.
• Verify with a member-scope key, not only the master key. Master-key requests bypass the member auth path, so they can pass even on a half-migrated schema while every member key 401s. See the checklist below.

Steps

1. Pick a target version from the LiteLLM changelog. Stay at or above v1.84.0 (the fix line for CVE-2026-35029 and CVE-2026-59822, the latter on CISA's KEV list).

2. Bump the pin in Dockerfile:

   FROM ghcr.io/berriai/litellm:v1.84.0
   #                              ^^^^^^ exact tag, never main-*-stable
   

3. Bump the manifest in CloudronManifest.json:

   "upstreamVersion": "1.84.0"
   

4. Redeploy:

   cloudron update --app <app-id>
   

   First boot after the version change runs prisma migrate deploy (or baselines + db push if the ledger is missing) before binding port 4000. Expect 15–20 min of ECONNREFUSED healthcheck errors — this is normal.

Verify after upgrading

1. GET https://gateway.inference.coop/health/liveliness → 200
2. Generate a temp key with the master key → if /key/generate 500s, the schema is incomplete; fix the migration before anything else
3. Use the temp key (not master) for GET /v1/models → full catalog
4. Use the temp key for a small POST /v1/chat/completions → content + usage
5. OPTIONS preflight: LITELLM_CORS_ORIGINS is set at the app level (* by co-op decision, co-op/support#38) — a preflight from any origin answers with the matching access-control-allow-origin header rather than a 400. Auth is the bearer key, not the origin.
6. Cloudron app health = healthy, then delete the temp key

Before updating

• Check the LiteLLM changelog for breaking changes
• The start.sh and config.yaml may need adjustment if LiteLLM changes its env-var or config schema
• Build-time patches ride on LiteLLM internals — verify each one after an upgrade. patch_budget_message.py, patch_image_cost.py (keeps image-gen pricing on the shared cost-map key) and patch_audio_request_cost.py (adds the declared per-request charge on the audio paths: Tinfoil bills voxtral-tts $0.002 and whisper-large-v3-turbo $0.01 per request, which the cost map cannot express). Each prints [coop-patch] in the build log. They are non-fatal by design: a missing anchor warns and the build still succeeds, so the behaviour can vanish silently on a version bump. Related: code/litellm#5.
• patch_passthrough_token_cost.py prices a generic pass-through route from the token count it reports in its own response body. A pass_through_endpoints target whose host matches none of vertex/anthropic/openai/gemini resolves to EndpointType.GENERIC: it is logged, but with no model on the route LiteLLM only knows the flat cost_per_request, which is a budget estimate, not an invoice. The patch bills any route listed in general_settings.coop_passthrough_cost_per_token (USD per input token, keyed by path) as usage.input_tokens × rate — Metis (/v1/systemone) at $0.12 per 1M input tokens — and records the same count on the row (see below). Same rules: non-fatal, [coop-patch] in the build log, verify after every upgrade. cost_per_request stays as the fallback for a listed route whose response carries no token count. The block prints [coop-passthrough-cost] input_tokens=… rate=… -> response_cost=… on each priced call, so the log is the field check that it fired.
• Pass-through spend rows also carry the token counts (Metis defect fix). Pricing the row from the body was not enough: the row recorded the correct spend but prompt_tokens/completion_tokens/total_tokens stayed 0, because the pass-through success handler hands the spend writer a StandardPassThroughResponseObject whose usage is empty and the generic branch builds no standard_logging_object usage. Anything summing tokens for the route (panel, dashboard, energy reporting) then under-reported. The same patch now sets combined_usage_object (a litellm.Usage) on the logging object — the override the streaming handler already uses for recovered partial usage — which the spend writer reads before the response object (spend_tracking_utils.get_logging_payload, StandardLoggingPayloadSetup.get_usage_as_dict). The row now records prompt_tokens = usage.input_tokens, completion_tokens = 0, total_tokens = usage.input_tokens, with spend unchanged.
• A zero-spend row with call_type = /v1/<path> is NOT a duplicate of the priced row — it is the 401 refusal. Every rejected pass-through request (missing/invalid key) is logged by upstream LiteLLM as a failure row whose call_type is the route path and whose spend is 0, via ProxyLogging._handle_logging_proxy_only_error (litellm/proxy/utils.py, original_function=route). That is upstream behaviour on the pristine image, not something our patches add (neither patch_passthrough_key_access.py nor patch_passthrough_token_cost.py writes a row). A single authenticated call produces exactly one row (call_type=pass_through_endpoint, the real spend). Measured: one 200 → one row; one unauthenticated 401 → one failure row. Count requests for the route by call_type = pass_through_endpoint, not by summing all rows carrying the model name.
• The CORS env var is LITELLM_CORS_ORIGINS (upstream-native since 1.84.0); the old LITELLM_CORS_ALLOWED_ORIGINS + sed patch are dead — don't re-add them
• Test on a staging instance if the version jump is large

---

Environment variables

Variable	Required	Purpose
TINFOIL_API_KEY	Yes	Tinfoil provider (TEE inference)
GREENPT_API_KEY	Yes	GreenPT provider (renewable energy)
REGOLO_API_KEY	Yes	Regolo provider (EU, renewable + ZDR)
LITELLM_CORS_ORIGINS	Set at app level	Currently * by co-op decision (co-op/support#38); start.sh falls back to the chat origin only if the app env omits it
LITELLM_MIGRATION_DIR	Set by start.sh	Persistent Prisma migrations ledger

Connecting chat clients

LibreChat (chat) does not talk to this gateway directly — it routes through the member portal, which authenticates the member and injects their personal LiteLLM key per request. Members building their own tools create API keys on the member dashboard and use:

• API Base URL: https://gateway.inference.coop/v1 (or the portal's proxy endpoint for chat-integrated tools)
• API Key: their own virtual key from the dashboard

Related repos

• code/member-portal — membership middleware: provisioning, billing webhooks, key injection
• code/member-dashboard — member-facing usage and API-key management
• code/admin-panel — admin overview
• co-op/website — the public landing page
• co-op/docs — documentation, including the member-facing model list
• co-op/design-assets — logos, fonts, brand materials

Scheduled tasks (scheduler addon)

The gateway pings the Tinfoil audio models every 10 minutes (audio_keepalive.py, declared in CloudronManifest.json → addons.scheduler) to keep the TEE audio enclave warm — voice mode then skips the 24s cold-start after idle. TTS-only: an STT ping would cost $0.01/request ($52/mo at 6/hr) versus $0.002 for TTS.

The script reads the master key from /app/data/.master_key (written by start.sh); the LITELLM_MASTER_KEY env var is only exported into the litellm process, not scheduler tasks. Verify after deploys: cloudron exec --app <id> -- python3 /app/code/audio_keepalive.py should exit 0 silently, and the deploy log must show setupAddons: ... scheduler ... audio_keepalive.