litellm/README.md

# LiteLLM Gateway — Cloudron App

A Cloudron app package for [LiteLLM](https://litellm.ai/), the open-source AI gateway, configured for the **Inference Cooperative**.

LiteLLM is the single gateway between members and the inference backends. It handles per-member API keys, usage metering, rate limiting (via budget caps), and model routing — all behind one OpenAI-compatible endpoint.

**Current base image:** LiteLLM **v1.103.2**, pinned to the exact tag (fix line for CVE-2026-35029 / CVE-2026-59822, CISA KEV-listed; also carries the streaming-usage fix for prompt_tokens_details). See [Updating LiteLLM](#updating-litellm).

## What this provides

- **LiteLLM proxy** with an OpenAI-compatible API on port 4000
- **Per-member virtual API keys** with budget caps and rate limits
- **Usage / spend tracking** per key, per model
- **Model routing** to multiple providers, chosen by member values (private / green / public)
- **Cloudron PostgreSQL** for keys, teams, and spend logs
- **Cloudron Redis** for rate limiting and caching
- **Automatic SSL, backups, and sandboxing** (Cloudron-managed)
- **Master key + salt key** auto-generated on first start, persisted in `/app/data`
- Hardened for the public internet: LiteLLM's built-in API docs surfaces are disabled (`NO_DOCS`/`NO_REDOC`/`NO_OPENAPI` → live 404s). The static `/ui/` admin-UI bundle **is served** on this version (see *Admin access*); logging into it still requires the master key. CORS is deliberately open (`LITELLM_CORS_ORIGINS=*`, co-op/support#38) — auth is the per-member bearer key, not origin checking.

## Architecture

```
Members → LibreChat (chat) / own tools (API) → member portal (key injection)
        → LiteLLM gateway (this app)
            ├→ Tinfoil (TEE enclaves — architectural privacy)
            ├→ GreenPT  (100% renewable energy)
            └→ Regolo   (EU-hosted, renewable + zero data retention)
```

LiteLLM is the control plane (keys, metering, routing). It does **not** run models itself — it forwards to providers, each chosen by members for a different value. A [tinfoil-proxy](https://github.com/tinfoilsh/tinfoil-proxy) sidecar provides EHBP end-to-end encryption to Tinfoil's enclaves.

## Files

- `CloudronManifest.json` — Cloudron app manifest (addons, ports, memory limit, metadata)
- `Dockerfile` — wraps LiteLLM's official image for Cloudron's environment (pinned to an exact tag)
- `start.sh` — startup script: wires Cloudron's Postgres/Redis, downloads the tinfoil-proxy sidecar, exports `LITELLM_MIGRATION_DIR`
- `config.yaml` — LiteLLM config: model catalog, provider routing, retry policy, custom per-token pricing
- `logo.png` — app icon

## Models

The catalog is defined in `config.yaml` and documented for members in [co-op/docs models](https://git.inference.coop/co-op/docs/src/branch/main/models.md) — that page is the single source of truth; this repo's `config.yaml` mirrors it. Models follow the `provider/model-name` convention so members can choose not just which model but which provider values it embodies (privacy, renewable energy, public development).

The catalog changes by member decision — expect it to evolve.

---

## Building and installing

This package uses Cloudron's **on-server build** (no local Docker, no registry). The Cloudron CLI uploads the source and Cloudron builds the image on the server.

### Prerequisites

- The [Cloudron CLI](https://docs.cloudron.io/packaging/cli/) installed on a machine with access to the Cloudron server
- A Cloudron instance with the `inference.coop` domain

### Install

```bash
# 1. Install the Cloudron CLI (once)
npm install -g cloudron-cli

# 2. Log in to the Cloudron
cloudron login my.inference.coop

# 3. From this directory, install (builds on the server)
cloudron install --location gateway
```

`--location gateway` installs the app at `gateway.inference.coop`. Omit `--location` to be prompted.

### Configure

After install, set the provider API keys in Cloudron → app → Settings → Environment Variables:

- `TINFOIL_API_KEY` — Tinfoil (https://tinfoil.sh/)
- `GREENPT_API_KEY` — GreenPT (https://greenpt.ai/)
- `REGOLO_API_KEY` — Regolo (regolo.ai; replaced PublicAI, removed 2026-10-16)

The following are auto-generated and should **not** be set manually:

- `LITELLM_MASTER_KEY` — generated on first start, stored in `/app/data/.master_key`
- `LITELLM_SALT_KEY` — generated on first start, stored in `/app/data/.salt_key`
- `DATABASE_URL` — from Cloudron's PostgreSQL addon
- `REDIS_HOST`, `REDIS_PORT`, `REDIS_PASSWORD` — from Cloudron's Redis addon

### Config deployment

`config.yaml` ships **with the image** and `start.sh` syncs it over the persistent
`/app/data/config.yaml` at boot whenever they differ. The volume copy would
otherwise shadow the image's file forever — which is exactly how a config change
can be merged, built and "deployed" while the gateway keeps serving the old
configuration. Change the repo and redeploy; never edit the volume copy directly.

### Admin access

The state of the admin surfaces on the pinned v1.103.2, wire-verified 2026-11-05:

- The built-in **API docs are disabled** (`NO_DOCS`/`NO_REDOC`/`NO_OPENAPI` in
  `start.sh`) — `/docs`, `/redoc` and `/openapi.json` return 404.
- The **static admin UI bundle is still served at `/ui/`** — upstream's
  `DISABLE_ADMIN_UI` env (which we set) only disables the UI's *SSO login
  redirect* routes in this version; nothing on v1.103.2 stops the unconditional
  `app.mount("/ui", …)`. So `GET /ui/` serves the login page. **Logging in
  still requires the master key** (auto-generated here, never published), and
  administrative operations normally run through the
  [member portal](https://git.inference.coop/code/member-portal), which holds
  the master key server-side.
- If the login surface itself should disappear, it needs a build-time patch
  (strip the `/ui` mount from `proxy_server` in the Dockerfile, the same
  non-fatal patch pattern the other 8 build patches use) — there is no env/flag
  for it on this version. Until then treat `/ui/` as a master-key-only
  endpoint, like any other control-plane route.

If direct LiteLLM admin access is ever needed, log in at `/ui/` with the
master key from `/app/data/.master_key` and lock it away again afterwards.

---

## Updating LiteLLM

This is the routine maintenance path. LiteLLM releases frequently. Upgrading is
a code change (pin a new version) plus a DB migration that runs on first boot —
expect a 15–20 min maintenance window and **never panic-restart mid-migration**
(each restart resets the clock).

### Rules (learned from the 2026-10-02 outage)

- **Pin the exact version tag, never a floating tag.** Registries silently move
  floating tags (`main-v1.74.0-stable`, `latest`) to newer digests, so a routine
  rebuild changes the deployed version with no pin change and triggers a full
  schema migration against an unprepared DB. Always
  `FROM ghcr.io/berriai/litellm:vX.Y.Z`.
- **Give the app a real memory limit.** Cloudron's default when `memoryLimit: 0`
  is a 256 MB cgroup cap; the proxy fits in 256 MB but Prisma's schema engine
  does NOT, and gets SIGKILL'd (OOM, exit 137) mid-migration. This package sets
  `memoryLimit: 2147483648` (2 GB) in `CloudronManifest.json` — don't lower it.
- **`LITELLM_MIGRATION_DIR=/app/data/migrations` is exported in `start.sh`.** The
  database was created by `prisma db push`, so it has no `_prisma_migrations`
  ledger; without a writable migration dir the boot migration hits P3005
  ("schema is not empty") and crash-loops. Keep it pointed at the persistent
  volume.
- **Verify with a member-scope key, not only the master key.** Master-key
  requests bypass the member auth path, so they can pass even on a half-migrated
  schema while every member key 401s. See the checklist below.

### Steps

1. **Pick a target version** from the [LiteLLM changelog](https://github.com/BerriAI/litellm/releases). Stay at or above v1.84.0 (the fix line for CVE-2026-35029 and CVE-2026-59822, the latter on CISA's KEV list).

2. **Bump the pin** in `Dockerfile`:

   ```dockerfile
   FROM ghcr.io/berriai/litellm:v1.84.0
   #                              ^^^^^^ exact tag, never main-*-stable
   ```

3. **Bump the manifest** in `CloudronManifest.json`:

   ```json
   "upstreamVersion": "1.84.0"
   ```

4. **Redeploy:**

   ```bash
   cloudron update --app <app-id>
   ```

   First boot after the version change runs `prisma migrate deploy` (or
   baselines + `db push` if the ledger is missing) before binding port 4000.
   Expect 15–20 min of ECONNREFUSED healthcheck errors — this is normal.

### Verify after upgrading

1. `GET https://gateway.inference.coop/health/liveliness` → 200
2. Generate a temp key with the master key → if `/key/generate` 500s, the schema is incomplete; fix the migration before anything else
3. Use the **temp key** (not master) for `GET /v1/models` → full catalog
4. Use the **temp key** for a small `POST /v1/chat/completions` → content + usage
5. OPTIONS preflight: `LITELLM_CORS_ORIGINS` is set at the app level (`*` by co-op decision, co-op/support#38) — a preflight from any origin answers with the matching `access-control-allow-origin` header rather than a 400. Auth is the bearer key, not the origin.
6. Cloudron app health = healthy, then delete the temp key

### Before updating

- Check the [LiteLLM changelog](https://github.com/BerriAI/litellm/releases) for breaking changes
- The `start.sh` and `config.yaml` may need adjustment if LiteLLM changes its env-var or config schema
- **Build-time patches ride on LiteLLM internals — verify each one after an upgrade.** `patch_budget_message.py`, `patch_image_cost.py` (keeps image-gen pricing on the shared cost-map key) and `patch_audio_request_cost.py` (adds the declared per-request charge on the audio paths: Tinfoil bills `voxtral-tts` $0.002 and `whisper-large-v3-turbo` $0.01 per request, which the cost map cannot express). Each prints `[coop-patch]` in the build log. They are non-fatal by design: a missing anchor warns and the build still succeeds, so the behaviour can vanish silently on a version bump. Related: code/litellm#5.
- **`patch_passthrough_token_cost.py`** prices a generic pass-through route from the token count it reports in its own response body. A `pass_through_endpoints` target whose host matches none of vertex/anthropic/openai/gemini resolves to `EndpointType.GENERIC`: it is logged, but with no model on the route LiteLLM only knows the flat `cost_per_request`, which is a budget estimate, not an invoice. The patch bills any route listed in `general_settings.coop_passthrough_cost_per_token` (USD per input token, keyed by path) as `usage.input_tokens × rate` — Metis (`/v1/systemone`) at $0.12 per 1M input tokens — and records the same count on the row (see below). Same rules: non-fatal, `[coop-patch]` in the build log, verify after every upgrade. `cost_per_request` stays as the fallback for a listed route whose response carries no token count. The block prints `[coop-passthrough-cost] input_tokens=… rate=… -> response_cost=…` on each priced call, so the log is the field check that it fired.
- **Pass-through spend rows also carry the token counts (Metis defect fix).** Pricing the row from the body was not enough: the row recorded the correct `spend` but `prompt_tokens`/`completion_tokens`/`total_tokens` stayed `0`, because the pass-through success handler hands the spend writer a `StandardPassThroughResponseObject` whose `usage` is empty and the generic branch builds no `standard_logging_object` usage. Anything summing tokens for the route (panel, dashboard, energy reporting) then under-reported. The same patch now sets `combined_usage_object` (a `litellm.Usage`) on the logging object — the override the streaming handler already uses for recovered partial usage — which the spend writer reads before the response object (`spend_tracking_utils.get_logging_payload`, `StandardLoggingPayloadSetup.get_usage_as_dict`). The row now records `prompt_tokens = usage.input_tokens`, `completion_tokens = 0`, `total_tokens = usage.input_tokens`, with spend unchanged.
- **A zero-spend row with `call_type = /v1/<path>` is NOT a duplicate of the priced row — it is the 401 refusal.** Every *rejected* pass-through request (missing/invalid key) is logged by upstream LiteLLM as a failure row whose `call_type` is the route path and whose `spend` is `0`, via `ProxyLogging._handle_logging_proxy_only_error` (`litellm/proxy/utils.py`, `original_function=route`). That is upstream behaviour on the pristine image, not something our patches add (neither `patch_passthrough_key_access.py` nor `patch_passthrough_token_cost.py` writes a row). A single *authenticated* call produces exactly **one** row (`call_type=pass_through_endpoint`, the real spend). Measured: one 200 → one row; one unauthenticated 401 → one failure row. Count requests for the route by `call_type = pass_through_endpoint`, not by summing all rows carrying the model name.
- The CORS env var is `LITELLM_CORS_ORIGINS` (upstream-native since 1.84.0); the old `LITELLM_CORS_ALLOWED_ORIGINS` + sed patch are dead — don't re-add them
- Test on a staging instance if the version jump is large

---

## Environment variables

| Variable | Required | Purpose |
|----------|----------|---------|
| `TINFOIL_API_KEY` | Yes | Tinfoil provider (TEE inference) |
| `GREENPT_API_KEY` | Yes | GreenPT provider (renewable energy) |
| `REGOLO_API_KEY` | Yes | Regolo provider (EU, renewable + ZDR) |
| `LITELLM_CORS_ORIGINS` | Set at app level | Currently `*` by co-op decision (co-op/support#38); `start.sh` falls back to the chat origin only if the app env omits it |
| `LITELLM_MIGRATION_DIR` | Set by start.sh | Persistent Prisma migrations ledger |

## Connecting chat clients

LibreChat (chat) does **not** talk to this gateway directly — it routes through the [member portal](https://git.inference.coop/code/member-portal), which authenticates the member and injects their personal LiteLLM key per request. Members building their own tools create API keys on the [member dashboard](https://dashboard.inference.coop) and use:

- **API Base URL:** `https://gateway.inference.coop/v1` (or the portal's proxy endpoint for chat-integrated tools)
- **API Key:** their own virtual key from the dashboard

## Related repos

- `code/member-portal` — membership middleware: provisioning, billing webhooks, key injection
- `code/member-dashboard` — member-facing usage and API-key management
- `code/admin-panel` — admin overview
- `co-op/website` — the public landing page
- `co-op/docs` — documentation, including the member-facing model list
- `co-op/design-assets` — logos, fonts, brand materials

## Scheduled tasks (scheduler addon)

The gateway pings the Tinfoil audio models every 10 minutes (`audio_keepalive.py`,
declared in `CloudronManifest.json` → `addons.scheduler`) to keep the TEE audio
enclave warm — voice mode then skips the ~24s cold-start after idle. TTS-only:
an STT ping would cost $0.01/request (~$52/mo at 6/hr) versus $0.002 for TTS.

The script reads the master key from `/app/data/.master_key` (written by
`start.sh`); the `LITELLM_MASTER_KEY` env var is only exported into the litellm
process, not scheduler tasks. Verify after deploys: `cloudron exec --app <id>
-- python3 /app/code/audio_keepalive.py` should exit 0 silently, and the deploy
log must show `setupAddons: ... scheduler ... audio_keepalive`.