Add upgrade runbook + bake memory limit and LITELLM_MIGRATION_DIR into packaging

The 2026-10-02 outage had three causes the runbook now prevents:
- floating tag drift (README now mandates exact-tag pinning)
- 256MB default cgroup cap OOM-killing the prisma migration engine
  (manifest now sets memoryLimit: 2147483648)
- no _prisma_migrations ledger -> P3005 crash-loop on boot
  (start.sh now exports LITELLM_MIGRATION_DIR=/app/data/migrations)

README 'Updating LiteLLM' section rewritten from the old (wrong)
floating-tag procedure into a runbook with rules + verify checklist.

CloudronManifest.json

@@ -5,9 +5,10 @@
   "description": "LiteLLM AI Gateway — OpenAI-compatible proxy with per-user API keys, usage tracking, rate limiting, and model routing. Packaged for Cloudron with SSO/OIDC and PostgreSQL.",
   "tagline": "AI gateway for cooperative inference",
   "version": "1.0.0",
-  "upstreamVersion": "1.74.0",
+  "upstreamVersion": "1.84.0",
   "healthCheckPath": "/health/readiness",
   "httpPort": 4000,
+  "memoryLimit": 2147483648,
   "manifestVersion": 2,
   "website": "https://litellm.ai",
   "contactEmail": "botbot@hermes",

Dockerfile

@@ -1,4 +1,9 @@
-FROM ghcr.io/berriai/litellm:main-v1.74.0-stable
+FROM ghcr.io/berriai/litellm:v1.84.0
+
+# v1.84.0 is the fix line for CVE-2026-35029 (auth bypass on
+# /config/update, fixed 1.83.0) and CVE-2026-59822 (MCP session auth
+# bypass, fixed 1.84.0, CISA KEV catalog). Pin the exact tag — do NOT
+# use a floating tag like main-v1.74.0-stable, which silently moved.
 
 # LiteLLM's official image runs as root. Cloudron's sandboxing (read-only
 # rootfs, AppArmor, dropped capabilities) still applies to root inside the
@@ -19,10 +24,7 @@ COPY config.yaml /app/data/config.yaml
 # startup script runs instead of being passed as an argument to litellm.
 ENTRYPOINT ["/app/code/start.sh"]
 
-# LiteLLM v1.74.0 hardcodes `origins = ["*"]` (with allow_credentials=True) in
-# proxy_server.py and ignores LITELLM_CORS_ALLOWED_ORIGINS entirely. Rewrite it
-# to honour the env var so we can lock CORS to the chat origin. Patch every copy
-# (site-packages is what the running CLI imports; /app/litellm is a dev copy).
-RUN sed -i 's/^origins = \["\*"\]$/import os; origins = os.getenv("LITELLM_CORS_ALLOWED_ORIGINS", "*").split(",")/' \
-      /usr/lib/python3.13/site-packages/litellm/proxy/proxy_server.py \
-      /app/litellm/proxy/proxy_server.py
+# No CORS patch needed since v1.84.0: upstream replaced the hardcoded
+# `origins = ["*"]` with a native LITELLM_CORS_ORIGINS env var (start.sh
+# exports it). The old sed patch targeted a v1.74.0 file layout that no
+# longer exists — the v1.84.0 rebuild self-skipped it correctly.

README.md

@@ -91,38 +91,72 @@ The following are auto-generated and should **not** be set manually:
 
 ## Updating LiteLLM
 
-This is the routine maintenance path. LiteLLM releases frequently; updating is a two-step change.
+This is the routine maintenance path. LiteLLM releases frequently. Upgrading is
+a code change (pin a new version) plus a DB migration that runs on first boot —
+expect a 15–20 min maintenance window and **never panic-restart mid-migration**
+(each restart resets the clock).
 
-### 1. Bump the base image tag
+### Rules (learned from the 2026-10-02 outage)
 
-In `Dockerfile`, change the pinned version:
+- **Pin the exact version tag, never a floating tag.** Registries silently move
+  floating tags (`main-v1.74.0-stable`, `latest`) to newer digests, so a routine
+  rebuild changes the deployed version with no pin change and triggers a full
+  schema migration against an unprepared DB. Always
+  `FROM ghcr.io/berriai/litellm:vX.Y.Z`.
+- **Give the app a real memory limit.** Cloudron's default when `memoryLimit: 0`
+  is a 256 MB cgroup cap; the proxy fits in 256 MB but Prisma's schema engine
+  does NOT, and gets SIGKILL'd (OOM, exit 137) mid-migration. This package sets
+  `memoryLimit: 2147483648` (2 GB) in `CloudronManifest.json` — don't lower it.
+- **`LITELLM_MIGRATION_DIR=/app/data/migrations` is exported in `start.sh`.** The
+  database was created by `prisma db push`, so it has no `_prisma_migrations`
+  ledger; without a writable migration dir the boot migration hits P3005
+  ("schema is not empty") and crash-loops. Keep it pointed at the persistent
+  volume.
+- **Verify with a member-scope key, not only the master key.** Master-key
+  requests bypass the member auth path, so they can pass even on a half-migrated
+  schema while every member key 401s. See the checklist below.
+
+### Steps
+
+1. **Pick a target version** from the [LiteLLM changelog](https://github.com/BerriAI/litellm/releases). Stay at or above v1.84.0 (the fix line for CVE-2026-35029 and CVE-2026-59822, the latter on CISA's KEV list).
+
+2. **Bump the pin** in `Dockerfile`:
 
    ```dockerfile
-FROM ghcr.io/berriai/litellm:main-v1.74.0-stable
-#                              ^^^^^^ bump this
+   FROM ghcr.io/berriai/litellm:v1.84.0
+   #                              ^^^^^^ exact tag, never main-*-stable
    ```
 
-### 2. Bump the manifest version
-
-In `CloudronManifest.json`, update:
+3. **Bump the manifest** in `CloudronManifest.json`:
 
    ```json
-"version": "1.0.0",
-"upstreamVersion": "1.74.0"
+   "upstreamVersion": "1.84.0"
    ```
 
-### 3. Redeploy
+4. **Redeploy:**
 
    ```bash
    cloudron update --app <app-id>
    ```
 
-Cloudron rebuilds from source and applies the update. Data (keys, spend logs) persists in PostgreSQL.
+   First boot after the version change runs `prisma migrate deploy` (or
+   baselines + `db push` if the ledger is missing) before binding port 4000.
+   Expect 15–20 min of ECONNREFUSED healthcheck errors — this is normal.
+
+### Verify after upgrading
+
+1. `GET https://gateway.inference.coop/health/liveliness` → 200
+2. Generate a temp key with the master key → if `/key/generate` 500s, the schema is incomplete; fix the migration before anything else
+3. Use the **temp key** (not master) for `GET /v1/models` → full catalog
+4. Use the **temp key** for a small `POST /v1/chat/completions` → content + usage
+5. OPTIONS preflight from a foreign origin → 400; from `chat.inference.coop` → allowed
+6. Cloudron app health = healthy, then delete the temp key
 
 ### Before updating
 
 - Check the [LiteLLM changelog](https://github.com/BerriAI/litellm/releases) for breaking changes
 - The `start.sh` and `config.yaml` may need adjustment if LiteLLM changes its env-var or config schema
+- The CORS env var is `LITELLM_CORS_ORIGINS` (upstream-native since 1.84.0); the old `LITELLM_CORS_ALLOWED_ORIGINS` + sed patch are dead — don't re-add them
 - Test on a staging instance if the version jump is large
 
 ---

start.sh

@@ -61,6 +61,13 @@ export STORE_MODEL_IN_DB="True"
 # (for single-instance deployment this is fine)
 export DISABLE_SCHEMA_UPDATE="false"
 
+# The DB was created by `prisma db push` (no _prisma_migrations ledger), so on
+# first boot after a version change LiteLLM hits P3005 ("schema is not empty")
+# and must baseline the existing schema. That baseline write needs a WRITABLE
+# directory — the package dir is read-only under Cloudron's rootfs, so point it
+# at the persistent volume. Without this the boot migration crash-loops.
+export LITELLM_MIGRATION_DIR="/app/data/migrations"
+
 # UI configuration — use Cloudron's domain
 export UI_USERNAME="admin"
 export UI_PASSWORD="${LITELLM_MASTER_KEY}"
@@ -77,11 +84,13 @@ export NO_OPENAPI="True"
 # Proxy settings for Cloudron's reverse proxy
 export LITELLM_PROXY_BASE_URL="https://${CLOUDRON_APP_DOMAIN}"
 
-# CORS: lock browser access to the co-op's own chat origin. The gateway is
-# called server-to-server (chat → portal → gateway) and by member API keys
-# (curl/SDKs, not browsers), so no wildcard is needed. A `*` here (the old
-# fallback) would let any website's JS hit the gateway.
-export LITELLM_CORS_ALLOWED_ORIGINS="https://chat.inference.coop"
+# CORS: LiteLLM v1.84.0 reads LITELLM_CORS_ORIGINS natively (upstream added
+# the env var in place of the old hardcoded `origins = ["*"]`). Lock browser
+# access to the co-op's own chat origin. The gateway is called
+# server-to-server (chat → portal → gateway) and by member API keys
+# (curl/SDKs, not browsers), so no wildcard is needed. A `*` here would let
+# any website's JS hit the gateway.
+export LITELLM_CORS_ORIGINS="https://chat.inference.coop"
 
 echo "PostgreSQL: ${CLOUDRON_POSTGRESQL_HOST}:${CLOUDRON_POSTGRESQL_PORT}"
 echo "Redis: ${CLOUDRON_REDIS_HOST}:${CLOUDRON_REDIS_PORT}"