ops-docs/maintenance/2026-11-08-litellm-1.108.0-librechat-1.4.8.md

Upgrade window — 2026-11-08 (Sunday evening): LiteLLM 1.103.2 → 1.108.0, LibreChat overlay 1.4.5 → 1.4.8

Both upgrades were prepared and validated in advance (2026-11-06); nothing has been deployed. This runbook is written to be followed by any operator during the member-watched maintenance window. Read the gotchas in §6 before touching anything — this app pair has a history of deploys that look successful while changing nothing.

Candidates (already built and validated, sitting in the registry):

App	Candidate	Rollback (currently deployed)
LiteLLM gateway (adababd4-2427-4bdd-9bf7-6bdad0302610, gateway.inference.coop)	branch upgrade-v1.108.0 in code/litellm; prebuilt image registry.inference.coop/local/ai.coop.litellm:1.0.0-20261106-9e25578 (STALE — see §0 and §3.1)	image registry.inference.coop/local/ai.coop.litellm:20261107-201000-407e751e2 — the tag running tonight, verified present in the registry. 1.0.0-20261105-e161db9 is two deploys older and is NOT the rollback target
LibreChat chat (4023ccbf-d57a-4500-97df-14de9cf3516e, chat.inference.coop)	branch bump-appstore-1.4.8 in co-op/librechat-custom; prebuilt overlay registry.inference.coop/co-op/librechat-custom:20261106-c2aec74	overlay registry.inference.coop/co-op/librechat-custom:20261031-2a98892 (verified present)

Prep evidence (what was already verified, so Sunday only re-confirms it):

• LiteLLM 1.108.0 candidate: all 8 build patches verified inside the running candidate (file markers + build-log markers, zero WARNs); litellm reports 1.108.0.
• patch_image_cost.py (image-gen spend): a real image generation through the candidate produced a nonzero success SpendLogs row — twice (fake OpenAI-shaped endpoint and the real Regolo Qwen-Image API): spend 0.0011403264 for 256×256 at $1.74e-8/px, call_type aimage_generation, status success.
• patch_passthrough_token_cost.py (Metis metering): a real /v1/systemone call logged [coop-passthrough-cost] input_tokens=148 rate=1.2e-07 -> response_cost=1.776e-05 and the SpendLogs row matched exactly (148 × $0.12/1M), tokens recorded.
• Member path: temp key generated with master auth → /v1/models (11 models) → real chat completion (greenpt/glm-5.3-flash, "pong", usage 19/3 tokens, spend row written).
• Migrations: 36 new between 1.103.2 and 1.108.0, all additive (no unguarded DROP/DELETE/UPDATE). A fresh DB ran the full 220-migration chain cleanly; the live ledger (1.103.2) will apply only the 36 new ones.
• budget_reset_at: column unchanged on LiteLLM_TeamTable; a direct-DB write (the portal's exact pattern) plus API read-back verified on the candidate.
• Portal API compatibility: /team/list (bare list) and /key/list ({keys, total_count, …}) shapes are unchanged; a bare /team/list call still returns ALL teams (pagination only applies when page/page_size are passed — verified with 11 teams); /team/update still drops budget_reset_at, so the portal's direct-DB write remains required. SpendLogs gains only an additive billing_agent_id column — digest queries are unaffected.
• LibreChat 1.4.8 (appstore ai.librechat.cloudronapp): changelog 1.4.5 → 1.4.8 is tooling only — uv 0.13.6/0.13.7 and Node.js 22.23.4 (security fix, 1.4.7); the LibreChat upstream stays 0.8.8. Every overlay patch anchor (title tag, manifest link, theme-color metas, webmanifest colors, "MCP Servers" / "Every AI for Everyone." strings, index.*.js, sw.js) was verified against the real 1.4.8 image — no overlay porting needed; the candidate overlay built with all markers (UI strings patched in 3 files, index/manifest branding applied).

0. State re-check — 2026-11-09 04:15 UTC (read live before the window)

Verified against the live box on the night of the window. Anything here that contradicts the prep notes above is newer and wins.

• Gateway: image registry.inference.coop/local/ai.coop.litellm:20261107-201000-407e751e2 (deployed 2026-11-07 20:10 UTC), in-container version 1.103.2, 8 expected models
  • extras, healthy. The rollback target is therefore THIS tag, not 1.0.0-20261105-e161db9 (see §3.3).
• Chat: overlay registry.inference.coop/co-op/librechat-custom:20261031-2a98892, appstore 1.4.5, healthy. master is 2 commits ahead of bump-appstore-1.4.8, but both touch only librechat.yaml — a persistent-volume file that is NOT baked into the image, so pinning the Nov-06 candidate does not revert them.
• Registry: all four tags this plan needs are present — local/ai.coop.litellm:{1.0.0-20261106-9e25578, 20261107-201000-407e751e2} and co-op/librechat-custom:{20261106-c2aec74, 20261031-2a98892} (listed from the registry app's own storage).
• upgrade-v1.108.0 is 5 commits BEHIND main. main gained patch_transcription_response_format.py, patch_audio_default_mp3.py and audio_keepalive.py (Dockerfile RUN/chmod lines) plus the STT/embedding model additions on 2026-11-06/07, all after the candidate branch was cut. Merging the branch into main (as §3 does) carries both sides — but the prebuilt candidate image does NOT contain them (see the §3.1 fallback note and the extended marker list).
• Version target: the branch pins 1.108.0; latest stable upstream is 1.108.3 (2026-11-07). The prep validates 1.108.0 only — ship 1.108.0 tonight unless someone re-runs the anchor/migration validation against 1.108.3 first.
• Backups are NOT healthy. Last successful Cloudron backup: 2026-11-08 21:18 UTC (box, encrypted v2, state normal). The scheduled 2026-11-09 04:00 UTC run FAILED (tarPack pipeline error … aborted — the same signature as the Infomaniak S3 quota rejections of 2026-11-06/07/08). The pre-flight backups in §2.1 are this window's undo and must be proven to land before anything is deployed.

1. Order: LiteLLM first, then LibreChat

LibreChat's post-upgrade checks (key sync, a real chat completion) run through the gateway — doing the gateway first means those checks also exercise the new gateway, and each upgrade stays an isolated abort point: if LiteLLM fails, roll it back and skip LibreChat (chat keeps serving on the old overlay); if LibreChat fails, the new gateway is already proven.

2. Pre-flight (both apps, ~5 min)

set -a; . ~/.hermes/.env; set +a
CR() { cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" "$@"; }

# 2.1 Fresh update-rollback backups (DB + volume snapshot each)
CR backup create --app adababd4-2427-4bdd-9bf7-6bdad0302610
CR backup create --app 4023ccbf-d57a-4500-97df-14de9cf3516e
CR backup list --app adababd4-2427-4bdd-9bf7-6bdad0302610 | tail -3   # note newest id
CR backup list --app 4023ccbf-d57a-4500-97df-14de9cf3516e | tail -3

# 2.1b PROVE the backup landed BEFORE deploying. The backup pipeline is failing on the
#      Infomaniak S3 plan quota (last success 2026-11-08 21:18 UTC; the 04:00 UTC run on
#      2026-11-09 failed again). Re-run `CR backup list --app <id>` and require a NEW id
#      with state normal — an unchanged newest id, or no new entry, means this window has
#      NO undo. Decide before deploying, not after.

# 2.2 Record the member-visible baseline for after: number of keys and teams
#     (use the MASTER key — it is in /app/data/.master_key inside the gateway:
#      CR exec --app gateway.inference.coop -- cat /app/data/.master_key
#      then GET /key/list and /team/list with it; count both. NEVER paste the key.)

3. LiteLLM 1.103.2 → 1.108.0 (~20–35 min)

cd ~/repos/litellm                       # ONE APP = ONE REPO DIR. Never run an update
git fetch origin && git checkout main && git pull   # for another app from this cwd
git merge upgrade-v1.108.0               # branch is pushed; approve the merge in Gitea
                                         # if you prefer the PR path
git log --oneline -1                     # note the sha

3.1 Deploy (on-server build — the repo manifest has no dockerImage pin, so cloudron update builds and swaps; do NOT pass --image):

CR update --app adababd4-2427-4bdd-9bf7-6bdad0302610

• The build output must contain all TEN patch markers (each is one line in the build log; a missing line = that patch silently did not apply — STOP and check):

  1. INFO: patched budget-exceeded messages in …/auth_checks.py
  2. INFO: patched image-cost registration in …/router.py
  3. [coop-patch] applied to …/cost_calculator.py
  4. [coop-patch] applied audio shared pricing to …/router.py
  5. [coop-patch] applied: provider response headers are no longer forwarded …
  6. expose-cost-headers: appended 20 cost header names to expose_headers …
  7. [coop-patch] applied: passthrough-key-access -> …/route_checks.py
  8. [coop-patch] applied: passthrough-token-cost -> …/success_handler.py
  9. [coop-patch] applied skip-verbose_json to … (patch_transcription_response_format.py)
  10. [coop-patch] applied TTS default mp3 to … (patch_audio_default_mp3.py)

• audio_keepalive.py must be copied and chmod +x'd in the build (start.sh calls it for the TTS warm path). Markers 9–10 and this file are the 2026-11-06/07 audio work; missing any of them means that work silently reverts to the pre-Nov-06 behaviour.

• Boot: LiteLLM runs prisma migrate deploy (the 36 additive migrations) before binding port 4000. Expect 502/ECONNREFUSED on health checks for a few minutes (10–20 in the bad case). Never restart mid-wait — a restart resets the migration clock. Once /health/liveliness returns 200, continue immediately ("up" ≠ "migrated"; the checks below are the proof).

• DO NOT use the prebuilt candidate as a flake fallback tonight. It exists (…ai.coop.litellm:1.0.0-20261106-9e25578, and --image would work because the repo manifest carries no dockerImage field), but it was built from the branch tree and therefore predates patch_transcription_response_format.py, patch_audio_default_mp3.py, audio_keepalive.py and the STT/embedding model additions. Deploying it reverts the 2026-11-06/07 audio and model work. If the on-server build flakes, fix the build or build the merged tree — the candidate image is not usable tonight.

3.2 Verify (in order — a failing early step means stop, not continue):

# 1. version inside the running container
CR exec --app gateway.inference.coop -- \
  /app/.venv/bin/python3 -c "from importlib.metadata import version; print(version('litellm'))"
#    → 1.108.0

# 2. patch markers inside the RUNNING container (not the build log alone)
CR exec --app gateway.inference.coop -- sh -c \
  'grep -c "CO-OP PATCH (2026-10-19)" /app/.venv/lib/python3.13/site-packages/litellm/router.py; \
   grep -c "x-litellm-response-cost-input" /app/.venv/lib/python3.13/site-packages/litellm/proxy/proxy_server.py'
#    → 1 and 1

# 2b. supply-chain IoC check (standing policy on every LiteLLM image change):
CR exec --app gateway.inference.coop -- sh -c \
  'ls /app/.venv/lib/python3.13/site-packages/litellm_init.pth 2>/dev/null; \
   ls /root/.config/sysmon 2>/dev/null; \
   ls /tmp/pglog 2>/dev/null; echo IOC-CLEAN'
#    → must print ONLY "IOC-CLEAN" (no such files). Any hit = stop, quarantine, rotate
#      the gateway env keys (master + providers live in that env).

# 3. MEMBER-SCOPE key path (master-key tests alone prove nothing — master bypasses
#    exactly the tables a broken migration empties):
#    POST /key/generate  {"alias":"upgrade-verify-temp"}   (master auth) → temp key
#    GET  /v1/models with the TEMP key                     → 11 models
#    POST /v1/chat/completions {"model":"greenpt/glm-5.3-flash",
#         "messages":[{"role":"user","content":"Reply with the single word: pong"}],
#         "max_tokens":16}                                  → content + usage
#    GET  /key/list (master)                               → same count as §2.2
#    POST /key/delete {"keys":["<temp>"]}                  → 200

# 4. CORS preflight (chat origin must be allowed):
#    OPTIONS /v1/models with Origin: https://chat.inference.coop → 200 + allow-origin

# 5. Cloudron state
curl -sS -H "Authorization: Bearer $CLOUDRON_TOKEN" \
  https://my.inference.coop/api/v1/apps/adababd4-2427-4bdd-9bf7-6bdad0302610 \
  | python3 -c "import sys,json; a=json.load(sys.stdin); a=a.get('app',a); print(a['health'], a['runState'], a['manifest'].get('dockerImage'))"
#    → healthy running registry.inference.coop/local/ai.coop.litellm:<new tag>

# 6. Watch 10–15 min of real member traffic before closing:
#    SpendLogs rows arriving with status 'success' (portal admin panel or
#    GET /spend/logs with master auth). Failure rows with call_type = a route
#    path and spend 0 are 401-refusals, not duplicates.

3.3 Rollback (one command; the additive migrations make old-code-on-new-schema safe):

CR update --app adababd4-2427-4bdd-9bf7-6bdad0302610 \
  --image registry.inference.coop/local/ai.coop.litellm:20261107-201000-407e751e2

If member keys 401 after any image change, the schema is the suspect: read the exact error text from a member key call — a missing-relation error names the failing path; do NOT reissue keys. (Full recipe: code/litellm README, "Verify after upgrading".)

4. LibreChat overlay 1.4.5 → 1.4.8 (~10–15 min)

cd ~/repos/librechat-custom              # again: one app = its own repo dir
git fetch origin && git checkout master && git pull
git merge bump-appstore-1.4.8            # overlay code is UNCHANGED apart from the
                                         # FROM pin — appstore 1.4.8 carries the same
                                         # LibreChat 0.8.8; only Node/uv/base bumped
set -a; . ~/.hermes/.env; set +a

# The candidate overlay is already built: pin + deploy via the repo's release tool
./release.sh pin 20261106-c2aec74        # writes dockerImage into CloudronManifest.json, commits, pushes
./release.sh deploy                      # cloudron update --force from THIS dir, then verify

4.1 Verify (release.sh verify runs the served-HTML checks; confirm the rest by hand):

./release.sh verify                      # title = Inference Co-op, x-coop-build meta present

# key sync task ran (manifest scheduler `librechat_key_sync`, every minute):
CR logs --app chat.inference.coop --lines 100 | grep -i "sync" | tail -5
#    → the sync script prints its own md5 prefix each run; it must be the new build's

# RAG service: the appstore package runs rag_api alongside; confirm no crash loop
curl -sS -o /dev/null -w "%{http_code}\n" https://chat.inference.coop/   # 200 (SSO shell)

# login via Cloudron SSO in a browser, then send one real chat completion through
# LibreChat (this exercises chat → NEW gateway → provider end-to-end)

4.2 Rollback:

cd ~/repos/librechat-custom
./release.sh pin 20261031-2a98892 && ./release.sh deploy

5. Window estimate

Phase	Time
Pre-flight + backups	5 min
LiteLLM build+swap (on-server build)	5–10 min
LiteLLM boot migrations + member-scope verification	10–20 min
LibreChat pin+deploy+verify (incl. an SSO login + one chat)	10–15 min
Post-watch on SpendLogs / close-out	10 min
Total	~45–60 min

6. Gotchas that have each bitten this stack once

1. One app = one cd, immediately before its update. cloudron update reads CloudronManifest.json from the CURRENT directory; updating app B from repo A's dir ships A's manifest to B — on 2026-11-05 that dropped the gateway's postgres addon and wiped the LiteLLM DB (~50 min member outage, restored from the pre-deploy snapshot). The pre-deploy backups in §2 are the undo; addons-block changes are DATA-BEARING.
2. A pinned manifest beats --image, silently. LibreChat's repo manifest pins dockerImage — always change the pin (release.sh pin), never pass --image to it. LiteLLM's repo manifest has NO pin — --image works there (used for its rollback and build-flake fallback above).
3. "App is updated." proves nothing. Verify the running image tag via the API AND a container-content marker (§3.2 steps 1–2, §4.1) before trusting any check.
4. Don't restart the gateway mid-migration-wait — it resets the boot clock.
5. Do not "fix" /team/update dropping budget_reset_at today — the portal's direct-DB write is the known, verified workaround; it still works on 1.108.0.
6. After both apps are up, app-inventory.md in this repo should be updated with the new pins in the same session (it mirrors live state).
7. The prebuilt candidate is not the tree you are deploying. The branch was cut five commits behind main; the merge supplies the rest, the prebuilt image does not. Verify the TEN build markers of §3.1 inside the running container, not the old eight.
8. A stale rollback reference is worse than none — the runbook pointed at a Nov-05 image while the box was running a Nov-07 build, so "roll back" would have silently dropped two days of member-visible fixes. Re-read the running tag (§0/§2.2) immediately before any rollback, and roll back to what was actually running.
9. The backup pipeline is itself an open incident. Assume no undo until a pre-flight backup shows state: normal with a new id.

7. If something looks wrong

• Member keys 401 with a missing-relation error → schema/migration issue; see §3.3 and the code/litellm README triage section. Restore path: the §2 backup, then re-pin the old image.
• Chat serves a stale shell after the overlay deploy → wait ~60 s (mid-swap) and re-check; release.sh verify already retries fetches.
• Anything else: halt at the failing step, note the exact command + output, roll back that app only, and leave the other app on its new version if it verified.

8. Executed — 2026-11-09, 04:41–05:05 UTC (what happened, and where it differed)

Order followed as planned: LiteLLM first, then LibreChat. Two deliberate changes:

1. Target 1.108.3, not 1.108.0 — latest stable, and it pins the same litellm-proxy-extras==0.4.110, so the migration set is exactly the one validated here (36 new, all additive). All ten build-patch anchors were re-validated against the real 1.108.3 sources first; patch_expose_cost_headers.py's 1.108.x anchor applied.
2. A pre-built, pre-verified image was deployed, not an on-server build — 20261109-c43d45c, built from upgrade-v1.108.3 (cut on top of main, so it carries the 2026-11-06/07 audio patches the original candidate predated) and exercised first as a throwaway app (gateway-dryrun.inference.coop, its own Postgres/Redis, no member data).

The one thing that did not go as written — the pre-update snapshot OOMs. A default cloudron update died at Backup - Snapshotting app gateway.inference.coop: task 1430 → Task 1430 ran out of memory or terminated, box log Memory peak: 401.8M, oom-kill. Cloudron's app-task service runs with a hard 400 MB cgroup limit (startTask … "memoryLimit":400 from the system log), and this gateway's snapshot now exceeds it. Production was never at risk — the app kept serving the old image; only installationState went to error. Workaround used: --no-backup, with the 04:41 pre-flight app backups as the undo — which is what §2.1 is for. Next time: take the app backups first, then cloudron update … --no-backup.

Also learned the hard way: --wait-timeout is a GLOBAL option, so cloudron --wait-timeout 45m update … works and cloudron update --wait-timeout … errors out.

What was verified live afterwards

• gateway …ai.coop.litellm:20261109-c43d45c — litellm 1.108.3, extras 0.4.110; all ten patch markers present in the running container; audio_keepalive.py executable; IoC check clean; migrations 185 → 221, none failed or rolled back; keys 210 / teams 111 unchanged; temp member key → /v1/models (15 models) → real completion → success SpendLogs row; CORS preflight 200. Member traffic within ten minutes: 73 success rows.
• chat …/librechat-custom:20261106-c2aec74 (appstore 1.4.8 / LibreChat 0.8.8) — release.sh verify all checks passed, key sync running every minute (108 member keys, revision matches the repo), Meili resync normal, served title still Inference Co‑op.

Migrations were rehearsed, not merely run

Before the window, the live gateway database was restored from the 2026-11-08 stopgap snapshot into a local PostgreSQL 18.6 (prod's exact version) and the image's own entrypoint (python -m prisma migrate deploy) was run against it: 185 → 221 applied in 12 s, row counts identical (210 keys / 111 teams / 138,422 spend rows), no table lost (90 → 110, the 20 additions all new-feature tables). That is what made --no-backup acceptable on the real box.

Not rehearsed

The LibreChat leg — a second chat app means ~2 GB of data plus its Meili/rag stack, for a change that is tooling-only (Node/uv; LibreChat upstream unchanged). Mitigation is the one-command rollback: ./release.sh pin 20261031-2a98892 && ./release.sh deploy.