Upgrade window — 2026-11-08 (Sunday evening): LiteLLM 1.103.2 → 1.108.0, LibreChat overlay 1.4.5 → 1.4.8

Both upgrades were prepared and validated in advance (2026-11-06); nothing has been deployed. This runbook is written to be followed by any operator during the member-watched maintenance window. Read the gotchas in §6 before touching anything — this app pair has a history of deploys that look successful while changing nothing.

Candidates (already built and validated, sitting in the registry):

App	Candidate	Rollback (currently deployed)
LiteLLM gateway (`adababd4-2427-4bdd-9bf7-6bdad0302610`, gateway.inference.coop)	branch `upgrade-v1.108.0` in `code/litellm`; prebuilt image `registry.inference.coop/local/ai.coop.litellm:1.0.0-20261106-9e25578`	image `registry.inference.coop/local/ai.coop.litellm:1.0.0-20261105-e161db9` (verified present in the registry)
LibreChat chat (`4023ccbf-d57a-4500-97df-14de9cf3516e`, chat.inference.coop)	branch `bump-appstore-1.4.8` in `co-op/librechat-custom`; prebuilt overlay `registry.inference.coop/co-op/librechat-custom:20261106-c2aec74`	overlay `registry.inference.coop/co-op/librechat-custom:20261031-2a98892` (verified present)

Prep evidence (what was already verified, so Sunday only re-confirms it):

LiteLLM 1.108.0 candidate: all 8 build patches verified inside the running candidate (file markers + build-log markers, zero WARNs); litellm reports 1.108.0.
patch_image_cost.py (image-gen spend): a real image generation through the candidate produced a nonzero success SpendLogs row — twice (fake OpenAI-shaped endpoint and the real Regolo Qwen-Image API): spend 0.0011403264 for 256×256 at $1.74e-8/px, call_type aimage_generation, status success.
patch_passthrough_token_cost.py (Metis metering): a real /v1/systemone call logged [coop-passthrough-cost] input_tokens=148 rate=1.2e-07 -> response_cost=1.776e-05 and the SpendLogs row matched exactly (148 × $0.12/1M), tokens recorded.
Member path: temp key generated with master auth → /v1/models (11 models) → real chat completion (greenpt/glm-5.3-flash, "pong", usage 19/3 tokens, spend row written).
Migrations: 36 new between 1.103.2 and 1.108.0, all additive (no unguarded DROP/DELETE/UPDATE). A fresh DB ran the full 220-migration chain cleanly; the live ledger (1.103.2) will apply only the 36 new ones.
budget_reset_at: column unchanged on LiteLLM_TeamTable; a direct-DB write (the portal's exact pattern) plus API read-back verified on the candidate.
Portal API compatibility: /team/list (bare list) and /key/list ({keys, total_count, …}) shapes are unchanged; a bare /team/list call still returns ALL teams (pagination only applies when page/page_size are passed — verified with 11 teams); /team/update still drops budget_reset_at, so the portal's direct-DB write remains required. SpendLogs gains only an additive billing_agent_id column — digest queries are unaffected.
LibreChat 1.4.8 (appstore ai.librechat.cloudronapp): changelog 1.4.5 → 1.4.8 is tooling only — uv 0.13.6/0.13.7 and Node.js 22.23.4 (security fix, 1.4.7); the LibreChat upstream stays 0.8.8. Every overlay patch anchor (title tag, manifest link, theme-color metas, webmanifest colors, "MCP Servers" / "Every AI for Everyone." strings, index.*.js, sw.js) was verified against the real 1.4.8 image — no overlay porting needed; the candidate overlay built with all markers (UI strings patched in 3 files, index/manifest branding applied).

1. Order: LiteLLM first, then LibreChat

LibreChat's post-upgrade checks (key sync, a real chat completion) run through the gateway — doing the gateway first means those checks also exercise the new gateway, and each upgrade stays an isolated abort point: if LiteLLM fails, roll it back and skip LibreChat (chat keeps serving on the old overlay); if LibreChat fails, the new gateway is already proven.

2. Pre-flight (both apps, ~5 min)

set -a; . ~/.hermes/.env; set +a
CR() { cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" "$@"; }

# 2.1 Fresh update-rollback backups (DB + volume snapshot each)
CR backup create --app adababd4-2427-4bdd-9bf7-6bdad0302610
CR backup create --app 4023ccbf-d57a-4500-97df-14de9cf3516e
CR backup list --app adababd4-2427-4bdd-9bf7-6bdad0302610 | tail -3   # note newest id
CR backup list --app 4023ccbf-d57a-4500-97df-14de9cf3516e | tail -3

# 2.2 Record the member-visible baseline for after: number of keys and teams
#     (use the MASTER key — it is in /app/data/.master_key inside the gateway:
#      CR exec --app gateway.inference.coop -- cat /app/data/.master_key
#      then GET /key/list and /team/list with it; count both. NEVER paste the key.)

3. LiteLLM 1.103.2 → 1.108.0 (~20–35 min)

cd ~/repos/litellm                       # ONE APP = ONE REPO DIR. Never run an update
git fetch origin && git checkout main && git pull   # for another app from this cwd
git merge upgrade-v1.108.0               # branch is pushed; approve the merge in Gitea
                                         # if you prefer the PR path
git log --oneline -1                     # note the sha

3.1 Deploy (on-server build — the repo manifest has no dockerImage pin, so cloudron update builds and swaps; do NOT pass --image):

CR update --app adababd4-2427-4bdd-9bf7-6bdad0302610

The build output must contain all eight patch markers (each is one line in the build log; a missing line = that patch silently did not apply — STOP and check):
1. INFO: patched budget-exceeded messages in …/auth_checks.py
2. INFO: patched image-cost registration in …/router.py
3. [coop-patch] applied to …/cost_calculator.py
4. [coop-patch] applied audio shared pricing to …/router.py
5. [coop-patch] applied: provider response headers are no longer forwarded …
6. expose-cost-headers: appended 20 cost header names to expose_headers …
7. [coop-patch] applied: passthrough-key-access -> …/route_checks.py
8. [coop-patch] applied: passthrough-token-cost -> …/success_handler.py
Boot: LiteLLM runs prisma migrate deploy (the 36 additive migrations) before binding port 4000. Expect 502/ECONNREFUSED on health checks for a few minutes (10–20 in the bad case). Never restart mid-wait — a restart resets the migration clock. Once /health/liveliness returns 200, continue immediately ("up" ≠ "migrated"; the checks below are the proof).
If the on-server build flakes for any reason, fall back to the pre-validated image: CR update --app adababd4-2427-4bdd-9bf7-6bdad0302610 --image registry.inference.coop/local/ai.coop.litellm:1.0.0-20261106-9e25578 (works precisely because the repo manifest carries no dockerImage field).

3.2 Verify (in order — a failing early step means stop, not continue):

# 1. version inside the running container
CR exec --app gateway.inference.coop -- \
  /app/.venv/bin/python3 -c "from importlib.metadata import version; print(version('litellm'))"
#    → 1.108.0

# 2. patch markers inside the RUNNING container (not the build log alone)
CR exec --app gateway.inference.coop -- sh -c \
  'grep -c "CO-OP PATCH (2026-10-19)" /app/.venv/lib/python3.13/site-packages/litellm/router.py; \
   grep -c "x-litellm-response-cost-input" /app/.venv/lib/python3.13/site-packages/litellm/proxy/proxy_server.py'
#    → 1 and 1

# 2b. supply-chain IoC check (standing policy on every LiteLLM image change):
CR exec --app gateway.inference.coop -- sh -c \
  'ls /app/.venv/lib/python3.13/site-packages/litellm_init.pth 2>/dev/null; \
   ls /root/.config/sysmon 2>/dev/null; \
   ls /tmp/pglog 2>/dev/null; echo IOC-CLEAN'
#    → must print ONLY "IOC-CLEAN" (no such files). Any hit = stop, quarantine, rotate
#      the gateway env keys (master + providers live in that env).

# 3. MEMBER-SCOPE key path (master-key tests alone prove nothing — master bypasses
#    exactly the tables a broken migration empties):
#    POST /key/generate  {"alias":"upgrade-verify-temp"}   (master auth) → temp key
#    GET  /v1/models with the TEMP key                     → 11 models
#    POST /v1/chat/completions {"model":"greenpt/glm-5.3-flash",
#         "messages":[{"role":"user","content":"Reply with the single word: pong"}],
#         "max_tokens":16}                                  → content + usage
#    GET  /key/list (master)                               → same count as §2.2
#    POST /key/delete {"keys":["<temp>"]}                  → 200

# 4. CORS preflight (chat origin must be allowed):
#    OPTIONS /v1/models with Origin: https://chat.inference.coop → 200 + allow-origin

# 5. Cloudron state
curl -sS -H "Authorization: Bearer $CLOUDRON_TOKEN" \
  https://my.inference.coop/api/v1/apps/adababd4-2427-4bdd-9bf7-6bdad0302610 \
  | python3 -c "import sys,json; a=json.load(sys.stdin); a=a.get('app',a); print(a['health'], a['runState'], a['manifest'].get('dockerImage'))"
#    → healthy running registry.inference.coop/local/ai.coop.litellm:<new tag>

# 6. Watch 10–15 min of real member traffic before closing:
#    SpendLogs rows arriving with status 'success' (portal admin panel or
#    GET /spend/logs with master auth). Failure rows with call_type = a route
#    path and spend 0 are 401-refusals, not duplicates.

3.3 Rollback (one command; the additive migrations make old-code-on-new-schema safe):

CR update --app adababd4-2427-4bdd-9bf7-6bdad0302610 \
  --image registry.inference.coop/local/ai.coop.litellm:1.0.0-20261105-e161db9

If member keys 401 after any image change, the schema is the suspect: read the exact error text from a member key call — a missing-relation error names the failing path; do NOT reissue keys. (Full recipe: code/litellm README, "Verify after upgrading".)

4. LibreChat overlay 1.4.5 → 1.4.8 (~10–15 min)

cd ~/repos/librechat-custom              # again: one app = its own repo dir
git fetch origin && git checkout master && git pull
git merge bump-appstore-1.4.8            # overlay code is UNCHANGED apart from the
                                         # FROM pin — appstore 1.4.8 carries the same
                                         # LibreChat 0.8.8; only Node/uv/base bumped
set -a; . ~/.hermes/.env; set +a

# The candidate overlay is already built: pin + deploy via the repo's release tool
./release.sh pin 20261106-c2aec74        # writes dockerImage into CloudronManifest.json, commits, pushes
./release.sh deploy                      # cloudron update --force from THIS dir, then verify

4.1 Verify (release.sh verify runs the served-HTML checks; confirm the rest by hand):

./release.sh verify                      # title = Inference Co-op, x-coop-build meta present

# key sync task ran (manifest scheduler `librechat_key_sync`, every minute):
CR logs --app chat.inference.coop --lines 100 | grep -i "sync" | tail -5
#    → the sync script prints its own md5 prefix each run; it must be the new build's

# RAG service: the appstore package runs rag_api alongside; confirm no crash loop
curl -sS -o /dev/null -w "%{http_code}\n" https://chat.inference.coop/   # 200 (SSO shell)

# login via Cloudron SSO in a browser, then send one real chat completion through
# LibreChat (this exercises chat → NEW gateway → provider end-to-end)

4.2 Rollback:

cd ~/repos/librechat-custom
./release.sh pin 20261031-2a98892 && ./release.sh deploy

5. Window estimate

Phase	Time
Pre-flight + backups	5 min
LiteLLM build+swap (on-server build)	5–10 min
LiteLLM boot migrations + member-scope verification	10–20 min
LibreChat pin+deploy+verify (incl. an SSO login + one chat)	10–15 min
Post-watch on SpendLogs / close-out	10 min
Total	~45–60 min

6. Gotchas that have each bitten this stack once

One app = one cd, immediately before its update. cloudron update reads CloudronManifest.json from the CURRENT directory; updating app B from repo A's dir ships A's manifest to B — on 2026-11-05 that dropped the gateway's postgres addon and wiped the LiteLLM DB (~50 min member outage, restored from the pre-deploy snapshot). The pre-deploy backups in §2 are the undo; addons-block changes are DATA-BEARING.
A pinned manifest beats --image, silently. LibreChat's repo manifest pins dockerImage — always change the pin (release.sh pin), never pass --image to it. LiteLLM's repo manifest has NO pin — --image works there (used for its rollback and build-flake fallback above).
"App is updated." proves nothing. Verify the running image tag via the API AND a container-content marker (§3.2 steps 1–2, §4.1) before trusting any check.
Don't restart the gateway mid-migration-wait — it resets the boot clock.
Do not "fix" /team/update dropping budget_reset_at today — the portal's direct-DB write is the known, verified workaround; it still works on 1.108.0.
After both apps are up, app-inventory.md in this repo should be updated with the new pins in the same session (it mirrors live state).

7. If something looks wrong

Member keys 401 with a missing-relation error → schema/migration issue; see §3.3 and the code/litellm README triage section. Restore path: the §2 backup, then re-pin the old image.
Chat serves a stale shell after the overlay deploy → wait ~60 s (mid-swap) and re-check; release.sh verify already retries fetches.
Anything else: halt at the failing step, note the exact command + output, roll back that app only, and leave the other app on its new version if it verified.

14 KiB Raw Blame History Unescape Escape