ops-docs/maintenance/2026-11-08-litellm-1.108.0-librechat-1.4.8.md

# Upgrade window — 2026-11-08 (Sunday evening): LiteLLM 1.103.2 → 1.108.0, LibreChat overlay 1.4.5 → 1.4.8

Both upgrades were **prepared and validated in advance** (2026-11-06); nothing has
been deployed. This runbook is written to be followed by any operator during the
member-watched maintenance window. Read the gotchas in §6 before touching anything —
this app pair has a history of deploys that look successful while changing nothing.

**Candidates (already built and validated, sitting in the registry):**

| App | Candidate | Rollback (currently deployed) |
|---|---|---|
| LiteLLM gateway (`adababd4-2427-4bdd-9bf7-6bdad0302610`, gateway.inference.coop) | branch `upgrade-v1.108.0` in `code/litellm`; prebuilt image `registry.inference.coop/local/ai.coop.litellm:1.0.0-20261106-9e25578` (STALE — see §0 and §3.1) | image `registry.inference.coop/local/ai.coop.litellm:20261107-201000-407e751e2` — the tag running tonight, verified present in the registry. `1.0.0-20261105-e161db9` is two deploys older and is NOT the rollback target |
| LibreChat chat (`4023ccbf-d57a-4500-97df-14de9cf3516e`, chat.inference.coop) | branch `bump-appstore-1.4.8` in `co-op/librechat-custom`; prebuilt overlay `registry.inference.coop/co-op/librechat-custom:20261106-c2aec74` | overlay `registry.inference.coop/co-op/librechat-custom:20261031-2a98892` (verified present) |

**Prep evidence (what was already verified, so Sunday only re-confirms it):**

- LiteLLM 1.108.0 candidate: all 8 build patches verified inside the running candidate
  (file markers + build-log markers, zero WARNs); litellm reports 1.108.0.
- `patch_image_cost.py` (image-gen spend): a real image generation through the
  candidate produced a **nonzero success SpendLogs row** — twice (fake OpenAI-shaped
  endpoint and the real Regolo Qwen-Image API): `spend 0.0011403264` for 256×256 at
  `$1.74e-8/px`, `call_type aimage_generation`, `status success`.
- `patch_passthrough_token_cost.py` (Metis metering): a real `/v1/systemone` call
  logged `[coop-passthrough-cost] input_tokens=148 rate=1.2e-07 -> response_cost=1.776e-05`
  and the SpendLogs row matched exactly (148 × $0.12/1M), tokens recorded.
- Member path: temp key generated with master auth → `/v1/models` (11 models) → real
  chat completion (greenpt/glm-5.3-flash, "pong", usage 19/3 tokens, spend row written).
- Migrations: 36 new between 1.103.2 and 1.108.0, **all additive** (no unguarded
  DROP/DELETE/UPDATE). A fresh DB ran the full 220-migration chain cleanly; the live
  ledger (1.103.2) will apply only the 36 new ones.
- `budget_reset_at`: column unchanged on `LiteLLM_TeamTable`; a direct-DB write
  (the portal's exact pattern) plus API read-back verified on the candidate.
- Portal API compatibility: `/team/list` (bare list) and `/key/list`
  (`{keys, total_count, …}`) shapes are **unchanged**; a bare `/team/list` call still
  returns ALL teams (pagination only applies when `page`/`page_size` are passed —
  verified with 11 teams); `/team/update` still drops `budget_reset_at`, so the
  portal's direct-DB write remains required. SpendLogs gains only an additive
  `billing_agent_id` column — digest queries are unaffected.
- LibreChat 1.4.8 (appstore `ai.librechat.cloudronapp`): changelog 1.4.5 → 1.4.8 is
  tooling only — uv 0.13.6/0.13.7 and Node.js 22.23.4 (security fix, 1.4.7); the
  LibreChat upstream stays 0.8.8. Every overlay patch anchor (title tag, manifest
  link, theme-color metas, webmanifest colors, "MCP Servers" / "Every AI for
  Everyone." strings, `index.*.js`, `sw.js`) was verified against the real 1.4.8
  image — **no overlay porting needed**; the candidate overlay built with all
  markers (UI strings patched in 3 files, index/manifest branding applied).

## 0. State re-check — 2026-11-09 04:15 UTC (read live before the window)

Verified against the live box on the night of the window. Anything here that
contradicts the prep notes above is newer and wins.

- **Gateway:** image `registry.inference.coop/local/ai.coop.litellm:20261107-201000-407e751e2`
  (deployed 2026-11-07 20:10 UTC), in-container version **1.103.2**, 8 expected models
  + extras, healthy. **The rollback target is therefore THIS tag, not
  `1.0.0-20261105-e161db9`** (see §3.3).
- **Chat:** overlay `registry.inference.coop/co-op/librechat-custom:20261031-2a98892`,
  appstore 1.4.5, healthy. `master` is 2 commits ahead of `bump-appstore-1.4.8`, but both
  touch only `librechat.yaml` — a persistent-volume file that is NOT baked into the image,
  so pinning the Nov-06 candidate does not revert them.
- **Registry:** all four tags this plan needs are present —
  `local/ai.coop.litellm:{1.0.0-20261106-9e25578, 20261107-201000-407e751e2}` and
  `co-op/librechat-custom:{20261106-c2aec74, 20261031-2a98892}` (listed from the registry
  app's own storage).
- **`upgrade-v1.108.0` is 5 commits BEHIND `main`.** main gained
  `patch_transcription_response_format.py`, `patch_audio_default_mp3.py` and
  `audio_keepalive.py` (Dockerfile RUN/chmod lines) plus the STT/embedding model additions
  on 2026-11-06/07, all after the candidate branch was cut. Merging the branch into main
  (as §3 does) carries both sides — but the **prebuilt candidate image does NOT contain
  them** (see the §3.1 fallback note and the extended marker list).
- **Version target:** the branch pins **1.108.0**; latest stable upstream is **1.108.3**
  (2026-11-07). The prep validates 1.108.0 only — ship 1.108.0 tonight unless someone
  re-runs the anchor/migration validation against 1.108.3 first.
- **Backups are NOT healthy.** Last successful Cloudron backup: **2026-11-08 21:18 UTC**
  (box, encrypted v2, state normal). The scheduled **2026-11-09 04:00 UTC run FAILED**
  (`tarPack pipeline error … aborted` — the same signature as the Infomaniak S3 quota
  rejections of 2026-11-06/07/08). The pre-flight backups in §2.1 are this window's undo
  and must be proven to land before anything is deployed.

## 1. Order: LiteLLM first, then LibreChat

LibreChat's post-upgrade checks (key sync, a real chat completion) run **through the
gateway** — doing the gateway first means those checks also exercise the new
gateway, and each upgrade stays an isolated abort point: if LiteLLM fails, roll it
back and skip LibreChat (chat keeps serving on the old overlay); if LibreChat fails,
the new gateway is already proven.

## 2. Pre-flight (both apps, ~5 min)

```bash
set -a; . ~/.hermes/.env; set +a
CR() { cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" "$@"; }

# 2.1 Fresh update-rollback backups (DB + volume snapshot each)
CR backup create --app adababd4-2427-4bdd-9bf7-6bdad0302610
CR backup create --app 4023ccbf-d57a-4500-97df-14de9cf3516e
CR backup list --app adababd4-2427-4bdd-9bf7-6bdad0302610 | tail -3   # note newest id
CR backup list --app 4023ccbf-d57a-4500-97df-14de9cf3516e | tail -3

# 2.1b PROVE the backup landed BEFORE deploying. The backup pipeline is failing on the
#      Infomaniak S3 plan quota (last success 2026-11-08 21:18 UTC; the 04:00 UTC run on
#      2026-11-09 failed again). Re-run `CR backup list --app <id>` and require a NEW id
#      with state normal — an unchanged newest id, or no new entry, means this window has
#      NO undo. Decide before deploying, not after.

# 2.2 Record the member-visible baseline for after: number of keys and teams
#     (use the MASTER key — it is in /app/data/.master_key inside the gateway:
#      CR exec --app gateway.inference.coop -- cat /app/data/.master_key
#      then GET /key/list and /team/list with it; count both. NEVER paste the key.)
```

## 3. LiteLLM 1.103.2 → 1.108.0 (~20–35 min)

```bash
cd ~/repos/litellm                       # ONE APP = ONE REPO DIR. Never run an update
git fetch origin && git checkout main && git pull   # for another app from this cwd
git merge upgrade-v1.108.0               # branch is pushed; approve the merge in Gitea
                                         # if you prefer the PR path
git log --oneline -1                     # note the sha
```

3.1 **Deploy (on-server build — the repo manifest has no `dockerImage` pin, so
`cloudron update` builds and swaps; do NOT pass `--image`):**

```bash
CR update --app adababd4-2427-4bdd-9bf7-6bdad0302610
```

- The build output must contain **all TEN** patch markers (each is one line in the
  build log; a missing line = that patch silently did not apply — STOP and check):

  1. `INFO: patched budget-exceeded messages in …/auth_checks.py`
  2. `INFO: patched image-cost registration in …/router.py`
  3. `[coop-patch] applied to …/cost_calculator.py`
  4. `[coop-patch] applied audio shared pricing to …/router.py`
  5. `[coop-patch] applied: provider response headers are no longer forwarded …`
  6. `expose-cost-headers: appended 20 cost header names to expose_headers …`
  7. `[coop-patch] applied: passthrough-key-access -> …/route_checks.py`
  8. `[coop-patch] applied: passthrough-token-cost -> …/success_handler.py`
  9. `[coop-patch] applied skip-verbose_json to …` (patch_transcription_response_format.py)
  10. `[coop-patch] applied TTS default mp3 to …` (patch_audio_default_mp3.py)

- `audio_keepalive.py` must be copied and `chmod +x`'d in the build (start.sh calls it for
  the TTS warm path). Markers 9–10 and this file are the 2026-11-06/07 audio work; missing
  any of them means that work silently reverts to the pre-Nov-06 behaviour.

- **Boot**: LiteLLM runs `prisma migrate deploy` (the 36 additive migrations) before
  binding port 4000. Expect `502`/`ECONNREFUSED` on health checks for a few minutes
  (10–20 in the bad case). **Never restart mid-wait** — a restart resets the
  migration clock. Once `/health/liveliness` returns 200, continue immediately
  ("up" ≠ "migrated"; the checks below are the proof).
- **DO NOT use the prebuilt candidate as a flake fallback tonight.** It exists
  (`…ai.coop.litellm:1.0.0-20261106-9e25578`, and `--image` would work because the repo
  manifest carries no `dockerImage` field), but it was built from the branch tree and
  therefore predates `patch_transcription_response_format.py`, `patch_audio_default_mp3.py`,
  `audio_keepalive.py` and the STT/embedding model additions. Deploying it reverts the
  2026-11-06/07 audio and model work. If the on-server build flakes, fix the build or build
  the merged tree — the candidate image is not usable tonight.

3.2 **Verify (in order — a failing early step means stop, not continue):**

```bash
# 1. version inside the running container
CR exec --app gateway.inference.coop -- \
  /app/.venv/bin/python3 -c "from importlib.metadata import version; print(version('litellm'))"
#    → 1.108.0

# 2. patch markers inside the RUNNING container (not the build log alone)
CR exec --app gateway.inference.coop -- sh -c \
  'grep -c "CO-OP PATCH (2026-10-19)" /app/.venv/lib/python3.13/site-packages/litellm/router.py; \
   grep -c "x-litellm-response-cost-input" /app/.venv/lib/python3.13/site-packages/litellm/proxy/proxy_server.py'
#    → 1 and 1

# 2b. supply-chain IoC check (standing policy on every LiteLLM image change):
CR exec --app gateway.inference.coop -- sh -c \
  'ls /app/.venv/lib/python3.13/site-packages/litellm_init.pth 2>/dev/null; \
   ls /root/.config/sysmon 2>/dev/null; \
   ls /tmp/pglog 2>/dev/null; echo IOC-CLEAN'
#    → must print ONLY "IOC-CLEAN" (no such files). Any hit = stop, quarantine, rotate
#      the gateway env keys (master + providers live in that env).

# 3. MEMBER-SCOPE key path (master-key tests alone prove nothing — master bypasses
#    exactly the tables a broken migration empties):
#    POST /key/generate  {"alias":"upgrade-verify-temp"}   (master auth) → temp key
#    GET  /v1/models with the TEMP key                     → 11 models
#    POST /v1/chat/completions {"model":"greenpt/glm-5.3-flash",
#         "messages":[{"role":"user","content":"Reply with the single word: pong"}],
#         "max_tokens":16}                                  → content + usage
#    GET  /key/list (master)                               → same count as §2.2
#    POST /key/delete {"keys":["<temp>"]}                  → 200

# 4. CORS preflight (chat origin must be allowed):
#    OPTIONS /v1/models with Origin: https://chat.inference.coop → 200 + allow-origin

# 5. Cloudron state
curl -sS -H "Authorization: Bearer $CLOUDRON_TOKEN" \
  https://my.inference.coop/api/v1/apps/adababd4-2427-4bdd-9bf7-6bdad0302610 \
  | python3 -c "import sys,json; a=json.load(sys.stdin); a=a.get('app',a); print(a['health'], a['runState'], a['manifest'].get('dockerImage'))"
#    → healthy running registry.inference.coop/local/ai.coop.litellm:<new tag>

# 6. Watch 10–15 min of real member traffic before closing:
#    SpendLogs rows arriving with status 'success' (portal admin panel or
#    GET /spend/logs with master auth). Failure rows with call_type = a route
#    path and spend 0 are 401-refusals, not duplicates.
```

3.3 **Rollback (one command; the additive migrations make old-code-on-new-schema safe):**

```bash
CR update --app adababd4-2427-4bdd-9bf7-6bdad0302610 \
  --image registry.inference.coop/local/ai.coop.litellm:20261107-201000-407e751e2
```

If member keys 401 after any image change, the schema is the suspect: read the
exact error text from a member key call — a missing-relation error names the failing
path; do NOT reissue keys. (Full recipe: `code/litellm` README, "Verify after
upgrading".)

## 4. LibreChat overlay 1.4.5 → 1.4.8 (~10–15 min)

```bash
cd ~/repos/librechat-custom              # again: one app = its own repo dir
git fetch origin && git checkout master && git pull
git merge bump-appstore-1.4.8            # overlay code is UNCHANGED apart from the
                                         # FROM pin — appstore 1.4.8 carries the same
                                         # LibreChat 0.8.8; only Node/uv/base bumped
set -a; . ~/.hermes/.env; set +a

# The candidate overlay is already built: pin + deploy via the repo's release tool
./release.sh pin 20261106-c2aec74        # writes dockerImage into CloudronManifest.json, commits, pushes
./release.sh deploy                      # cloudron update --force from THIS dir, then verify
```

4.1 **Verify** (release.sh verify runs the served-HTML checks; confirm the rest by hand):

```bash
./release.sh verify                      # title = Inference Co-op, x-coop-build meta present

# key sync task ran (manifest scheduler `librechat_key_sync`, every minute):
CR logs --app chat.inference.coop --lines 100 | grep -i "sync" | tail -5
#    → the sync script prints its own md5 prefix each run; it must be the new build's

# RAG service: the appstore package runs rag_api alongside; confirm no crash loop
curl -sS -o /dev/null -w "%{http_code}\n" https://chat.inference.coop/   # 200 (SSO shell)

# login via Cloudron SSO in a browser, then send one real chat completion through
# LibreChat (this exercises chat → NEW gateway → provider end-to-end)
```

4.2 **Rollback:**

```bash
cd ~/repos/librechat-custom
./release.sh pin 20261031-2a98892 && ./release.sh deploy
```

## 5. Window estimate

| Phase | Time |
|---|---|
| Pre-flight + backups | 5 min |
| LiteLLM build+swap (on-server build) | 5–10 min |
| LiteLLM boot migrations + member-scope verification | 10–20 min |
| LibreChat pin+deploy+verify (incl. an SSO login + one chat) | 10–15 min |
| Post-watch on SpendLogs / close-out | 10 min |
| **Total** | **~45–60 min** |

## 6. Gotchas that have each bitten this stack once

1. **One app = one `cd`, immediately before its update.** `cloudron update` reads
   `CloudronManifest.json` from the CURRENT directory; updating app B from repo A's
   dir ships A's manifest to B — on 2026-11-05 that dropped the gateway's postgres
   addon and wiped the LiteLLM DB (~50 min member outage, restored from the
   pre-deploy snapshot). The pre-deploy backups in §2 are the undo; addons-block
   changes are DATA-BEARING.
2. **A pinned manifest beats `--image`, silently.** LibreChat's repo manifest pins
   `dockerImage` — always change the pin (release.sh pin), never pass `--image` to
   it. LiteLLM's repo manifest has NO pin — `--image` works there (used for its
   rollback and build-flake fallback above).
3. **"App is updated." proves nothing.** Verify the running image tag via the API
   AND a container-content marker (§3.2 steps 1–2, §4.1) before trusting any check.
4. **Don't restart the gateway mid-migration-wait** — it resets the boot clock.
5. **Do not "fix" `/team/update` dropping `budget_reset_at` today** — the portal's
   direct-DB write is the known, verified workaround; it still works on 1.108.0.
6. After both apps are up, `app-inventory.md` in this repo should be updated with
   the new pins in the same session (it mirrors live state).
7. **The prebuilt candidate is not the tree you are deploying.** The branch was cut five
   commits behind `main`; the merge supplies the rest, the prebuilt image does not. Verify
   the TEN build markers of §3.1 inside the running container, not the old eight.
8. **A stale rollback reference is worse than none** — the runbook pointed at a Nov-05
   image while the box was running a Nov-07 build, so "roll back" would have silently
   dropped two days of member-visible fixes. Re-read the running tag (§0/§2.2) immediately
   before any rollback, and roll back to what was actually running.
9. **The backup pipeline is itself an open incident.** Assume no undo until a pre-flight
   backup shows `state: normal` with a new id.

## 7. If something looks wrong

- Member keys 401 with a missing-relation error → schema/migration issue; see
  §3.3 and the `code/litellm` README triage section. Restore path: the §2 backup,
  then re-pin the old image.
- Chat serves a stale shell after the overlay deploy → wait ~60 s (mid-swap) and
  re-check; release.sh verify already retries fetches.
- Anything else: halt at the failing step, note the exact command + output, roll
  back that app only, and leave the other app on its new version if it verified.

## 8. Executed — 2026-11-09, 04:41–05:05 UTC (what happened, and where it differed)

Order followed as planned: LiteLLM first, then LibreChat. Two deliberate changes:

1. **Target 1.108.3, not 1.108.0** — latest stable, and it pins the *same*
   `litellm-proxy-extras==0.4.110`, so the migration set is exactly the one validated here
   (36 new, all additive). All ten build-patch anchors were re-validated against the real
   1.108.3 sources first; `patch_expose_cost_headers.py`'s 1.108.x anchor applied.
2. **A pre-built, pre-verified image was deployed, not an on-server build** —
   `20261109-c43d45c`, built from `upgrade-v1.108.3` (cut on top of `main`, so it carries the
   2026-11-06/07 audio patches the original candidate predated) and exercised first as a
   throwaway app (`gateway-dryrun.inference.coop`, its own Postgres/Redis, no member data).

**The one thing that did not go as written — the pre-update snapshot OOMs.**
A default `cloudron update` died at `Backup - Snapshotting app gateway.inference.coop`:
task 1430 → `Task 1430 ran out of memory or terminated`, box log `Memory peak: 401.8M`,
`oom-kill`. Cloudron's app-task service runs with a **hard 400 MB cgroup limit**
(`startTask … "memoryLimit":400` from the system log), and this gateway's snapshot now
exceeds it. Production was never at risk — the app kept serving the old image; only
`installationState` went to `error`.
**Workaround used: `--no-backup`**, with the 04:41 pre-flight app backups as the undo —
which is what §2.1 is for. Next time: take the app backups first, then
`cloudron update … --no-backup`.

**Also learned the hard way:** `--wait-timeout` is a GLOBAL option, so
`cloudron --wait-timeout 45m update …` works and `cloudron update --wait-timeout …` errors out.

### What was verified live afterwards

- **gateway** `…ai.coop.litellm:20261109-c43d45c` — litellm **1.108.3**, extras 0.4.110;
  **all ten** patch markers present in the *running* container; `audio_keepalive.py`
  executable; IoC check clean; migrations **185 → 221**, none failed or rolled back;
  keys 210 / teams 111 unchanged; temp member key → `/v1/models` (15 models) → real
  completion → **success SpendLogs row**; CORS preflight 200. Member traffic within ten
  minutes: 73 success rows.
- **chat** `…/librechat-custom:20261106-c2aec74` (appstore 1.4.8 / LibreChat 0.8.8) —
  `release.sh verify` all checks passed, key sync running every minute (108 member keys,
  revision matches the repo), Meili resync normal, served title still `Inference Co‑op`.

### Migrations were rehearsed, not merely run

Before the window, the live gateway database was restored from the 2026-11-08 stopgap
snapshot into a local PostgreSQL **18.6** (prod's exact version) and the image's own
entrypoint (`python -m prisma migrate deploy`) was run against it: **185 → 221 applied in
12 s**, row counts identical (210 keys / 111 teams / 138,422 spend rows), **no table lost**
(90 → 110, the 20 additions all new-feature tables). That is what made `--no-backup`
acceptable on the real box.

### Not rehearsed

The LibreChat leg — a second chat app means ~2 GB of data plus its Meili/rag stack, for a
change that is tooling-only (Node/uv; LibreChat upstream unchanged). Mitigation is the
one-command rollback: `./release.sh pin 20261031-2a98892 && ./release.sh deploy`.