# Upgrade window — 2026-11-08 (Sunday evening): LiteLLM 1.103.2 → 1.108.0, LibreChat overlay 1.4.5 → 1.4.8

Both upgrades were **prepared and validated in advance** (2026-11-06); nothing has
been deployed. This runbook is written to be followed by any operator during the
member-watched maintenance window. Read the gotchas in §6 before touching anything —
this app pair has a history of deploys that look successful while changing nothing.

**Candidates (already built and validated, sitting in the registry):**

| App | Candidate | Rollback (currently deployed) |
|---|---|---|
| LiteLLM gateway (`adababd4-2427-4bdd-9bf7-6bdad0302610`, gateway.inference.coop) | branch `upgrade-v1.108.0` in `code/litellm`; prebuilt image `registry.inference.coop/local/ai.coop.litellm:1.0.0-20261106-9e25578` | image `registry.inference.coop/local/ai.coop.litellm:1.0.0-20261105-e161db9` (verified present in the registry) |
| LibreChat chat (`4023ccbf-d57a-4500-97df-14de9cf3516e`, chat.inference.coop) | branch `bump-appstore-1.4.8` in `co-op/librechat-custom`; prebuilt overlay `registry.inference.coop/co-op/librechat-custom:20261106-c2aec74` | overlay `registry.inference.coop/co-op/librechat-custom:20261031-2a98892` (verified present) |

**Prep evidence (what was already verified, so Sunday only re-confirms it):**

- LiteLLM 1.108.0 candidate: all 8 build patches verified inside the running candidate
  (file markers + build-log markers, zero WARNs); litellm reports 1.108.0.
- `patch_image_cost.py` (image-gen spend): a real image generation through the
  candidate produced a **nonzero success SpendLogs row** — twice (fake OpenAI-shaped
  endpoint and the real Regolo Qwen-Image API): `spend 0.0011403264` for 256×256 at
  `$1.74e-8/px`, `call_type aimage_generation`, `status success`.
- `patch_passthrough_token_cost.py` (Metis metering): a real `/v1/systemone` call
  logged `[coop-passthrough-cost] input_tokens=148 rate=1.2e-07 -> response_cost=1.776e-05`
  and the SpendLogs row matched exactly (148 × $0.12/1M), tokens recorded.
- Member path: temp key generated with master auth → `/v1/models` (11 models) → real
  chat completion (greenpt/glm-5.3-flash, "pong", usage 19/3 tokens, spend row written).
- Migrations: 36 new between 1.103.2 and 1.108.0, **all additive** (no unguarded
  DROP/DELETE/UPDATE). A fresh DB ran the full 220-migration chain cleanly; the live
  ledger (1.103.2) will apply only the 36 new ones.
- `budget_reset_at`: column unchanged on `LiteLLM_TeamTable`; a direct-DB write
  (the portal's exact pattern) plus API read-back verified on the candidate.
- Portal API compatibility: `/team/list` (bare list) and `/key/list`
  (`{keys, total_count, …}`) shapes are **unchanged**; a bare `/team/list` call still
  returns ALL teams (pagination only applies when `page`/`page_size` are passed —
  verified with 11 teams); `/team/update` still drops `budget_reset_at`, so the
  portal's direct-DB write remains required. SpendLogs gains only an additive
  `billing_agent_id` column — digest queries are unaffected.
- LibreChat 1.4.8 (appstore `ai.librechat.cloudronapp`): changelog 1.4.5 → 1.4.8 is
  tooling only — uv 0.13.6/0.13.7 and Node.js 22.23.4 (security fix, 1.4.7); the
  LibreChat upstream stays 0.8.8. Every overlay patch anchor (title tag, manifest
  link, theme-color metas, webmanifest colors, "MCP Servers" / "Every AI for
  Everyone." strings, `index.*.js`, `sw.js`) was verified against the real 1.4.8
  image — **no overlay porting needed**; the candidate overlay built with all
  markers (UI strings patched in 3 files, index/manifest branding applied).

## 1. Order: LiteLLM first, then LibreChat

LibreChat's post-upgrade checks (key sync, a real chat completion) run **through the
gateway** — doing the gateway first means those checks also exercise the new
gateway, and each upgrade stays an isolated abort point: if LiteLLM fails, roll it
back and skip LibreChat (chat keeps serving on the old overlay); if LibreChat fails,
the new gateway is already proven.

## 2. Pre-flight (both apps, ~5 min)

```bash
set -a; . ~/.hermes/.env; set +a
CR() { cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" "$@"; }

# 2.1 Fresh update-rollback backups (DB + volume snapshot each)
CR backup create --app adababd4-2427-4bdd-9bf7-6bdad0302610
CR backup create --app 4023ccbf-d57a-4500-97df-14de9cf3516e
CR backup list --app adababd4-2427-4bdd-9bf7-6bdad0302610 | tail -3   # note newest id
CR backup list --app 4023ccbf-d57a-4500-97df-14de9cf3516e | tail -3

# 2.2 Record the member-visible baseline for after: number of keys and teams
#     (use the MASTER key — it is in /app/data/.master_key inside the gateway:
#      CR exec --app gateway.inference.coop -- cat /app/data/.master_key
#      then GET /key/list and /team/list with it; count both. NEVER paste the key.)
```

## 3. LiteLLM 1.103.2 → 1.108.0 (~20–35 min)

```bash
cd ~/repos/litellm                       # ONE APP = ONE REPO DIR. Never run an update
git fetch origin && git checkout main && git pull   # for another app from this cwd
git merge upgrade-v1.108.0               # branch is pushed; approve the merge in Gitea
                                         # if you prefer the PR path
git log --oneline -1                     # note the sha
```

3.1 **Deploy (on-server build — the repo manifest has no `dockerImage` pin, so
`cloudron update` builds and swaps; do NOT pass `--image`):**

```bash
CR update --app adababd4-2427-4bdd-9bf7-6bdad0302610
```

- The build output must contain **all eight** patch markers (each is one line in the
  build log; a missing line = that patch silently did not apply — STOP and check):

  1. `INFO: patched budget-exceeded messages in …/auth_checks.py`
  2. `INFO: patched image-cost registration in …/router.py`
  3. `[coop-patch] applied to …/cost_calculator.py`
  4. `[coop-patch] applied audio shared pricing to …/router.py`
  5. `[coop-patch] applied: provider response headers are no longer forwarded …`
  6. `expose-cost-headers: appended 20 cost header names to expose_headers …`
  7. `[coop-patch] applied: passthrough-key-access -> …/route_checks.py`
  8. `[coop-patch] applied: passthrough-token-cost -> …/success_handler.py`

- **Boot**: LiteLLM runs `prisma migrate deploy` (the 36 additive migrations) before
  binding port 4000. Expect `502`/`ECONNREFUSED` on health checks for a few minutes
  (10–20 in the bad case). **Never restart mid-wait** — a restart resets the
  migration clock. Once `/health/liveliness` returns 200, continue immediately
  ("up" ≠ "migrated"; the checks below are the proof).
- If the on-server build flakes for any reason, fall back to the pre-validated
  image: `CR update --app adababd4-2427-4bdd-9bf7-6bdad0302610 --image registry.inference.coop/local/ai.coop.litellm:1.0.0-20261106-9e25578`
  (works precisely because the repo manifest carries no `dockerImage` field).

3.2 **Verify (in order — a failing early step means stop, not continue):**

```bash
# 1. version inside the running container
CR exec --app gateway.inference.coop -- \
  /app/.venv/bin/python3 -c "from importlib.metadata import version; print(version('litellm'))"
#    → 1.108.0

# 2. patch markers inside the RUNNING container (not the build log alone)
CR exec --app gateway.inference.coop -- sh -c \
  'grep -c "CO-OP PATCH (2026-10-19)" /app/.venv/lib/python3.13/site-packages/litellm/router.py; \
   grep -c "x-litellm-response-cost-input" /app/.venv/lib/python3.13/site-packages/litellm/proxy/proxy_server.py'
#    → 1 and 1

# 3. MEMBER-SCOPE key path (master-key tests alone prove nothing — master bypasses
#    exactly the tables a broken migration empties):
#    POST /key/generate  {"alias":"upgrade-verify-temp"}   (master auth) → temp key
#    GET  /v1/models with the TEMP key                     → 11 models
#    POST /v1/chat/completions {"model":"greenpt/glm-5.3-flash",
#         "messages":[{"role":"user","content":"Reply with the single word: pong"}],
#         "max_tokens":16}                                  → content + usage
#    GET  /key/list (master)                               → same count as §2.2
#    POST /key/delete {"keys":["<temp>"]}                  → 200

# 4. CORS preflight (chat origin must be allowed):
#    OPTIONS /v1/models with Origin: https://chat.inference.coop → 200 + allow-origin

# 5. Cloudron state
curl -sS -H "Authorization: Bearer $CLOUDRON_TOKEN" \
  https://my.inference.coop/api/v1/apps/adababd4-2427-4bdd-9bf7-6bdad0302610 \
  | python3 -c "import sys,json; a=json.load(sys.stdin); a=a.get('app',a); print(a['health'], a['runState'], a['manifest'].get('dockerImage'))"
#    → healthy running registry.inference.coop/local/ai.coop.litellm:<new tag>

# 6. Watch 10–15 min of real member traffic before closing:
#    SpendLogs rows arriving with status 'success' (portal admin panel or
#    GET /spend/logs with master auth). Failure rows with call_type = a route
#    path and spend 0 are 401-refusals, not duplicates.
```

3.3 **Rollback (one command; the additive migrations make old-code-on-new-schema safe):**

```bash
CR update --app adababd4-2427-4bdd-9bf7-6bdad0302610 \
  --image registry.inference.coop/local/ai.coop.litellm:1.0.0-20261105-e161db9
```

If member keys 401 after any image change, the schema is the suspect: read the
exact error text from a member key call — a missing-relation error names the failing
path; do NOT reissue keys. (Full recipe: `code/litellm` README, "Verify after
upgrading".)

## 4. LibreChat overlay 1.4.5 → 1.4.8 (~10–15 min)

```bash
cd ~/repos/librechat-custom              # again: one app = its own repo dir
git fetch origin && git checkout master && git pull
git merge bump-appstore-1.4.8            # overlay code is UNCHANGED apart from the
                                         # FROM pin — appstore 1.4.8 carries the same
                                         # LibreChat 0.8.8; only Node/uv/base bumped
set -a; . ~/.hermes/.env; set +a

# The candidate overlay is already built: pin + deploy via the repo's release tool
./release.sh pin 20261106-c2aec74        # writes dockerImage into CloudronManifest.json, commits, pushes
./release.sh deploy                      # cloudron update --force from THIS dir, then verify
```

4.1 **Verify** (release.sh verify runs the served-HTML checks; confirm the rest by hand):

```bash
./release.sh verify                      # title = Inference Co-op, x-coop-build meta present

# key sync task ran (manifest scheduler `librechat_key_sync`, every minute):
CR logs --app chat.inference.coop --lines 100 | grep -i "sync" | tail -5
#    → the sync script prints its own md5 prefix each run; it must be the new build's

# RAG service: the appstore package runs rag_api alongside; confirm no crash loop
curl -sS -o /dev/null -w "%{http_code}\n" https://chat.inference.coop/   # 200 (SSO shell)

# login via Cloudron SSO in a browser, then send one real chat completion through
# LibreChat (this exercises chat → NEW gateway → provider end-to-end)
```

4.2 **Rollback:**

```bash
cd ~/repos/librechat-custom
./release.sh pin 20261031-2a98892 && ./release.sh deploy
```

## 5. Window estimate

| Phase | Time |
|---|---|
| Pre-flight + backups | 5 min |
| LiteLLM build+swap (on-server build) | 5–10 min |
| LiteLLM boot migrations + member-scope verification | 10–20 min |
| LibreChat pin+deploy+verify (incl. an SSO login + one chat) | 10–15 min |
| Post-watch on SpendLogs / close-out | 10 min |
| **Total** | **~45–60 min** |

## 6. Gotchas that have each bitten this stack once

1. **One app = one `cd`, immediately before its update.** `cloudron update` reads
   `CloudronManifest.json` from the CURRENT directory; updating app B from repo A's
   dir ships A's manifest to B — on 2026-11-05 that dropped the gateway's postgres
   addon and wiped the LiteLLM DB (~50 min member outage, restored from the
   pre-deploy snapshot). The pre-deploy backups in §2 are the undo; addons-block
   changes are DATA-BEARING.
2. **A pinned manifest beats `--image`, silently.** LibreChat's repo manifest pins
   `dockerImage` — always change the pin (release.sh pin), never pass `--image` to
   it. LiteLLM's repo manifest has NO pin — `--image` works there (used for its
   rollback and build-flake fallback above).
3. **"App is updated." proves nothing.** Verify the running image tag via the API
   AND a container-content marker (§3.2 steps 1–2, §4.1) before trusting any check.
4. **Don't restart the gateway mid-migration-wait** — it resets the boot clock.
5. **Do not "fix" `/team/update` dropping `budget_reset_at` today** — the portal's
   direct-DB write is the known, verified workaround; it still works on 1.108.0.
6. After both apps are up, `app-inventory.md` in this repo should be updated with
   the new pins in the same session (it mirrors live state).

## 7. If something looks wrong

- Member keys 401 with a missing-relation error → schema/migration issue; see
  §3.3 and the `code/litellm` README triage section. Restore path: the §2 backup,
  then re-pin the old image.
- Chat serves a stale shell after the overlay deploy → wait ~60 s (mid-swap) and
  re-check; release.sh verify already retries fetches.
- Anything else: halt at the failing step, note the exact command + output, roll
  back that app only, and leave the other app on its new version if it verified.