248 lines
14 KiB
Markdown
248 lines
14 KiB
Markdown
# Upgrade window — 2026-11-08 (Sunday evening): LiteLLM 1.103.2 → 1.108.0, LibreChat overlay 1.4.5 → 1.4.8
|
||
|
||
Both upgrades were **prepared and validated in advance** (2026-11-06); nothing has
|
||
been deployed. This runbook is written to be followed by any operator during the
|
||
member-watched maintenance window. Read the gotchas in §6 before touching anything —
|
||
this app pair has a history of deploys that look successful while changing nothing.
|
||
|
||
**Candidates (already built and validated, sitting in the registry):**
|
||
|
||
| App | Candidate | Rollback (currently deployed) |
|
||
|---|---|---|
|
||
| LiteLLM gateway (`adababd4-2427-4bdd-9bf7-6bdad0302610`, gateway.inference.coop) | branch `upgrade-v1.108.0` in `code/litellm`; prebuilt image `registry.inference.coop/local/ai.coop.litellm:1.0.0-20261106-9e25578` | image `registry.inference.coop/local/ai.coop.litellm:1.0.0-20261105-e161db9` (verified present in the registry) |
|
||
| LibreChat chat (`4023ccbf-d57a-4500-97df-14de9cf3516e`, chat.inference.coop) | branch `bump-appstore-1.4.8` in `co-op/librechat-custom`; prebuilt overlay `registry.inference.coop/co-op/librechat-custom:20261106-c2aec74` | overlay `registry.inference.coop/co-op/librechat-custom:20261031-2a98892` (verified present) |
|
||
|
||
**Prep evidence (what was already verified, so Sunday only re-confirms it):**
|
||
|
||
- LiteLLM 1.108.0 candidate: all 8 build patches verified inside the running candidate
|
||
(file markers + build-log markers, zero WARNs); litellm reports 1.108.0.
|
||
- `patch_image_cost.py` (image-gen spend): a real image generation through the
|
||
candidate produced a **nonzero success SpendLogs row** — twice (fake OpenAI-shaped
|
||
endpoint and the real Regolo Qwen-Image API): `spend 0.0011403264` for 256×256 at
|
||
`$1.74e-8/px`, `call_type aimage_generation`, `status success`.
|
||
- `patch_passthrough_token_cost.py` (Metis metering): a real `/v1/systemone` call
|
||
logged `[coop-passthrough-cost] input_tokens=148 rate=1.2e-07 -> response_cost=1.776e-05`
|
||
and the SpendLogs row matched exactly (148 × $0.12/1M), tokens recorded.
|
||
- Member path: temp key generated with master auth → `/v1/models` (11 models) → real
|
||
chat completion (greenpt/glm-5.3-flash, "pong", usage 19/3 tokens, spend row written).
|
||
- Migrations: 36 new between 1.103.2 and 1.108.0, **all additive** (no unguarded
|
||
DROP/DELETE/UPDATE). A fresh DB ran the full 220-migration chain cleanly; the live
|
||
ledger (1.103.2) will apply only the 36 new ones.
|
||
- `budget_reset_at`: column unchanged on `LiteLLM_TeamTable`; a direct-DB write
|
||
(the portal's exact pattern) plus API read-back verified on the candidate.
|
||
- Portal API compatibility: `/team/list` (bare list) and `/key/list`
|
||
(`{keys, total_count, …}`) shapes are **unchanged**; a bare `/team/list` call still
|
||
returns ALL teams (pagination only applies when `page`/`page_size` are passed —
|
||
verified with 11 teams); `/team/update` still drops `budget_reset_at`, so the
|
||
portal's direct-DB write remains required. SpendLogs gains only an additive
|
||
`billing_agent_id` column — digest queries are unaffected.
|
||
- LibreChat 1.4.8 (appstore `ai.librechat.cloudronapp`): changelog 1.4.5 → 1.4.8 is
|
||
tooling only — uv 0.13.6/0.13.7 and Node.js 22.23.4 (security fix, 1.4.7); the
|
||
LibreChat upstream stays 0.8.8. Every overlay patch anchor (title tag, manifest
|
||
link, theme-color metas, webmanifest colors, "MCP Servers" / "Every AI for
|
||
Everyone." strings, `index.*.js`, `sw.js`) was verified against the real 1.4.8
|
||
image — **no overlay porting needed**; the candidate overlay built with all
|
||
markers (UI strings patched in 3 files, index/manifest branding applied).
|
||
|
||
## 1. Order: LiteLLM first, then LibreChat
|
||
|
||
LibreChat's post-upgrade checks (key sync, a real chat completion) run **through the
|
||
gateway** — doing the gateway first means those checks also exercise the new
|
||
gateway, and each upgrade stays an isolated abort point: if LiteLLM fails, roll it
|
||
back and skip LibreChat (chat keeps serving on the old overlay); if LibreChat fails,
|
||
the new gateway is already proven.
|
||
|
||
## 2. Pre-flight (both apps, ~5 min)
|
||
|
||
```bash
|
||
set -a; . ~/.hermes/.env; set +a
|
||
CR() { cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" "$@"; }
|
||
|
||
# 2.1 Fresh update-rollback backups (DB + volume snapshot each)
|
||
CR backup create --app adababd4-2427-4bdd-9bf7-6bdad0302610
|
||
CR backup create --app 4023ccbf-d57a-4500-97df-14de9cf3516e
|
||
CR backup list --app adababd4-2427-4bdd-9bf7-6bdad0302610 | tail -3 # note newest id
|
||
CR backup list --app 4023ccbf-d57a-4500-97df-14de9cf3516e | tail -3
|
||
|
||
# 2.2 Record the member-visible baseline for after: number of keys and teams
|
||
# (use the MASTER key — it is in /app/data/.master_key inside the gateway:
|
||
# CR exec --app gateway.inference.coop -- cat /app/data/.master_key
|
||
# then GET /key/list and /team/list with it; count both. NEVER paste the key.)
|
||
```
|
||
|
||
## 3. LiteLLM 1.103.2 → 1.108.0 (~20–35 min)
|
||
|
||
```bash
|
||
cd ~/repos/litellm # ONE APP = ONE REPO DIR. Never run an update
|
||
git fetch origin && git checkout main && git pull # for another app from this cwd
|
||
git merge upgrade-v1.108.0 # branch is pushed; approve the merge in Gitea
|
||
# if you prefer the PR path
|
||
git log --oneline -1 # note the sha
|
||
```
|
||
|
||
3.1 **Deploy (on-server build — the repo manifest has no `dockerImage` pin, so
|
||
`cloudron update` builds and swaps; do NOT pass `--image`):**
|
||
|
||
```bash
|
||
CR update --app adababd4-2427-4bdd-9bf7-6bdad0302610
|
||
```
|
||
|
||
- The build output must contain **all eight** patch markers (each is one line in the
|
||
build log; a missing line = that patch silently did not apply — STOP and check):
|
||
|
||
1. `INFO: patched budget-exceeded messages in …/auth_checks.py`
|
||
2. `INFO: patched image-cost registration in …/router.py`
|
||
3. `[coop-patch] applied to …/cost_calculator.py`
|
||
4. `[coop-patch] applied audio shared pricing to …/router.py`
|
||
5. `[coop-patch] applied: provider response headers are no longer forwarded …`
|
||
6. `expose-cost-headers: appended 20 cost header names to expose_headers …`
|
||
7. `[coop-patch] applied: passthrough-key-access -> …/route_checks.py`
|
||
8. `[coop-patch] applied: passthrough-token-cost -> …/success_handler.py`
|
||
|
||
- **Boot**: LiteLLM runs `prisma migrate deploy` (the 36 additive migrations) before
|
||
binding port 4000. Expect `502`/`ECONNREFUSED` on health checks for a few minutes
|
||
(10–20 in the bad case). **Never restart mid-wait** — a restart resets the
|
||
migration clock. Once `/health/liveliness` returns 200, continue immediately
|
||
("up" ≠ "migrated"; the checks below are the proof).
|
||
- If the on-server build flakes for any reason, fall back to the pre-validated
|
||
image: `CR update --app adababd4-2427-4bdd-9bf7-6bdad0302610 --image registry.inference.coop/local/ai.coop.litellm:1.0.0-20261106-9e25578`
|
||
(works precisely because the repo manifest carries no `dockerImage` field).
|
||
|
||
3.2 **Verify (in order — a failing early step means stop, not continue):**
|
||
|
||
```bash
|
||
# 1. version inside the running container
|
||
CR exec --app gateway.inference.coop -- \
|
||
/app/.venv/bin/python3 -c "from importlib.metadata import version; print(version('litellm'))"
|
||
# → 1.108.0
|
||
|
||
# 2. patch markers inside the RUNNING container (not the build log alone)
|
||
CR exec --app gateway.inference.coop -- sh -c \
|
||
'grep -c "CO-OP PATCH (2026-10-19)" /app/.venv/lib/python3.13/site-packages/litellm/router.py; \
|
||
grep -c "x-litellm-response-cost-input" /app/.venv/lib/python3.13/site-packages/litellm/proxy/proxy_server.py'
|
||
# → 1 and 1
|
||
|
||
# 2b. supply-chain IoC check (standing policy on every LiteLLM image change):
|
||
CR exec --app gateway.inference.coop -- sh -c \
|
||
'ls /app/.venv/lib/python3.13/site-packages/litellm_init.pth 2>/dev/null; \
|
||
ls /root/.config/sysmon 2>/dev/null; \
|
||
ls /tmp/pglog 2>/dev/null; echo IOC-CLEAN'
|
||
# → must print ONLY "IOC-CLEAN" (no such files). Any hit = stop, quarantine, rotate
|
||
# the gateway env keys (master + providers live in that env).
|
||
|
||
# 3. MEMBER-SCOPE key path (master-key tests alone prove nothing — master bypasses
|
||
# exactly the tables a broken migration empties):
|
||
# POST /key/generate {"alias":"upgrade-verify-temp"} (master auth) → temp key
|
||
# GET /v1/models with the TEMP key → 11 models
|
||
# POST /v1/chat/completions {"model":"greenpt/glm-5.3-flash",
|
||
# "messages":[{"role":"user","content":"Reply with the single word: pong"}],
|
||
# "max_tokens":16} → content + usage
|
||
# GET /key/list (master) → same count as §2.2
|
||
# POST /key/delete {"keys":["<temp>"]} → 200
|
||
|
||
# 4. CORS preflight (chat origin must be allowed):
|
||
# OPTIONS /v1/models with Origin: https://chat.inference.coop → 200 + allow-origin
|
||
|
||
# 5. Cloudron state
|
||
curl -sS -H "Authorization: Bearer $CLOUDRON_TOKEN" \
|
||
https://my.inference.coop/api/v1/apps/adababd4-2427-4bdd-9bf7-6bdad0302610 \
|
||
| python3 -c "import sys,json; a=json.load(sys.stdin); a=a.get('app',a); print(a['health'], a['runState'], a['manifest'].get('dockerImage'))"
|
||
# → healthy running registry.inference.coop/local/ai.coop.litellm:<new tag>
|
||
|
||
# 6. Watch 10–15 min of real member traffic before closing:
|
||
# SpendLogs rows arriving with status 'success' (portal admin panel or
|
||
# GET /spend/logs with master auth). Failure rows with call_type = a route
|
||
# path and spend 0 are 401-refusals, not duplicates.
|
||
```
|
||
|
||
3.3 **Rollback (one command; the additive migrations make old-code-on-new-schema safe):**
|
||
|
||
```bash
|
||
CR update --app adababd4-2427-4bdd-9bf7-6bdad0302610 \
|
||
--image registry.inference.coop/local/ai.coop.litellm:1.0.0-20261105-e161db9
|
||
```
|
||
|
||
If member keys 401 after any image change, the schema is the suspect: read the
|
||
exact error text from a member key call — a missing-relation error names the failing
|
||
path; do NOT reissue keys. (Full recipe: `code/litellm` README, "Verify after
|
||
upgrading".)
|
||
|
||
## 4. LibreChat overlay 1.4.5 → 1.4.8 (~10–15 min)
|
||
|
||
```bash
|
||
cd ~/repos/librechat-custom # again: one app = its own repo dir
|
||
git fetch origin && git checkout master && git pull
|
||
git merge bump-appstore-1.4.8 # overlay code is UNCHANGED apart from the
|
||
# FROM pin — appstore 1.4.8 carries the same
|
||
# LibreChat 0.8.8; only Node/uv/base bumped
|
||
set -a; . ~/.hermes/.env; set +a
|
||
|
||
# The candidate overlay is already built: pin + deploy via the repo's release tool
|
||
./release.sh pin 20261106-c2aec74 # writes dockerImage into CloudronManifest.json, commits, pushes
|
||
./release.sh deploy # cloudron update --force from THIS dir, then verify
|
||
```
|
||
|
||
4.1 **Verify** (release.sh verify runs the served-HTML checks; confirm the rest by hand):
|
||
|
||
```bash
|
||
./release.sh verify # title = Inference Co-op, x-coop-build meta present
|
||
|
||
# key sync task ran (manifest scheduler `librechat_key_sync`, every minute):
|
||
CR logs --app chat.inference.coop --lines 100 | grep -i "sync" | tail -5
|
||
# → the sync script prints its own md5 prefix each run; it must be the new build's
|
||
|
||
# RAG service: the appstore package runs rag_api alongside; confirm no crash loop
|
||
curl -sS -o /dev/null -w "%{http_code}\n" https://chat.inference.coop/ # 200 (SSO shell)
|
||
|
||
# login via Cloudron SSO in a browser, then send one real chat completion through
|
||
# LibreChat (this exercises chat → NEW gateway → provider end-to-end)
|
||
```
|
||
|
||
4.2 **Rollback:**
|
||
|
||
```bash
|
||
cd ~/repos/librechat-custom
|
||
./release.sh pin 20261031-2a98892 && ./release.sh deploy
|
||
```
|
||
|
||
## 5. Window estimate
|
||
|
||
| Phase | Time |
|
||
|---|---|
|
||
| Pre-flight + backups | 5 min |
|
||
| LiteLLM build+swap (on-server build) | 5–10 min |
|
||
| LiteLLM boot migrations + member-scope verification | 10–20 min |
|
||
| LibreChat pin+deploy+verify (incl. an SSO login + one chat) | 10–15 min |
|
||
| Post-watch on SpendLogs / close-out | 10 min |
|
||
| **Total** | **~45–60 min** |
|
||
|
||
## 6. Gotchas that have each bitten this stack once
|
||
|
||
1. **One app = one `cd`, immediately before its update.** `cloudron update` reads
|
||
`CloudronManifest.json` from the CURRENT directory; updating app B from repo A's
|
||
dir ships A's manifest to B — on 2026-11-05 that dropped the gateway's postgres
|
||
addon and wiped the LiteLLM DB (~50 min member outage, restored from the
|
||
pre-deploy snapshot). The pre-deploy backups in §2 are the undo; addons-block
|
||
changes are DATA-BEARING.
|
||
2. **A pinned manifest beats `--image`, silently.** LibreChat's repo manifest pins
|
||
`dockerImage` — always change the pin (release.sh pin), never pass `--image` to
|
||
it. LiteLLM's repo manifest has NO pin — `--image` works there (used for its
|
||
rollback and build-flake fallback above).
|
||
3. **"App is updated." proves nothing.** Verify the running image tag via the API
|
||
AND a container-content marker (§3.2 steps 1–2, §4.1) before trusting any check.
|
||
4. **Don't restart the gateway mid-migration-wait** — it resets the boot clock.
|
||
5. **Do not "fix" `/team/update` dropping `budget_reset_at` today** — the portal's
|
||
direct-DB write is the known, verified workaround; it still works on 1.108.0.
|
||
6. After both apps are up, `app-inventory.md` in this repo should be updated with
|
||
the new pins in the same session (it mirrors live state).
|
||
|
||
## 7. If something looks wrong
|
||
|
||
- Member keys 401 with a missing-relation error → schema/migration issue; see
|
||
§3.3 and the `code/litellm` README triage section. Restore path: the §2 backup,
|
||
then re-pin the old image.
|
||
- Chat serves a stale shell after the overlay deploy → wait ~60 s (mid-swap) and
|
||
re-check; release.sh verify already retries fetches.
|
||
- Anything else: halt at the failing step, note the exact command + output, roll
|
||
back that app only, and leave the other app on its new version if it verified.
|