From 816961fb2dad5698efad1d734afbec5100e0b8bb Mon Sep 17 00:00:00 2001 From: Inferencebot Date: Fri, 6 Nov 2026 04:11:01 +0000 Subject: [PATCH] =?UTF-8?q?runbook:=202026-11-08=20upgrade=20window=20?= =?UTF-8?q?=E2=80=94=20LiteLLM=201.103.2=E2=86=921.108.0=20+=20LibreChat?= =?UTF-8?q?=20overlay=201.4.5=E2=86=921.4.8?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Prepared and validated on 2026-11-06; nothing deployed. Candidates: - code/litellm branch upgrade-v1.108.0 → image 1.0.0-20261106-9e25578 (8/8 patches verified in the built image and in a live sandbox run; image-gen spend row verified nonzero twice — fake + real Regolo probe; 36 additive migrations; budget_reset_at + portal API shapes unchanged). - co-op/librechat-custom branch bump-appstore-1.4.8 → overlay 20261106-c2aec74 (appstore 1.4.8 = tooling bumps on the same 0.8.8 upstream; all overlay anchors verified against the real 1.4.8 image — no porting needed). Runbook: LiteLLM first, member-scope verification, rollback = previous tag per app, window estimate 45-60 min. --- README.md | 1 + ...6-11-08-litellm-1.108.0-librechat-1.4.8.md | 239 ++++++++++++++++++ 2 files changed, 240 insertions(+) create mode 100644 maintenance/2026-11-08-litellm-1.108.0-librechat-1.4.8.md diff --git a/README.md b/README.md index fbcba65..644ed24 100644 --- a/README.md +++ b/README.md @@ -30,6 +30,7 @@ Read it on the web here: | `maintenance/librechat-sandbox.md` | Operating the LibreChat evaluation instance (config, overlay image, key sync, pitfalls) | | `maintenance/backups.md` | The two backup systems (agent-host restic + Cloudron app backup): schedules, retention, the cache split and the grouping fix | | `maintenance/woodpecker-ci.md` | The self-hosted CI server and agent: build/pin, manual-only trigger, git-lfs, and its not-yet-green state | +| `maintenance/2026-11-08-litellm-1.108.0-librechat-1.4.8.md` | The Sunday-evening upgrade-window runbook (gateway → 1.108.0, chat overlay → appstore 1.4.8): candidates, order, exact commands, per-step verification, rollback, window estimate | | `incidents/` | Post-incident write-ups, one file per incident | | `migration/` | The Cloudron server-migration record: the plan, the DNS change sheet, and the captured pre-migration state | | `provider-pricing/` | What each model provider charges, and when it changed (GreenPT, Tinfoil, Regolo, and the decision-model route) | diff --git a/maintenance/2026-11-08-litellm-1.108.0-librechat-1.4.8.md b/maintenance/2026-11-08-litellm-1.108.0-librechat-1.4.8.md new file mode 100644 index 0000000..5b93c5e --- /dev/null +++ b/maintenance/2026-11-08-litellm-1.108.0-librechat-1.4.8.md @@ -0,0 +1,239 @@ +# Upgrade window — 2026-11-08 (Sunday evening): LiteLLM 1.103.2 → 1.108.0, LibreChat overlay 1.4.5 → 1.4.8 + +Both upgrades were **prepared and validated in advance** (2026-11-06); nothing has +been deployed. This runbook is written to be followed by any operator during the +member-watched maintenance window. Read the gotchas in §6 before touching anything — +this app pair has a history of deploys that look successful while changing nothing. + +**Candidates (already built and validated, sitting in the registry):** + +| App | Candidate | Rollback (currently deployed) | +|---|---|---| +| LiteLLM gateway (`adababd4-2427-4bdd-9bf7-6bdad0302610`, gateway.inference.coop) | branch `upgrade-v1.108.0` in `code/litellm`; prebuilt image `registry.inference.coop/local/ai.coop.litellm:1.0.0-20261106-9e25578` | image `registry.inference.coop/local/ai.coop.litellm:1.0.0-20261105-e161db9` (verified present in the registry) | +| LibreChat chat (`4023ccbf-d57a-4500-97df-14de9cf3516e`, chat.inference.coop) | branch `bump-appstore-1.4.8` in `co-op/librechat-custom`; prebuilt overlay `registry.inference.coop/co-op/librechat-custom:20261106-c2aec74` | overlay `registry.inference.coop/co-op/librechat-custom:20261031-2a98892` (verified present) | + +**Prep evidence (what was already verified, so Sunday only re-confirms it):** + +- LiteLLM 1.108.0 candidate: all 8 build patches verified inside the running candidate + (file markers + build-log markers, zero WARNs); litellm reports 1.108.0. +- `patch_image_cost.py` (image-gen spend): a real image generation through the + candidate produced a **nonzero success SpendLogs row** — twice (fake OpenAI-shaped + endpoint and the real Regolo Qwen-Image API): `spend 0.0011403264` for 256×256 at + `$1.74e-8/px`, `call_type aimage_generation`, `status success`. +- `patch_passthrough_token_cost.py` (Metis metering): a real `/v1/systemone` call + logged `[coop-passthrough-cost] input_tokens=148 rate=1.2e-07 -> response_cost=1.776e-05` + and the SpendLogs row matched exactly (148 × $0.12/1M), tokens recorded. +- Member path: temp key generated with master auth → `/v1/models` (11 models) → real + chat completion (greenpt/glm-5.3-flash, "pong", usage 19/3 tokens, spend row written). +- Migrations: 36 new between 1.103.2 and 1.108.0, **all additive** (no unguarded + DROP/DELETE/UPDATE). A fresh DB ran the full 220-migration chain cleanly; the live + ledger (1.103.2) will apply only the 36 new ones. +- `budget_reset_at`: column unchanged on `LiteLLM_TeamTable`; a direct-DB write + (the portal's exact pattern) plus API read-back verified on the candidate. +- Portal API compatibility: `/team/list` (bare list) and `/key/list` + (`{keys, total_count, …}`) shapes are **unchanged**; a bare `/team/list` call still + returns ALL teams (pagination only applies when `page`/`page_size` are passed — + verified with 11 teams); `/team/update` still drops `budget_reset_at`, so the + portal's direct-DB write remains required. SpendLogs gains only an additive + `billing_agent_id` column — digest queries are unaffected. +- LibreChat 1.4.8 (appstore `ai.librechat.cloudronapp`): changelog 1.4.5 → 1.4.8 is + tooling only — uv 0.13.6/0.13.7 and Node.js 22.23.4 (security fix, 1.4.7); the + LibreChat upstream stays 0.8.8. Every overlay patch anchor (title tag, manifest + link, theme-color metas, webmanifest colors, "MCP Servers" / "Every AI for + Everyone." strings, `index.*.js`, `sw.js`) was verified against the real 1.4.8 + image — **no overlay porting needed**; the candidate overlay built with all + markers (UI strings patched in 3 files, index/manifest branding applied). + +## 1. Order: LiteLLM first, then LibreChat + +LibreChat's post-upgrade checks (key sync, a real chat completion) run **through the +gateway** — doing the gateway first means those checks also exercise the new +gateway, and each upgrade stays an isolated abort point: if LiteLLM fails, roll it +back and skip LibreChat (chat keeps serving on the old overlay); if LibreChat fails, +the new gateway is already proven. + +## 2. Pre-flight (both apps, ~5 min) + +```bash +set -a; . ~/.hermes/.env; set +a +CR() { cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" "$@"; } + +# 2.1 Fresh update-rollback backups (DB + volume snapshot each) +CR backup create --app adababd4-2427-4bdd-9bf7-6bdad0302610 +CR backup create --app 4023ccbf-d57a-4500-97df-14de9cf3516e +CR backup list --app adababd4-2427-4bdd-9bf7-6bdad0302610 | tail -3 # note newest id +CR backup list --app 4023ccbf-d57a-4500-97df-14de9cf3516e | tail -3 + +# 2.2 Record the member-visible baseline for after: number of keys and teams +# (use the MASTER key — it is in /app/data/.master_key inside the gateway: +# CR exec --app gateway.inference.coop -- cat /app/data/.master_key +# then GET /key/list and /team/list with it; count both. NEVER paste the key.) +``` + +## 3. LiteLLM 1.103.2 → 1.108.0 (~20–35 min) + +```bash +cd ~/repos/litellm # ONE APP = ONE REPO DIR. Never run an update +git fetch origin && git checkout main && git pull # for another app from this cwd +git merge upgrade-v1.108.0 # branch is pushed; approve the merge in Gitea + # if you prefer the PR path +git log --oneline -1 # note the sha +``` + +3.1 **Deploy (on-server build — the repo manifest has no `dockerImage` pin, so +`cloudron update` builds and swaps; do NOT pass `--image`):** + +```bash +CR update --app adababd4-2427-4bdd-9bf7-6bdad0302610 +``` + +- The build output must contain **all eight** patch markers (each is one line in the + build log; a missing line = that patch silently did not apply — STOP and check): + + 1. `INFO: patched budget-exceeded messages in …/auth_checks.py` + 2. `INFO: patched image-cost registration in …/router.py` + 3. `[coop-patch] applied to …/cost_calculator.py` + 4. `[coop-patch] applied audio shared pricing to …/router.py` + 5. `[coop-patch] applied: provider response headers are no longer forwarded …` + 6. `expose-cost-headers: appended 20 cost header names to expose_headers …` + 7. `[coop-patch] applied: passthrough-key-access -> …/route_checks.py` + 8. `[coop-patch] applied: passthrough-token-cost -> …/success_handler.py` + +- **Boot**: LiteLLM runs `prisma migrate deploy` (the 36 additive migrations) before + binding port 4000. Expect `502`/`ECONNREFUSED` on health checks for a few minutes + (10–20 in the bad case). **Never restart mid-wait** — a restart resets the + migration clock. Once `/health/liveliness` returns 200, continue immediately + ("up" ≠ "migrated"; the checks below are the proof). +- If the on-server build flakes for any reason, fall back to the pre-validated + image: `CR update --app adababd4-2427-4bdd-9bf7-6bdad0302610 --image registry.inference.coop/local/ai.coop.litellm:1.0.0-20261106-9e25578` + (works precisely because the repo manifest carries no `dockerImage` field). + +3.2 **Verify (in order — a failing early step means stop, not continue):** + +```bash +# 1. version inside the running container +CR exec --app gateway.inference.coop -- \ + /app/.venv/bin/python3 -c "from importlib.metadata import version; print(version('litellm'))" +# → 1.108.0 + +# 2. patch markers inside the RUNNING container (not the build log alone) +CR exec --app gateway.inference.coop -- sh -c \ + 'grep -c "CO-OP PATCH (2026-10-19)" /app/.venv/lib/python3.13/site-packages/litellm/router.py; \ + grep -c "x-litellm-response-cost-input" /app/.venv/lib/python3.13/site-packages/litellm/proxy/proxy_server.py' +# → 1 and 1 + +# 3. MEMBER-SCOPE key path (master-key tests alone prove nothing — master bypasses +# exactly the tables a broken migration empties): +# POST /key/generate {"alias":"upgrade-verify-temp"} (master auth) → temp key +# GET /v1/models with the TEMP key → 11 models +# POST /v1/chat/completions {"model":"greenpt/glm-5.3-flash", +# "messages":[{"role":"user","content":"Reply with the single word: pong"}], +# "max_tokens":16} → content + usage +# GET /key/list (master) → same count as §2.2 +# POST /key/delete {"keys":[""]} → 200 + +# 4. CORS preflight (chat origin must be allowed): +# OPTIONS /v1/models with Origin: https://chat.inference.coop → 200 + allow-origin + +# 5. Cloudron state +curl -sS -H "Authorization: Bearer $CLOUDRON_TOKEN" \ + https://my.inference.coop/api/v1/apps/adababd4-2427-4bdd-9bf7-6bdad0302610 \ + | python3 -c "import sys,json; a=json.load(sys.stdin); a=a.get('app',a); print(a['health'], a['runState'], a['manifest'].get('dockerImage'))" +# → healthy running registry.inference.coop/local/ai.coop.litellm: + +# 6. Watch 10–15 min of real member traffic before closing: +# SpendLogs rows arriving with status 'success' (portal admin panel or +# GET /spend/logs with master auth). Failure rows with call_type = a route +# path and spend 0 are 401-refusals, not duplicates. +``` + +3.3 **Rollback (one command; the additive migrations make old-code-on-new-schema safe):** + +```bash +CR update --app adababd4-2427-4bdd-9bf7-6bdad0302610 \ + --image registry.inference.coop/local/ai.coop.litellm:1.0.0-20261105-e161db9 +``` + +If member keys 401 after any image change, the schema is the suspect: read the +exact error text from a member key call — a missing-relation error names the failing +path; do NOT reissue keys. (Full recipe: `code/litellm` README, "Verify after +upgrading".) + +## 4. LibreChat overlay 1.4.5 → 1.4.8 (~10–15 min) + +```bash +cd ~/repos/librechat-custom # again: one app = its own repo dir +git fetch origin && git checkout master && git pull +git merge bump-appstore-1.4.8 # overlay code is UNCHANGED apart from the + # FROM pin — appstore 1.4.8 carries the same + # LibreChat 0.8.8; only Node/uv/base bumped +set -a; . ~/.hermes/.env; set +a + +# The candidate overlay is already built: pin + deploy via the repo's release tool +./release.sh pin 20261106-c2aec74 # writes dockerImage into CloudronManifest.json, commits, pushes +./release.sh deploy # cloudron update --force from THIS dir, then verify +``` + +4.1 **Verify** (release.sh verify runs the served-HTML checks; confirm the rest by hand): + +```bash +./release.sh verify # title = Inference Co-op, x-coop-build meta present + +# key sync task ran (manifest scheduler `librechat_key_sync`, every minute): +CR logs --app chat.inference.coop --lines 100 | grep -i "sync" | tail -5 +# → the sync script prints its own md5 prefix each run; it must be the new build's + +# RAG service: the appstore package runs rag_api alongside; confirm no crash loop +curl -sS -o /dev/null -w "%{http_code}\n" https://chat.inference.coop/ # 200 (SSO shell) + +# login via Cloudron SSO in a browser, then send one real chat completion through +# LibreChat (this exercises chat → NEW gateway → provider end-to-end) +``` + +4.2 **Rollback:** + +```bash +cd ~/repos/librechat-custom +./release.sh pin 20261031-2a98892 && ./release.sh deploy +``` + +## 5. Window estimate + +| Phase | Time | +|---|---| +| Pre-flight + backups | 5 min | +| LiteLLM build+swap (on-server build) | 5–10 min | +| LiteLLM boot migrations + member-scope verification | 10–20 min | +| LibreChat pin+deploy+verify (incl. an SSO login + one chat) | 10–15 min | +| Post-watch on SpendLogs / close-out | 10 min | +| **Total** | **~45–60 min** | + +## 6. Gotchas that have each bitten this stack once + +1. **One app = one `cd`, immediately before its update.** `cloudron update` reads + `CloudronManifest.json` from the CURRENT directory; updating app B from repo A's + dir ships A's manifest to B — on 2026-11-05 that dropped the gateway's postgres + addon and wiped the LiteLLM DB (~50 min member outage, restored from the + pre-deploy snapshot). The pre-deploy backups in §2 are the undo; addons-block + changes are DATA-BEARING. +2. **A pinned manifest beats `--image`, silently.** LibreChat's repo manifest pins + `dockerImage` — always change the pin (release.sh pin), never pass `--image` to + it. LiteLLM's repo manifest has NO pin — `--image` works there (used for its + rollback and build-flake fallback above). +3. **"App is updated." proves nothing.** Verify the running image tag via the API + AND a container-content marker (§3.2 steps 1–2, §4.1) before trusting any check. +4. **Don't restart the gateway mid-migration-wait** — it resets the boot clock. +5. **Do not "fix" `/team/update` dropping `budget_reset_at` today** — the portal's + direct-DB write is the known, verified workaround; it still works on 1.108.0. +6. After both apps are up, `app-inventory.md` in this repo should be updated with + the new pins in the same session (it mirrors live state). + +## 7. If something looks wrong + +- Member keys 401 with a missing-relation error → schema/migration issue; see + §3.3 and the `code/litellm` README triage section. Restore path: the §2 backup, + then re-pin the old image. +- Chat serves a stale shell after the overlay deploy → wait ~60 s (mid-swap) and + re-check; release.sh verify already retries fetches. +- Anything else: halt at the failing step, note the exact command + output, roll + back that app only, and leave the other app on its new version if it verified.