runbook: 2026-11-08 upgrade window — LiteLLM 1.103.2→1.108.0 + LibreChat overlay 1.4.5→1.4.8

Prepared and validated on 2026-11-06; nothing deployed. Candidates:
- code/litellm branch upgrade-v1.108.0 → image 1.0.0-20261106-9e25578
  (8/8 patches verified in the built image and in a live sandbox run;
  image-gen spend row verified nonzero twice — fake + real Regolo probe;
  36 additive migrations; budget_reset_at + portal API shapes unchanged).
- co-op/librechat-custom branch bump-appstore-1.4.8 → overlay 20261106-c2aec74
  (appstore 1.4.8 = tooling bumps on the same 0.8.8 upstream; all overlay
  anchors verified against the real 1.4.8 image — no porting needed).
Runbook: LiteLLM first, member-scope verification, rollback = previous tag
per app, window estimate 45-60 min.

README.md

@@ -30,6 +30,7 @@ Read it on the web here: <https://git.inference.coop/co-op/ops-docs>
 | `maintenance/librechat-sandbox.md` | Operating the LibreChat evaluation instance (config, overlay image, key sync, pitfalls) |
 | `maintenance/backups.md` | The two backup systems (agent-host restic + Cloudron app backup): schedules, retention, the cache split and the grouping fix |
 | `maintenance/woodpecker-ci.md` | The self-hosted CI server and agent: build/pin, manual-only trigger, git-lfs, and its not-yet-green state |
+| `maintenance/2026-11-08-litellm-1.108.0-librechat-1.4.8.md` | The Sunday-evening upgrade-window runbook (gateway → 1.108.0, chat overlay → appstore 1.4.8): candidates, order, exact commands, per-step verification, rollback, window estimate |
 | `incidents/` | Post-incident write-ups, one file per incident |
 | `migration/` | The Cloudron server-migration record: the plan, the DNS change sheet, and the captured pre-migration state |
 | `provider-pricing/` | What each model provider charges, and when it changed (GreenPT, Tinfoil, Regolo, and the decision-model route) |

maintenance/2026-11-08-litellm-1.108.0-librechat-1.4.8.md

@@ -0,0 +1,239 @@
+# Upgrade window — 2026-11-08 (Sunday evening): LiteLLM 1.103.2 → 1.108.0, LibreChat overlay 1.4.5 → 1.4.8
+
+Both upgrades were **prepared and validated in advance** (2026-11-06); nothing has
+been deployed. This runbook is written to be followed by any operator during the
+member-watched maintenance window. Read the gotchas in §6 before touching anything —
+this app pair has a history of deploys that look successful while changing nothing.
+
+**Candidates (already built and validated, sitting in the registry):**
+
+| App | Candidate | Rollback (currently deployed) |
+|---|---|---|
+| LiteLLM gateway (`adababd4-2427-4bdd-9bf7-6bdad0302610`, gateway.inference.coop) | branch `upgrade-v1.108.0` in `code/litellm`; prebuilt image `registry.inference.coop/local/ai.coop.litellm:1.0.0-20261106-9e25578` | image `registry.inference.coop/local/ai.coop.litellm:1.0.0-20261105-e161db9` (verified present in the registry) |
+| LibreChat chat (`4023ccbf-d57a-4500-97df-14de9cf3516e`, chat.inference.coop) | branch `bump-appstore-1.4.8` in `co-op/librechat-custom`; prebuilt overlay `registry.inference.coop/co-op/librechat-custom:20261106-c2aec74` | overlay `registry.inference.coop/co-op/librechat-custom:20261031-2a98892` (verified present) |
+
+**Prep evidence (what was already verified, so Sunday only re-confirms it):**
+
+- LiteLLM 1.108.0 candidate: all 8 build patches verified inside the running candidate
+  (file markers + build-log markers, zero WARNs); litellm reports 1.108.0.
+- `patch_image_cost.py` (image-gen spend): a real image generation through the
+  candidate produced a **nonzero success SpendLogs row** — twice (fake OpenAI-shaped
+  endpoint and the real Regolo Qwen-Image API): `spend 0.0011403264` for 256×256 at
+  `$1.74e-8/px`, `call_type aimage_generation`, `status success`.
+- `patch_passthrough_token_cost.py` (Metis metering): a real `/v1/systemone` call
+  logged `[coop-passthrough-cost] input_tokens=148 rate=1.2e-07 -> response_cost=1.776e-05`
+  and the SpendLogs row matched exactly (148 × $0.12/1M), tokens recorded.
+- Member path: temp key generated with master auth → `/v1/models` (11 models) → real
+  chat completion (greenpt/glm-5.3-flash, "pong", usage 19/3 tokens, spend row written).
+- Migrations: 36 new between 1.103.2 and 1.108.0, **all additive** (no unguarded
+  DROP/DELETE/UPDATE). A fresh DB ran the full 220-migration chain cleanly; the live
+  ledger (1.103.2) will apply only the 36 new ones.
+- `budget_reset_at`: column unchanged on `LiteLLM_TeamTable`; a direct-DB write
+  (the portal's exact pattern) plus API read-back verified on the candidate.
+- Portal API compatibility: `/team/list` (bare list) and `/key/list`
+  (`{keys, total_count, …}`) shapes are **unchanged**; a bare `/team/list` call still
+  returns ALL teams (pagination only applies when `page`/`page_size` are passed —
+  verified with 11 teams); `/team/update` still drops `budget_reset_at`, so the
+  portal's direct-DB write remains required. SpendLogs gains only an additive
+  `billing_agent_id` column — digest queries are unaffected.
+- LibreChat 1.4.8 (appstore `ai.librechat.cloudronapp`): changelog 1.4.5 → 1.4.8 is
+  tooling only — uv 0.13.6/0.13.7 and Node.js 22.23.4 (security fix, 1.4.7); the
+  LibreChat upstream stays 0.8.8. Every overlay patch anchor (title tag, manifest
+  link, theme-color metas, webmanifest colors, "MCP Servers" / "Every AI for
+  Everyone." strings, `index.*.js`, `sw.js`) was verified against the real 1.4.8
+  image — **no overlay porting needed**; the candidate overlay built with all
+  markers (UI strings patched in 3 files, index/manifest branding applied).
+
+## 1. Order: LiteLLM first, then LibreChat
+
+LibreChat's post-upgrade checks (key sync, a real chat completion) run **through the
+gateway** — doing the gateway first means those checks also exercise the new
+gateway, and each upgrade stays an isolated abort point: if LiteLLM fails, roll it
+back and skip LibreChat (chat keeps serving on the old overlay); if LibreChat fails,
+the new gateway is already proven.
+
+## 2. Pre-flight (both apps, ~5 min)
+
+```bash
+set -a; . ~/.hermes/.env; set +a
+CR() { cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" "$@"; }
+
+# 2.1 Fresh update-rollback backups (DB + volume snapshot each)
+CR backup create --app adababd4-2427-4bdd-9bf7-6bdad0302610
+CR backup create --app 4023ccbf-d57a-4500-97df-14de9cf3516e
+CR backup list --app adababd4-2427-4bdd-9bf7-6bdad0302610 | tail -3   # note newest id
+CR backup list --app 4023ccbf-d57a-4500-97df-14de9cf3516e | tail -3
+
+# 2.2 Record the member-visible baseline for after: number of keys and teams
+#     (use the MASTER key — it is in /app/data/.master_key inside the gateway:
+#      CR exec --app gateway.inference.coop -- cat /app/data/.master_key
+#      then GET /key/list and /team/list with it; count both. NEVER paste the key.)
+```
+
+## 3. LiteLLM 1.103.2 → 1.108.0 (~20–35 min)
+
+```bash
+cd ~/repos/litellm                       # ONE APP = ONE REPO DIR. Never run an update
+git fetch origin && git checkout main && git pull   # for another app from this cwd
+git merge upgrade-v1.108.0               # branch is pushed; approve the merge in Gitea
+                                         # if you prefer the PR path
+git log --oneline -1                     # note the sha
+```
+
+3.1 **Deploy (on-server build — the repo manifest has no `dockerImage` pin, so
+`cloudron update` builds and swaps; do NOT pass `--image`):**
+
+```bash
+CR update --app adababd4-2427-4bdd-9bf7-6bdad0302610
+```
+
+- The build output must contain **all eight** patch markers (each is one line in the
+  build log; a missing line = that patch silently did not apply — STOP and check):
+
+  1. `INFO: patched budget-exceeded messages in …/auth_checks.py`
+  2. `INFO: patched image-cost registration in …/router.py`
+  3. `[coop-patch] applied to …/cost_calculator.py`
+  4. `[coop-patch] applied audio shared pricing to …/router.py`
+  5. `[coop-patch] applied: provider response headers are no longer forwarded …`
+  6. `expose-cost-headers: appended 20 cost header names to expose_headers …`
+  7. `[coop-patch] applied: passthrough-key-access -> …/route_checks.py`
+  8. `[coop-patch] applied: passthrough-token-cost -> …/success_handler.py`
+
+- **Boot**: LiteLLM runs `prisma migrate deploy` (the 36 additive migrations) before
+  binding port 4000. Expect `502`/`ECONNREFUSED` on health checks for a few minutes
+  (10–20 in the bad case). **Never restart mid-wait** — a restart resets the
+  migration clock. Once `/health/liveliness` returns 200, continue immediately
+  ("up" ≠ "migrated"; the checks below are the proof).
+- If the on-server build flakes for any reason, fall back to the pre-validated
+  image: `CR update --app adababd4-2427-4bdd-9bf7-6bdad0302610 --image registry.inference.coop/local/ai.coop.litellm:1.0.0-20261106-9e25578`
+  (works precisely because the repo manifest carries no `dockerImage` field).
+
+3.2 **Verify (in order — a failing early step means stop, not continue):**
+
+```bash
+# 1. version inside the running container
+CR exec --app gateway.inference.coop -- \
+  /app/.venv/bin/python3 -c "from importlib.metadata import version; print(version('litellm'))"
+#    → 1.108.0
+
+# 2. patch markers inside the RUNNING container (not the build log alone)
+CR exec --app gateway.inference.coop -- sh -c \
+  'grep -c "CO-OP PATCH (2026-10-19)" /app/.venv/lib/python3.13/site-packages/litellm/router.py; \
+   grep -c "x-litellm-response-cost-input" /app/.venv/lib/python3.13/site-packages/litellm/proxy/proxy_server.py'
+#    → 1 and 1
+
+# 3. MEMBER-SCOPE key path (master-key tests alone prove nothing — master bypasses
+#    exactly the tables a broken migration empties):
+#    POST /key/generate  {"alias":"upgrade-verify-temp"}   (master auth) → temp key
+#    GET  /v1/models with the TEMP key                     → 11 models
+#    POST /v1/chat/completions {"model":"greenpt/glm-5.3-flash",
+#         "messages":[{"role":"user","content":"Reply with the single word: pong"}],
+#         "max_tokens":16}                                  → content + usage
+#    GET  /key/list (master)                               → same count as §2.2
+#    POST /key/delete {"keys":["<temp>"]}                  → 200
+
+# 4. CORS preflight (chat origin must be allowed):
+#    OPTIONS /v1/models with Origin: https://chat.inference.coop → 200 + allow-origin
+
+# 5. Cloudron state
+curl -sS -H "Authorization: Bearer $CLOUDRON_TOKEN" \
+  https://my.inference.coop/api/v1/apps/adababd4-2427-4bdd-9bf7-6bdad0302610 \
+  | python3 -c "import sys,json; a=json.load(sys.stdin); a=a.get('app',a); print(a['health'], a['runState'], a['manifest'].get('dockerImage'))"
+#    → healthy running registry.inference.coop/local/ai.coop.litellm:<new tag>
+
+# 6. Watch 10–15 min of real member traffic before closing:
+#    SpendLogs rows arriving with status 'success' (portal admin panel or
+#    GET /spend/logs with master auth). Failure rows with call_type = a route
+#    path and spend 0 are 401-refusals, not duplicates.
+```
+
+3.3 **Rollback (one command; the additive migrations make old-code-on-new-schema safe):**
+
+```bash
+CR update --app adababd4-2427-4bdd-9bf7-6bdad0302610 \
+  --image registry.inference.coop/local/ai.coop.litellm:1.0.0-20261105-e161db9
+```
+
+If member keys 401 after any image change, the schema is the suspect: read the
+exact error text from a member key call — a missing-relation error names the failing
+path; do NOT reissue keys. (Full recipe: `code/litellm` README, "Verify after
+upgrading".)
+
+## 4. LibreChat overlay 1.4.5 → 1.4.8 (~10–15 min)
+
+```bash
+cd ~/repos/librechat-custom              # again: one app = its own repo dir
+git fetch origin && git checkout master && git pull
+git merge bump-appstore-1.4.8            # overlay code is UNCHANGED apart from the
+                                         # FROM pin — appstore 1.4.8 carries the same
+                                         # LibreChat 0.8.8; only Node/uv/base bumped
+set -a; . ~/.hermes/.env; set +a
+
+# The candidate overlay is already built: pin + deploy via the repo's release tool
+./release.sh pin 20261106-c2aec74        # writes dockerImage into CloudronManifest.json, commits, pushes
+./release.sh deploy                      # cloudron update --force from THIS dir, then verify
+```
+
+4.1 **Verify** (release.sh verify runs the served-HTML checks; confirm the rest by hand):
+
+```bash
+./release.sh verify                      # title = Inference Co-op, x-coop-build meta present
+
+# key sync task ran (manifest scheduler `librechat_key_sync`, every minute):
+CR logs --app chat.inference.coop --lines 100 | grep -i "sync" | tail -5
+#    → the sync script prints its own md5 prefix each run; it must be the new build's
+
+# RAG service: the appstore package runs rag_api alongside; confirm no crash loop
+curl -sS -o /dev/null -w "%{http_code}\n" https://chat.inference.coop/   # 200 (SSO shell)
+
+# login via Cloudron SSO in a browser, then send one real chat completion through
+# LibreChat (this exercises chat → NEW gateway → provider end-to-end)
+```
+
+4.2 **Rollback:**
+
+```bash
+cd ~/repos/librechat-custom
+./release.sh pin 20261031-2a98892 && ./release.sh deploy
+```
+
+## 5. Window estimate
+
+| Phase | Time |
+|---|---|
+| Pre-flight + backups | 5 min |
+| LiteLLM build+swap (on-server build) | 5–10 min |
+| LiteLLM boot migrations + member-scope verification | 10–20 min |
+| LibreChat pin+deploy+verify (incl. an SSO login + one chat) | 10–15 min |
+| Post-watch on SpendLogs / close-out | 10 min |
+| **Total** | **~45–60 min** |
+
+## 6. Gotchas that have each bitten this stack once
+
+1. **One app = one `cd`, immediately before its update.** `cloudron update` reads
+   `CloudronManifest.json` from the CURRENT directory; updating app B from repo A's
+   dir ships A's manifest to B — on 2026-11-05 that dropped the gateway's postgres
+   addon and wiped the LiteLLM DB (~50 min member outage, restored from the
+   pre-deploy snapshot). The pre-deploy backups in §2 are the undo; addons-block
+   changes are DATA-BEARING.
+2. **A pinned manifest beats `--image`, silently.** LibreChat's repo manifest pins
+   `dockerImage` — always change the pin (release.sh pin), never pass `--image` to
+   it. LiteLLM's repo manifest has NO pin — `--image` works there (used for its
+   rollback and build-flake fallback above).
+3. **"App is updated." proves nothing.** Verify the running image tag via the API
+   AND a container-content marker (§3.2 steps 1–2, §4.1) before trusting any check.
+4. **Don't restart the gateway mid-migration-wait** — it resets the boot clock.
+5. **Do not "fix" `/team/update` dropping `budget_reset_at` today** — the portal's
+   direct-DB write is the known, verified workaround; it still works on 1.108.0.
+6. After both apps are up, `app-inventory.md` in this repo should be updated with
+   the new pins in the same session (it mirrors live state).
+
+## 7. If something looks wrong
+
+- Member keys 401 with a missing-relation error → schema/migration issue; see
+  §3.3 and the `code/litellm` README triage section. Restore path: the §2 backup,
+  then re-pin the old image.
+- Chat serves a stale shell after the overlay deploy → wait ~60 s (mid-swap) and
+  re-check; release.sh verify already retries fetches.
+- Anything else: halt at the failing step, note the exact command + output, roll
+  back that app only, and leave the other app on its new version if it verified.