co-op/ops-docs
2
0 Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

runbook: 2026-11-08 upgrade window — LiteLLM 1.103.2→1.108.0 + LibreChat overlay 1.4.5→1.4.8

Browse Source 
Prepared and validated on 2026-11-06; nothing deployed. Candidates:
- code/litellm branch upgrade-v1.108.0 → image 1.0.0-20261106-9e25578
  (8/8 patches verified in the built image and in a live sandbox run;
  image-gen spend row verified nonzero twice — fake + real Regolo probe;
  36 additive migrations; budget_reset_at + portal API shapes unchanged).
- co-op/librechat-custom branch bump-appstore-1.4.8 → overlay 20261106-c2aec74
  (appstore 1.4.8 = tooling bumps on the same 0.8.8 upstream; all overlay
  anchors verified against the real 1.4.8 image — no porting needed).
Runbook: LiteLLM first, member-scope verification, rollback = previous tag
per app, window estimate 45-60 min.
This commit is contained in:
inference-bot committed 2026-11-06 04:11:01 +00:00
1 parent fcd4ac6943
commit 816961fb2d
2 files changed
+240
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

README.md
+1
View File
@@ -30,6 +30,7 @@ Read it on the web here: <https://git.inference.coop/co-op/ops-docs>
| `maintenance/librechat-sandbox.md` | Operating the LibreChat evaluation instance (config, overlay image, key sync, pitfalls) |
| `maintenance/backups.md` | The two backup systems (agent-host restic + Cloudron app backup): schedules, retention, the cache split and the grouping fix |
| `maintenance/woodpecker-ci.md` | The self-hosted CI server and agent: build/pin, manual-only trigger, git-lfs, and its not-yet-green state |
| `maintenance/2026-11-08-litellm-1.108.0-librechat-1.4.8.md` | The Sunday-evening upgrade-window runbook (gateway → 1.108.0, chat overlay → appstore 1.4.8): candidates, order, exact commands, per-step verification, rollback, window estimate |
| `incidents/` | Post-incident write-ups, one file per incident |
| `migration/` | The Cloudron server-migration record: the plan, the DNS change sheet, and the captured pre-migration state |
| `provider-pricing/` | What each model provider charges, and when it changed (GreenPT, Tinfoil, Regolo, and the decision-model route) |
maintenance/2026-11-08-litellm-1.108.0-librechat-1.4.8.md
+239
View File
@@ -0,0 +1,239 @@
# Upgrade window — 2026-11-08 (Sunday evening): LiteLLM 1.103.2 → 1.108.0, LibreChat overlay 1.4.5 → 1.4.8
Both upgrades were **prepared and validated in advance** (2026-11-06); nothing has
been deployed. This runbook is written to be followed by any operator during the
member-watched maintenance window. Read the gotchas in §6 before touching anything —
this app pair has a history of deploys that look successful while changing nothing.
**Candidates (already built and validated, sitting in the registry):**
| App | Candidate | Rollback (currently deployed) |
|---|---|---|
| LiteLLM gateway (`adababd4-2427-4bdd-9bf7-6bdad0302610`, gateway.inference.coop) | branch `upgrade-v1.108.0` in `code/litellm`; prebuilt image `registry.inference.coop/local/ai.coop.litellm:1.0.0-20261106-9e25578` | image `registry.inference.coop/local/ai.coop.litellm:1.0.0-20261105-e161db9` (verified present in the registry) |
| LibreChat chat (`4023ccbf-d57a-4500-97df-14de9cf3516e`, chat.inference.coop) | branch `bump-appstore-1.4.8` in `co-op/librechat-custom`; prebuilt overlay `registry.inference.coop/co-op/librechat-custom:20261106-c2aec74` | overlay `registry.inference.coop/co-op/librechat-custom:20261031-2a98892` (verified present) |
**Prep evidence (what was already verified, so Sunday only re-confirms it):**
- LiteLLM 1.108.0 candidate: all 8 build patches verified inside the running candidate
(file markers + build-log markers, zero WARNs); litellm reports 1.108.0.
- `patch_image_cost.py` (image-gen spend): a real image generation through the
candidate produced a **nonzero success SpendLogs row** — twice (fake OpenAI-shaped
endpoint and the real Regolo Qwen-Image API): `spend 0.0011403264` for 256×256 at
`$1.74e-8/px`, `call_type aimage_generation`, `status success`.
- `patch_passthrough_token_cost.py` (Metis metering): a real `/v1/systemone` call
logged `[coop-passthrough-cost] input_tokens=148 rate=1.2e-07 -> response_cost=1.776e-05`
and the SpendLogs row matched exactly (148 × $0.12/1M), tokens recorded.
- Member path: temp key generated with master auth → `/v1/models` (11 models) → real
chat completion (greenpt/glm-5.3-flash, "pong", usage 19/3 tokens, spend row written).
- Migrations: 36 new between 1.103.2 and 1.108.0, **all additive** (no unguarded
DROP/DELETE/UPDATE). A fresh DB ran the full 220-migration chain cleanly; the live
ledger (1.103.2) will apply only the 36 new ones.
- `budget_reset_at`: column unchanged on `LiteLLM_TeamTable`; a direct-DB write
(the portal's exact pattern) plus API read-back verified on the candidate.
- Portal API compatibility: `/team/list` (bare list) and `/key/list`
(`{keys, total_count, …}`) shapes are **unchanged**; a bare `/team/list` call still
returns ALL teams (pagination only applies when `page`/`page_size` are passed —
verified with 11 teams); `/team/update` still drops `budget_reset_at`, so the
portal's direct-DB write remains required. SpendLogs gains only an additive
`billing_agent_id` column — digest queries are unaffected.
- LibreChat 1.4.8 (appstore `ai.librechat.cloudronapp`): changelog 1.4.5 → 1.4.8 is
tooling only — uv 0.13.6/0.13.7 and Node.js 22.23.4 (security fix, 1.4.7); the
LibreChat upstream stays 0.8.8. Every overlay patch anchor (title tag, manifest
link, theme-color metas, webmanifest colors, "MCP Servers" / "Every AI for
Everyone." strings, `index.*.js`, `sw.js`) was verified against the real 1.4.8
image — **no overlay porting needed**; the candidate overlay built with all
markers (UI strings patched in 3 files, index/manifest branding applied).
## 1. Order: LiteLLM first, then LibreChat
LibreChat's post-upgrade checks (key sync, a real chat completion) run **through the
gateway** — doing the gateway first means those checks also exercise the new
gateway, and each upgrade stays an isolated abort point: if LiteLLM fails, roll it
back and skip LibreChat (chat keeps serving on the old overlay); if LibreChat fails,
the new gateway is already proven.
## 2. Pre-flight (both apps, ~5 min)
```bash
set -a; . ~/.hermes/.env; set +a
CR() { cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" "$@"; }
# 2.1 Fresh update-rollback backups (DB + volume snapshot each)
CR backup create --app adababd4-2427-4bdd-9bf7-6bdad0302610
CR backup create --app 4023ccbf-d57a-4500-97df-14de9cf3516e
CR backup list --app adababd4-2427-4bdd-9bf7-6bdad0302610 | tail -3 # note newest id
CR backup list --app 4023ccbf-d57a-4500-97df-14de9cf3516e | tail -3
# 2.2 Record the member-visible baseline for after: number of keys and teams
# (use the MASTER key — it is in /app/data/.master_key inside the gateway:
# CR exec --app gateway.inference.coop -- cat /app/data/.master_key
# then GET /key/list and /team/list with it; count both. NEVER paste the key.)
```
## 3. LiteLLM 1.103.2 → 1.108.0 (~20–35 min)
```bash
cd ~/repos/litellm # ONE APP = ONE REPO DIR. Never run an update
git fetch origin && git checkout main && git pull # for another app from this cwd
git merge upgrade-v1.108.0 # branch is pushed; approve the merge in Gitea
# if you prefer the PR path
git log --oneline -1 # note the sha
```
3.1 **Deploy (on-server build — the repo manifest has no `dockerImage` pin, so
`cloudron update` builds and swaps; do NOT pass `--image`):**
```bash
CR update --app adababd4-2427-4bdd-9bf7-6bdad0302610
```
- The build output must contain **all eight** patch markers (each is one line in the
build log; a missing line = that patch silently did not apply — STOP and check):
1. `INFO: patched budget-exceeded messages in …/auth_checks.py`
2. `INFO: patched image-cost registration in …/router.py`
3. `[coop-patch] applied to …/cost_calculator.py`
4. `[coop-patch] applied audio shared pricing to …/router.py`
5. `[coop-patch] applied: provider response headers are no longer forwarded …`
6. `expose-cost-headers: appended 20 cost header names to expose_headers …`
7. `[coop-patch] applied: passthrough-key-access -> …/route_checks.py`
8. `[coop-patch] applied: passthrough-token-cost -> …/success_handler.py`
- **Boot**: LiteLLM runs `prisma migrate deploy` (the 36 additive migrations) before
binding port 4000. Expect `502`/`ECONNREFUSED` on health checks for a few minutes
(10–20 in the bad case). **Never restart mid-wait** — a restart resets the
migration clock. Once `/health/liveliness` returns 200, continue immediately
("up" ≠ "migrated"; the checks below are the proof).
- If the on-server build flakes for any reason, fall back to the pre-validated
image: `CR update --app adababd4-2427-4bdd-9bf7-6bdad0302610 --image registry.inference.coop/local/ai.coop.litellm:1.0.0-20261106-9e25578`
(works precisely because the repo manifest carries no `dockerImage` field).
3.2 **Verify (in order — a failing early step means stop, not continue):**
```bash
# 1. version inside the running container
CR exec --app gateway.inference.coop -- \
/app/.venv/bin/python3 -c "from importlib.metadata import version; print(version('litellm'))"
# → 1.108.0
# 2. patch markers inside the RUNNING container (not the build log alone)
CR exec --app gateway.inference.coop -- sh -c \
'grep -c "CO-OP PATCH (2026-10-19)" /app/.venv/lib/python3.13/site-packages/litellm/router.py; \
grep -c "x-litellm-response-cost-input" /app/.venv/lib/python3.13/site-packages/litellm/proxy/proxy_server.py'
# → 1 and 1
# 3. MEMBER-SCOPE key path (master-key tests alone prove nothing — master bypasses
# exactly the tables a broken migration empties):
# POST /key/generate {"alias":"upgrade-verify-temp"} (master auth) → temp key
# GET /v1/models with the TEMP key → 11 models
# POST /v1/chat/completions {"model":"greenpt/glm-5.3-flash",
# "messages":[{"role":"user","content":"Reply with the single word: pong"}],
# "max_tokens":16} → content + usage
# GET /key/list (master) → same count as §2.2
# POST /key/delete {"keys":["<temp>"]} → 200
# 4. CORS preflight (chat origin must be allowed):
# OPTIONS /v1/models with Origin: https://chat.inference.coop → 200 + allow-origin
# 5. Cloudron state
curl -sS -H "Authorization: Bearer $CLOUDRON_TOKEN" \
https://my.inference.coop/api/v1/apps/adababd4-2427-4bdd-9bf7-6bdad0302610 \
| python3 -c "import sys,json; a=json.load(sys.stdin); a=a.get('app',a); print(a['health'], a['runState'], a['manifest'].get('dockerImage'))"
# → healthy running registry.inference.coop/local/ai.coop.litellm:<new tag>
# 6. Watch 10–15 min of real member traffic before closing:
# SpendLogs rows arriving with status 'success' (portal admin panel or
# GET /spend/logs with master auth). Failure rows with call_type = a route
# path and spend 0 are 401-refusals, not duplicates.
```
3.3 **Rollback (one command; the additive migrations make old-code-on-new-schema safe):**
```bash
CR update --app adababd4-2427-4bdd-9bf7-6bdad0302610 \
--image registry.inference.coop/local/ai.coop.litellm:1.0.0-20261105-e161db9
```
If member keys 401 after any image change, the schema is the suspect: read the
exact error text from a member key call — a missing-relation error names the failing
path; do NOT reissue keys. (Full recipe: `code/litellm` README, "Verify after
upgrading".)
## 4. LibreChat overlay 1.4.5 → 1.4.8 (~10–15 min)
```bash
cd ~/repos/librechat-custom # again: one app = its own repo dir
git fetch origin && git checkout master && git pull
git merge bump-appstore-1.4.8 # overlay code is UNCHANGED apart from the
# FROM pin — appstore 1.4.8 carries the same
# LibreChat 0.8.8; only Node/uv/base bumped
set -a; . ~/.hermes/.env; set +a
# The candidate overlay is already built: pin + deploy via the repo's release tool
./release.sh pin 20261106-c2aec74 # writes dockerImage into CloudronManifest.json, commits, pushes
./release.sh deploy # cloudron update --force from THIS dir, then verify
```
4.1 **Verify** (release.sh verify runs the served-HTML checks; confirm the rest by hand):
```bash
./release.sh verify # title = Inference Co-op, x-coop-build meta present
# key sync task ran (manifest scheduler `librechat_key_sync`, every minute):
CR logs --app chat.inference.coop --lines 100 | grep -i "sync" | tail -5
# → the sync script prints its own md5 prefix each run; it must be the new build's
# RAG service: the appstore package runs rag_api alongside; confirm no crash loop
curl -sS -o /dev/null -w "%{http_code}\n" https://chat.inference.coop/ # 200 (SSO shell)
# login via Cloudron SSO in a browser, then send one real chat completion through
# LibreChat (this exercises chat → NEW gateway → provider end-to-end)
```
4.2 **Rollback:**
```bash
cd ~/repos/librechat-custom
./release.sh pin 20261031-2a98892 && ./release.sh deploy
```
## 5. Window estimate
| Phase | Time |
|---|---|
| Pre-flight + backups | 5 min |
| LiteLLM build+swap (on-server build) | 5–10 min |
| LiteLLM boot migrations + member-scope verification | 10–20 min |
| LibreChat pin+deploy+verify (incl. an SSO login + one chat) | 10–15 min |
| Post-watch on SpendLogs / close-out | 10 min |
| **Total** | **~45–60 min** |
## 6. Gotchas that have each bitten this stack once
1. **One app = one `cd`, immediately before its update.** `cloudron update` reads
`CloudronManifest.json` from the CURRENT directory; updating app B from repo A's
dir ships A's manifest to B — on 2026-11-05 that dropped the gateway's postgres
addon and wiped the LiteLLM DB (~50 min member outage, restored from the
pre-deploy snapshot). The pre-deploy backups in §2 are the undo; addons-block
changes are DATA-BEARING.
2. **A pinned manifest beats `--image`, silently.** LibreChat's repo manifest pins
`dockerImage` — always change the pin (release.sh pin), never pass `--image` to
it. LiteLLM's repo manifest has NO pin — `--image` works there (used for its
rollback and build-flake fallback above).
3. **"App is updated." proves nothing.** Verify the running image tag via the API
AND a container-content marker (§3.2 steps 1–2, §4.1) before trusting any check.
4. **Don't restart the gateway mid-migration-wait** — it resets the boot clock.
5. **Do not "fix" `/team/update` dropping `budget_reset_at` today** — the portal's
direct-DB write is the known, verified workaround; it still works on 1.108.0.
6. After both apps are up, `app-inventory.md` in this repo should be updated with
the new pins in the same session (it mirrors live state).
## 7. If something looks wrong
- Member keys 401 with a missing-relation error → schema/migration issue; see
§3.3 and the `code/litellm` README triage section. Restore path: the §2 backup,
then re-pin the old image.
- Chat serves a stale shell after the overlay deploy → wait ~60 s (mid-swap) and
re-check; release.sh verify already retries fetches.
- Anything else: halt at the failing step, note the exact command + output, roll
back that app only, and leave the other app on its new version if it verified.