upgrade window 2026-11-09: executed (gateway 1.108.3, chat overlay 1.4.8) — record the outcome, the 400 MB task-limit OOM and its --no-backup remedy, and refresh the live image pins

maintenance/2026-11-08-litellm-1.108.0-librechat-1.4.8.md

@@ -302,3 +302,57 @@ cd ~/repos/librechat-custom
   re-check; release.sh verify already retries fetches.
 - Anything else: halt at the failing step, note the exact command + output, roll
   back that app only, and leave the other app on its new version if it verified.
+
+## 8. Executed — 2026-11-09, 04:41–05:05 UTC (what happened, and where it differed)
+
+Order followed as planned: LiteLLM first, then LibreChat. Two deliberate changes:
+
+1. **Target 1.108.3, not 1.108.0** — latest stable, and it pins the *same*
+   `litellm-proxy-extras==0.4.110`, so the migration set is exactly the one validated here
+   (36 new, all additive). All ten build-patch anchors were re-validated against the real
+   1.108.3 sources first; `patch_expose_cost_headers.py`'s 1.108.x anchor applied.
+2. **A pre-built, pre-verified image was deployed, not an on-server build** —
+   `20261109-c43d45c`, built from `upgrade-v1.108.3` (cut on top of `main`, so it carries the
+   2026-11-06/07 audio patches the original candidate predated) and exercised first as a
+   throwaway app (`gateway-dryrun.inference.coop`, its own Postgres/Redis, no member data).
+
+**The one thing that did not go as written — the pre-update snapshot OOMs.**
+A default `cloudron update` died at `Backup - Snapshotting app gateway.inference.coop`:
+task 1430 → `Task 1430 ran out of memory or terminated`, box log `Memory peak: 401.8M`,
+`oom-kill`. Cloudron's app-task service runs with a **hard 400 MB cgroup limit**
+(`startTask … "memoryLimit":400` from the system log), and this gateway's snapshot now
+exceeds it. Production was never at risk — the app kept serving the old image; only
+`installationState` went to `error`.
+**Workaround used: `--no-backup`**, with the 04:41 pre-flight app backups as the undo —
+which is what §2.1 is for. Next time: take the app backups first, then
+`cloudron update … --no-backup`.
+
+**Also learned the hard way:** `--wait-timeout` is a GLOBAL option, so
+`cloudron --wait-timeout 45m update …` works and `cloudron update --wait-timeout …` errors out.
+
+### What was verified live afterwards
+
+- **gateway** `…ai.coop.litellm:20261109-c43d45c` — litellm **1.108.3**, extras 0.4.110;
+  **all ten** patch markers present in the *running* container; `audio_keepalive.py`
+  executable; IoC check clean; migrations **185 → 221**, none failed or rolled back;
+  keys 210 / teams 111 unchanged; temp member key → `/v1/models` (15 models) → real
+  completion → **success SpendLogs row**; CORS preflight 200. Member traffic within ten
+  minutes: 73 success rows.
+- **chat** `…/librechat-custom:20261106-c2aec74` (appstore 1.4.8 / LibreChat 0.8.8) —
+  `release.sh verify` all checks passed, key sync running every minute (108 member keys,
+  revision matches the repo), Meili resync normal, served title still `Inference Co‑op`.
+
+### Migrations were rehearsed, not merely run
+
+Before the window, the live gateway database was restored from the 2026-11-08 stopgap
+snapshot into a local PostgreSQL **18.6** (prod's exact version) and the image's own
+entrypoint (`python -m prisma migrate deploy`) was run against it: **185 → 221 applied in
+12 s**, row counts identical (210 keys / 111 teams / 138,422 spend rows), **no table lost**
+(90 → 110, the 20 additions all new-feature tables). That is what made `--no-backup`
+acceptable on the real box.
+
+### Not rehearsed
+
+The LibreChat leg — a second chat app means ~2 GB of data plus its Meili/rag stack, for a
+change that is tooling-only (Node/uv; LibreChat upstream unchanged). Mitigation is the
+one-command rollback: `./release.sh pin 20261031-2a98892 && ./release.sh deploy`.