upgrade window 2026-11-09: executed (gateway 1.108.3, chat overlay 1.4.8) — record the outcome, the 400 MB task-limit OOM and its --no-backup remedy, and refresh the live image pins

app-inventory.md

@@ -20,17 +20,17 @@ does not hold it.
 | FQDN | App id | Label | Image (pin) | Install |
 |---|---|---|---|---|
 | `inference.coop` | `55f5f904-3233-44de-ad7b-5dc7c8533b75` | — | `cloudron/io.cloudron.surfer:202611021342220000` | store (Surfer — marketing site) |
-| `chat.inference.coop` | `4023ccbf-d57a-4500-97df-14de9cf3516e` | AI Chat | `registry.inference.coop/co-op/librechat-custom:20261031-2a98892` | **custom** (LibreChat overlay) |
-| `chat-legacy.inference.coop` | `50ae6c97-e851-4d17-b013-51ca7a4e5be0` | Legacy AI Chat | `cloudron/com.openwebui.cloudronapp:202611030420300000` | store (Open WebUI) |
-| `portal.inference.coop` | `d4f05e4b-75c6-492e-9faf-7afb0712fc94` | — | `registry.inference.coop/code/member-portal:20261104-d076729` | **custom** |
-| `dashboard.inference.coop` | `daf8750e-2683-4104-83ea-f77d1b4601e8` | Dashboard | `registry.inference.coop/code/member-dashboard:20261103-3453131` | **custom** |
-| `panel.inference.coop` | `ae187071-7a43-4a6e-8a02-986609a66a1a` | Admin Panel | `local/coop.inference.admin-panel:0.1.0-1793634279942` | **custom** |
-| `gateway.inference.coop` | `adababd4-2427-4bdd-9bf7-6bdad0302610` | LiteLLM | `registry.inference.coop/local/ai.coop.litellm:1.0.0-20261103-d6cfe78` | **custom** |
+| `chat.inference.coop` | `4023ccbf-d57a-4500-97df-14de9cf3516e` | AI Chat | `registry.inference.coop/co-op/librechat-custom:20261106-c2aec74` | **custom** (LibreChat overlay) |
+| `chat-legacy.inference.coop` | `50ae6c97-e851-4d17-b013-51ca7a4e5be0` | Legacy AI Chat | `cloudron/com.openwebui.cloudronapp:202611040420410000` | store (Open WebUI) |
+| `portal.inference.coop` | `d4f05e4b-75c6-492e-9faf-7afb0712fc94` | — | `registry.inference.coop/code/member-portal:20261108-2954fe60` | **custom** |
+| `dashboard.inference.coop` | `daf8750e-2683-4104-83ea-f77d1b4601e8` | Dashboard | `registry.inference.coop/code/member-dashboard:20261108-1e6e9e48` | **custom** |
+| `panel.inference.coop` | `ae187071-7a43-4a6e-8a02-986609a66a1a` | Admin Panel | `registry.inference.coop/code/admin-panel:20261108-b23cc67` | **custom** |
+| `gateway.inference.coop` | `adababd4-2427-4bdd-9bf7-6bdad0302610` | LiteLLM | `registry.inference.coop/local/ai.coop.litellm:20261109-c43d45c` | **custom** |
 | `matrix.inference.coop` | `aa6f4597-6206-4509-b32e-c89750b2c1d8` | Matrix Chat | `cloudron/org.matrix.synapse:202611030719520000` | store (Synapse) |
 | `git.inference.coop` | `c91c13aa-30ed-4af9-9599-7a87ab4f221d` | Code | `cloudron/io.gitea.cloudronapp:202611011719500000` | store (Gitea) |
 | `registry.inference.coop` | `c257ba3c-6737-4375-ad51-ea19586bf3f0` | — | `cloudron/io.cloudron.container-registry:202610221511140000` | store (container registry) |
 | `vault.inference.coop` | `cb838bcf-f2e2-4670-b117-dca04b0ee957` | — | `cloudron/com.github.bitwardenrs:202611020820270000` | store (Vaultwarden) |
-| `forum.inference.coop` | `8b1a972c-08af-4e92-852c-d573c37b01ad` | Forum | `cloudron/org.loomio.cloudronapp:202611010321240000` | store (Loomio) |
+| `forum.inference.coop` | `8b1a972c-08af-4e92-852c-d573c37b01ad` | Forum | `cloudron/org.loomio.cloudronapp:202611050321160000` | store (Loomio) |
 | `newsletter.inference.coop` | `eed2217c-b165-430e-bc29-7ccfef9b11e4` | — | `cloudron/app.listmonk.cloudronapp:202610220221130000` | store (Listmonk) |
 | `search.inference.coop` | `d7c73fff-ee42-4db0-98c0-4ce7a8ab35cc` | — | `cloudron/io.github.ascimoo.searx:202611012319290000` | store (SearXNG) |
 | `cloud.inference.coop` | `fd261d67-b790-424f-9e82-f74883ee8585` | — | `cloudron/com.nextcloud.cloudronapp:202610211606270000` | store (Nextcloud) |

maintenance/2026-11-08-litellm-1.108.0-librechat-1.4.8.md

@@ -302,3 +302,57 @@ cd ~/repos/librechat-custom
   re-check; release.sh verify already retries fetches.
 - Anything else: halt at the failing step, note the exact command + output, roll
   back that app only, and leave the other app on its new version if it verified.
+
+## 8. Executed — 2026-11-09, 04:41–05:05 UTC (what happened, and where it differed)
+
+Order followed as planned: LiteLLM first, then LibreChat. Two deliberate changes:
+
+1. **Target 1.108.3, not 1.108.0** — latest stable, and it pins the *same*
+   `litellm-proxy-extras==0.4.110`, so the migration set is exactly the one validated here
+   (36 new, all additive). All ten build-patch anchors were re-validated against the real
+   1.108.3 sources first; `patch_expose_cost_headers.py`'s 1.108.x anchor applied.
+2. **A pre-built, pre-verified image was deployed, not an on-server build** —
+   `20261109-c43d45c`, built from `upgrade-v1.108.3` (cut on top of `main`, so it carries the
+   2026-11-06/07 audio patches the original candidate predated) and exercised first as a
+   throwaway app (`gateway-dryrun.inference.coop`, its own Postgres/Redis, no member data).
+
+**The one thing that did not go as written — the pre-update snapshot OOMs.**
+A default `cloudron update` died at `Backup - Snapshotting app gateway.inference.coop`:
+task 1430 → `Task 1430 ran out of memory or terminated`, box log `Memory peak: 401.8M`,
+`oom-kill`. Cloudron's app-task service runs with a **hard 400 MB cgroup limit**
+(`startTask … "memoryLimit":400` from the system log), and this gateway's snapshot now
+exceeds it. Production was never at risk — the app kept serving the old image; only
+`installationState` went to `error`.
+**Workaround used: `--no-backup`**, with the 04:41 pre-flight app backups as the undo —
+which is what §2.1 is for. Next time: take the app backups first, then
+`cloudron update … --no-backup`.
+
+**Also learned the hard way:** `--wait-timeout` is a GLOBAL option, so
+`cloudron --wait-timeout 45m update …` works and `cloudron update --wait-timeout …` errors out.
+
+### What was verified live afterwards
+
+- **gateway** `…ai.coop.litellm:20261109-c43d45c` — litellm **1.108.3**, extras 0.4.110;
+  **all ten** patch markers present in the *running* container; `audio_keepalive.py`
+  executable; IoC check clean; migrations **185 → 221**, none failed or rolled back;
+  keys 210 / teams 111 unchanged; temp member key → `/v1/models` (15 models) → real
+  completion → **success SpendLogs row**; CORS preflight 200. Member traffic within ten
+  minutes: 73 success rows.
+- **chat** `…/librechat-custom:20261106-c2aec74` (appstore 1.4.8 / LibreChat 0.8.8) —
+  `release.sh verify` all checks passed, key sync running every minute (108 member keys,
+  revision matches the repo), Meili resync normal, served title still `Inference Co‑op`.
+
+### Migrations were rehearsed, not merely run
+
+Before the window, the live gateway database was restored from the 2026-11-08 stopgap
+snapshot into a local PostgreSQL **18.6** (prod's exact version) and the image's own
+entrypoint (`python -m prisma migrate deploy`) was run against it: **185 → 221 applied in
+12 s**, row counts identical (210 keys / 111 teams / 138,422 spend rows), **no table lost**
+(90 → 110, the 20 additions all new-feature tables). That is what made `--no-backup`
+acceptable on the real box.
+
+### Not rehearsed
+
+The LibreChat leg — a second chat app means ~2 GB of data plus its Meili/rag stack, for a
+change that is tooling-only (Node/uv; LibreChat upstream unchanged). Mitigation is the
+one-command rollback: `./release.sh pin 20261031-2a98892 && ./release.sh deploy`.