co-op/ops-docs
2
0 Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

upgrade window 2026-11-09: executed (gateway 1.108.3, chat overlay 1.4.8) — record the outcome, the 400 MB task-limit OOM and its --no-backup remedy, and refresh the live image pins

Browse Source
This commit is contained in:
inference-bot committed 2026-11-09 05:04:27 +00:00
1 parent 52e3c9d502
commit 2c38f4d2c3
2 files changed
+61 -7
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

app-inventory.md
+7 -7
View File
@@ -20,17 +20,17 @@ does not hold it.
| FQDN | App id | Label | Image (pin) | Install |
|---|---|---|---|---|
| `inference.coop` | `55f5f904-3233-44de-ad7b-5dc7c8533b75` | — | `cloudron/io.cloudron.surfer:202611021342220000` | store (Surfer — marketing site) |
| `chat.inference.coop` | `4023ccbf-d57a-4500-97df-14de9cf3516e` | AI Chat | `registry.inference.coop/co-op/librechat-custom:20261031-2a98892` | **custom** (LibreChat overlay) |
| `chat-legacy.inference.coop` | `50ae6c97-e851-4d17-b013-51ca7a4e5be0` | Legacy AI Chat | `cloudron/com.openwebui.cloudronapp:202611030420300000` | store (Open WebUI) |
| `portal.inference.coop` | `d4f05e4b-75c6-492e-9faf-7afb0712fc94` | — | `registry.inference.coop/code/member-portal:20261104-d076729` | **custom** |
| `dashboard.inference.coop` | `daf8750e-2683-4104-83ea-f77d1b4601e8` | Dashboard | `registry.inference.coop/code/member-dashboard:20261103-3453131` | **custom** |
| `panel.inference.coop` | `ae187071-7a43-4a6e-8a02-986609a66a1a` | Admin Panel | `local/coop.inference.admin-panel:0.1.0-1793634279942` | **custom** |
| `gateway.inference.coop` | `adababd4-2427-4bdd-9bf7-6bdad0302610` | LiteLLM | `registry.inference.coop/local/ai.coop.litellm:1.0.0-20261103-d6cfe78` | **custom** |
| `chat.inference.coop` | `4023ccbf-d57a-4500-97df-14de9cf3516e` | AI Chat | `registry.inference.coop/co-op/librechat-custom:20261106-c2aec74` | **custom** (LibreChat overlay) |
| `chat-legacy.inference.coop` | `50ae6c97-e851-4d17-b013-51ca7a4e5be0` | Legacy AI Chat | `cloudron/com.openwebui.cloudronapp:202611040420410000` | store (Open WebUI) |
| `portal.inference.coop` | `d4f05e4b-75c6-492e-9faf-7afb0712fc94` | — | `registry.inference.coop/code/member-portal:20261108-2954fe60` | **custom** |
| `dashboard.inference.coop` | `daf8750e-2683-4104-83ea-f77d1b4601e8` | Dashboard | `registry.inference.coop/code/member-dashboard:20261108-1e6e9e48` | **custom** |
| `panel.inference.coop` | `ae187071-7a43-4a6e-8a02-986609a66a1a` | Admin Panel | `registry.inference.coop/code/admin-panel:20261108-b23cc67` | **custom** |
| `gateway.inference.coop` | `adababd4-2427-4bdd-9bf7-6bdad0302610` | LiteLLM | `registry.inference.coop/local/ai.coop.litellm:20261109-c43d45c` | **custom** |
| `matrix.inference.coop` | `aa6f4597-6206-4509-b32e-c89750b2c1d8` | Matrix Chat | `cloudron/org.matrix.synapse:202611030719520000` | store (Synapse) |
| `git.inference.coop` | `c91c13aa-30ed-4af9-9599-7a87ab4f221d` | Code | `cloudron/io.gitea.cloudronapp:202611011719500000` | store (Gitea) |
| `registry.inference.coop` | `c257ba3c-6737-4375-ad51-ea19586bf3f0` | — | `cloudron/io.cloudron.container-registry:202610221511140000` | store (container registry) |
| `vault.inference.coop` | `cb838bcf-f2e2-4670-b117-dca04b0ee957` | — | `cloudron/com.github.bitwardenrs:202611020820270000` | store (Vaultwarden) |
| `forum.inference.coop` | `8b1a972c-08af-4e92-852c-d573c37b01ad` | Forum | `cloudron/org.loomio.cloudronapp:202611010321240000` | store (Loomio) |
| `forum.inference.coop` | `8b1a972c-08af-4e92-852c-d573c37b01ad` | Forum | `cloudron/org.loomio.cloudronapp:202611050321160000` | store (Loomio) |
| `newsletter.inference.coop` | `eed2217c-b165-430e-bc29-7ccfef9b11e4` | — | `cloudron/app.listmonk.cloudronapp:202610220221130000` | store (Listmonk) |
| `search.inference.coop` | `d7c73fff-ee42-4db0-98c0-4ce7a8ab35cc` | — | `cloudron/io.github.ascimoo.searx:202611012319290000` | store (SearXNG) |
| `cloud.inference.coop` | `fd261d67-b790-424f-9e82-f74883ee8585` | — | `cloudron/com.nextcloud.cloudronapp:202610211606270000` | store (Nextcloud) |
maintenance/2026-11-08-litellm-1.108.0-librechat-1.4.8.md
+54
View File
@@ -302,3 +302,57 @@ cd ~/repos/librechat-custom
re-check; release.sh verify already retries fetches.
- Anything else: halt at the failing step, note the exact command + output, roll
back that app only, and leave the other app on its new version if it verified.
## 8. Executed — 2026-11-09, 04:41–05:05 UTC (what happened, and where it differed)
Order followed as planned: LiteLLM first, then LibreChat. Two deliberate changes:
1. **Target 1.108.3, not 1.108.0** — latest stable, and it pins the *same*
`litellm-proxy-extras==0.4.110`, so the migration set is exactly the one validated here
(36 new, all additive). All ten build-patch anchors were re-validated against the real
1.108.3 sources first; `patch_expose_cost_headers.py`'s 1.108.x anchor applied.
2. **A pre-built, pre-verified image was deployed, not an on-server build** —
`20261109-c43d45c`, built from `upgrade-v1.108.3` (cut on top of `main`, so it carries the
2026-11-06/07 audio patches the original candidate predated) and exercised first as a
throwaway app (`gateway-dryrun.inference.coop`, its own Postgres/Redis, no member data).
**The one thing that did not go as written — the pre-update snapshot OOMs.**
A default `cloudron update` died at `Backup - Snapshotting app gateway.inference.coop`:
task 1430 → `Task 1430 ran out of memory or terminated`, box log `Memory peak: 401.8M`,
`oom-kill`. Cloudron's app-task service runs with a **hard 400 MB cgroup limit**
(`startTask … "memoryLimit":400` from the system log), and this gateway's snapshot now
exceeds it. Production was never at risk — the app kept serving the old image; only
`installationState` went to `error`.
**Workaround used: `--no-backup`**, with the 04:41 pre-flight app backups as the undo —
which is what §2.1 is for. Next time: take the app backups first, then
`cloudron update … --no-backup`.
**Also learned the hard way:** `--wait-timeout` is a GLOBAL option, so
`cloudron --wait-timeout 45m update …` works and `cloudron update --wait-timeout …` errors out.
### What was verified live afterwards
- **gateway** `…ai.coop.litellm:20261109-c43d45c` — litellm **1.108.3**, extras 0.4.110;
**all ten** patch markers present in the *running* container; `audio_keepalive.py`
executable; IoC check clean; migrations **185 → 221**, none failed or rolled back;
keys 210 / teams 111 unchanged; temp member key → `/v1/models` (15 models) → real
completion → **success SpendLogs row**; CORS preflight 200. Member traffic within ten
minutes: 73 success rows.
- **chat** `…/librechat-custom:20261106-c2aec74` (appstore 1.4.8 / LibreChat 0.8.8) —
`release.sh verify` all checks passed, key sync running every minute (108 member keys,
revision matches the repo), Meili resync normal, served title still `Inference Co‑op`.
### Migrations were rehearsed, not merely run
Before the window, the live gateway database was restored from the 2026-11-08 stopgap
snapshot into a local PostgreSQL **18.6** (prod's exact version) and the image's own
entrypoint (`python -m prisma migrate deploy`) was run against it: **185 → 221 applied in
12 s**, row counts identical (210 keys / 111 teams / 138,422 spend rows), **no table lost**
(90 → 110, the 20 additions all new-feature tables). That is what made `--no-backup`
acceptable on the real box.
### Not rehearsed
The LibreChat leg — a second chat app means ~2 GB of data plus its Meili/rag stack, for a
change that is tooling-only (Node/uv; LibreChat upstream unchanged). Mitigation is the
one-command rollback: `./release.sh pin 20261031-2a98892 && ./release.sh deploy`.