diff --git a/app-inventory.md b/app-inventory.md index 85d373a..140b7ba 100644 --- a/app-inventory.md +++ b/app-inventory.md @@ -20,17 +20,17 @@ does not hold it. | FQDN | App id | Label | Image (pin) | Install | |---|---|---|---|---| | `inference.coop` | `55f5f904-3233-44de-ad7b-5dc7c8533b75` | — | `cloudron/io.cloudron.surfer:202611021342220000` | store (Surfer — marketing site) | -| `chat.inference.coop` | `4023ccbf-d57a-4500-97df-14de9cf3516e` | AI Chat | `registry.inference.coop/co-op/librechat-custom:20261031-2a98892` | **custom** (LibreChat overlay) | -| `chat-legacy.inference.coop` | `50ae6c97-e851-4d17-b013-51ca7a4e5be0` | Legacy AI Chat | `cloudron/com.openwebui.cloudronapp:202611030420300000` | store (Open WebUI) | -| `portal.inference.coop` | `d4f05e4b-75c6-492e-9faf-7afb0712fc94` | — | `registry.inference.coop/code/member-portal:20261104-d076729` | **custom** | -| `dashboard.inference.coop` | `daf8750e-2683-4104-83ea-f77d1b4601e8` | Dashboard | `registry.inference.coop/code/member-dashboard:20261103-3453131` | **custom** | -| `panel.inference.coop` | `ae187071-7a43-4a6e-8a02-986609a66a1a` | Admin Panel | `local/coop.inference.admin-panel:0.1.0-1793634279942` | **custom** | -| `gateway.inference.coop` | `adababd4-2427-4bdd-9bf7-6bdad0302610` | LiteLLM | `registry.inference.coop/local/ai.coop.litellm:1.0.0-20261103-d6cfe78` | **custom** | +| `chat.inference.coop` | `4023ccbf-d57a-4500-97df-14de9cf3516e` | AI Chat | `registry.inference.coop/co-op/librechat-custom:20261106-c2aec74` | **custom** (LibreChat overlay) | +| `chat-legacy.inference.coop` | `50ae6c97-e851-4d17-b013-51ca7a4e5be0` | Legacy AI Chat | `cloudron/com.openwebui.cloudronapp:202611040420410000` | store (Open WebUI) | +| `portal.inference.coop` | `d4f05e4b-75c6-492e-9faf-7afb0712fc94` | — | `registry.inference.coop/code/member-portal:20261108-2954fe60` | **custom** | +| `dashboard.inference.coop` | `daf8750e-2683-4104-83ea-f77d1b4601e8` | Dashboard | `registry.inference.coop/code/member-dashboard:20261108-1e6e9e48` | **custom** | +| `panel.inference.coop` | `ae187071-7a43-4a6e-8a02-986609a66a1a` | Admin Panel | `registry.inference.coop/code/admin-panel:20261108-b23cc67` | **custom** | +| `gateway.inference.coop` | `adababd4-2427-4bdd-9bf7-6bdad0302610` | LiteLLM | `registry.inference.coop/local/ai.coop.litellm:20261109-c43d45c` | **custom** | | `matrix.inference.coop` | `aa6f4597-6206-4509-b32e-c89750b2c1d8` | Matrix Chat | `cloudron/org.matrix.synapse:202611030719520000` | store (Synapse) | | `git.inference.coop` | `c91c13aa-30ed-4af9-9599-7a87ab4f221d` | Code | `cloudron/io.gitea.cloudronapp:202611011719500000` | store (Gitea) | | `registry.inference.coop` | `c257ba3c-6737-4375-ad51-ea19586bf3f0` | — | `cloudron/io.cloudron.container-registry:202610221511140000` | store (container registry) | | `vault.inference.coop` | `cb838bcf-f2e2-4670-b117-dca04b0ee957` | — | `cloudron/com.github.bitwardenrs:202611020820270000` | store (Vaultwarden) | -| `forum.inference.coop` | `8b1a972c-08af-4e92-852c-d573c37b01ad` | Forum | `cloudron/org.loomio.cloudronapp:202611010321240000` | store (Loomio) | +| `forum.inference.coop` | `8b1a972c-08af-4e92-852c-d573c37b01ad` | Forum | `cloudron/org.loomio.cloudronapp:202611050321160000` | store (Loomio) | | `newsletter.inference.coop` | `eed2217c-b165-430e-bc29-7ccfef9b11e4` | — | `cloudron/app.listmonk.cloudronapp:202610220221130000` | store (Listmonk) | | `search.inference.coop` | `d7c73fff-ee42-4db0-98c0-4ce7a8ab35cc` | — | `cloudron/io.github.ascimoo.searx:202611012319290000` | store (SearXNG) | | `cloud.inference.coop` | `fd261d67-b790-424f-9e82-f74883ee8585` | — | `cloudron/com.nextcloud.cloudronapp:202610211606270000` | store (Nextcloud) | diff --git a/maintenance/2026-11-08-litellm-1.108.0-librechat-1.4.8.md b/maintenance/2026-11-08-litellm-1.108.0-librechat-1.4.8.md index 7bd1c7d..b6b6bd0 100644 --- a/maintenance/2026-11-08-litellm-1.108.0-librechat-1.4.8.md +++ b/maintenance/2026-11-08-litellm-1.108.0-librechat-1.4.8.md @@ -302,3 +302,57 @@ cd ~/repos/librechat-custom re-check; release.sh verify already retries fetches. - Anything else: halt at the failing step, note the exact command + output, roll back that app only, and leave the other app on its new version if it verified. + +## 8. Executed — 2026-11-09, 04:41–05:05 UTC (what happened, and where it differed) + +Order followed as planned: LiteLLM first, then LibreChat. Two deliberate changes: + +1. **Target 1.108.3, not 1.108.0** — latest stable, and it pins the *same* + `litellm-proxy-extras==0.4.110`, so the migration set is exactly the one validated here + (36 new, all additive). All ten build-patch anchors were re-validated against the real + 1.108.3 sources first; `patch_expose_cost_headers.py`'s 1.108.x anchor applied. +2. **A pre-built, pre-verified image was deployed, not an on-server build** — + `20261109-c43d45c`, built from `upgrade-v1.108.3` (cut on top of `main`, so it carries the + 2026-11-06/07 audio patches the original candidate predated) and exercised first as a + throwaway app (`gateway-dryrun.inference.coop`, its own Postgres/Redis, no member data). + +**The one thing that did not go as written — the pre-update snapshot OOMs.** +A default `cloudron update` died at `Backup - Snapshotting app gateway.inference.coop`: +task 1430 → `Task 1430 ran out of memory or terminated`, box log `Memory peak: 401.8M`, +`oom-kill`. Cloudron's app-task service runs with a **hard 400 MB cgroup limit** +(`startTask … "memoryLimit":400` from the system log), and this gateway's snapshot now +exceeds it. Production was never at risk — the app kept serving the old image; only +`installationState` went to `error`. +**Workaround used: `--no-backup`**, with the 04:41 pre-flight app backups as the undo — +which is what §2.1 is for. Next time: take the app backups first, then +`cloudron update … --no-backup`. + +**Also learned the hard way:** `--wait-timeout` is a GLOBAL option, so +`cloudron --wait-timeout 45m update …` works and `cloudron update --wait-timeout …` errors out. + +### What was verified live afterwards + +- **gateway** `…ai.coop.litellm:20261109-c43d45c` — litellm **1.108.3**, extras 0.4.110; + **all ten** patch markers present in the *running* container; `audio_keepalive.py` + executable; IoC check clean; migrations **185 → 221**, none failed or rolled back; + keys 210 / teams 111 unchanged; temp member key → `/v1/models` (15 models) → real + completion → **success SpendLogs row**; CORS preflight 200. Member traffic within ten + minutes: 73 success rows. +- **chat** `…/librechat-custom:20261106-c2aec74` (appstore 1.4.8 / LibreChat 0.8.8) — + `release.sh verify` all checks passed, key sync running every minute (108 member keys, + revision matches the repo), Meili resync normal, served title still `Inference Co‑op`. + +### Migrations were rehearsed, not merely run + +Before the window, the live gateway database was restored from the 2026-11-08 stopgap +snapshot into a local PostgreSQL **18.6** (prod's exact version) and the image's own +entrypoint (`python -m prisma migrate deploy`) was run against it: **185 → 221 applied in +12 s**, row counts identical (210 keys / 111 teams / 138,422 spend rows), **no table lost** +(90 → 110, the 20 additions all new-feature tables). That is what made `--no-backup` +acceptable on the real box. + +### Not rehearsed + +The LibreChat leg — a second chat app means ~2 GB of data plus its Meili/rag stack, for a +change that is tooling-only (Node/uv; LibreChat upstream unchanged). Mitigation is the +one-command rollback: `./release.sh pin 20261031-2a98892 && ./release.sh deploy`.