runbook: re-check the upgrade window against live state — correct the rollback tag, mark the prebuilt candidate stale, extend the marker list to ten, and flag the failing backups as the open risk

maintenance/2026-11-08-litellm-1.108.0-librechat-1.4.8.md

@@ -9,7 +9,7 @@ this app pair has a history of deploys that look successful while changing nothi
 
 | App | Candidate | Rollback (currently deployed) |
 |---|---|---|
-| LiteLLM gateway (`adababd4-2427-4bdd-9bf7-6bdad0302610`, gateway.inference.coop) | branch `upgrade-v1.108.0` in `code/litellm`; prebuilt image `registry.inference.coop/local/ai.coop.litellm:1.0.0-20261106-9e25578` | image `registry.inference.coop/local/ai.coop.litellm:1.0.0-20261105-e161db9` (verified present in the registry) |
+| LiteLLM gateway (`adababd4-2427-4bdd-9bf7-6bdad0302610`, gateway.inference.coop) | branch `upgrade-v1.108.0` in `code/litellm`; prebuilt image `registry.inference.coop/local/ai.coop.litellm:1.0.0-20261106-9e25578` (STALE — see §0 and §3.1) | image `registry.inference.coop/local/ai.coop.litellm:20261107-201000-407e751e2` — the tag running tonight, verified present in the registry. `1.0.0-20261105-e161db9` is two deploys older and is NOT the rollback target |
 | LibreChat chat (`4023ccbf-d57a-4500-97df-14de9cf3516e`, chat.inference.coop) | branch `bump-appstore-1.4.8` in `co-op/librechat-custom`; prebuilt overlay `registry.inference.coop/co-op/librechat-custom:20261106-c2aec74` | overlay `registry.inference.coop/co-op/librechat-custom:20261031-2a98892` (verified present) |
 
 **Prep evidence (what was already verified, so Sunday only re-confirms it):**
@@ -44,6 +44,38 @@ this app pair has a history of deploys that look successful while changing nothi
   image — **no overlay porting needed**; the candidate overlay built with all
   markers (UI strings patched in 3 files, index/manifest branding applied).
 
+## 0. State re-check — 2026-11-09 04:15 UTC (read live before the window)
+
+Verified against the live box on the night of the window. Anything here that
+contradicts the prep notes above is newer and wins.
+
+- **Gateway:** image `registry.inference.coop/local/ai.coop.litellm:20261107-201000-407e751e2`
+  (deployed 2026-11-07 20:10 UTC), in-container version **1.103.2**, 8 expected models
+  + extras, healthy. **The rollback target is therefore THIS tag, not
+  `1.0.0-20261105-e161db9`** (see §3.3).
+- **Chat:** overlay `registry.inference.coop/co-op/librechat-custom:20261031-2a98892`,
+  appstore 1.4.5, healthy. `master` is 2 commits ahead of `bump-appstore-1.4.8`, but both
+  touch only `librechat.yaml` — a persistent-volume file that is NOT baked into the image,
+  so pinning the Nov-06 candidate does not revert them.
+- **Registry:** all four tags this plan needs are present —
+  `local/ai.coop.litellm:{1.0.0-20261106-9e25578, 20261107-201000-407e751e2}` and
+  `co-op/librechat-custom:{20261106-c2aec74, 20261031-2a98892}` (listed from the registry
+  app's own storage).
+- **`upgrade-v1.108.0` is 5 commits BEHIND `main`.** main gained
+  `patch_transcription_response_format.py`, `patch_audio_default_mp3.py` and
+  `audio_keepalive.py` (Dockerfile RUN/chmod lines) plus the STT/embedding model additions
+  on 2026-11-06/07, all after the candidate branch was cut. Merging the branch into main
+  (as §3 does) carries both sides — but the **prebuilt candidate image does NOT contain
+  them** (see the §3.1 fallback note and the extended marker list).
+- **Version target:** the branch pins **1.108.0**; latest stable upstream is **1.108.3**
+  (2026-11-07). The prep validates 1.108.0 only — ship 1.108.0 tonight unless someone
+  re-runs the anchor/migration validation against 1.108.3 first.
+- **Backups are NOT healthy.** Last successful Cloudron backup: **2026-11-08 21:18 UTC**
+  (box, encrypted v2, state normal). The scheduled **2026-11-09 04:00 UTC run FAILED**
+  (`tarPack pipeline error … aborted` — the same signature as the Infomaniak S3 quota
+  rejections of 2026-11-06/07/08). The pre-flight backups in §2.1 are this window's undo
+  and must be proven to land before anything is deployed.
+
 ## 1. Order: LiteLLM first, then LibreChat
 
 LibreChat's post-upgrade checks (key sync, a real chat completion) run **through the
@@ -64,6 +96,12 @@ CR backup create --app 4023ccbf-d57a-4500-97df-14de9cf3516e
 CR backup list --app adababd4-2427-4bdd-9bf7-6bdad0302610 | tail -3   # note newest id
 CR backup list --app 4023ccbf-d57a-4500-97df-14de9cf3516e | tail -3
 
+# 2.1b PROVE the backup landed BEFORE deploying. The backup pipeline is failing on the
+#      Infomaniak S3 plan quota (last success 2026-11-08 21:18 UTC; the 04:00 UTC run on
+#      2026-11-09 failed again). Re-run `CR backup list --app <id>` and require a NEW id
+#      with state normal — an unchanged newest id, or no new entry, means this window has
+#      NO undo. Decide before deploying, not after.
+
 # 2.2 Record the member-visible baseline for after: number of keys and teams
 #     (use the MASTER key — it is in /app/data/.master_key inside the gateway:
 #      CR exec --app gateway.inference.coop -- cat /app/data/.master_key
@@ -87,7 +125,7 @@ git log --oneline -1                     # note the sha
 CR update --app adababd4-2427-4bdd-9bf7-6bdad0302610
 ```
 
-- The build output must contain **all eight** patch markers (each is one line in the
+- The build output must contain **all TEN** patch markers (each is one line in the
   build log; a missing line = that patch silently did not apply — STOP and check):
 
   1. `INFO: patched budget-exceeded messages in …/auth_checks.py`
@@ -98,15 +136,25 @@ CR update --app adababd4-2427-4bdd-9bf7-6bdad0302610
   6. `expose-cost-headers: appended 20 cost header names to expose_headers …`
   7. `[coop-patch] applied: passthrough-key-access -> …/route_checks.py`
   8. `[coop-patch] applied: passthrough-token-cost -> …/success_handler.py`
+  9. `[coop-patch] applied skip-verbose_json to …` (patch_transcription_response_format.py)
+  10. `[coop-patch] applied TTS default mp3 to …` (patch_audio_default_mp3.py)
+
+- `audio_keepalive.py` must be copied and `chmod +x`'d in the build (start.sh calls it for
+  the TTS warm path). Markers 9–10 and this file are the 2026-11-06/07 audio work; missing
+  any of them means that work silently reverts to the pre-Nov-06 behaviour.
 
 - **Boot**: LiteLLM runs `prisma migrate deploy` (the 36 additive migrations) before
   binding port 4000. Expect `502`/`ECONNREFUSED` on health checks for a few minutes
   (10–20 in the bad case). **Never restart mid-wait** — a restart resets the
   migration clock. Once `/health/liveliness` returns 200, continue immediately
   ("up" ≠ "migrated"; the checks below are the proof).
-- If the on-server build flakes for any reason, fall back to the pre-validated
+- **DO NOT use the prebuilt candidate as a flake fallback tonight.** It exists
-  image: `CR update --app adababd4-2427-4bdd-9bf7-6bdad0302610 --image registry.inference.coop/local/ai.coop.litellm:1.0.0-20261106-9e25578`
+  (`…ai.coop.litellm:1.0.0-20261106-9e25578`, and `--image` would work because the repo
-  (works precisely because the repo manifest carries no `dockerImage` field).
+  manifest carries no `dockerImage` field), but it was built from the branch tree and
+  therefore predates `patch_transcription_response_format.py`, `patch_audio_default_mp3.py`,
+  `audio_keepalive.py` and the STT/embedding model additions. Deploying it reverts the
+  2026-11-06/07 audio and model work. If the on-server build flakes, fix the build or build
+  the merged tree — the candidate image is not usable tonight.
 
 3.2 **Verify (in order — a failing early step means stop, not continue):**
 
@@ -159,7 +207,7 @@ curl -sS -H "Authorization: Bearer $CLOUDRON_TOKEN" \
 
 ```bash
 CR update --app adababd4-2427-4bdd-9bf7-6bdad0302610 \
-  --image registry.inference.coop/local/ai.coop.litellm:1.0.0-20261105-e161db9
+  --image registry.inference.coop/local/ai.coop.litellm:20261107-201000-407e751e2
 ```
 
 If member keys 401 after any image change, the schema is the suspect: read the
@@ -235,6 +283,15 @@ cd ~/repos/librechat-custom
    direct-DB write is the known, verified workaround; it still works on 1.108.0.
 6. After both apps are up, `app-inventory.md` in this repo should be updated with
    the new pins in the same session (it mirrors live state).
+7. **The prebuilt candidate is not the tree you are deploying.** The branch was cut five
+   commits behind `main`; the merge supplies the rest, the prebuilt image does not. Verify
+   the TEN build markers of §3.1 inside the running container, not the old eight.
+8. **A stale rollback reference is worse than none** — the runbook pointed at a Nov-05
+   image while the box was running a Nov-07 build, so "roll back" would have silently
+   dropped two days of member-visible fixes. Re-read the running tag (§0/§2.2) immediately
+   before any rollback, and roll back to what was actually running.
+9. **The backup pipeline is itself an open incident.** Assume no undo until a pre-flight
+   backup shows `state: normal` with a new id.
 
 ## 7. If something looks wrong