incident: 2026-11-05 gateway addon loss during multi-app deploy — wiped LiteLLM DB, restored from the pre-deploy snapshot the same hour

Full mechanics, impact, detection and prevention in the file. Portal
member data was never at risk.

incidents/2026-11-05-gateway-addon-loss.md

@@ -0,0 +1,95 @@
+# Incident 2026-11-05 — gateway/panel briefly deployed with the portal's manifest; gateway LiteLLM database wiped and restored
+
+Self-reported by the audit-fix session (inferencebot), 2026-11-05 ~18:00 UTC.
+Damage fully recovered the same hour. Nathan: see "What you need to know" at
+the end.
+
+## What happened
+
+During the audit fix-order deploys, three apps (member-portal, admin-panel,
+gateway) were updated in one shell loop that stayed `cd`'d into the
+**member-portal repo**. `cloudron update` reads `CloudronManifest.json` from
+the CURRENT DIRECTORY, and the portal manifest carries a `dockerImage` pin —
+which wins over any `--image` argument. So:
+
+1. The **admin-panel** app received the portal's manifest (portal image
+   `20261105-ca3b8db`, portal manifest id `coop.inference.member-portal`,
+   portal addons) for ~10 minutes.
+2. The **gateway** app received the portal's manifest too. The portal manifest
+   has **no postgresql addon** → Cloudron **dropped the gateway's Postgres
+   database** (the LiteLLM key/team/spend store). The panel recovery and the
+   gateway image recovery ran ~10 and ~25 minutes later; the gateway's addon
+   was re-created on recovery as an EMPTY database with the same name.
+3. Result: from ~16:29 to ~17:20 UTC every member key 401'd (184 keys gone
+   from the gateway's view), chat was down for the membership. The portal app
+   itself was never affected (its own manifest was the one deployed).
+
+## Impact
+
+- Gateway member inference: **down ~50 minutes** (16:29–17:20 UTC). Master-key
+  surfaces (/v1/models with the master key) worked, which made the outage look
+  partial at first glance.
+- Gateway LiteLLM Postgres: keys 184→0, teams 100→1, SpendLogs 125k→17 at the
+  moment of detection.
+- Spend rows for live traffic between the 16:28:12 backup and the database
+  wipe (a few minutes) are unrecoverable; everything after the wipe was
+  refused calls (no spend to lose).
+- Panel: wrong image for ~10 min, no data loss (stateless; no durable addon
+  data).
+- No member data (portal SQLite) was ever at risk — that lives in the portal
+  app's own localstorage volume, which was never touched.
+
+## Detection
+
+Post-deploy verification battery failed exactly where it should: the
+member-scope chat call returned 401, the panel's container md5 matched the
+PORTAL's main.py, and `/key/list` returned 0 keys. All three surfaced within
+minutes of the recovery deploy.
+
+## Resolution (all verified)
+
+1. Redeployed panel and gateway **from their own repo directories** with their
+   correct manifests/images (16:47, 16:52 UTC).
+2. Gateway database confirmed wiped (0 keys) → located the deploy-time app
+   snapshot `app_…_v1.0.0_5aad8953` (16:28:12 UTC — taken seconds before the
+   damage, contains the addon dump).
+3. `cloudron restore --app <gateway> --backup …_5aad8953` (17:2x UTC) →
+   **184 keys, 100 teams, 125,106 spend rows restored**; /v1/models 200.
+4. Re-applied the intended gateway image update `1.0.0-20261105-e161db9` from
+   the litellm repo dir (17:3x UTC) → verified DB still intact after the
+   redeploy (184/100/125,110 rows — the +4 spend rows are live traffic billing
+   again), member-scope chat completes for real (tinfoil/glm-5.3, tokens
+   billed), contactEmail + corrected start.sh live, audio_keepalive scheduler
+   restored.
+5. Panel pin bumped to its own new image `20261105-0713551` and deployed from
+   its own repo dir; container md5 matches the repo; panel 200.
+
+## Root cause
+
+Running `cloudron update` for multiple apps from a shared shell whose working
+directory belonged to a DIFFERENT app's repo. The CLI's behavior (CWD manifest
+wins; `--image` silently ignored when a `dockerImage` pin exists) is documented
+in the co-op skill, but nothing in the loop enforced "one app, one repo, one
+CWD".
+
+## Prevention (recorded in the deploy skill as a hard rule)
+
+**Never batch `cloudron update` for different apps in one shell without
+`cd`-ing to each app's own repo per command, and never run an update for an
+app while CWD holds another app's `CloudronManifest.json`.** Verify after
+EVERY deploy with both facts: the app's own `manifest.dockerImage` from the
+API (never the CLI's output) and a container-content check (md5/marker).
+Additionally: a manifest change that alters the `addons` block of an
+app with a database addon drops/reprovisions that addon's data — treat addon
+blocks as data-bearing, not metadata.
+
+## What you need to know (Nathan)
+
+- Chat was down for members ~50 minutes this afternoon (UTC); it is verified
+  working again with a real member key and real billed tokens.
+- The gateway database was restored from the automatic pre-deploy snapshot;
+  only a few minutes' worth of spend rows from before the outage are missing.
+  No member key, team, or balance data was lost.
+- The portal's member database was never touched.
+- The audit fix order otherwise completed; a separate summary has the full
+  stage-by-stage record.