diff --git a/incidents/2026-11-05-gateway-addon-loss.md b/incidents/2026-11-05-gateway-addon-loss.md new file mode 100644 index 0000000..eb99dc0 --- /dev/null +++ b/incidents/2026-11-05-gateway-addon-loss.md @@ -0,0 +1,95 @@ +# Incident 2026-11-05 — gateway/panel briefly deployed with the portal's manifest; gateway LiteLLM database wiped and restored + +Self-reported by the audit-fix session (inferencebot), 2026-11-05 ~18:00 UTC. +Damage fully recovered the same hour. Nathan: see "What you need to know" at +the end. + +## What happened + +During the audit fix-order deploys, three apps (member-portal, admin-panel, +gateway) were updated in one shell loop that stayed `cd`'d into the +**member-portal repo**. `cloudron update` reads `CloudronManifest.json` from +the CURRENT DIRECTORY, and the portal manifest carries a `dockerImage` pin — +which wins over any `--image` argument. So: + +1. The **admin-panel** app received the portal's manifest (portal image + `20261105-ca3b8db`, portal manifest id `coop.inference.member-portal`, + portal addons) for ~10 minutes. +2. The **gateway** app received the portal's manifest too. The portal manifest + has **no postgresql addon** → Cloudron **dropped the gateway's Postgres + database** (the LiteLLM key/team/spend store). The panel recovery and the + gateway image recovery ran ~10 and ~25 minutes later; the gateway's addon + was re-created on recovery as an EMPTY database with the same name. +3. Result: from ~16:29 to ~17:20 UTC every member key 401'd (184 keys gone + from the gateway's view), chat was down for the membership. The portal app + itself was never affected (its own manifest was the one deployed). + +## Impact + +- Gateway member inference: **down ~50 minutes** (16:29–17:20 UTC). Master-key + surfaces (/v1/models with the master key) worked, which made the outage look + partial at first glance. +- Gateway LiteLLM Postgres: keys 184→0, teams 100→1, SpendLogs 125k→17 at the + moment of detection. +- Spend rows for live traffic between the 16:28:12 backup and the database + wipe (a few minutes) are unrecoverable; everything after the wipe was + refused calls (no spend to lose). +- Panel: wrong image for ~10 min, no data loss (stateless; no durable addon + data). +- No member data (portal SQLite) was ever at risk — that lives in the portal + app's own localstorage volume, which was never touched. + +## Detection + +Post-deploy verification battery failed exactly where it should: the +member-scope chat call returned 401, the panel's container md5 matched the +PORTAL's main.py, and `/key/list` returned 0 keys. All three surfaced within +minutes of the recovery deploy. + +## Resolution (all verified) + +1. Redeployed panel and gateway **from their own repo directories** with their + correct manifests/images (16:47, 16:52 UTC). +2. Gateway database confirmed wiped (0 keys) → located the deploy-time app + snapshot `app_…_v1.0.0_5aad8953` (16:28:12 UTC — taken seconds before the + damage, contains the addon dump). +3. `cloudron restore --app --backup …_5aad8953` (17:2x UTC) → + **184 keys, 100 teams, 125,106 spend rows restored**; /v1/models 200. +4. Re-applied the intended gateway image update `1.0.0-20261105-e161db9` from + the litellm repo dir (17:3x UTC) → verified DB still intact after the + redeploy (184/100/125,110 rows — the +4 spend rows are live traffic billing + again), member-scope chat completes for real (tinfoil/glm-5.3, tokens + billed), contactEmail + corrected start.sh live, audio_keepalive scheduler + restored. +5. Panel pin bumped to its own new image `20261105-0713551` and deployed from + its own repo dir; container md5 matches the repo; panel 200. + +## Root cause + +Running `cloudron update` for multiple apps from a shared shell whose working +directory belonged to a DIFFERENT app's repo. The CLI's behavior (CWD manifest +wins; `--image` silently ignored when a `dockerImage` pin exists) is documented +in the co-op skill, but nothing in the loop enforced "one app, one repo, one +CWD". + +## Prevention (recorded in the deploy skill as a hard rule) + +**Never batch `cloudron update` for different apps in one shell without +`cd`-ing to each app's own repo per command, and never run an update for an +app while CWD holds another app's `CloudronManifest.json`.** Verify after +EVERY deploy with both facts: the app's own `manifest.dockerImage` from the +API (never the CLI's output) and a container-content check (md5/marker). +Additionally: a manifest change that alters the `addons` block of an +app with a database addon drops/reprovisions that addon's data — treat addon +blocks as data-bearing, not metadata. + +## What you need to know (Nathan) + +- Chat was down for members ~50 minutes this afternoon (UTC); it is verified + working again with a real member key and real billed tokens. +- The gateway database was restored from the automatic pre-deploy snapshot; + only a few minutes' worth of spend rows from before the outage are missing. + No member key, team, or balance data was lost. +- The portal's member database was never touched. +- The audit fix order otherwise completed; a separate summary has the full + stage-by-stage record.