ops-docs/incidents/2026-11-05-gateway-addon-loss.md

Incident 2026-11-05 — gateway/panel briefly deployed with the portal's manifest; gateway LiteLLM database wiped and restored

Self-reported by the audit-fix session (inferencebot), 2026-11-05 ~18:00 UTC. Damage fully recovered the same hour. Nathan: see "What you need to know" at the end.

What happened

During the audit fix-order deploys, three apps (member-portal, admin-panel, gateway) were updated in one shell loop that stayed cd'd into the member-portal repo. cloudron update reads CloudronManifest.json from the CURRENT DIRECTORY, and the portal manifest carries a dockerImage pin — which wins over any --image argument. So:

1. The admin-panel app received the portal's manifest (portal image 20261105-ca3b8db, portal manifest id coop.inference.member-portal, portal addons) for ~10 minutes.
2. The gateway app received the portal's manifest too. The portal manifest has no postgresql addon → Cloudron dropped the gateway's Postgres database (the LiteLLM key/team/spend store). The panel recovery and the gateway image recovery ran ~10 and ~25 minutes later; the gateway's addon was re-created on recovery as an EMPTY database with the same name.
3. Result: from ~16:29 to ~17:20 UTC every member key 401'd (184 keys gone from the gateway's view), chat was down for the membership. The portal app itself was never affected (its own manifest was the one deployed).

Impact

• Gateway member inference: down ~50 minutes (16:29–17:20 UTC). Master-key surfaces (/v1/models with the master key) worked, which made the outage look partial at first glance.
• Gateway LiteLLM Postgres: keys 184→0, teams 100→1, SpendLogs 125k→17 at the moment of detection.
• Spend rows for live traffic between the 16:28:12 backup and the database wipe (a few minutes) are unrecoverable; everything after the wipe was refused calls (no spend to lose).
• Panel: wrong image for ~10 min, no data loss (stateless; no durable addon data).
• No member data (portal SQLite) was ever at risk — that lives in the portal app's own localstorage volume, which was never touched.

Detection

Post-deploy verification battery failed exactly where it should: the member-scope chat call returned 401, the panel's container md5 matched the PORTAL's main.py, and /key/list returned 0 keys. All three surfaced within minutes of the recovery deploy.

Resolution (all verified)

1. Redeployed panel and gateway from their own repo directories with their correct manifests/images (16:47, 16:52 UTC).
2. Gateway database confirmed wiped (0 keys) → located the deploy-time app snapshot app_…_v1.0.0_5aad8953 (16:28:12 UTC — taken seconds before the damage, contains the addon dump).
3. cloudron restore --app <gateway> --backup …_5aad8953 (17:2x UTC) → 184 keys, 100 teams, 125,106 spend rows restored; /v1/models 200.
4. Re-applied the intended gateway image update 1.0.0-20261105-e161db9 from the litellm repo dir (17:3x UTC) → verified DB still intact after the redeploy (184/100/125,110 rows — the +4 spend rows are live traffic billing again), member-scope chat completes for real (tinfoil/glm-5.3, tokens billed), contactEmail + corrected start.sh live, audio_keepalive scheduler restored.
5. Panel pin bumped to its own new image 20261105-0713551 and deployed from its own repo dir; container md5 matches the repo; panel 200.

Root cause

Running cloudron update for multiple apps from a shared shell whose working directory belonged to a DIFFERENT app's repo. The CLI's behavior (CWD manifest wins; --image silently ignored when a dockerImage pin exists) is documented in the co-op skill, but nothing in the loop enforced "one app, one repo, one CWD".

Prevention (recorded in the deploy skill as a hard rule)

Never batch cloudron update for different apps in one shell without cd-ing to each app's own repo per command, and never run an update for an app while CWD holds another app's CloudronManifest.json. Verify after EVERY deploy with both facts: the app's own manifest.dockerImage from the API (never the CLI's output) and a container-content check (md5/marker). Additionally: a manifest change that alters the addons block of an app with a database addon drops/reprovisions that addon's data — treat addon blocks as data-bearing, not metadata.

What you need to know (Nathan)

• Chat was down for members ~50 minutes this afternoon (UTC); it is verified working again with a real member key and real billed tokens.
• The gateway database was restored from the automatic pre-deploy snapshot; only a few minutes' worth of spend rows from before the outage are missing. No member key, team, or balance data was lost.
• The portal's member database was never touched.
• The audit fix order otherwise completed; a separate summary has the full stage-by-stage record.