inference-bot
bb10d93412
Full mechanics, impact, detection and prevention in the file. Portal member data was never at risk.
96 lines
4.8 KiB
Markdown
96 lines
4.8 KiB
Markdown
# Incident 2026-11-05 — gateway/panel briefly deployed with the portal's manifest; gateway LiteLLM database wiped and restored
|
||
|
||
Self-reported by the audit-fix session (inferencebot), 2026-11-05 ~18:00 UTC.
|
||
Damage fully recovered the same hour. Nathan: see "What you need to know" at
|
||
the end.
|
||
|
||
## What happened
|
||
|
||
During the audit fix-order deploys, three apps (member-portal, admin-panel,
|
||
gateway) were updated in one shell loop that stayed `cd`'d into the
|
||
**member-portal repo**. `cloudron update` reads `CloudronManifest.json` from
|
||
the CURRENT DIRECTORY, and the portal manifest carries a `dockerImage` pin —
|
||
which wins over any `--image` argument. So:
|
||
|
||
1. The **admin-panel** app received the portal's manifest (portal image
|
||
`20261105-ca3b8db`, portal manifest id `coop.inference.member-portal`,
|
||
portal addons) for ~10 minutes.
|
||
2. The **gateway** app received the portal's manifest too. The portal manifest
|
||
has **no postgresql addon** → Cloudron **dropped the gateway's Postgres
|
||
database** (the LiteLLM key/team/spend store). The panel recovery and the
|
||
gateway image recovery ran ~10 and ~25 minutes later; the gateway's addon
|
||
was re-created on recovery as an EMPTY database with the same name.
|
||
3. Result: from ~16:29 to ~17:20 UTC every member key 401'd (184 keys gone
|
||
from the gateway's view), chat was down for the membership. The portal app
|
||
itself was never affected (its own manifest was the one deployed).
|
||
|
||
## Impact
|
||
|
||
- Gateway member inference: **down ~50 minutes** (16:29–17:20 UTC). Master-key
|
||
surfaces (/v1/models with the master key) worked, which made the outage look
|
||
partial at first glance.
|
||
- Gateway LiteLLM Postgres: keys 184→0, teams 100→1, SpendLogs 125k→17 at the
|
||
moment of detection.
|
||
- Spend rows for live traffic between the 16:28:12 backup and the database
|
||
wipe (a few minutes) are unrecoverable; everything after the wipe was
|
||
refused calls (no spend to lose).
|
||
- Panel: wrong image for ~10 min, no data loss (stateless; no durable addon
|
||
data).
|
||
- No member data (portal SQLite) was ever at risk — that lives in the portal
|
||
app's own localstorage volume, which was never touched.
|
||
|
||
## Detection
|
||
|
||
Post-deploy verification battery failed exactly where it should: the
|
||
member-scope chat call returned 401, the panel's container md5 matched the
|
||
PORTAL's main.py, and `/key/list` returned 0 keys. All three surfaced within
|
||
minutes of the recovery deploy.
|
||
|
||
## Resolution (all verified)
|
||
|
||
1. Redeployed panel and gateway **from their own repo directories** with their
|
||
correct manifests/images (16:47, 16:52 UTC).
|
||
2. Gateway database confirmed wiped (0 keys) → located the deploy-time app
|
||
snapshot `app_…_v1.0.0_5aad8953` (16:28:12 UTC — taken seconds before the
|
||
damage, contains the addon dump).
|
||
3. `cloudron restore --app <gateway> --backup …_5aad8953` (17:2x UTC) →
|
||
**184 keys, 100 teams, 125,106 spend rows restored**; /v1/models 200.
|
||
4. Re-applied the intended gateway image update `1.0.0-20261105-e161db9` from
|
||
the litellm repo dir (17:3x UTC) → verified DB still intact after the
|
||
redeploy (184/100/125,110 rows — the +4 spend rows are live traffic billing
|
||
again), member-scope chat completes for real (tinfoil/glm-5.3, tokens
|
||
billed), contactEmail + corrected start.sh live, audio_keepalive scheduler
|
||
restored.
|
||
5. Panel pin bumped to its own new image `20261105-0713551` and deployed from
|
||
its own repo dir; container md5 matches the repo; panel 200.
|
||
|
||
## Root cause
|
||
|
||
Running `cloudron update` for multiple apps from a shared shell whose working
|
||
directory belonged to a DIFFERENT app's repo. The CLI's behavior (CWD manifest
|
||
wins; `--image` silently ignored when a `dockerImage` pin exists) is documented
|
||
in the co-op skill, but nothing in the loop enforced "one app, one repo, one
|
||
CWD".
|
||
|
||
## Prevention (recorded in the deploy skill as a hard rule)
|
||
|
||
**Never batch `cloudron update` for different apps in one shell without
|
||
`cd`-ing to each app's own repo per command, and never run an update for an
|
||
app while CWD holds another app's `CloudronManifest.json`.** Verify after
|
||
EVERY deploy with both facts: the app's own `manifest.dockerImage` from the
|
||
API (never the CLI's output) and a container-content check (md5/marker).
|
||
Additionally: a manifest change that alters the `addons` block of an
|
||
app with a database addon drops/reprovisions that addon's data — treat addon
|
||
blocks as data-bearing, not metadata.
|
||
|
||
## What you need to know (Nathan)
|
||
|
||
- Chat was down for members ~50 minutes this afternoon (UTC); it is verified
|
||
working again with a real member key and real billed tokens.
|
||
- The gateway database was restored from the automatic pre-deploy snapshot;
|
||
only a few minutes' worth of spend rows from before the outage are missing.
|
||
No member key, team, or balance data was lost.
|
||
- The portal's member database was never touched.
|
||
- The audit fix order otherwise completed; a separate summary has the full
|
||
stage-by-stage record.
|