incident: portal's copied LiteLLM DB credential rotted after the 2026-11-05 gateway restore; all-time spend silently read $0 — timeline, fix, the digest spend-consistency guard, and the two-copies-of-one-secret dependency flagged for the next rotation
This commit is contained in:
inference-bot committed
1 parent
bb10d93412
commit
fcd4ac6943
1 file changed
+100
@@ -0,0 +1,100 @@
|
|||||||
|
# Incident 2026-11-05 — the portal's copied LiteLLM DB credential rotted after the gateway restore; all-time spend silently read $0
|
||||||
|
|
||||||
|
Companion to [`2026-11-05-gateway-addon-loss.md`](2026-11-05-gateway-addon-loss.md)
|
||||||
|
(the deploy mistake that made the gateway restore necessary). This document
|
||||||
|
covers the second-order failure that restore caused. No spend data was lost at
|
||||||
|
any point; the failure was in the portal's **read path**, and it failed in a
|
||||||
|
way that looked like clean data.
|
||||||
|
|
||||||
|
## What broke
|
||||||
|
|
||||||
|
The portal aggregates all-time spend, model usage and the leaderboard by
|
||||||
|
querying LiteLLM's Postgres directly, using a **copy** of the gateway's
|
||||||
|
postgres addon credential: the portal app's `LITELLM_DB_*` env mirrors the
|
||||||
|
gateway's `CLOUDRON_POSTGRESQL_*` runtime env. Two things then combined:
|
||||||
|
|
||||||
|
1. **The copy rotted.** Restoring a Cloudron app re-provisions its postgres
|
||||||
|
addon and **rotates its credential**. When the gateway was restored
|
||||||
|
(16:32:35 UTC, see timeline), the gateway got the new credential re-injected
|
||||||
|
at runtime — but the portal's copy went stale at that exact moment.
|
||||||
|
2. **The failure degraded silently.** The portal's aggregation catches a DB
|
||||||
|
auth failure, logs a WARNING, and degrades to an empty result — all-time
|
||||||
|
spend read `$0.00`. The panel and member dashboard rendered that $0 while
|
||||||
|
the *cycle* figures (read from LiteLLM's API, unaffected) still looked
|
||||||
|
plausible, so nothing on screen looked broken. The daily digest had no
|
||||||
|
spend-consistency check, so it also reported nothing.
|
||||||
|
|
||||||
|
## Timeline (UTC, 2026-11-05 — restore/configure times from the platform eventlog)
|
||||||
|
|
||||||
|
- **16:29:01** — during the audit-fix deploys, the gateway received the
|
||||||
|
portal's manifest; its postgres addon was dropped. (Sibling incident.)
|
||||||
|
- **16:32:35** — the gateway was restored from the pre-deploy snapshot to
|
||||||
|
recover the sibling incident. The restore re-provisioned the postgres addon
|
||||||
|
and **rotated its credential** — the moment this incident began.
|
||||||
|
- **16:32:35 → 18:51** — every portal aggregation query against LiteLLM's
|
||||||
|
Postgres failed authentication and degraded to empty: panel + dashboard
|
||||||
|
all-time spend read $0. LiteLLM itself kept serving and billing throughout;
|
||||||
|
the portal log carried only WARNING lines ("all-time spend query failed"),
|
||||||
|
easy to read as noise.
|
||||||
|
- **18:51:22** — the portal's `LITELLM_DB_PASSWORD` was re-synced from the
|
||||||
|
gateway's runtime env (value staged file→argv through the Cloudron API;
|
||||||
|
never printed, never committed). Aggregation restored immediately.
|
||||||
|
- **Same day, verification:** the source of truth (LiteLLM's own Postgres, read
|
||||||
|
gateway-side) held 125,385 rows / **$655.13** all-time; member teams 97 /
|
||||||
|
$652.88 — **no gap through the incident window**, because the gateway's own
|
||||||
|
writes were never affected. The member dashboard was verified restored (a
|
||||||
|
sample member's all-time figure matched the source exactly).
|
||||||
|
- **Same day, panel verification (final):** panel running image
|
||||||
|
`code/admin-panel:20261105-0713551` (matches the repo pin; code marker
|
||||||
|
confirmed in the container). Its data path is `GET /api/overview` → portal
|
||||||
|
`GET /admin/overview` (server-side, `X-Admin-Token: ADMIN_TOKEN`) — replayed
|
||||||
|
live from the panel container, HTTP 200. Displayed all-time total
|
||||||
|
**$653.08** with the co-op's own-accounts split **$170.73**, both matching an
|
||||||
|
independent SQL read of the source exactly; four sample members' all-time
|
||||||
|
figures matched the source to the cent. The full-table total ($655.33–655.41
|
||||||
|
while verifying — the ledger accrues live) decomposes exactly: members +
|
||||||
|
operators $653.08 + agent teams $1.18 (shown in the panel's Agents section) +
|
||||||
|
sandbox/unattributed rows $1.07 (displayed nowhere, accepted knowingly).
|
||||||
|
|
||||||
|
## Fix
|
||||||
|
|
||||||
|
1. **Immediate:** re-sync the portal's `LITELLM_DB_PASSWORD` from the gateway's
|
||||||
|
runtime env (18:51:22). Verified by re-reading the portal's aggregation
|
||||||
|
against the source, not by the absence of log lines.
|
||||||
|
2. **Standing guard (added the same day):** the daily digest
|
||||||
|
(cron job `95e5b8fece86`) gained a **SPEND CONSISTENCY** section: it runs
|
||||||
|
`~/.hermes/scripts/spend_consistency_check.py`, which compares the portal's
|
||||||
|
reported all-time spend (`/admin/overview` `totals.total_all_time_spend` +
|
||||||
|
the `agents[]` rows) against LiteLLM's own SpendLogs total — read from the
|
||||||
|
**gateway** app, the credential owner, so the source does not depend on the
|
||||||
|
copied credential. A $0 portal figure, a missing field, a mismatch beyond a
|
||||||
|
$5 tolerance (the deliberately-unattributed sandbox rows are ~$1), or an
|
||||||
|
unreadable source is a FAILED check reported near the top of the digest —
|
||||||
|
never as clean. All three failure branches were exercised (unreadable / $0 /
|
||||||
|
mismatch) and fire with exit 1; the live comparison passes with both numbers
|
||||||
|
printed. Do not substitute LiteLLM's `/global/spend` for the source: it
|
||||||
|
returns a ~30-day figure (~$576 when the table holds ~$655), not all-time.
|
||||||
|
3. **Practice (recorded here because it bit):** after any `app.restore` on this
|
||||||
|
box, diff the credential **fingerprint** (`sha256`[:8] of the value — never
|
||||||
|
the value) between the owner app and every app holding a copy, and re-sync
|
||||||
|
any that differ.
|
||||||
|
|
||||||
|
## Architectural dependency — read before rotating anything
|
||||||
|
|
||||||
|
**The portal does not hold its own LiteLLM DB credential. It aggregates from
|
||||||
|
LiteLLM's Postgres using a COPY of the gateway's addon credential.** They are
|
||||||
|
two copies of one secret with no shared source, and Cloudron rotates the addon
|
||||||
|
secret on every restore (and on any addon re-provision). Therefore:
|
||||||
|
|
||||||
|
- **A gateway DB credential rotation or restore MUST be followed by syncing the
|
||||||
|
portal's copy.** The rotation itself announces nothing; the only symptoms are
|
||||||
|
the portal's WARNING lines and $0 figures downstream.
|
||||||
|
- **Long-term fix (flagged, deliberately not done now):** both apps should
|
||||||
|
source the credential from ONE place — a shared secret location — or the
|
||||||
|
portal should not hold a DB credential at all (e.g. the gateway exposes a
|
||||||
|
read-only aggregation endpoint the portal calls). Either removes the
|
||||||
|
two-copies-of-one-secret failure mode.
|
||||||
|
- Until that lands, the digest's SPEND CONSISTENCY check is the drift alarm:
|
||||||
|
if it ever reports FAILED with a $0 or mismatched portal figure, check the
|
||||||
|
portal's `LITELLM_DB_PASSWORD` against the gateway's current addon credential
|
||||||
|
first, before theorising about code.
|
||||||
Reference in new issue
Block a user