diff --git a/incidents/2026-11-05-portal-alltime-spend-zero.md b/incidents/2026-11-05-portal-alltime-spend-zero.md new file mode 100644 index 0000000..d7ac6b2 --- /dev/null +++ b/incidents/2026-11-05-portal-alltime-spend-zero.md @@ -0,0 +1,100 @@ +# Incident 2026-11-05 — the portal's copied LiteLLM DB credential rotted after the gateway restore; all-time spend silently read $0 + +Companion to [`2026-11-05-gateway-addon-loss.md`](2026-11-05-gateway-addon-loss.md) +(the deploy mistake that made the gateway restore necessary). This document +covers the second-order failure that restore caused. No spend data was lost at +any point; the failure was in the portal's **read path**, and it failed in a +way that looked like clean data. + +## What broke + +The portal aggregates all-time spend, model usage and the leaderboard by +querying LiteLLM's Postgres directly, using a **copy** of the gateway's +postgres addon credential: the portal app's `LITELLM_DB_*` env mirrors the +gateway's `CLOUDRON_POSTGRESQL_*` runtime env. Two things then combined: + +1. **The copy rotted.** Restoring a Cloudron app re-provisions its postgres + addon and **rotates its credential**. When the gateway was restored + (16:32:35 UTC, see timeline), the gateway got the new credential re-injected + at runtime — but the portal's copy went stale at that exact moment. +2. **The failure degraded silently.** The portal's aggregation catches a DB + auth failure, logs a WARNING, and degrades to an empty result — all-time + spend read `$0.00`. The panel and member dashboard rendered that $0 while + the *cycle* figures (read from LiteLLM's API, unaffected) still looked + plausible, so nothing on screen looked broken. The daily digest had no + spend-consistency check, so it also reported nothing. + +## Timeline (UTC, 2026-11-05 — restore/configure times from the platform eventlog) + +- **16:29:01** — during the audit-fix deploys, the gateway received the + portal's manifest; its postgres addon was dropped. (Sibling incident.) +- **16:32:35** — the gateway was restored from the pre-deploy snapshot to + recover the sibling incident. The restore re-provisioned the postgres addon + and **rotated its credential** — the moment this incident began. +- **16:32:35 → 18:51** — every portal aggregation query against LiteLLM's + Postgres failed authentication and degraded to empty: panel + dashboard + all-time spend read $0. LiteLLM itself kept serving and billing throughout; + the portal log carried only WARNING lines ("all-time spend query failed"), + easy to read as noise. +- **18:51:22** — the portal's `LITELLM_DB_PASSWORD` was re-synced from the + gateway's runtime env (value staged file→argv through the Cloudron API; + never printed, never committed). Aggregation restored immediately. +- **Same day, verification:** the source of truth (LiteLLM's own Postgres, read + gateway-side) held 125,385 rows / **$655.13** all-time; member teams 97 / + $652.88 — **no gap through the incident window**, because the gateway's own + writes were never affected. The member dashboard was verified restored (a + sample member's all-time figure matched the source exactly). +- **Same day, panel verification (final):** panel running image + `code/admin-panel:20261105-0713551` (matches the repo pin; code marker + confirmed in the container). Its data path is `GET /api/overview` → portal + `GET /admin/overview` (server-side, `X-Admin-Token: ADMIN_TOKEN`) — replayed + live from the panel container, HTTP 200. Displayed all-time total + **$653.08** with the co-op's own-accounts split **$170.73**, both matching an + independent SQL read of the source exactly; four sample members' all-time + figures matched the source to the cent. The full-table total ($655.33–655.41 + while verifying — the ledger accrues live) decomposes exactly: members + + operators $653.08 + agent teams $1.18 (shown in the panel's Agents section) + + sandbox/unattributed rows $1.07 (displayed nowhere, accepted knowingly). + +## Fix + +1. **Immediate:** re-sync the portal's `LITELLM_DB_PASSWORD` from the gateway's + runtime env (18:51:22). Verified by re-reading the portal's aggregation + against the source, not by the absence of log lines. +2. **Standing guard (added the same day):** the daily digest + (cron job `95e5b8fece86`) gained a **SPEND CONSISTENCY** section: it runs + `~/.hermes/scripts/spend_consistency_check.py`, which compares the portal's + reported all-time spend (`/admin/overview` `totals.total_all_time_spend` + + the `agents[]` rows) against LiteLLM's own SpendLogs total — read from the + **gateway** app, the credential owner, so the source does not depend on the + copied credential. A $0 portal figure, a missing field, a mismatch beyond a + $5 tolerance (the deliberately-unattributed sandbox rows are ~$1), or an + unreadable source is a FAILED check reported near the top of the digest — + never as clean. All three failure branches were exercised (unreadable / $0 / + mismatch) and fire with exit 1; the live comparison passes with both numbers + printed. Do not substitute LiteLLM's `/global/spend` for the source: it + returns a ~30-day figure (~$576 when the table holds ~$655), not all-time. +3. **Practice (recorded here because it bit):** after any `app.restore` on this + box, diff the credential **fingerprint** (`sha256`[:8] of the value — never + the value) between the owner app and every app holding a copy, and re-sync + any that differ. + +## Architectural dependency — read before rotating anything + +**The portal does not hold its own LiteLLM DB credential. It aggregates from +LiteLLM's Postgres using a COPY of the gateway's addon credential.** They are +two copies of one secret with no shared source, and Cloudron rotates the addon +secret on every restore (and on any addon re-provision). Therefore: + +- **A gateway DB credential rotation or restore MUST be followed by syncing the + portal's copy.** The rotation itself announces nothing; the only symptoms are + the portal's WARNING lines and $0 figures downstream. +- **Long-term fix (flagged, deliberately not done now):** both apps should + source the credential from ONE place — a shared secret location — or the + portal should not hold a DB credential at all (e.g. the gateway exposes a + read-only aggregation endpoint the portal calls). Either removes the + two-copies-of-one-secret failure mode. +- Until that lands, the digest's SPEND CONSISTENCY check is the drift alarm: + if it ever reports FAILED with a $0 or mismatched portal figure, check the + portal's `LITELLM_DB_PASSWORD` against the gateway's current addon credential + first, before theorising about code.