incident: portal's copied LiteLLM DB credential rotted after the 2026-11-05 gateway restore; all-time spend silently read $0 — timeline, fix, the digest spend-consistency guard, and the two-copies-of-one-secret dependency flagged for the next rotation

incidents/2026-11-05-portal-alltime-spend-zero.md

@@ -0,0 +1,100 @@
+# Incident 2026-11-05 — the portal's copied LiteLLM DB credential rotted after the gateway restore; all-time spend silently read $0
+
+Companion to [`2026-11-05-gateway-addon-loss.md`](2026-11-05-gateway-addon-loss.md)
+(the deploy mistake that made the gateway restore necessary). This document
+covers the second-order failure that restore caused. No spend data was lost at
+any point; the failure was in the portal's **read path**, and it failed in a
+way that looked like clean data.
+
+## What broke
+
+The portal aggregates all-time spend, model usage and the leaderboard by
+querying LiteLLM's Postgres directly, using a **copy** of the gateway's
+postgres addon credential: the portal app's `LITELLM_DB_*` env mirrors the
+gateway's `CLOUDRON_POSTGRESQL_*` runtime env. Two things then combined:
+
+1. **The copy rotted.** Restoring a Cloudron app re-provisions its postgres
+   addon and **rotates its credential**. When the gateway was restored
+   (16:32:35 UTC, see timeline), the gateway got the new credential re-injected
+   at runtime — but the portal's copy went stale at that exact moment.
+2. **The failure degraded silently.** The portal's aggregation catches a DB
+   auth failure, logs a WARNING, and degrades to an empty result — all-time
+   spend read `$0.00`. The panel and member dashboard rendered that $0 while
+   the *cycle* figures (read from LiteLLM's API, unaffected) still looked
+   plausible, so nothing on screen looked broken. The daily digest had no
+   spend-consistency check, so it also reported nothing.
+
+## Timeline (UTC, 2026-11-05 — restore/configure times from the platform eventlog)
+
+- **16:29:01** — during the audit-fix deploys, the gateway received the
+  portal's manifest; its postgres addon was dropped. (Sibling incident.)
+- **16:32:35** — the gateway was restored from the pre-deploy snapshot to
+  recover the sibling incident. The restore re-provisioned the postgres addon
+  and **rotated its credential** — the moment this incident began.
+- **16:32:35 → 18:51** — every portal aggregation query against LiteLLM's
+  Postgres failed authentication and degraded to empty: panel + dashboard
+  all-time spend read $0. LiteLLM itself kept serving and billing throughout;
+  the portal log carried only WARNING lines ("all-time spend query failed"),
+  easy to read as noise.
+- **18:51:22** — the portal's `LITELLM_DB_PASSWORD` was re-synced from the
+  gateway's runtime env (value staged file→argv through the Cloudron API;
+  never printed, never committed). Aggregation restored immediately.
+- **Same day, verification:** the source of truth (LiteLLM's own Postgres, read
+  gateway-side) held 125,385 rows / **$655.13** all-time; member teams 97 /
+  $652.88 — **no gap through the incident window**, because the gateway's own
+  writes were never affected. The member dashboard was verified restored (a
+  sample member's all-time figure matched the source exactly).
+- **Same day, panel verification (final):** panel running image
+  `code/admin-panel:20261105-0713551` (matches the repo pin; code marker
+  confirmed in the container). Its data path is `GET /api/overview` → portal
+  `GET /admin/overview` (server-side, `X-Admin-Token: ADMIN_TOKEN`) — replayed
+  live from the panel container, HTTP 200. Displayed all-time total
+  **$653.08** with the co-op's own-accounts split **$170.73**, both matching an
+  independent SQL read of the source exactly; four sample members' all-time
+  figures matched the source to the cent. The full-table total ($655.33–655.41
+  while verifying — the ledger accrues live) decomposes exactly: members +
+  operators $653.08 + agent teams $1.18 (shown in the panel's Agents section) +
+  sandbox/unattributed rows $1.07 (displayed nowhere, accepted knowingly).
+
+## Fix
+
+1. **Immediate:** re-sync the portal's `LITELLM_DB_PASSWORD` from the gateway's
+   runtime env (18:51:22). Verified by re-reading the portal's aggregation
+   against the source, not by the absence of log lines.
+2. **Standing guard (added the same day):** the daily digest
+   (cron job `95e5b8fece86`) gained a **SPEND CONSISTENCY** section: it runs
+   `~/.hermes/scripts/spend_consistency_check.py`, which compares the portal's
+   reported all-time spend (`/admin/overview` `totals.total_all_time_spend` +
+   the `agents[]` rows) against LiteLLM's own SpendLogs total — read from the
+   **gateway** app, the credential owner, so the source does not depend on the
+   copied credential. A $0 portal figure, a missing field, a mismatch beyond a
+   $5 tolerance (the deliberately-unattributed sandbox rows are ~$1), or an
+   unreadable source is a FAILED check reported near the top of the digest —
+   never as clean. All three failure branches were exercised (unreadable / $0 /
+   mismatch) and fire with exit 1; the live comparison passes with both numbers
+   printed. Do not substitute LiteLLM's `/global/spend` for the source: it
+   returns a ~30-day figure (~$576 when the table holds ~$655), not all-time.
+3. **Practice (recorded here because it bit):** after any `app.restore` on this
+   box, diff the credential **fingerprint** (`sha256`[:8] of the value — never
+   the value) between the owner app and every app holding a copy, and re-sync
+   any that differ.
+
+## Architectural dependency — read before rotating anything
+
+**The portal does not hold its own LiteLLM DB credential. It aggregates from
+LiteLLM's Postgres using a COPY of the gateway's addon credential.** They are
+two copies of one secret with no shared source, and Cloudron rotates the addon
+secret on every restore (and on any addon re-provision). Therefore:
+
+- **A gateway DB credential rotation or restore MUST be followed by syncing the
+  portal's copy.** The rotation itself announces nothing; the only symptoms are
+  the portal's WARNING lines and $0 figures downstream.
+- **Long-term fix (flagged, deliberately not done now):** both apps should
+  source the credential from ONE place — a shared secret location — or the
+  portal should not hold a DB credential at all (e.g. the gateway exposes a
+  read-only aggregation endpoint the portal calls). Either removes the
+  two-copies-of-one-secret failure mode.
+- Until that lands, the digest's SPEND CONSISTENCY check is the drift alarm:
+  if it ever reports FAILED with a $0 or mismatched portal figure, check the
+  portal's `LITELLM_DB_PASSWORD` against the gateway's current addon credential
+  first, before theorising about code.