# Incident 2026-11-05 — the portal's copied LiteLLM DB credential rotted after the gateway restore; all-time spend silently read $0 Companion to [`2026-11-05-gateway-addon-loss.md`](2026-11-05-gateway-addon-loss.md) (the deploy mistake that made the gateway restore necessary). This document covers the second-order failure that restore caused. No spend data was lost at any point; the failure was in the portal's **read path**, and it failed in a way that looked like clean data. ## What broke The portal aggregates all-time spend, model usage and the leaderboard by querying LiteLLM's Postgres directly, using a **copy** of the gateway's postgres addon credential: the portal app's `LITELLM_DB_*` env mirrors the gateway's `CLOUDRON_POSTGRESQL_*` runtime env. Two things then combined: 1. **The copy rotted.** Restoring a Cloudron app re-provisions its postgres addon and **rotates its credential**. When the gateway was restored (16:32:35 UTC, see timeline), the gateway got the new credential re-injected at runtime — but the portal's copy went stale at that exact moment. 2. **The failure degraded silently.** The portal's aggregation catches a DB auth failure, logs a WARNING, and degrades to an empty result — all-time spend read `$0.00`. The panel and member dashboard rendered that $0 while the *cycle* figures (read from LiteLLM's API, unaffected) still looked plausible, so nothing on screen looked broken. The daily digest had no spend-consistency check, so it also reported nothing. ## Timeline (UTC, 2026-11-05 — restore/configure times from the platform eventlog) - **16:29:01** — during the audit-fix deploys, the gateway received the portal's manifest; its postgres addon was dropped. (Sibling incident.) - **16:32:35** — the gateway was restored from the pre-deploy snapshot to recover the sibling incident. The restore re-provisioned the postgres addon and **rotated its credential** — the moment this incident began. - **16:32:35 → 18:51** — every portal aggregation query against LiteLLM's Postgres failed authentication and degraded to empty: panel + dashboard all-time spend read $0. LiteLLM itself kept serving and billing throughout; the portal log carried only WARNING lines ("all-time spend query failed"), easy to read as noise. - **18:51:22** — the portal's `LITELLM_DB_PASSWORD` was re-synced from the gateway's runtime env (value staged file→argv through the Cloudron API; never printed, never committed). Aggregation restored immediately. - **Same day, verification:** the source of truth (LiteLLM's own Postgres, read gateway-side) held 125,385 rows / **$655.13** all-time; member teams 97 / $652.88 — **no gap through the incident window**, because the gateway's own writes were never affected. The member dashboard was verified restored (a sample member's all-time figure matched the source exactly). - **Same day, panel verification (final):** panel running image `code/admin-panel:20261105-0713551` (matches the repo pin; code marker confirmed in the container). Its data path is `GET /api/overview` → portal `GET /admin/overview` (server-side, `X-Admin-Token: ADMIN_TOKEN`) — replayed live from the panel container, HTTP 200. Displayed all-time total **$653.08** with the co-op's own-accounts split **$170.73**, both matching an independent SQL read of the source exactly; four sample members' all-time figures matched the source to the cent. The full-table total ($655.33–655.41 while verifying — the ledger accrues live) decomposes exactly: members + operators $653.08 + agent teams $1.18 (shown in the panel's Agents section) + sandbox/unattributed rows $1.07 (displayed nowhere, accepted knowingly). ## Fix 1. **Immediate:** re-sync the portal's `LITELLM_DB_PASSWORD` from the gateway's runtime env (18:51:22). Verified by re-reading the portal's aggregation against the source, not by the absence of log lines. 2. **Standing guard (added the same day):** the daily digest (cron job `95e5b8fece86`) gained a **SPEND CONSISTENCY** section: it runs `~/.hermes/scripts/spend_consistency_check.py`, which compares the portal's reported all-time spend (`/admin/overview` `totals.total_all_time_spend` + the `agents[]` rows) against LiteLLM's own SpendLogs total — read from the **gateway** app, the credential owner, so the source does not depend on the copied credential. A $0 portal figure, a missing field, a mismatch beyond a $5 tolerance (the deliberately-unattributed sandbox rows are ~$1), or an unreadable source is a FAILED check reported near the top of the digest — never as clean. All three failure branches were exercised (unreadable / $0 / mismatch) and fire with exit 1; the live comparison passes with both numbers printed. Do not substitute LiteLLM's `/global/spend` for the source: it returns a ~30-day figure (~$576 when the table holds ~$655), not all-time. 3. **Practice (recorded here because it bit):** after any `app.restore` on this box, diff the credential **fingerprint** (`sha256`[:8] of the value — never the value) between the owner app and every app holding a copy, and re-sync any that differ. ## Architectural dependency — read before rotating anything **The portal does not hold its own LiteLLM DB credential. It aggregates from LiteLLM's Postgres using a COPY of the gateway's addon credential.** They are two copies of one secret with no shared source, and Cloudron rotates the addon secret on every restore (and on any addon re-provision). Therefore: - **A gateway DB credential rotation or restore MUST be followed by syncing the portal's copy.** The rotation itself announces nothing; the only symptoms are the portal's WARNING lines and $0 figures downstream. - **Long-term fix (flagged, deliberately not done now):** both apps should source the credential from ONE place — a shared secret location — or the portal should not hold a DB credential at all (e.g. the gateway exposes a read-only aggregation endpoint the portal calls). Either removes the two-copies-of-one-secret failure mode. - Until that lands, the digest's SPEND CONSISTENCY check is the drift alarm: if it ever reports FAILED with a $0 or mismatched portal figure, check the portal's `LITELLM_DB_PASSWORD` against the gateway's current addon credential first, before theorising about code.