ops-docs/incidents/2026-11-05-portal-alltime-spend-zero.md

Incident 2026-11-05 — the portal's copied LiteLLM DB credential rotted after the gateway restore; all-time spend silently read $0

Companion to 2026-11-05-gateway-addon-loss.md (the deploy mistake that made the gateway restore necessary). This document covers the second-order failure that restore caused. No spend data was lost at any point; the failure was in the portal's read path, and it failed in a way that looked like clean data.

What broke

The portal aggregates all-time spend, model usage and the leaderboard by querying LiteLLM's Postgres directly, using a copy of the gateway's postgres addon credential: the portal app's LITELLM_DB_* env mirrors the gateway's CLOUDRON_POSTGRESQL_* runtime env. Two things then combined:

1. The copy rotted. Restoring a Cloudron app re-provisions its postgres addon and rotates its credential. When the gateway was restored (16:32:35 UTC, see timeline), the gateway got the new credential re-injected at runtime — but the portal's copy went stale at that exact moment.
2. The failure degraded silently. The portal's aggregation catches a DB auth failure, logs a WARNING, and degrades to an empty result — all-time spend read $0.00. The panel and member dashboard rendered that $0 while the cycle figures (read from LiteLLM's API, unaffected) still looked plausible, so nothing on screen looked broken. The daily digest had no spend-consistency check, so it also reported nothing.

Timeline (UTC, 2026-11-05 — restore/configure times from the platform eventlog)

• 16:29:01 — during the audit-fix deploys, the gateway received the portal's manifest; its postgres addon was dropped. (Sibling incident.)
• 16:32:35 — the gateway was restored from the pre-deploy snapshot to recover the sibling incident. The restore re-provisioned the postgres addon and rotated its credential — the moment this incident began.
• 16:32:35 → 18:51 — every portal aggregation query against LiteLLM's Postgres failed authentication and degraded to empty: panel + dashboard all-time spend read $0. LiteLLM itself kept serving and billing throughout; the portal log carried only WARNING lines ("all-time spend query failed"), easy to read as noise.
• 18:51:22 — the portal's LITELLM_DB_PASSWORD was re-synced from the gateway's runtime env (value staged file→argv through the Cloudron API; never printed, never committed). Aggregation restored immediately.
• Same day, verification: the source of truth (LiteLLM's own Postgres, read gateway-side) held 125,385 rows / $655.13 all-time; member teams 97 / $652.88 — no gap through the incident window, because the gateway's own writes were never affected. The member dashboard was verified restored (a sample member's all-time figure matched the source exactly).
• Same day, panel verification (final): panel running image code/admin-panel:20261105-0713551 (matches the repo pin; code marker confirmed in the container). Its data path is GET /api/overview → portal GET /admin/overview (server-side, X-Admin-Token: ADMIN_TOKEN) — replayed live from the panel container, HTTP 200. Displayed all-time total $653.08 with the co-op's own-accounts split $170.73, both matching an independent SQL read of the source exactly; four sample members' all-time figures matched the source to the cent. The full-table total ($655.33–655.41 while verifying — the ledger accrues live) decomposes exactly: members + operators $653.08 + agent teams $1.18 (shown in the panel's Agents section) + sandbox/unattributed rows $1.07 (displayed nowhere, accepted knowingly).

Fix

1. Immediate: re-sync the portal's LITELLM_DB_PASSWORD from the gateway's runtime env (18:51:22). Verified by re-reading the portal's aggregation against the source, not by the absence of log lines.
2. Standing guard (added the same day): the daily digest (cron job 95e5b8fece86) gained a SPEND CONSISTENCY section: it runs ~/.hermes/scripts/spend_consistency_check.py, which compares the portal's reported all-time spend (/admin/overview totals.total_all_time_spend + the agents[] rows) against LiteLLM's own SpendLogs total — read from the gateway app, the credential owner, so the source does not depend on the copied credential. A $0 portal figure, a missing field, a mismatch beyond a $5 tolerance (the deliberately-unattributed sandbox rows are ~$1), or an unreadable source is a FAILED check reported near the top of the digest — never as clean. All three failure branches were exercised (unreadable / $0 / mismatch) and fire with exit 1; the live comparison passes with both numbers printed. Do not substitute LiteLLM's /global/spend for the source: it returns a 30-day figure ($576 when the table holds ~$655), not all-time.
3. Practice (recorded here because it bit): after any app.restore on this box, diff the credential fingerprint (sha256[:8] of the value — never the value) between the owner app and every app holding a copy, and re-sync any that differ.

Architectural dependency — read before rotating anything

The portal does not hold its own LiteLLM DB credential. It aggregates from LiteLLM's Postgres using a COPY of the gateway's addon credential. They are two copies of one secret with no shared source, and Cloudron rotates the addon secret on every restore (and on any addon re-provision). Therefore:

• A gateway DB credential rotation or restore MUST be followed by syncing the portal's copy. The rotation itself announces nothing; the only symptoms are the portal's WARNING lines and $0 figures downstream.
• Long-term fix (flagged, deliberately not done now): both apps should source the credential from ONE place — a shared secret location — or the portal should not hold a DB credential at all (e.g. the gateway exposes a read-only aggregation endpoint the portal calls). Either removes the two-copies-of-one-secret failure mode.
• Until that lands, the digest's SPEND CONSISTENCY check is the drift alarm: if it ever reports FAILED with a $0 or mismatched portal figure, check the portal's LITELLM_DB_PASSWORD against the gateway's current addon credential first, before theorising about code.