101 lines
6.3 KiB
Markdown
101 lines
6.3 KiB
Markdown
# Incident 2026-11-05 — the portal's copied LiteLLM DB credential rotted after the gateway restore; all-time spend silently read $0
|
||
|
||
Companion to [`2026-11-05-gateway-addon-loss.md`](2026-11-05-gateway-addon-loss.md)
|
||
(the deploy mistake that made the gateway restore necessary). This document
|
||
covers the second-order failure that restore caused. No spend data was lost at
|
||
any point; the failure was in the portal's **read path**, and it failed in a
|
||
way that looked like clean data.
|
||
|
||
## What broke
|
||
|
||
The portal aggregates all-time spend, model usage and the leaderboard by
|
||
querying LiteLLM's Postgres directly, using a **copy** of the gateway's
|
||
postgres addon credential: the portal app's `LITELLM_DB_*` env mirrors the
|
||
gateway's `CLOUDRON_POSTGRESQL_*` runtime env. Two things then combined:
|
||
|
||
1. **The copy rotted.** Restoring a Cloudron app re-provisions its postgres
|
||
addon and **rotates its credential**. When the gateway was restored
|
||
(16:32:35 UTC, see timeline), the gateway got the new credential re-injected
|
||
at runtime — but the portal's copy went stale at that exact moment.
|
||
2. **The failure degraded silently.** The portal's aggregation catches a DB
|
||
auth failure, logs a WARNING, and degrades to an empty result — all-time
|
||
spend read `$0.00`. The panel and member dashboard rendered that $0 while
|
||
the *cycle* figures (read from LiteLLM's API, unaffected) still looked
|
||
plausible, so nothing on screen looked broken. The daily digest had no
|
||
spend-consistency check, so it also reported nothing.
|
||
|
||
## Timeline (UTC, 2026-11-05 — restore/configure times from the platform eventlog)
|
||
|
||
- **16:29:01** — during the audit-fix deploys, the gateway received the
|
||
portal's manifest; its postgres addon was dropped. (Sibling incident.)
|
||
- **16:32:35** — the gateway was restored from the pre-deploy snapshot to
|
||
recover the sibling incident. The restore re-provisioned the postgres addon
|
||
and **rotated its credential** — the moment this incident began.
|
||
- **16:32:35 → 18:51** — every portal aggregation query against LiteLLM's
|
||
Postgres failed authentication and degraded to empty: panel + dashboard
|
||
all-time spend read $0. LiteLLM itself kept serving and billing throughout;
|
||
the portal log carried only WARNING lines ("all-time spend query failed"),
|
||
easy to read as noise.
|
||
- **18:51:22** — the portal's `LITELLM_DB_PASSWORD` was re-synced from the
|
||
gateway's runtime env (value staged file→argv through the Cloudron API;
|
||
never printed, never committed). Aggregation restored immediately.
|
||
- **Same day, verification:** the source of truth (LiteLLM's own Postgres, read
|
||
gateway-side) held 125,385 rows / **$655.13** all-time; member teams 97 /
|
||
$652.88 — **no gap through the incident window**, because the gateway's own
|
||
writes were never affected. The member dashboard was verified restored (a
|
||
sample member's all-time figure matched the source exactly).
|
||
- **Same day, panel verification (final):** panel running image
|
||
`code/admin-panel:20261105-0713551` (matches the repo pin; code marker
|
||
confirmed in the container). Its data path is `GET /api/overview` → portal
|
||
`GET /admin/overview` (server-side, `X-Admin-Token: ADMIN_TOKEN`) — replayed
|
||
live from the panel container, HTTP 200. Displayed all-time total
|
||
**$653.08** with the co-op's own-accounts split **$170.73**, both matching an
|
||
independent SQL read of the source exactly; four sample members' all-time
|
||
figures matched the source to the cent. The full-table total ($655.33–655.41
|
||
while verifying — the ledger accrues live) decomposes exactly: members +
|
||
operators $653.08 + agent teams $1.18 (shown in the panel's Agents section) +
|
||
sandbox/unattributed rows $1.07 (displayed nowhere, accepted knowingly).
|
||
|
||
## Fix
|
||
|
||
1. **Immediate:** re-sync the portal's `LITELLM_DB_PASSWORD` from the gateway's
|
||
runtime env (18:51:22). Verified by re-reading the portal's aggregation
|
||
against the source, not by the absence of log lines.
|
||
2. **Standing guard (added the same day):** the daily digest
|
||
(cron job `95e5b8fece86`) gained a **SPEND CONSISTENCY** section: it runs
|
||
`~/.hermes/scripts/spend_consistency_check.py`, which compares the portal's
|
||
reported all-time spend (`/admin/overview` `totals.total_all_time_spend` +
|
||
the `agents[]` rows) against LiteLLM's own SpendLogs total — read from the
|
||
**gateway** app, the credential owner, so the source does not depend on the
|
||
copied credential. A $0 portal figure, a missing field, a mismatch beyond a
|
||
$5 tolerance (the deliberately-unattributed sandbox rows are ~$1), or an
|
||
unreadable source is a FAILED check reported near the top of the digest —
|
||
never as clean. All three failure branches were exercised (unreadable / $0 /
|
||
mismatch) and fire with exit 1; the live comparison passes with both numbers
|
||
printed. Do not substitute LiteLLM's `/global/spend` for the source: it
|
||
returns a ~30-day figure (~$576 when the table holds ~$655), not all-time.
|
||
3. **Practice (recorded here because it bit):** after any `app.restore` on this
|
||
box, diff the credential **fingerprint** (`sha256`[:8] of the value — never
|
||
the value) between the owner app and every app holding a copy, and re-sync
|
||
any that differ.
|
||
|
||
## Architectural dependency — read before rotating anything
|
||
|
||
**The portal does not hold its own LiteLLM DB credential. It aggregates from
|
||
LiteLLM's Postgres using a COPY of the gateway's addon credential.** They are
|
||
two copies of one secret with no shared source, and Cloudron rotates the addon
|
||
secret on every restore (and on any addon re-provision). Therefore:
|
||
|
||
- **A gateway DB credential rotation or restore MUST be followed by syncing the
|
||
portal's copy.** The rotation itself announces nothing; the only symptoms are
|
||
the portal's WARNING lines and $0 figures downstream.
|
||
- **Long-term fix (flagged, deliberately not done now):** both apps should
|
||
source the credential from ONE place — a shared secret location — or the
|
||
portal should not hold a DB credential at all (e.g. the gateway exposes a
|
||
read-only aggregation endpoint the portal calls). Either removes the
|
||
two-copies-of-one-secret failure mode.
|
||
- Until that lands, the digest's SPEND CONSISTENCY check is the drift alarm:
|
||
if it ever reports FAILED with a $0 or mismatched portal figure, check the
|
||
portal's `LITELLM_DB_PASSWORD` against the gateway's current addon credential
|
||
first, before theorising about code.
|