spend check: read the field LiteLLM actually sends (total_spend), and fail loudly either way
ci/woodpecker/push/woodpecker Pipeline was canceled

The first live run FAILED with api $0.00 over 306 keys against the portal's $935.35.
The disagreement was real, the verdict was right, and the cause was mine: LiteLLM's
/global/spend/keys returns each key's all-time spend as `total_spend`, not `spend`.
A checker that trusts its own parse then reports "the portal is wrong" — the exact
mirror of the failure this check exists to catch, which is why the guard now names
both possibilities when the independent side reads zero against real portal spend.

Parser accepts either field name; a regression test exercises the real function
against a stubbed httpx (the module attribute is stubbed for the other checks, so
the real one is captured at import). 24 checks; all 20 suites green.
This commit is contained in:
inference-admin committed 2026-11-13 20:43:41 +00:00
1 parent 1c1fced5ba
commit c0ef046b0d
2 files changed
+61 -4

No files matched your search

+21 -4
View File
@@ -8282,10 +8282,18 @@ async def litellm_api_total_spend() -> tuple[float, int]:
raise RuntimeError(f"LiteLLM spend API returned {type(rows).__name__}, not a list")
total = 0.0
for row in rows:
try:
total += float((row or {}).get("spend") or 0)
except (TypeError, ValueError):
continue
row = row or {}
# LiteLLM reports each key's all-time spend as `total_spend`; older builds
# called it `spend`. Reading only `spend` returned $0 for all 306 keys on
# 2026-11-13 — a failure that a naive checker reports as "the portal is wrong".
for field in ("total_spend", "spend"):
try:
val = float(row.get(field) or 0)
except (TypeError, ValueError):
val = 0.0
if val:
total += val
break
return round(total, 6), len(rows)
@@ -8326,6 +8334,15 @@ async def check_spend_consistency(notify: bool = True) -> dict:
elif api_keys == 0 and portal_rows > 0:
reason = (f"LiteLLM's spend API returned no keys while the portal read {portal_rows} "
f"SpendLogs rows — the comparison has no independent side")
elif api_total == 0 and portal_total > 0:
# Both directions matter: an independent side that reads zero while the portal
# reads real spend is a disagreement, and on 2026-11-13 it was OUR parse (the
# API calls the field `total_spend`, not `spend`). Say both possibilities.
reason = (f"LiteLLM's spend API reports ${api_total:.2f} across {api_keys} keys while "
f"the portal reads ${portal_total:.2f} over {portal_rows} rows — either "
f"LiteLLM's own accounting is broken or this check is reading the wrong "
f"field (it must accept `total_spend`); do not believe either side until "
f"that is settled")
elif abs(delta) > SPEND_CHECK_TOLERANCE:
reason = (f"portal SpendLogs total ${portal_total:.2f} ({portal_rows} rows) vs LiteLLM "
f"API ${api_total:.2f} over {api_keys} keys: delta ${delta:+.2f} exceeds the "