metrics: self-heal missing OC balance days from the OC ledger
The 2026-11-12 hole was NOT a missed sample: the 03:15 run recorded oc_balance_usd=1357.39, and the evening repair backfill's full-row upsert of the then-current day — whose dict carries no balance key — nulled it on conflict. Nothing ever revisited a day's balance, so the hole was permanent. Two fixes: - upsert() can no longer erase a balance it did not measure: the conflict clause COALESCEs, so a NULL (a failed OC read) keeps the previously recorded value. - the nightly snapshot heals any recent day (within a week, after the series' first recorded balance) whose balance is missing: the value is reconstructed from Open Collective's transaction ledger (balance(cutoff) = current - net flow after the 03:15 sample moment, ledger window clamped to the balance read), matching the daily sample semantics exactly. It is written balance-column-only, so membership and usage series are untouched. Days before 2026-11-08 stay NULL by design. Fail-soft: an OC failure prints and retries next night. The reconstruction was verified to the cent against OC's own ledger: full-ledger signed sum == stats.balance (1470.44 on 2026-11-13), and 1470.44 - 113.05 (net flow 2026-11-12 03:15 -> 2026-11-13 03:15:58) == 1357.39, exactly the sample the 03:15 run had recorded.
This commit is contained in:
1 parent
8405fd9157
commit
cad5897e9a
2 files changed
+385
-3
No files matched your search
+190
-3
@@ -11,6 +11,12 @@ dashboard can chart trends from one source of truth.
|
||||
repair recent days' usage only (membership
|
||||
columns are left untouched)
|
||||
|
||||
The nightly run also SELF-HEALS the OC balance line: any recent day (within a
|
||||
week, after the series' first recorded balance) whose `oc_balance_usd` is NULL
|
||||
is reconstructed from Open Collective's transaction ledger and written into the
|
||||
balance column only, so one failed night no longer leaves a permanent hole in
|
||||
the money chart. See heal_missing_balances().
|
||||
|
||||
Rules this file follows deliberately:
|
||||
|
||||
- **Fail closed per metric.** A source that cannot be read yields NULL for its
|
||||
@@ -94,8 +100,10 @@ def connect() -> sqlite3.Connection:
|
||||
("factor_revision", "TEXT"),
|
||||
# The collective's Open Collective balance, read once a day. The OC API
|
||||
# exposes only the CURRENT balance — there is no historical series — so
|
||||
# a real balance history can only accrue from these daily snapshots
|
||||
# (from 2026-11-08 onward; earlier days stay NULL by design).
|
||||
# a real balance history accrues from these daily snapshots, and a day
|
||||
# whose sample failed is RECONSTRUCTED from OC's transaction ledger the
|
||||
# next night (heal_missing_balances; from 2026-11-08 onward — earlier
|
||||
# days stay NULL by design, a gap in the chart, never a zero).
|
||||
("oc_balance_usd", "REAL"),
|
||||
):
|
||||
try:
|
||||
@@ -583,6 +591,167 @@ def oc_collective_balance() -> float | None:
|
||||
return None
|
||||
|
||||
|
||||
def oc_fetch_transactions(date_from: str, date_to: str) -> list[dict]:
|
||||
"""Every transaction in the collective's OC ledger within [date_from, date_to].
|
||||
|
||||
The raw building block for the balance self-heal. Raises on failure — the
|
||||
heal caller prints the hole and lets the next night retry; an empty result
|
||||
must stay distinguishable from a failed read, so nothing is swallowed here.
|
||||
OC stores DEBIT amounts NEGATIVE and CREDIT positive; callers must not
|
||||
re-sign them. Same browser-UA requirement as oc_collective_balance above
|
||||
(the default Python UA is Cloudflare-blocked, 403/1010).
|
||||
"""
|
||||
token = os.environ.get("OC_PERSONAL_TOKEN", "")
|
||||
slug = os.environ.get("OC_COLLECTIVE_SLUG", "inference-cooperative")
|
||||
if not token:
|
||||
raise RuntimeError("OC_PERSONAL_TOKEN not set")
|
||||
query = ("query TxW($slug: String, $from: DateTime!, $to: DateTime!, "
|
||||
"$limit: Int!, $offset: Int!) { collective(slug: $slug) { "
|
||||
"transactions(dateFrom: $from, dateTo: $to, limit: $limit, offset: $offset) "
|
||||
"{ totalCount nodes { createdAt type amount { valueInCents } } } } }")
|
||||
nodes: list[dict] = []
|
||||
offset = 0
|
||||
for _page in range(40): # hard cap, mirroring main.py's money pagination
|
||||
req = urllib.request.Request(
|
||||
OC_GRAPHQL_URL,
|
||||
data=json.dumps({"query": query, "variables": {
|
||||
"slug": slug, "from": date_from, "to": date_to,
|
||||
"limit": 500, "offset": offset,
|
||||
}}).encode("utf-8"),
|
||||
headers={
|
||||
"Content-Type": "application/json",
|
||||
"Authorization": f"Bearer {token}",
|
||||
"Api-Version": "2022-01-01",
|
||||
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64)",
|
||||
},
|
||||
)
|
||||
with urllib.request.urlopen(req, timeout=30) as resp:
|
||||
data = json.loads(resp.read() or b"{}")
|
||||
if data.get("errors"):
|
||||
raise RuntimeError(f"OC transactions read failed: {str(data['errors'])[:200]}")
|
||||
coll = ((data.get("data") or {}).get("collective") or {}).get("transactions") or {}
|
||||
batch = coll.get("nodes") or []
|
||||
nodes.extend(batch)
|
||||
offset += len(batch)
|
||||
if not batch or offset >= int(coll.get("totalCount") or 0):
|
||||
break
|
||||
return nodes
|
||||
|
||||
|
||||
def oc_historical_balances(current_balance: float, read_at: str, days: list[str]) -> dict[str, float]:
|
||||
"""Reconstruct missing days' balances from OC's transaction ledger.
|
||||
|
||||
OC exposes only the CURRENT balance, but the collective's balance is exactly
|
||||
the signed sum of its transactions (verified to the cent: the full-ledger
|
||||
sum equals stats.balance), so the balance as of a past cutoff C is:
|
||||
|
||||
current_balance - sum(signed amounts of transactions in (C, read_at])
|
||||
|
||||
The daily sample semantics are preserved: each day's value is the balance as
|
||||
of that day's 03:15 UTC snapshot moment, matching what the nightly read
|
||||
would have recorded. `read_at` is the moment `current_balance` was fetched;
|
||||
the ledger window is clamped to it (dateTo) so a payment landing between the
|
||||
balance read and the ledger read cannot distort the reconstruction. Days
|
||||
with no transactions after their cutoff simply carry the current balance.
|
||||
"""
|
||||
flows: dict[str, float] = {d: 0.0 for d in days} # in CENTS
|
||||
if days:
|
||||
earliest = min(days) + "T03:15:00.000Z"
|
||||
nodes = oc_fetch_transactions(earliest, read_at)
|
||||
for n in nodes:
|
||||
created = str(n.get("createdAt") or "")
|
||||
cents = int((n.get("amount") or {}).get("valueInCents") or 0)
|
||||
signed = cents if (n.get("type") or "").upper() == "CREDIT" else -abs(cents)
|
||||
for d in days:
|
||||
if d + "T03:15:00.000Z" < created <= read_at:
|
||||
flows[d] += signed
|
||||
return {
|
||||
# cents -> USD before touching the balance, never after
|
||||
d: round(current_balance - flows.get(d, 0.0) / 100.0, 2)
|
||||
for d in days
|
||||
}
|
||||
|
||||
|
||||
def missing_balance_days(today_key: str, window_days: int = 7) -> list[str]:
|
||||
"""Recent days whose balance sample is missing — holes in the line, not
|
||||
pre-history. A day qualifies when it is AFTER the series' first recorded
|
||||
balance (days before 2026-11-08 stay NULL by design: the chart shows a gap
|
||||
there, never a zero) and within the last `window_days` days, which bounds
|
||||
the nightly repair work and keeps a long outage from triggering a huge
|
||||
backfill. Only PAST days are returned: today's row is written by this very
|
||||
run."""
|
||||
conn = connect()
|
||||
try:
|
||||
first = conn.execute(
|
||||
"SELECT MIN(day) FROM metrics_daily WHERE oc_balance_usd IS NOT NULL"
|
||||
).fetchone()[0]
|
||||
rows = conn.execute(
|
||||
"SELECT day FROM metrics_daily WHERE oc_balance_usd IS NULL AND day < ? ORDER BY day",
|
||||
(today_key,),
|
||||
).fetchall()
|
||||
finally:
|
||||
conn.close()
|
||||
if not first:
|
||||
return []
|
||||
today_d = dt.date.fromisoformat(today_key)
|
||||
return [
|
||||
d for (d,) in rows
|
||||
if d > first and (today_d - dt.date.fromisoformat(d)).days <= window_days
|
||||
]
|
||||
|
||||
|
||||
def write_balance(day: str, value: float) -> None:
|
||||
"""Write ONLY oc_balance_usd for a day — the balance-column twin of
|
||||
upsert_usage. A missing row is INSERTed with every other column NULL
|
||||
(fail-closed: "balance known, membership not measured that day", never a
|
||||
zero); an existing row keeps all its other columns untouched. This is the
|
||||
only way a healed day's value reaches the store, so no heal can ever
|
||||
corrupt the membership or usage series."""
|
||||
conn = connect()
|
||||
try:
|
||||
conn.execute(
|
||||
"""INSERT INTO metrics_daily (day, oc_balance_usd, recorded_at)
|
||||
VALUES (?, ?, ?)
|
||||
ON CONFLICT(day) DO UPDATE SET
|
||||
oc_balance_usd = excluded.oc_balance_usd,
|
||||
recorded_at = excluded.recorded_at""",
|
||||
(
|
||||
day,
|
||||
float(value),
|
||||
dt.datetime.now(dt.timezone.utc).isoformat(timespec="seconds"),
|
||||
),
|
||||
)
|
||||
conn.commit()
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def heal_missing_balances(balance: float | None, read_at: str) -> None:
|
||||
"""Self-heal the money chart: fill RECENT days whose balance sample is gone.
|
||||
|
||||
A missed night (OC unreachable at 03:15, a scheduler run killed mid-deploy,
|
||||
a repair backfill that predated the no-erase rule below) used to leave a
|
||||
PERMANENT hole in the balance line, because nothing ever revisited a day's
|
||||
balance. The next night now reconstructs each missing day from OC's own
|
||||
transaction ledger (same source the nightly sample reads — no guessing by
|
||||
arithmetic from a remembered number) and writes it balance-column-only.
|
||||
Fail-soft end to end: any OC read failure prints and leaves the hole for
|
||||
the next run; nothing here can raise into the snapshot itself.
|
||||
"""
|
||||
if balance is None:
|
||||
return # OC read failed this run — today's heal would have no anchor
|
||||
try:
|
||||
days = missing_balance_days(today())
|
||||
if not days:
|
||||
return
|
||||
healed = oc_historical_balances(balance, read_at, days)
|
||||
for d in sorted(healed): # oldest first: a later failure leaves earlier days done
|
||||
write_balance(d, healed[d])
|
||||
print(f"metrics: healed missing balance day(s) -> {json.dumps(healed)}")
|
||||
except Exception as exc: # noqa: BLE001 - the hole waits for the next night
|
||||
print(f"metrics: balance heal failed ({exc}) — missing day(s) left for the next run")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- writes
|
||||
|
||||
# The columns of metrics_daily that describe THAT DAY'S TRAFFIC and its impact,
|
||||
@@ -693,7 +862,13 @@ def upsert(day: str, row: dict) -> None:
|
||||
tokens_measured=excluded.tokens_measured,
|
||||
tokens_estimated=excluded.tokens_estimated,
|
||||
factor_revision=excluded.factor_revision,
|
||||
oc_balance_usd=excluded.oc_balance_usd,
|
||||
-- A NULL oc_balance_usd means "the OC read failed this run",
|
||||
-- never "the balance is gone": keep the previously recorded
|
||||
-- value instead of erasing it. (The Nov 2026 chart hole was
|
||||
-- exactly this: a backfill's full-row upsert whose dict
|
||||
-- carried no balance NULLed a sample the 03:15 run had
|
||||
-- recorded hours earlier.)
|
||||
oc_balance_usd=COALESCE(excluded.oc_balance_usd, metrics_daily.oc_balance_usd),
|
||||
recorded_at=excluded.recorded_at""",
|
||||
(
|
||||
day,
|
||||
@@ -735,6 +910,14 @@ def snapshot(day: str | None = None) -> dict:
|
||||
op_spend = operator_spend_by_day(since=since)
|
||||
tokens = tokens_by_day_model(since=since)
|
||||
balance = oc_collective_balance()
|
||||
# The moment the balance above was read. The self-heal clamps its ledger
|
||||
# window to this instant, so a payment landing between the balance read and
|
||||
# the ledger read is excluded from the reconstruction (it is not in the
|
||||
# anchor balance either). Millisecond precision: OC stamps createdAt with
|
||||
# millis, and the comparison is lexicographic on matching ISO shapes.
|
||||
balance_read_at = (
|
||||
dt.datetime.now(dt.timezone.utc).strftime("%Y-%m-%dT%H:%M:%S.%f")[:-3] + "Z"
|
||||
)
|
||||
# 1. Complete the day that just ENDED. The window above already contains
|
||||
# yesterday's full aggregates — the old code fetched them and threw them
|
||||
# away, keeping only today's first ~3 hours and never revisiting the
|
||||
@@ -760,6 +943,10 @@ def snapshot(day: str | None = None) -> dict:
|
||||
oc_balance_usd=balance,
|
||||
)
|
||||
upsert(day, today_row)
|
||||
# 3. Self-heal: any RECENT day whose balance sample is gone (one failed
|
||||
# night used to leave a permanent hole in the money chart) is
|
||||
# reconstructed from OC's own transaction ledger, balance column only.
|
||||
heal_missing_balances(balance, balance_read_at)
|
||||
print(f"metrics: completed {yesterday} -> {json.dumps(yesterday_usage)}")
|
||||
print(f"metrics: recorded {day} -> {json.dumps(today_row)}")
|
||||
return today_row
|
||||
|
||||
Reference in new issue
Block a user