reconcile paid credit packs against the ledger, report-first (issue #7)
ci/woodpecker/push/woodpecker Pipeline was canceled

Credits have one path in: the OC order.processed webhook. On 2026-11-13 two paid
packs got nothing through it — a $50 whose delivery 500'd on httpx's 5s default,
and a $10 with no delivery in the retained log at all — and nothing on a
schedule noticed. This adds a pass that reads OC's own order list and compares it
with our books over a day-based window (not "whatever the log still holds": the
$10 gap was 11h old and the log had lost the whole morning).

Report-first: CREDIT_RECONCILE defaults to 'report' (log findings, change
nothing) and grants only when set to 'on'. Grants go through add_credits() with
credit_pack_reference(), the same per-order key the webhook writes, so a late
delivery and this pass are idempotent against each other in both directions. A
buyer with no active membership is reported, never credited — those keep the
hold-and-email path.

credit_pack_accounted() encodes what our own rows actually look like: a v2 id, a
hyphenated legacy id, an empty-suffix legacy reference, and held_credits rows. A
bare oc:<slug>:<order.id> match read 21 credited purchases as missing during
development, so the fallback demands a second signal (member + tip-free amount +
date) before calling an order credited.

POST /admin/credit-pack-reconcile (X-Admin-Token) runs it on demand, dry unless
?apply=1 and the policy is on. 33 new checks; all 18 suites green.
This commit is contained in:
inference-admin committed 2026-11-13 20:16:24 +00:00
1 parent 32e744d9ee
commit 5868ca82e8
3 files changed
+620 -1

No files matched your search

+280
View File
@@ -7661,6 +7661,262 @@ def shortfall_email(name: str, amount: float, prev: str, cur: str,
return subject, text, html
# --- Credit-pack reconciliation (issue #7) --------------------------------- #
# Credits are granted on exactly ONE path: the OC `order.processed` webhook. If
# that delivery fails, or never arrives, the co-op holds a member's money and
# nothing on a schedule notices. 2026-11-13 produced two of those in one day — a
# $50 pack whose delivery 500'd on httpx's 5-second default, and a $10 pack with
# no delivery in the retained log at all, found only by reading OC's own order
# list against our books. This pass makes that routine.
#
# CREDIT_RECONCILE: report (default) | on | off
# report — find and LOG unaccounted packs; change nothing
# on — grant them, with the SAME per-order reference the webhook writes, so
# the webhook and this pass are idempotent against each other in both
# directions (a late delivery is a no-op; a pass after a delivery is a
# no-op too)
# off — skip the pass entirely
CREDIT_RECONCILE = (os.environ.get("CREDIT_RECONCILE", "report") or "report").strip().lower()
# How far back to look — DAYS off OC's own data, not "whatever our log still
# holds": the $10 gap was 11 hours old and the portal's log had already lost the
# whole morning, so a log-derived window cannot see the failures it exists for.
CREDIT_RECONCILE_DAYS = int(os.environ.get("CREDIT_RECONCILE_DAYS", "30") or "30")
CREDIT_RECONCILE_MAX_PAGES = int(os.environ.get("CREDIT_RECONCILE_MAX_PAGES", "3") or "3")
async def fetch_collective_orders(per_page: int = 100, max_pages: int = 3) -> list[dict]:
"""Recent orders for the collective, newest first. Raises on any failure.
Every failure raises rather than returning a short list: a caller that reads
a partial read as a complete one would conclude "nothing missing" from a
request that never answered. Needs the `orders` scope on OC_PERSONAL_TOKEN
(granted 2026-11-13; before that this answered `The Personal Token is not
allowed for operations in scope "orders"`).
"""
if not OC_PERSONAL_TOKEN:
raise RuntimeError("OC_PERSONAL_TOKEN not set; cannot fetch orders")
query = (
'{ collective(slug: "%s") { orders(limit: %d, offset: %d) { totalCount nodes { '
"id status createdAt processedAt frequency amount { value currency } "
"platformTipAmount { value } totalAmount { value } "
"tier { name slug } fromAccount { slug name type } } } } }"
)
out: list[dict] = []
async with httpx.AsyncClient(timeout=30.0) as client:
for page in range(max(1, max_pages)):
r = await client.post(
OC_GRAPHQL_URL,
headers={"Content-Type": "application/json",
"Authorization": f"Bearer {OC_PERSONAL_TOKEN}",
"Api-Version": "2022-01-01",
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64)"},
json={"query": query % (OC_COLLECTIVE_SLUG, per_page, page * per_page)},
)
if r.status_code != 200:
raise RuntimeError(f"OC order fetch failed: HTTP {r.status_code}")
body = r.json()
if body.get("errors"):
raise RuntimeError(f"OC order fetch errors: {str(body['errors'])[:200]}")
conn_ = ((body.get("data") or {}).get("collective") or {}).get("orders") or {}
nodes = conn_.get("nodes") or []
out.extend(nodes)
if len(nodes) < per_page or len(out) >= (conn_.get("totalCount") or 0):
break
return out
def _parse_ts(s) -> datetime | None:
"""OC sends ISO-8601 (…Z); our ledger rows are 'YYYY-MM-DD HH:MM:SS' (UTC)."""
if not s:
return None
t = str(s).strip()
if t.endswith("Z"):
t = t[:-1] + "+00:00"
if "T" not in t:
t = t.replace(" ", "T", 1)
try:
d = datetime.fromisoformat(t)
except ValueError:
return None
return d if d.tzinfo else d.replace(tzinfo=timezone.utc)
def credit_pack_accounted(order: dict, ledger_rows: list, held_rows: list) -> str | None:
"""Evidence that this order is ALREADY in our books, else None.
The reference is NOT uniform across our own rows, and a bare
`oc:<slug>:<order.id>` match therefore reports credited purchases as missing
— it read 21 of them that way on 2026-11-13, all false:
* newer ledger rows `oc:nightrose:or_apShJowSoPPj1dlM3ExTH` (the v2 id)
* older ledger rows `oc:benlo_0:a47byg9n-xozdp84r-…` (a hyphenated id)
* held rows `oc:madrush:` (EMPTY suffix)
So match the exact reference when the row carries one, and otherwise demand a
SECOND signal — same member, same tip-free amount, within a few days — before
calling an order credited. A held row counts as accounted for: the money was
recorded on the hold path, which is what non-member purchases are supposed to
take.
"""
slug = ((order.get("fromAccount") or {}).get("slug") or "").strip()
oid = (order.get("id") or "").strip()
try:
amount = float((order.get("amount") or {}).get("value") or 0)
except (TypeError, ValueError):
amount = 0.0
when = _parse_ts(order.get("createdAt"))
if oid and slug:
exact = f"oc:{slug}:{oid}"
for r in ledger_rows:
if (r.get("reference") or "") == exact:
return f"ledger {exact}"
for h in held_rows:
if (h.get("oc_reference") or "") == exact:
return f"held #{h.get('id')} {exact}"
for rows, what in ((ledger_rows, "ledger"), (held_rows, "held")):
for r in rows:
ref = (r.get("reference") or r.get("oc_reference") or "")
email = (r.get("email") or "").lower()
if slug and slug.lower() not in ref.lower() and slug.lower() not in email:
continue
try:
amt = float(r.get("delta") if what == "ledger" else r.get("amount") or 0)
except (TypeError, ValueError):
continue
if abs(amt - amount) > 0.005:
continue
rt = _parse_ts(r.get("created_at"))
if when and rt and abs((rt - when).total_seconds()) <= 5 * 86400:
return f"{what} {ref or '(no ref)'} — matched on member + amount + date"
return None
async def reconcile_credit_packs(apply: bool = False) -> dict:
"""Credit PAID credit packs that never reached our books (issue #7).
`apply=False` (report) is the default and changes NOTHING; granting also
requires CREDIT_RECONCILE == "on", so the pass can run on every sweep without
moving money until it has been read once in production. Every grant goes
through `add_credits` with `credit_pack_reference()` — the same key the
webhook writes — never a hand-set balance, so a late OC delivery and this
pass cannot both credit the same purchase.
"""
if CREDIT_RECONCILE == "off":
return {"status": "skipped", "reason": "credit_reconcile_off"}
try:
orders = await fetch_collective_orders(max_pages=CREDIT_RECONCILE_MAX_PAGES)
except Exception as e: # noqa: BLE001
# Never "nothing missing": an unreadable order list has to read as UNKNOWN.
logger.warning("Credit reconcile: order fetch failed; skipping (fail-safe): %s", e)
return {"status": "skipped", "reason": "order_fetch_failed", "detail": str(e)[:200]}
try:
index = await fetch_member_index()
except Exception as e: # noqa: BLE001
logger.warning("Credit reconcile: member index fetch failed; skipping: %s", e)
return {"status": "skipped", "reason": "member_index_failed", "detail": str(e)[:200]}
# Orders are matched on the slug from the LIVE OC list, never on the stored
# `members.slug` column, which can hold a stale `guest-…` from provisioning.
slug2email: dict[str, str] = {}
for email, rec in index.items():
s = (rec.get("slug") or "").strip()
if s:
slug2email.setdefault(s, email)
conn = get_db()
# Positional rows: get_db() hands back a plain connection (no row_factory),
# which is how the rest of this module reads it too.
ledger_rows = [
{"email": r[0], "delta": r[1], "reference": r[2], "created_at": r[3]}
for r in conn.execute(
"SELECT email, delta, reference, created_at FROM credit_ledger")]
held_rows = [
{"id": r[0], "slug": r[1], "email": r[2], "amount": r[3],
"oc_reference": r[4], "status": r[5], "created_at": r[6]}
for r in conn.execute(
"SELECT id, slug, email, amount, oc_reference, status, created_at "
"FROM held_credits")]
active = {r[0].strip().lower() for r in conn.execute("SELECT email FROM members WHERE active = 1")}
conn.close()
cutoff = datetime.now(timezone.utc) - timedelta(days=CREDIT_RECONCILE_DAYS)
packs = 0
unaccounted: list[dict] = []
for o in orders:
if (o.get("status") or "").upper() != "PAID":
continue
if (o.get("frequency") or "").upper() != "ONETIME":
continue
if "credit" not in (((o.get("tier") or {}).get("name") or "").lower()):
continue
when = _parse_ts(o.get("createdAt"))
if when and when < cutoff:
continue
packs += 1
if not credit_pack_accounted(o, ledger_rows, held_rows):
unaccounted.append(o)
acting = bool(apply and CREDIT_RECONCILE == "on")
candidates: list[dict] = []
applied: list[dict] = []
needs_hold: list[dict] = []
for o in unaccounted:
slug = ((o.get("fromAccount") or {}).get("slug") or "").strip()
rec = {"slug": slug,
"order_id": o.get("id"),
"amount": float((o.get("amount") or {}).get("value") or 0),
"created_at": str(o.get("createdAt") or "")[:19],
"tier": (o.get("tier") or {}).get("name")}
email = slug2email.get(slug, "")
if not email or email.lower() not in active:
# A purchase by someone who is not an active member keeps the
# hold-and-email path. This pass must never mint credits for a buyer
# who has no membership to spend them under.
rec["reason"] = "no active member for this OC slug — hold path, not a grant"
needs_hold.append(rec)
continue
rec["email"] = email
ref = credit_pack_reference(slug, {"idV2": o.get("id")})
rec["reference"] = ref
candidates.append(rec)
if not acting or credit_reference_exists(ref):
continue
new_balance = add_credits(email, rec["amount"],
reason="credit pack purchase (Open Collective)",
reference=ref)
try:
await sync_team_budget_from_balance(email)
rec["result"] = f"granted (balance {new_balance})"
except Exception as e: # noqa: BLE001
# The ledger row is the record that matters; the budget push is
# retried by the next sweep rather than failing the grant.
logger.warning("Credit reconcile: budget sync failed for %s: %s", email, e)
rec["result"] = f"granted (balance {new_balance}); budget sync failed: {str(e)[:80]}"
applied.append(rec)
logger.warning("Credit reconcile: GRANTED $%.2f to %s for order %s (was missing)",
rec["amount"], email, rec["order_id"])
if candidates and not acting:
logger.warning(
"Credit reconcile: %d PAID credit pack(s) with no record in our books "
"(report mode — nothing changed): %s", len(candidates),
"; ".join(f"${c['amount']:.2f} {c['slug']} {c['created_at']}" for c in candidates))
return {
"status": "ok",
"dry_run": not acting,
"window_days": CREDIT_RECONCILE_DAYS,
"orders_read": len(orders),
"packs_in_window": packs,
"unaccounted": len(candidates) + len(needs_hold),
"candidates": candidates,
"needs_hold": needs_hold,
"applied": applied,
}
async def reconcile_allowance_shortfalls(send: bool = False) -> dict:
"""Credit members whose PAID cycle Open Collective's billing-day alignment truncated.
@@ -7860,9 +8116,33 @@ async def admin_sweep(request: Request):
except Exception as exc: # noqa: BLE001
result["shortfall_credits"] = {"error": str(exc)[:200]}
# Paid credit packs with no record in our books (issue #7). REPORT-ONLY
# unless CREDIT_RECONCILE=on; wrapped so it can never break reconciliation.
try:
result["credit_reconcile"] = await reconcile_credit_packs(
apply=(CREDIT_RECONCILE == "on"))
except Exception as exc: # noqa: BLE001
result["credit_reconcile"] = {"error": str(exc)[:200]}
return result
@app.post("/admin/credit-pack-reconcile")
@limiter.limit("20/minute")
async def admin_credit_pack_reconcile(request: Request, apply: int = 0):
"""Reconcile PAID credit packs against our books — DRY RUN by default.
Authenticated by the X-Admin-Token header, like the other /admin/* routes.
`?apply=1` asks for the grants to be applied, and is honoured only when
CREDIT_RECONCILE=on: the point of the default is that an operator can read
this pass's output in production before any member's money is touched. Every
grant uses the same per-order reference the webhook writes, so running it
twice — or running it after a late delivery — credits a purchase once.
"""
_verify_admin_token(request)
return await reconcile_credit_packs(apply=bool(apply))
def _service_sync_pending() -> list[str]:
"""Active members still waiting on their Loomio invitation.