reconcile paid credit packs against the ledger, report-first (issue #7)
ci/woodpecker/push/woodpecker Pipeline was canceled
ci/woodpecker/push/woodpecker Pipeline was canceled
Credits have one path in: the OC order.processed webhook. On 2026-11-13 two paid packs got nothing through it — a $50 whose delivery 500'd on httpx's 5s default, and a $10 with no delivery in the retained log at all — and nothing on a schedule noticed. This adds a pass that reads OC's own order list and compares it with our books over a day-based window (not "whatever the log still holds": the $10 gap was 11h old and the log had lost the whole morning). Report-first: CREDIT_RECONCILE defaults to 'report' (log findings, change nothing) and grants only when set to 'on'. Grants go through add_credits() with credit_pack_reference(), the same per-order key the webhook writes, so a late delivery and this pass are idempotent against each other in both directions. A buyer with no active membership is reported, never credited — those keep the hold-and-email path. credit_pack_accounted() encodes what our own rows actually look like: a v2 id, a hyphenated legacy id, an empty-suffix legacy reference, and held_credits rows. A bare oc:<slug>:<order.id> match read 21 credited purchases as missing during development, so the fallback demands a second signal (member + tip-free amount + date) before calling an order credited. POST /admin/credit-pack-reconcile (X-Admin-Token) runs it on demand, dry unless ?apply=1 and the policy is on. 33 new checks; all 18 suites green.
This commit is contained in:
1 parent
32e744d9ee
commit
5868ca82e8
3 files changed
+620
-1
No files matched your search
+280
@@ -7661,6 +7661,262 @@ def shortfall_email(name: str, amount: float, prev: str, cur: str,
|
||||
return subject, text, html
|
||||
|
||||
|
||||
# --- Credit-pack reconciliation (issue #7) --------------------------------- #
|
||||
# Credits are granted on exactly ONE path: the OC `order.processed` webhook. If
|
||||
# that delivery fails, or never arrives, the co-op holds a member's money and
|
||||
# nothing on a schedule notices. 2026-11-13 produced two of those in one day — a
|
||||
# $50 pack whose delivery 500'd on httpx's 5-second default, and a $10 pack with
|
||||
# no delivery in the retained log at all, found only by reading OC's own order
|
||||
# list against our books. This pass makes that routine.
|
||||
#
|
||||
# CREDIT_RECONCILE: report (default) | on | off
|
||||
# report — find and LOG unaccounted packs; change nothing
|
||||
# on — grant them, with the SAME per-order reference the webhook writes, so
|
||||
# the webhook and this pass are idempotent against each other in both
|
||||
# directions (a late delivery is a no-op; a pass after a delivery is a
|
||||
# no-op too)
|
||||
# off — skip the pass entirely
|
||||
CREDIT_RECONCILE = (os.environ.get("CREDIT_RECONCILE", "report") or "report").strip().lower()
|
||||
# How far back to look — DAYS off OC's own data, not "whatever our log still
|
||||
# holds": the $10 gap was 11 hours old and the portal's log had already lost the
|
||||
# whole morning, so a log-derived window cannot see the failures it exists for.
|
||||
CREDIT_RECONCILE_DAYS = int(os.environ.get("CREDIT_RECONCILE_DAYS", "30") or "30")
|
||||
CREDIT_RECONCILE_MAX_PAGES = int(os.environ.get("CREDIT_RECONCILE_MAX_PAGES", "3") or "3")
|
||||
|
||||
|
||||
async def fetch_collective_orders(per_page: int = 100, max_pages: int = 3) -> list[dict]:
|
||||
"""Recent orders for the collective, newest first. Raises on any failure.
|
||||
|
||||
Every failure raises rather than returning a short list: a caller that reads
|
||||
a partial read as a complete one would conclude "nothing missing" from a
|
||||
request that never answered. Needs the `orders` scope on OC_PERSONAL_TOKEN
|
||||
(granted 2026-11-13; before that this answered `The Personal Token is not
|
||||
allowed for operations in scope "orders"`).
|
||||
"""
|
||||
if not OC_PERSONAL_TOKEN:
|
||||
raise RuntimeError("OC_PERSONAL_TOKEN not set; cannot fetch orders")
|
||||
query = (
|
||||
'{ collective(slug: "%s") { orders(limit: %d, offset: %d) { totalCount nodes { '
|
||||
"id status createdAt processedAt frequency amount { value currency } "
|
||||
"platformTipAmount { value } totalAmount { value } "
|
||||
"tier { name slug } fromAccount { slug name type } } } } }"
|
||||
)
|
||||
out: list[dict] = []
|
||||
async with httpx.AsyncClient(timeout=30.0) as client:
|
||||
for page in range(max(1, max_pages)):
|
||||
r = await client.post(
|
||||
OC_GRAPHQL_URL,
|
||||
headers={"Content-Type": "application/json",
|
||||
"Authorization": f"Bearer {OC_PERSONAL_TOKEN}",
|
||||
"Api-Version": "2022-01-01",
|
||||
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64)"},
|
||||
json={"query": query % (OC_COLLECTIVE_SLUG, per_page, page * per_page)},
|
||||
)
|
||||
if r.status_code != 200:
|
||||
raise RuntimeError(f"OC order fetch failed: HTTP {r.status_code}")
|
||||
body = r.json()
|
||||
if body.get("errors"):
|
||||
raise RuntimeError(f"OC order fetch errors: {str(body['errors'])[:200]}")
|
||||
conn_ = ((body.get("data") or {}).get("collective") or {}).get("orders") or {}
|
||||
nodes = conn_.get("nodes") or []
|
||||
out.extend(nodes)
|
||||
if len(nodes) < per_page or len(out) >= (conn_.get("totalCount") or 0):
|
||||
break
|
||||
return out
|
||||
|
||||
|
||||
def _parse_ts(s) -> datetime | None:
|
||||
"""OC sends ISO-8601 (…Z); our ledger rows are 'YYYY-MM-DD HH:MM:SS' (UTC)."""
|
||||
if not s:
|
||||
return None
|
||||
t = str(s).strip()
|
||||
if t.endswith("Z"):
|
||||
t = t[:-1] + "+00:00"
|
||||
if "T" not in t:
|
||||
t = t.replace(" ", "T", 1)
|
||||
try:
|
||||
d = datetime.fromisoformat(t)
|
||||
except ValueError:
|
||||
return None
|
||||
return d if d.tzinfo else d.replace(tzinfo=timezone.utc)
|
||||
|
||||
|
||||
def credit_pack_accounted(order: dict, ledger_rows: list, held_rows: list) -> str | None:
|
||||
"""Evidence that this order is ALREADY in our books, else None.
|
||||
|
||||
The reference is NOT uniform across our own rows, and a bare
|
||||
`oc:<slug>:<order.id>` match therefore reports credited purchases as missing
|
||||
— it read 21 of them that way on 2026-11-13, all false:
|
||||
|
||||
* newer ledger rows `oc:nightrose:or_apShJowSoPPj1dlM3ExTH` (the v2 id)
|
||||
* older ledger rows `oc:benlo_0:a47byg9n-xozdp84r-…` (a hyphenated id)
|
||||
* held rows `oc:madrush:` (EMPTY suffix)
|
||||
|
||||
So match the exact reference when the row carries one, and otherwise demand a
|
||||
SECOND signal — same member, same tip-free amount, within a few days — before
|
||||
calling an order credited. A held row counts as accounted for: the money was
|
||||
recorded on the hold path, which is what non-member purchases are supposed to
|
||||
take.
|
||||
"""
|
||||
slug = ((order.get("fromAccount") or {}).get("slug") or "").strip()
|
||||
oid = (order.get("id") or "").strip()
|
||||
try:
|
||||
amount = float((order.get("amount") or {}).get("value") or 0)
|
||||
except (TypeError, ValueError):
|
||||
amount = 0.0
|
||||
when = _parse_ts(order.get("createdAt"))
|
||||
|
||||
if oid and slug:
|
||||
exact = f"oc:{slug}:{oid}"
|
||||
for r in ledger_rows:
|
||||
if (r.get("reference") or "") == exact:
|
||||
return f"ledger {exact}"
|
||||
for h in held_rows:
|
||||
if (h.get("oc_reference") or "") == exact:
|
||||
return f"held #{h.get('id')} {exact}"
|
||||
|
||||
for rows, what in ((ledger_rows, "ledger"), (held_rows, "held")):
|
||||
for r in rows:
|
||||
ref = (r.get("reference") or r.get("oc_reference") or "")
|
||||
email = (r.get("email") or "").lower()
|
||||
if slug and slug.lower() not in ref.lower() and slug.lower() not in email:
|
||||
continue
|
||||
try:
|
||||
amt = float(r.get("delta") if what == "ledger" else r.get("amount") or 0)
|
||||
except (TypeError, ValueError):
|
||||
continue
|
||||
if abs(amt - amount) > 0.005:
|
||||
continue
|
||||
rt = _parse_ts(r.get("created_at"))
|
||||
if when and rt and abs((rt - when).total_seconds()) <= 5 * 86400:
|
||||
return f"{what} {ref or '(no ref)'} — matched on member + amount + date"
|
||||
return None
|
||||
|
||||
|
||||
async def reconcile_credit_packs(apply: bool = False) -> dict:
|
||||
"""Credit PAID credit packs that never reached our books (issue #7).
|
||||
|
||||
`apply=False` (report) is the default and changes NOTHING; granting also
|
||||
requires CREDIT_RECONCILE == "on", so the pass can run on every sweep without
|
||||
moving money until it has been read once in production. Every grant goes
|
||||
through `add_credits` with `credit_pack_reference()` — the same key the
|
||||
webhook writes — never a hand-set balance, so a late OC delivery and this
|
||||
pass cannot both credit the same purchase.
|
||||
"""
|
||||
if CREDIT_RECONCILE == "off":
|
||||
return {"status": "skipped", "reason": "credit_reconcile_off"}
|
||||
try:
|
||||
orders = await fetch_collective_orders(max_pages=CREDIT_RECONCILE_MAX_PAGES)
|
||||
except Exception as e: # noqa: BLE001
|
||||
# Never "nothing missing": an unreadable order list has to read as UNKNOWN.
|
||||
logger.warning("Credit reconcile: order fetch failed; skipping (fail-safe): %s", e)
|
||||
return {"status": "skipped", "reason": "order_fetch_failed", "detail": str(e)[:200]}
|
||||
try:
|
||||
index = await fetch_member_index()
|
||||
except Exception as e: # noqa: BLE001
|
||||
logger.warning("Credit reconcile: member index fetch failed; skipping: %s", e)
|
||||
return {"status": "skipped", "reason": "member_index_failed", "detail": str(e)[:200]}
|
||||
|
||||
# Orders are matched on the slug from the LIVE OC list, never on the stored
|
||||
# `members.slug` column, which can hold a stale `guest-…` from provisioning.
|
||||
slug2email: dict[str, str] = {}
|
||||
for email, rec in index.items():
|
||||
s = (rec.get("slug") or "").strip()
|
||||
if s:
|
||||
slug2email.setdefault(s, email)
|
||||
|
||||
conn = get_db()
|
||||
# Positional rows: get_db() hands back a plain connection (no row_factory),
|
||||
# which is how the rest of this module reads it too.
|
||||
ledger_rows = [
|
||||
{"email": r[0], "delta": r[1], "reference": r[2], "created_at": r[3]}
|
||||
for r in conn.execute(
|
||||
"SELECT email, delta, reference, created_at FROM credit_ledger")]
|
||||
held_rows = [
|
||||
{"id": r[0], "slug": r[1], "email": r[2], "amount": r[3],
|
||||
"oc_reference": r[4], "status": r[5], "created_at": r[6]}
|
||||
for r in conn.execute(
|
||||
"SELECT id, slug, email, amount, oc_reference, status, created_at "
|
||||
"FROM held_credits")]
|
||||
active = {r[0].strip().lower() for r in conn.execute("SELECT email FROM members WHERE active = 1")}
|
||||
conn.close()
|
||||
|
||||
cutoff = datetime.now(timezone.utc) - timedelta(days=CREDIT_RECONCILE_DAYS)
|
||||
packs = 0
|
||||
unaccounted: list[dict] = []
|
||||
for o in orders:
|
||||
if (o.get("status") or "").upper() != "PAID":
|
||||
continue
|
||||
if (o.get("frequency") or "").upper() != "ONETIME":
|
||||
continue
|
||||
if "credit" not in (((o.get("tier") or {}).get("name") or "").lower()):
|
||||
continue
|
||||
when = _parse_ts(o.get("createdAt"))
|
||||
if when and when < cutoff:
|
||||
continue
|
||||
packs += 1
|
||||
if not credit_pack_accounted(o, ledger_rows, held_rows):
|
||||
unaccounted.append(o)
|
||||
|
||||
acting = bool(apply and CREDIT_RECONCILE == "on")
|
||||
candidates: list[dict] = []
|
||||
applied: list[dict] = []
|
||||
needs_hold: list[dict] = []
|
||||
for o in unaccounted:
|
||||
slug = ((o.get("fromAccount") or {}).get("slug") or "").strip()
|
||||
rec = {"slug": slug,
|
||||
"order_id": o.get("id"),
|
||||
"amount": float((o.get("amount") or {}).get("value") or 0),
|
||||
"created_at": str(o.get("createdAt") or "")[:19],
|
||||
"tier": (o.get("tier") or {}).get("name")}
|
||||
email = slug2email.get(slug, "")
|
||||
if not email or email.lower() not in active:
|
||||
# A purchase by someone who is not an active member keeps the
|
||||
# hold-and-email path. This pass must never mint credits for a buyer
|
||||
# who has no membership to spend them under.
|
||||
rec["reason"] = "no active member for this OC slug — hold path, not a grant"
|
||||
needs_hold.append(rec)
|
||||
continue
|
||||
rec["email"] = email
|
||||
ref = credit_pack_reference(slug, {"idV2": o.get("id")})
|
||||
rec["reference"] = ref
|
||||
candidates.append(rec)
|
||||
if not acting or credit_reference_exists(ref):
|
||||
continue
|
||||
new_balance = add_credits(email, rec["amount"],
|
||||
reason="credit pack purchase (Open Collective)",
|
||||
reference=ref)
|
||||
try:
|
||||
await sync_team_budget_from_balance(email)
|
||||
rec["result"] = f"granted (balance {new_balance})"
|
||||
except Exception as e: # noqa: BLE001
|
||||
# The ledger row is the record that matters; the budget push is
|
||||
# retried by the next sweep rather than failing the grant.
|
||||
logger.warning("Credit reconcile: budget sync failed for %s: %s", email, e)
|
||||
rec["result"] = f"granted (balance {new_balance}); budget sync failed: {str(e)[:80]}"
|
||||
applied.append(rec)
|
||||
logger.warning("Credit reconcile: GRANTED $%.2f to %s for order %s (was missing)",
|
||||
rec["amount"], email, rec["order_id"])
|
||||
|
||||
if candidates and not acting:
|
||||
logger.warning(
|
||||
"Credit reconcile: %d PAID credit pack(s) with no record in our books "
|
||||
"(report mode — nothing changed): %s", len(candidates),
|
||||
"; ".join(f"${c['amount']:.2f} {c['slug']} {c['created_at']}" for c in candidates))
|
||||
|
||||
return {
|
||||
"status": "ok",
|
||||
"dry_run": not acting,
|
||||
"window_days": CREDIT_RECONCILE_DAYS,
|
||||
"orders_read": len(orders),
|
||||
"packs_in_window": packs,
|
||||
"unaccounted": len(candidates) + len(needs_hold),
|
||||
"candidates": candidates,
|
||||
"needs_hold": needs_hold,
|
||||
"applied": applied,
|
||||
}
|
||||
|
||||
|
||||
async def reconcile_allowance_shortfalls(send: bool = False) -> dict:
|
||||
"""Credit members whose PAID cycle Open Collective's billing-day alignment truncated.
|
||||
|
||||
@@ -7860,9 +8116,33 @@ async def admin_sweep(request: Request):
|
||||
except Exception as exc: # noqa: BLE001
|
||||
result["shortfall_credits"] = {"error": str(exc)[:200]}
|
||||
|
||||
# Paid credit packs with no record in our books (issue #7). REPORT-ONLY
|
||||
# unless CREDIT_RECONCILE=on; wrapped so it can never break reconciliation.
|
||||
try:
|
||||
result["credit_reconcile"] = await reconcile_credit_packs(
|
||||
apply=(CREDIT_RECONCILE == "on"))
|
||||
except Exception as exc: # noqa: BLE001
|
||||
result["credit_reconcile"] = {"error": str(exc)[:200]}
|
||||
|
||||
return result
|
||||
|
||||
|
||||
@app.post("/admin/credit-pack-reconcile")
|
||||
@limiter.limit("20/minute")
|
||||
async def admin_credit_pack_reconcile(request: Request, apply: int = 0):
|
||||
"""Reconcile PAID credit packs against our books — DRY RUN by default.
|
||||
|
||||
Authenticated by the X-Admin-Token header, like the other /admin/* routes.
|
||||
`?apply=1` asks for the grants to be applied, and is honoured only when
|
||||
CREDIT_RECONCILE=on: the point of the default is that an operator can read
|
||||
this pass's output in production before any member's money is touched. Every
|
||||
grant uses the same per-order reference the webhook writes, so running it
|
||||
twice — or running it after a late delivery — credits a purchase once.
|
||||
"""
|
||||
_verify_admin_token(request)
|
||||
return await reconcile_credit_packs(apply=bool(apply))
|
||||
|
||||
|
||||
def _service_sync_pending() -> list[str]:
|
||||
"""Active members still waiting on their Loomio invitation.
|
||||
|
||||
|
||||
Reference in new issue
Block a user