reconcile paid credit packs against the ledger, report-first (issue #7)
ci/woodpecker/push/woodpecker Pipeline was canceled
ci/woodpecker/push/woodpecker Pipeline was canceled
Credits have one path in: the OC order.processed webhook. On 2026-11-13 two paid packs got nothing through it — a $50 whose delivery 500'd on httpx's 5s default, and a $10 with no delivery in the retained log at all — and nothing on a schedule noticed. This adds a pass that reads OC's own order list and compares it with our books over a day-based window (not "whatever the log still holds": the $10 gap was 11h old and the log had lost the whole morning). Report-first: CREDIT_RECONCILE defaults to 'report' (log findings, change nothing) and grants only when set to 'on'. Grants go through add_credits() with credit_pack_reference(), the same per-order key the webhook writes, so a late delivery and this pass are idempotent against each other in both directions. A buyer with no active membership is reported, never credited — those keep the hold-and-email path. credit_pack_accounted() encodes what our own rows actually look like: a v2 id, a hyphenated legacy id, an empty-suffix legacy reference, and held_credits rows. A bare oc:<slug>:<order.id> match read 21 credited purchases as missing during development, so the fallback demands a second signal (member + tip-free amount + date) before calling an order credited. POST /admin/credit-pack-reconcile (X-Admin-Token) runs it on demand, dry unless ?apply=1 and the policy is on. 33 new checks; all 18 suites green.
This commit is contained in:
1 parent
32e744d9ee
commit
5868ca82e8
3 files changed
+620
-1
No files matched your search
@@ -191,7 +191,31 @@ Four jobs run inside the container via `CloudronManifest.json` → `addons.sched
|
|||||||
against the live Open Collective list: provisions missed signups, deactivates
|
against the live Open Collective list: provisions missed signups, deactivates
|
||||||
lapsed members, syncs Loomio + Matrix. Idempotent; a missed webhook
|
lapsed members, syncs Loomio + Matrix. Idempotent; a missed webhook
|
||||||
self-heals within a day. The same sweep also runs the payment-driven
|
self-heals within a day. The same sweep also runs the payment-driven
|
||||||
allowance reconciliation (below).
|
allowance reconciliation (below) and the credit-pack reconciliation.
|
||||||
|
|
||||||
|
**Credit-pack reconciliation** (`CREDIT_RECONCILE`, default `report`) — credits
|
||||||
|
are granted on exactly one path, the OC `order.processed` webhook, and that
|
||||||
|
path can fail AFTER the money has moved: on 2026-11-13 a $50 pack's delivery
|
||||||
|
500'd on httpx's invisible 5-second default, and a $10 pack had no delivery in
|
||||||
|
the retained log at all. Neither left a ledger row and nothing noticed. This
|
||||||
|
pass reads OC's OWN order list (`fetch_collective_orders`, needs the `orders`
|
||||||
|
scope on `OC_PERSONAL_TOKEN`, granted 2026-11-13) and compares it with our
|
||||||
|
books over `CREDIT_RECONCILE_DAYS` (default 30 — days, not "whatever our log
|
||||||
|
still holds": the $10 gap was 11 hours old and the log had already lost the
|
||||||
|
whole morning). In `report` mode it logs findings and changes NOTHING; grants
|
||||||
|
happen only when `CREDIT_RECONCILE=on`. Every grant uses
|
||||||
|
`credit_pack_reference()` — the same per-order key the webhook writes — so a
|
||||||
|
late delivery and this pass cannot both credit one purchase. A buyer who is not
|
||||||
|
an active member is reported, never credited: those keep the hold-and-email
|
||||||
|
path. On demand: `POST /admin/credit-pack-reconcile` (`X-Admin-Token`), a dry
|
||||||
|
run unless `?apply=1` and the policy is `on`.
|
||||||
|
|
||||||
|
Matching caution: our own references are NOT uniform, so a bare
|
||||||
|
`oc:<slug>:<order.id>` test reports credited purchases as missing (it read 21
|
||||||
|
of them that way, all false). `credit_pack_accounted()` matches the exact
|
||||||
|
reference when one exists and otherwise requires a second signal — same member,
|
||||||
|
same tip-free amount, within days — and counts a `held_credits` row as
|
||||||
|
accounted for.
|
||||||
|
|
||||||
**Service-invite triggers:** the sweep is the backstop, not the primary path.
|
**Service-invite triggers:** the sweep is the backstop, not the primary path.
|
||||||
Two faster triggers exist, because a member who has just set up their account
|
Two faster triggers exist, because a member who has just set up their account
|
||||||
|
|||||||
+280
@@ -7661,6 +7661,262 @@ def shortfall_email(name: str, amount: float, prev: str, cur: str,
|
|||||||
return subject, text, html
|
return subject, text, html
|
||||||
|
|
||||||
|
|
||||||
|
# --- Credit-pack reconciliation (issue #7) --------------------------------- #
|
||||||
|
# Credits are granted on exactly ONE path: the OC `order.processed` webhook. If
|
||||||
|
# that delivery fails, or never arrives, the co-op holds a member's money and
|
||||||
|
# nothing on a schedule notices. 2026-11-13 produced two of those in one day — a
|
||||||
|
# $50 pack whose delivery 500'd on httpx's 5-second default, and a $10 pack with
|
||||||
|
# no delivery in the retained log at all, found only by reading OC's own order
|
||||||
|
# list against our books. This pass makes that routine.
|
||||||
|
#
|
||||||
|
# CREDIT_RECONCILE: report (default) | on | off
|
||||||
|
# report — find and LOG unaccounted packs; change nothing
|
||||||
|
# on — grant them, with the SAME per-order reference the webhook writes, so
|
||||||
|
# the webhook and this pass are idempotent against each other in both
|
||||||
|
# directions (a late delivery is a no-op; a pass after a delivery is a
|
||||||
|
# no-op too)
|
||||||
|
# off — skip the pass entirely
|
||||||
|
CREDIT_RECONCILE = (os.environ.get("CREDIT_RECONCILE", "report") or "report").strip().lower()
|
||||||
|
# How far back to look — DAYS off OC's own data, not "whatever our log still
|
||||||
|
# holds": the $10 gap was 11 hours old and the portal's log had already lost the
|
||||||
|
# whole morning, so a log-derived window cannot see the failures it exists for.
|
||||||
|
CREDIT_RECONCILE_DAYS = int(os.environ.get("CREDIT_RECONCILE_DAYS", "30") or "30")
|
||||||
|
CREDIT_RECONCILE_MAX_PAGES = int(os.environ.get("CREDIT_RECONCILE_MAX_PAGES", "3") or "3")
|
||||||
|
|
||||||
|
|
||||||
|
async def fetch_collective_orders(per_page: int = 100, max_pages: int = 3) -> list[dict]:
|
||||||
|
"""Recent orders for the collective, newest first. Raises on any failure.
|
||||||
|
|
||||||
|
Every failure raises rather than returning a short list: a caller that reads
|
||||||
|
a partial read as a complete one would conclude "nothing missing" from a
|
||||||
|
request that never answered. Needs the `orders` scope on OC_PERSONAL_TOKEN
|
||||||
|
(granted 2026-11-13; before that this answered `The Personal Token is not
|
||||||
|
allowed for operations in scope "orders"`).
|
||||||
|
"""
|
||||||
|
if not OC_PERSONAL_TOKEN:
|
||||||
|
raise RuntimeError("OC_PERSONAL_TOKEN not set; cannot fetch orders")
|
||||||
|
query = (
|
||||||
|
'{ collective(slug: "%s") { orders(limit: %d, offset: %d) { totalCount nodes { '
|
||||||
|
"id status createdAt processedAt frequency amount { value currency } "
|
||||||
|
"platformTipAmount { value } totalAmount { value } "
|
||||||
|
"tier { name slug } fromAccount { slug name type } } } } }"
|
||||||
|
)
|
||||||
|
out: list[dict] = []
|
||||||
|
async with httpx.AsyncClient(timeout=30.0) as client:
|
||||||
|
for page in range(max(1, max_pages)):
|
||||||
|
r = await client.post(
|
||||||
|
OC_GRAPHQL_URL,
|
||||||
|
headers={"Content-Type": "application/json",
|
||||||
|
"Authorization": f"Bearer {OC_PERSONAL_TOKEN}",
|
||||||
|
"Api-Version": "2022-01-01",
|
||||||
|
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64)"},
|
||||||
|
json={"query": query % (OC_COLLECTIVE_SLUG, per_page, page * per_page)},
|
||||||
|
)
|
||||||
|
if r.status_code != 200:
|
||||||
|
raise RuntimeError(f"OC order fetch failed: HTTP {r.status_code}")
|
||||||
|
body = r.json()
|
||||||
|
if body.get("errors"):
|
||||||
|
raise RuntimeError(f"OC order fetch errors: {str(body['errors'])[:200]}")
|
||||||
|
conn_ = ((body.get("data") or {}).get("collective") or {}).get("orders") or {}
|
||||||
|
nodes = conn_.get("nodes") or []
|
||||||
|
out.extend(nodes)
|
||||||
|
if len(nodes) < per_page or len(out) >= (conn_.get("totalCount") or 0):
|
||||||
|
break
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_ts(s) -> datetime | None:
|
||||||
|
"""OC sends ISO-8601 (…Z); our ledger rows are 'YYYY-MM-DD HH:MM:SS' (UTC)."""
|
||||||
|
if not s:
|
||||||
|
return None
|
||||||
|
t = str(s).strip()
|
||||||
|
if t.endswith("Z"):
|
||||||
|
t = t[:-1] + "+00:00"
|
||||||
|
if "T" not in t:
|
||||||
|
t = t.replace(" ", "T", 1)
|
||||||
|
try:
|
||||||
|
d = datetime.fromisoformat(t)
|
||||||
|
except ValueError:
|
||||||
|
return None
|
||||||
|
return d if d.tzinfo else d.replace(tzinfo=timezone.utc)
|
||||||
|
|
||||||
|
|
||||||
|
def credit_pack_accounted(order: dict, ledger_rows: list, held_rows: list) -> str | None:
|
||||||
|
"""Evidence that this order is ALREADY in our books, else None.
|
||||||
|
|
||||||
|
The reference is NOT uniform across our own rows, and a bare
|
||||||
|
`oc:<slug>:<order.id>` match therefore reports credited purchases as missing
|
||||||
|
— it read 21 of them that way on 2026-11-13, all false:
|
||||||
|
|
||||||
|
* newer ledger rows `oc:nightrose:or_apShJowSoPPj1dlM3ExTH` (the v2 id)
|
||||||
|
* older ledger rows `oc:benlo_0:a47byg9n-xozdp84r-…` (a hyphenated id)
|
||||||
|
* held rows `oc:madrush:` (EMPTY suffix)
|
||||||
|
|
||||||
|
So match the exact reference when the row carries one, and otherwise demand a
|
||||||
|
SECOND signal — same member, same tip-free amount, within a few days — before
|
||||||
|
calling an order credited. A held row counts as accounted for: the money was
|
||||||
|
recorded on the hold path, which is what non-member purchases are supposed to
|
||||||
|
take.
|
||||||
|
"""
|
||||||
|
slug = ((order.get("fromAccount") or {}).get("slug") or "").strip()
|
||||||
|
oid = (order.get("id") or "").strip()
|
||||||
|
try:
|
||||||
|
amount = float((order.get("amount") or {}).get("value") or 0)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
amount = 0.0
|
||||||
|
when = _parse_ts(order.get("createdAt"))
|
||||||
|
|
||||||
|
if oid and slug:
|
||||||
|
exact = f"oc:{slug}:{oid}"
|
||||||
|
for r in ledger_rows:
|
||||||
|
if (r.get("reference") or "") == exact:
|
||||||
|
return f"ledger {exact}"
|
||||||
|
for h in held_rows:
|
||||||
|
if (h.get("oc_reference") or "") == exact:
|
||||||
|
return f"held #{h.get('id')} {exact}"
|
||||||
|
|
||||||
|
for rows, what in ((ledger_rows, "ledger"), (held_rows, "held")):
|
||||||
|
for r in rows:
|
||||||
|
ref = (r.get("reference") or r.get("oc_reference") or "")
|
||||||
|
email = (r.get("email") or "").lower()
|
||||||
|
if slug and slug.lower() not in ref.lower() and slug.lower() not in email:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
amt = float(r.get("delta") if what == "ledger" else r.get("amount") or 0)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
continue
|
||||||
|
if abs(amt - amount) > 0.005:
|
||||||
|
continue
|
||||||
|
rt = _parse_ts(r.get("created_at"))
|
||||||
|
if when and rt and abs((rt - when).total_seconds()) <= 5 * 86400:
|
||||||
|
return f"{what} {ref or '(no ref)'} — matched on member + amount + date"
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
async def reconcile_credit_packs(apply: bool = False) -> dict:
|
||||||
|
"""Credit PAID credit packs that never reached our books (issue #7).
|
||||||
|
|
||||||
|
`apply=False` (report) is the default and changes NOTHING; granting also
|
||||||
|
requires CREDIT_RECONCILE == "on", so the pass can run on every sweep without
|
||||||
|
moving money until it has been read once in production. Every grant goes
|
||||||
|
through `add_credits` with `credit_pack_reference()` — the same key the
|
||||||
|
webhook writes — never a hand-set balance, so a late OC delivery and this
|
||||||
|
pass cannot both credit the same purchase.
|
||||||
|
"""
|
||||||
|
if CREDIT_RECONCILE == "off":
|
||||||
|
return {"status": "skipped", "reason": "credit_reconcile_off"}
|
||||||
|
try:
|
||||||
|
orders = await fetch_collective_orders(max_pages=CREDIT_RECONCILE_MAX_PAGES)
|
||||||
|
except Exception as e: # noqa: BLE001
|
||||||
|
# Never "nothing missing": an unreadable order list has to read as UNKNOWN.
|
||||||
|
logger.warning("Credit reconcile: order fetch failed; skipping (fail-safe): %s", e)
|
||||||
|
return {"status": "skipped", "reason": "order_fetch_failed", "detail": str(e)[:200]}
|
||||||
|
try:
|
||||||
|
index = await fetch_member_index()
|
||||||
|
except Exception as e: # noqa: BLE001
|
||||||
|
logger.warning("Credit reconcile: member index fetch failed; skipping: %s", e)
|
||||||
|
return {"status": "skipped", "reason": "member_index_failed", "detail": str(e)[:200]}
|
||||||
|
|
||||||
|
# Orders are matched on the slug from the LIVE OC list, never on the stored
|
||||||
|
# `members.slug` column, which can hold a stale `guest-…` from provisioning.
|
||||||
|
slug2email: dict[str, str] = {}
|
||||||
|
for email, rec in index.items():
|
||||||
|
s = (rec.get("slug") or "").strip()
|
||||||
|
if s:
|
||||||
|
slug2email.setdefault(s, email)
|
||||||
|
|
||||||
|
conn = get_db()
|
||||||
|
# Positional rows: get_db() hands back a plain connection (no row_factory),
|
||||||
|
# which is how the rest of this module reads it too.
|
||||||
|
ledger_rows = [
|
||||||
|
{"email": r[0], "delta": r[1], "reference": r[2], "created_at": r[3]}
|
||||||
|
for r in conn.execute(
|
||||||
|
"SELECT email, delta, reference, created_at FROM credit_ledger")]
|
||||||
|
held_rows = [
|
||||||
|
{"id": r[0], "slug": r[1], "email": r[2], "amount": r[3],
|
||||||
|
"oc_reference": r[4], "status": r[5], "created_at": r[6]}
|
||||||
|
for r in conn.execute(
|
||||||
|
"SELECT id, slug, email, amount, oc_reference, status, created_at "
|
||||||
|
"FROM held_credits")]
|
||||||
|
active = {r[0].strip().lower() for r in conn.execute("SELECT email FROM members WHERE active = 1")}
|
||||||
|
conn.close()
|
||||||
|
|
||||||
|
cutoff = datetime.now(timezone.utc) - timedelta(days=CREDIT_RECONCILE_DAYS)
|
||||||
|
packs = 0
|
||||||
|
unaccounted: list[dict] = []
|
||||||
|
for o in orders:
|
||||||
|
if (o.get("status") or "").upper() != "PAID":
|
||||||
|
continue
|
||||||
|
if (o.get("frequency") or "").upper() != "ONETIME":
|
||||||
|
continue
|
||||||
|
if "credit" not in (((o.get("tier") or {}).get("name") or "").lower()):
|
||||||
|
continue
|
||||||
|
when = _parse_ts(o.get("createdAt"))
|
||||||
|
if when and when < cutoff:
|
||||||
|
continue
|
||||||
|
packs += 1
|
||||||
|
if not credit_pack_accounted(o, ledger_rows, held_rows):
|
||||||
|
unaccounted.append(o)
|
||||||
|
|
||||||
|
acting = bool(apply and CREDIT_RECONCILE == "on")
|
||||||
|
candidates: list[dict] = []
|
||||||
|
applied: list[dict] = []
|
||||||
|
needs_hold: list[dict] = []
|
||||||
|
for o in unaccounted:
|
||||||
|
slug = ((o.get("fromAccount") or {}).get("slug") or "").strip()
|
||||||
|
rec = {"slug": slug,
|
||||||
|
"order_id": o.get("id"),
|
||||||
|
"amount": float((o.get("amount") or {}).get("value") or 0),
|
||||||
|
"created_at": str(o.get("createdAt") or "")[:19],
|
||||||
|
"tier": (o.get("tier") or {}).get("name")}
|
||||||
|
email = slug2email.get(slug, "")
|
||||||
|
if not email or email.lower() not in active:
|
||||||
|
# A purchase by someone who is not an active member keeps the
|
||||||
|
# hold-and-email path. This pass must never mint credits for a buyer
|
||||||
|
# who has no membership to spend them under.
|
||||||
|
rec["reason"] = "no active member for this OC slug — hold path, not a grant"
|
||||||
|
needs_hold.append(rec)
|
||||||
|
continue
|
||||||
|
rec["email"] = email
|
||||||
|
ref = credit_pack_reference(slug, {"idV2": o.get("id")})
|
||||||
|
rec["reference"] = ref
|
||||||
|
candidates.append(rec)
|
||||||
|
if not acting or credit_reference_exists(ref):
|
||||||
|
continue
|
||||||
|
new_balance = add_credits(email, rec["amount"],
|
||||||
|
reason="credit pack purchase (Open Collective)",
|
||||||
|
reference=ref)
|
||||||
|
try:
|
||||||
|
await sync_team_budget_from_balance(email)
|
||||||
|
rec["result"] = f"granted (balance {new_balance})"
|
||||||
|
except Exception as e: # noqa: BLE001
|
||||||
|
# The ledger row is the record that matters; the budget push is
|
||||||
|
# retried by the next sweep rather than failing the grant.
|
||||||
|
logger.warning("Credit reconcile: budget sync failed for %s: %s", email, e)
|
||||||
|
rec["result"] = f"granted (balance {new_balance}); budget sync failed: {str(e)[:80]}"
|
||||||
|
applied.append(rec)
|
||||||
|
logger.warning("Credit reconcile: GRANTED $%.2f to %s for order %s (was missing)",
|
||||||
|
rec["amount"], email, rec["order_id"])
|
||||||
|
|
||||||
|
if candidates and not acting:
|
||||||
|
logger.warning(
|
||||||
|
"Credit reconcile: %d PAID credit pack(s) with no record in our books "
|
||||||
|
"(report mode — nothing changed): %s", len(candidates),
|
||||||
|
"; ".join(f"${c['amount']:.2f} {c['slug']} {c['created_at']}" for c in candidates))
|
||||||
|
|
||||||
|
return {
|
||||||
|
"status": "ok",
|
||||||
|
"dry_run": not acting,
|
||||||
|
"window_days": CREDIT_RECONCILE_DAYS,
|
||||||
|
"orders_read": len(orders),
|
||||||
|
"packs_in_window": packs,
|
||||||
|
"unaccounted": len(candidates) + len(needs_hold),
|
||||||
|
"candidates": candidates,
|
||||||
|
"needs_hold": needs_hold,
|
||||||
|
"applied": applied,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
async def reconcile_allowance_shortfalls(send: bool = False) -> dict:
|
async def reconcile_allowance_shortfalls(send: bool = False) -> dict:
|
||||||
"""Credit members whose PAID cycle Open Collective's billing-day alignment truncated.
|
"""Credit members whose PAID cycle Open Collective's billing-day alignment truncated.
|
||||||
|
|
||||||
@@ -7860,9 +8116,33 @@ async def admin_sweep(request: Request):
|
|||||||
except Exception as exc: # noqa: BLE001
|
except Exception as exc: # noqa: BLE001
|
||||||
result["shortfall_credits"] = {"error": str(exc)[:200]}
|
result["shortfall_credits"] = {"error": str(exc)[:200]}
|
||||||
|
|
||||||
|
# Paid credit packs with no record in our books (issue #7). REPORT-ONLY
|
||||||
|
# unless CREDIT_RECONCILE=on; wrapped so it can never break reconciliation.
|
||||||
|
try:
|
||||||
|
result["credit_reconcile"] = await reconcile_credit_packs(
|
||||||
|
apply=(CREDIT_RECONCILE == "on"))
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
result["credit_reconcile"] = {"error": str(exc)[:200]}
|
||||||
|
|
||||||
return result
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/admin/credit-pack-reconcile")
|
||||||
|
@limiter.limit("20/minute")
|
||||||
|
async def admin_credit_pack_reconcile(request: Request, apply: int = 0):
|
||||||
|
"""Reconcile PAID credit packs against our books — DRY RUN by default.
|
||||||
|
|
||||||
|
Authenticated by the X-Admin-Token header, like the other /admin/* routes.
|
||||||
|
`?apply=1` asks for the grants to be applied, and is honoured only when
|
||||||
|
CREDIT_RECONCILE=on: the point of the default is that an operator can read
|
||||||
|
this pass's output in production before any member's money is touched. Every
|
||||||
|
grant uses the same per-order reference the webhook writes, so running it
|
||||||
|
twice — or running it after a late delivery — credits a purchase once.
|
||||||
|
"""
|
||||||
|
_verify_admin_token(request)
|
||||||
|
return await reconcile_credit_packs(apply=bool(apply))
|
||||||
|
|
||||||
|
|
||||||
def _service_sync_pending() -> list[str]:
|
def _service_sync_pending() -> list[str]:
|
||||||
"""Active members still waiting on their Loomio invitation.
|
"""Active members still waiting on their Loomio invitation.
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,315 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Regression test: credit-pack reconciliation (issue #7).
|
||||||
|
|
||||||
|
WHY THIS EXISTS
|
||||||
|
Credits are granted on exactly ONE path — the OC `order.processed` webhook —
|
||||||
|
and that path can fail AFTER the money has moved. On 2026-11-13 two purchases
|
||||||
|
did: a $50 pack whose delivery 500'd (httpx's invisible 5-second default), and
|
||||||
|
a $10 pack with no delivery in the retained log at all. Neither produced a
|
||||||
|
ledger row, and nothing on a schedule noticed. The reconcile pass reads OC's
|
||||||
|
OWN order list and compares it with our books.
|
||||||
|
|
||||||
|
The thing that makes this dangerous rather than merely absent is the matching:
|
||||||
|
our own stored references are NOT uniform, so the obvious
|
||||||
|
`oc:<slug>:<order.id>` test reported 21 already-credited purchases as missing
|
||||||
|
on its first run. A pass that "fixes" those would double-credit members. This
|
||||||
|
test pins down, case by case, what counts as accounted for:
|
||||||
|
|
||||||
|
* newer ledger rows carry the v2 id `oc:alice:or_…`
|
||||||
|
* older ledger rows carry a HYPHENATED id `oc:bob:8k03reyd-…`
|
||||||
|
* legacy rows carry an EMPTY suffix `oc:erin:`
|
||||||
|
* non-member purchases live in `held_credits`, not the ledger, and are
|
||||||
|
accounted for — they took the hold path on purpose
|
||||||
|
|
||||||
|
and that a purchase with NO such record is reported (dry run) or granted ONCE
|
||||||
|
(apply), using the same reference the webhook writes, so the two paths are
|
||||||
|
idempotent against each other in both directions.
|
||||||
|
|
||||||
|
Network is STUBBED; assertions are on our own recorded state.
|
||||||
|
|
||||||
|
Runnable: python3 tests/test_credit_pack_reconcile.py (from the repo root)
|
||||||
|
"""
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import asyncio
|
||||||
|
import tempfile
|
||||||
|
from datetime import datetime, timedelta, timezone
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
|
||||||
|
os.environ["DB_PATH"] = os.path.join(tempfile.mkdtemp(), "fixture.db")
|
||||||
|
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||||
|
import app.main as m # noqa: E402
|
||||||
|
|
||||||
|
m.DB_PATH = os.environ["DB_PATH"]
|
||||||
|
m.ADMIN_TOKEN = "test-admin-token"
|
||||||
|
m.ADMIN_TOKEN_LEGACY = ""
|
||||||
|
m.CREDIT_RECONCILE = "report" # the shipped default
|
||||||
|
m.CREDIT_RECONCILE_DAYS = 30
|
||||||
|
|
||||||
|
NOW = datetime.now(timezone.utc)
|
||||||
|
|
||||||
|
|
||||||
|
def iso(days_ago: float) -> str:
|
||||||
|
return (NOW - timedelta(days=days_ago)).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||||
|
|
||||||
|
|
||||||
|
def stamp(days_ago: float) -> str:
|
||||||
|
return (NOW - timedelta(days=days_ago)).strftime("%Y-%m-%d %H:%M:%S")
|
||||||
|
|
||||||
|
|
||||||
|
def order(oid, slug, amount, days_ago=1.0, tier="Credit pack", status="PAID",
|
||||||
|
frequency="ONETIME"):
|
||||||
|
return {"id": oid, "status": status, "createdAt": iso(days_ago),
|
||||||
|
"frequency": frequency, "amount": {"value": amount, "currency": "USD"},
|
||||||
|
"platformTipAmount": {"value": 0}, "totalAmount": {"value": amount},
|
||||||
|
"tier": {"name": tier, "slug": (tier or "").lower().replace(" ", "-")},
|
||||||
|
"fromAccount": {"slug": slug, "name": slug.title(), "type": "INDIVIDUAL"}}
|
||||||
|
|
||||||
|
|
||||||
|
ORDERS = [
|
||||||
|
# accounted by the EXACT v2 reference (and the ledger amount deliberately
|
||||||
|
# disagrees, to prove the exact match does not need the fuzzy second signal)
|
||||||
|
order("or_AAA", "alice", 50, 2.0),
|
||||||
|
# accounted only by the SECOND signal: older rows carry a hyphenated id
|
||||||
|
order("or_BBB", "bob", 50, 3.0),
|
||||||
|
# accounted only by the SECOND signal: a legacy EMPTY-suffix reference
|
||||||
|
order("or_EEE", "erin", 5, 4.0),
|
||||||
|
# accounted as HELD — a non-member purchase recorded on the hold path
|
||||||
|
order("or_HHH", "holdguy", 10, 5.0),
|
||||||
|
# GENUINELY UNACCOUNTED, and by an active member → the one real candidate
|
||||||
|
order("or_CCC", "carol", 10, 0.5),
|
||||||
|
# unaccounted, but the buyer is not an active member → hold path, not a grant
|
||||||
|
order("or_DDD", "dave", 10, 0.7),
|
||||||
|
# unaccounted and active, but outside the window → not this pass's business
|
||||||
|
order("or_FFF", "frank", 10, 100.0),
|
||||||
|
# not a credit pack / not paid / not one-time → never examined as a pack
|
||||||
|
order("or_G1", "carol", 15, 1.0, tier="Membership"),
|
||||||
|
order("or_G2", "carol", 10, 1.0, status="PENDING"),
|
||||||
|
order("or_G3", "carol", 10, 1.0, status="REFUNDED"),
|
||||||
|
order("or_G4", "carol", 10, 1.0, frequency="MONTHLY"),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
class Boom(Exception):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
async def fake_orders(per_page=100, max_pages=3):
|
||||||
|
if FAIL_ORDERS:
|
||||||
|
raise Boom("OC said 500")
|
||||||
|
return list(ORDERS)
|
||||||
|
|
||||||
|
|
||||||
|
async def fake_index():
|
||||||
|
if FAIL_INDEX:
|
||||||
|
raise Boom("member index unavailable")
|
||||||
|
return {e: {"slug": e.split("@")[0]} for e in
|
||||||
|
("alice@x.org", "bob@x.org", "carol@x.org", "dave@x.org",
|
||||||
|
"erin@x.org", "frank@x.org")}
|
||||||
|
|
||||||
|
|
||||||
|
BUDGET_PUSHES = []
|
||||||
|
|
||||||
|
|
||||||
|
async def fake_budget(email):
|
||||||
|
BUDGET_PUSHES.append(email)
|
||||||
|
return 25.0
|
||||||
|
|
||||||
|
|
||||||
|
FAIL_ORDERS = False
|
||||||
|
FAIL_INDEX = False
|
||||||
|
|
||||||
|
results = []
|
||||||
|
|
||||||
|
|
||||||
|
def check(name, ok):
|
||||||
|
results.append((name, bool(ok)))
|
||||||
|
|
||||||
|
|
||||||
|
def ledger_rows():
|
||||||
|
conn = m.get_db()
|
||||||
|
conn.row_factory = __import__("sqlite3").Row
|
||||||
|
rows = [dict(r) for r in conn.execute(
|
||||||
|
"SELECT email, delta, reference, created_at FROM credit_ledger ORDER BY id")]
|
||||||
|
conn.close()
|
||||||
|
return rows
|
||||||
|
|
||||||
|
|
||||||
|
def balance(email):
|
||||||
|
return m.get_credit_balance(email)
|
||||||
|
|
||||||
|
|
||||||
|
def seed():
|
||||||
|
conn = m.get_db()
|
||||||
|
conn.execute("DELETE FROM members")
|
||||||
|
conn.execute("DELETE FROM credit_ledger")
|
||||||
|
conn.execute("DELETE FROM held_credits")
|
||||||
|
for email, slug, active in (("alice@x.org", "alice", 1), ("bob@x.org", "bob", 1),
|
||||||
|
("carol@x.org", "carol", 1), ("dave@x.org", "dave", 0),
|
||||||
|
("erin@x.org", "erin", 1), ("frank@x.org", "frank", 1)):
|
||||||
|
conn.execute(
|
||||||
|
"INSERT INTO members (email, slug, active, account_type, credit_balance) "
|
||||||
|
"VALUES (?, ?, ?, 'member', 0)", (email, slug, active))
|
||||||
|
# alice: the v2 form, with an amount that DISAGREES with the order ($48 ≠ $50)
|
||||||
|
conn.execute("INSERT INTO credit_ledger (email, delta, reason, reference, created_at) "
|
||||||
|
"VALUES ('alice@x.org', 48.0, 'credit pack', ?, ?)",
|
||||||
|
("oc:alice:or_AAA", stamp(2.0)))
|
||||||
|
# bob: the hyphenated legacy id for the SAME order (or_BBB)
|
||||||
|
conn.execute("INSERT INTO credit_ledger (email, delta, reason, reference, created_at) "
|
||||||
|
"VALUES ('bob@x.org', 50.0, 'credit pack', ?, ?)",
|
||||||
|
("oc:bob:8k03reyd-5agmq5x4-4nkplbwo-z7j4nxv9", stamp(3.0)))
|
||||||
|
# erin: the empty-suffix legacy form
|
||||||
|
conn.execute("INSERT INTO credit_ledger (email, delta, reason, reference, created_at) "
|
||||||
|
"VALUES ('erin@x.org', 5.0, 'credit pack', 'oc:erin:', ?)", (stamp(4.0),))
|
||||||
|
# holdguy: a non-member purchase held, never a ledger row
|
||||||
|
conn.execute("INSERT INTO held_credits (slug, email, name, amount, oc_reference, "
|
||||||
|
"status, created_at) VALUES ('holdguy', 'hold@x.org', 'Hold Guy', 10.0, "
|
||||||
|
"'oc:holdguy:', 'pending', ?)", (stamp(5.0),))
|
||||||
|
conn.commit()
|
||||||
|
conn.close()
|
||||||
|
|
||||||
|
|
||||||
|
def names(res):
|
||||||
|
return sorted(c["slug"] for c in res["candidates"])
|
||||||
|
|
||||||
|
|
||||||
|
async def main():
|
||||||
|
global FAIL_ORDERS, FAIL_INDEX
|
||||||
|
m.fetch_collective_orders = fake_orders
|
||||||
|
m.fetch_member_index = fake_index
|
||||||
|
m.sync_team_budget_from_balance = fake_budget
|
||||||
|
|
||||||
|
# ---------- dry run: report only, change nothing --------------------------
|
||||||
|
seed()
|
||||||
|
before = ledger_rows()
|
||||||
|
res = await m.reconcile_credit_packs(apply=False)
|
||||||
|
check(f"dry run reports ok (got {res.get('status')})", res.get("status") == "ok")
|
||||||
|
check("dry run is marked dry_run", res.get("dry_run") is True)
|
||||||
|
check(f"packs in window counted (got {res.get('packs_in_window')})",
|
||||||
|
res.get("packs_in_window") == 6)
|
||||||
|
check(f"only the unaccounted active-member pack is a candidate (got {names(res)})",
|
||||||
|
names(res) == ["carol"])
|
||||||
|
check(f"the inactive member's pack is a hold, not a grant (got {res['needs_hold']})",
|
||||||
|
[c["slug"] for c in res["needs_hold"]] == ["dave"])
|
||||||
|
check("dry run applied nothing", res.get("applied") == [])
|
||||||
|
check("dry run wrote no ledger row", ledger_rows() == before)
|
||||||
|
check(f"dry run left carol's balance alone (got {balance('carol@x.org')})",
|
||||||
|
balance("carol@x.org") == 0.0)
|
||||||
|
check("dry run pushed no budget", BUDGET_PUSHES == [])
|
||||||
|
|
||||||
|
# the three accounting shapes that a v2-only match would have called missing
|
||||||
|
for oid, slug, why in (("or_AAA", "alice", "exact v2 reference"),
|
||||||
|
("or_BBB", "bob", "hyphenated legacy id"),
|
||||||
|
("or_EEE", "erin", "empty-suffix legacy reference"),
|
||||||
|
("or_HHH", "holdguy", "held (non-member) purchase")):
|
||||||
|
o = next(x for x in ORDERS if x["id"] == oid)
|
||||||
|
ev = m.credit_pack_accounted(o, ledger_rows(), [
|
||||||
|
dict(r) for r in
|
||||||
|
conn_rows("SELECT id, slug, email, amount, oc_reference, status, created_at FROM held_credits")
|
||||||
|
])
|
||||||
|
check(f"accounted: {why} ({oid}) -> {ev}", ev is not None)
|
||||||
|
check("a genuinely unaccounted order stays unaccounted",
|
||||||
|
m.credit_pack_accounted(next(x for x in ORDERS if x["id"] == "or_CCC"),
|
||||||
|
ledger_rows(), []) is None)
|
||||||
|
|
||||||
|
# ---------- a read that fails is UNKNOWN, never 'nothing missing' ---------
|
||||||
|
FAIL_ORDERS = True
|
||||||
|
res = await m.reconcile_credit_packs(apply=False)
|
||||||
|
check(f"order fetch failure is skipped/order_fetch_failed (got {res})",
|
||||||
|
res.get("status") == "skipped" and res.get("reason") == "order_fetch_failed")
|
||||||
|
FAIL_ORDERS = False
|
||||||
|
FAIL_INDEX = True
|
||||||
|
res = await m.reconcile_credit_packs(apply=False)
|
||||||
|
check(f"member index failure is skipped/member_index_failed (got {res})",
|
||||||
|
res.get("status") == "skipped" and res.get("reason") == "member_index_failed")
|
||||||
|
FAIL_INDEX = False
|
||||||
|
|
||||||
|
# ---------- apply=1 while the policy is still 'report' --------------------
|
||||||
|
seed()
|
||||||
|
res = await m.reconcile_credit_packs(apply=True)
|
||||||
|
check("apply is IGNORED while CREDIT_RECONCILE != 'on'",
|
||||||
|
res.get("dry_run") is True and res.get("applied") == [])
|
||||||
|
check(f"and still nothing was credited (got {balance('carol@x.org')})",
|
||||||
|
balance("carol@x.org") == 0.0)
|
||||||
|
|
||||||
|
# ---------- CREDIT_RECONCILE=on: grant exactly once ----------------------
|
||||||
|
seed()
|
||||||
|
m.CREDIT_RECONCILE = "on"
|
||||||
|
res = await m.reconcile_credit_packs(apply=True)
|
||||||
|
check(f"apply grants the one candidate (got {res.get('applied')})",
|
||||||
|
[c["slug"] for c in res.get("applied", [])] == ["carol"])
|
||||||
|
check(f"carol credited $10 (got {balance('carol@x.org')})",
|
||||||
|
balance("carol@x.org") == 10.0)
|
||||||
|
ref = f"oc:carol:or_CCC"
|
||||||
|
check(f"the grant used the webhook's own reference ({ref})",
|
||||||
|
any(r["reference"] == ref for r in ledger_rows()))
|
||||||
|
check(f"exactly one ledger row was added (got {len(ledger_rows())})",
|
||||||
|
len(ledger_rows()) == 4)
|
||||||
|
check("the budget was pushed once", BUDGET_PUSHES == ["carol@x.org"])
|
||||||
|
check("the inactive member was NOT credited",
|
||||||
|
balance("dave@x.org") == 0.0)
|
||||||
|
check("the out-of-window member was NOT credited",
|
||||||
|
balance("frank@x.org") == 0.0)
|
||||||
|
|
||||||
|
# the webhook and this pass must dedupe each other, whichever runs first
|
||||||
|
check("the webhook would now answer already_credited for this order",
|
||||||
|
m.credit_reference_exists(m.credit_pack_reference("carol", {"idV2": "or_CCC"})))
|
||||||
|
|
||||||
|
res = await m.reconcile_credit_packs(apply=True)
|
||||||
|
check(f"a second apply changes nothing (got {res.get('applied')})",
|
||||||
|
res.get("applied") == [] and res.get("candidates") == [])
|
||||||
|
check(f"carol's balance unchanged by the re-run (got {balance('carol@x.org')})",
|
||||||
|
balance("carol@x.org") == 10.0)
|
||||||
|
check(f"still one row for the order (got {len(ledger_rows())})",
|
||||||
|
len(ledger_rows()) == 4)
|
||||||
|
|
||||||
|
# ---------- the endpoint -------------------------------------------------
|
||||||
|
# Re-seed: the direct apply above already credited carol, so the endpoint
|
||||||
|
# would have no candidate and the check would pass for the wrong reason.
|
||||||
|
seed()
|
||||||
|
m.CREDIT_RECONCILE = "report"
|
||||||
|
|
||||||
|
async def endpoint_checks():
|
||||||
|
transport = httpx.ASGITransport(app=m.app)
|
||||||
|
async with httpx.AsyncClient(transport=transport, base_url="http://t") as c:
|
||||||
|
r = await c.post("/admin/credit-pack-reconcile")
|
||||||
|
check(f"endpoint needs the admin token (got {r.status_code})",
|
||||||
|
r.status_code in (401, 403))
|
||||||
|
r = await c.post("/admin/credit-pack-reconcile",
|
||||||
|
headers={"X-Admin-Token": "test-admin-token"})
|
||||||
|
body = r.json()
|
||||||
|
check(f"endpoint defaults to a DRY RUN (got {body})",
|
||||||
|
r.status_code == 200 and body.get("dry_run") is True)
|
||||||
|
r = await c.post("/admin/credit-pack-reconcile?apply=1",
|
||||||
|
headers={"X-Admin-Token": "test-admin-token"})
|
||||||
|
body = r.json()
|
||||||
|
check("endpoint ?apply=1 is still a dry run while the policy is 'report'",
|
||||||
|
body.get("dry_run") is True and body.get("applied") == [])
|
||||||
|
m.CREDIT_RECONCILE = "on"
|
||||||
|
r = await c.post("/admin/credit-pack-reconcile?apply=1",
|
||||||
|
headers={"X-Admin-Token": "test-admin-token"})
|
||||||
|
body = r.json()
|
||||||
|
check(f"endpoint ?apply=1 grants once the policy is 'on' (got {body})",
|
||||||
|
body.get("dry_run") is False
|
||||||
|
and [x["slug"] for x in body.get("applied", [])] == ["carol"])
|
||||||
|
m.CREDIT_RECONCILE = "report"
|
||||||
|
|
||||||
|
await endpoint_checks()
|
||||||
|
|
||||||
|
|
||||||
|
def conn_rows(sql):
|
||||||
|
conn = m.get_db()
|
||||||
|
conn.row_factory = __import__("sqlite3").Row
|
||||||
|
rows = conn.execute(sql).fetchall()
|
||||||
|
conn.close()
|
||||||
|
return rows
|
||||||
|
|
||||||
|
|
||||||
|
asyncio.run(main())
|
||||||
|
|
||||||
|
fails = sum(not ok for _, ok in results)
|
||||||
|
for name, ok in results:
|
||||||
|
print(f" [{'PASS' if ok else 'FAIL'}] {name}")
|
||||||
|
print(f"\n{len(results) - fails}/{len(results)} passed")
|
||||||
|
sys.exit(1 if fails else 0)
|
||||||
Reference in new issue
Block a user