reconcile paid credit packs against the ledger, report-first (issue #7)
ci/woodpecker/push/woodpecker Pipeline was canceled
ci/woodpecker/push/woodpecker Pipeline was canceled
Credits have one path in: the OC order.processed webhook. On 2026-11-13 two paid packs got nothing through it — a $50 whose delivery 500'd on httpx's 5s default, and a $10 with no delivery in the retained log at all — and nothing on a schedule noticed. This adds a pass that reads OC's own order list and compares it with our books over a day-based window (not "whatever the log still holds": the $10 gap was 11h old and the log had lost the whole morning). Report-first: CREDIT_RECONCILE defaults to 'report' (log findings, change nothing) and grants only when set to 'on'. Grants go through add_credits() with credit_pack_reference(), the same per-order key the webhook writes, so a late delivery and this pass are idempotent against each other in both directions. A buyer with no active membership is reported, never credited — those keep the hold-and-email path. credit_pack_accounted() encodes what our own rows actually look like: a v2 id, a hyphenated legacy id, an empty-suffix legacy reference, and held_credits rows. A bare oc:<slug>:<order.id> match read 21 credited purchases as missing during development, so the fallback demands a second signal (member + tip-free amount + date) before calling an order credited. POST /admin/credit-pack-reconcile (X-Admin-Token) runs it on demand, dry unless ?apply=1 and the policy is on. 33 new checks; all 18 suites green.
This commit is contained in:
1 parent
32e744d9ee
commit
5868ca82e8
3 files changed
+620
-1
No files matched your search
@@ -191,7 +191,31 @@ Four jobs run inside the container via `CloudronManifest.json` → `addons.sched
|
||||
against the live Open Collective list: provisions missed signups, deactivates
|
||||
lapsed members, syncs Loomio + Matrix. Idempotent; a missed webhook
|
||||
self-heals within a day. The same sweep also runs the payment-driven
|
||||
allowance reconciliation (below).
|
||||
allowance reconciliation (below) and the credit-pack reconciliation.
|
||||
|
||||
**Credit-pack reconciliation** (`CREDIT_RECONCILE`, default `report`) — credits
|
||||
are granted on exactly one path, the OC `order.processed` webhook, and that
|
||||
path can fail AFTER the money has moved: on 2026-11-13 a $50 pack's delivery
|
||||
500'd on httpx's invisible 5-second default, and a $10 pack had no delivery in
|
||||
the retained log at all. Neither left a ledger row and nothing noticed. This
|
||||
pass reads OC's OWN order list (`fetch_collective_orders`, needs the `orders`
|
||||
scope on `OC_PERSONAL_TOKEN`, granted 2026-11-13) and compares it with our
|
||||
books over `CREDIT_RECONCILE_DAYS` (default 30 — days, not "whatever our log
|
||||
still holds": the $10 gap was 11 hours old and the log had already lost the
|
||||
whole morning). In `report` mode it logs findings and changes NOTHING; grants
|
||||
happen only when `CREDIT_RECONCILE=on`. Every grant uses
|
||||
`credit_pack_reference()` — the same per-order key the webhook writes — so a
|
||||
late delivery and this pass cannot both credit one purchase. A buyer who is not
|
||||
an active member is reported, never credited: those keep the hold-and-email
|
||||
path. On demand: `POST /admin/credit-pack-reconcile` (`X-Admin-Token`), a dry
|
||||
run unless `?apply=1` and the policy is `on`.
|
||||
|
||||
Matching caution: our own references are NOT uniform, so a bare
|
||||
`oc:<slug>:<order.id>` test reports credited purchases as missing (it read 21
|
||||
of them that way, all false). `credit_pack_accounted()` matches the exact
|
||||
reference when one exists and otherwise requires a second signal — same member,
|
||||
same tip-free amount, within days — and counts a `held_credits` row as
|
||||
accounted for.
|
||||
|
||||
**Service-invite triggers:** the sweep is the backstop, not the primary path.
|
||||
Two faster triggers exist, because a member who has just set up their account
|
||||
|
||||
+280
@@ -7661,6 +7661,262 @@ def shortfall_email(name: str, amount: float, prev: str, cur: str,
|
||||
return subject, text, html
|
||||
|
||||
|
||||
# --- Credit-pack reconciliation (issue #7) --------------------------------- #
|
||||
# Credits are granted on exactly ONE path: the OC `order.processed` webhook. If
|
||||
# that delivery fails, or never arrives, the co-op holds a member's money and
|
||||
# nothing on a schedule notices. 2026-11-13 produced two of those in one day — a
|
||||
# $50 pack whose delivery 500'd on httpx's 5-second default, and a $10 pack with
|
||||
# no delivery in the retained log at all, found only by reading OC's own order
|
||||
# list against our books. This pass makes that routine.
|
||||
#
|
||||
# CREDIT_RECONCILE: report (default) | on | off
|
||||
# report — find and LOG unaccounted packs; change nothing
|
||||
# on — grant them, with the SAME per-order reference the webhook writes, so
|
||||
# the webhook and this pass are idempotent against each other in both
|
||||
# directions (a late delivery is a no-op; a pass after a delivery is a
|
||||
# no-op too)
|
||||
# off — skip the pass entirely
|
||||
CREDIT_RECONCILE = (os.environ.get("CREDIT_RECONCILE", "report") or "report").strip().lower()
|
||||
# How far back to look — DAYS off OC's own data, not "whatever our log still
|
||||
# holds": the $10 gap was 11 hours old and the portal's log had already lost the
|
||||
# whole morning, so a log-derived window cannot see the failures it exists for.
|
||||
CREDIT_RECONCILE_DAYS = int(os.environ.get("CREDIT_RECONCILE_DAYS", "30") or "30")
|
||||
CREDIT_RECONCILE_MAX_PAGES = int(os.environ.get("CREDIT_RECONCILE_MAX_PAGES", "3") or "3")
|
||||
|
||||
|
||||
async def fetch_collective_orders(per_page: int = 100, max_pages: int = 3) -> list[dict]:
|
||||
"""Recent orders for the collective, newest first. Raises on any failure.
|
||||
|
||||
Every failure raises rather than returning a short list: a caller that reads
|
||||
a partial read as a complete one would conclude "nothing missing" from a
|
||||
request that never answered. Needs the `orders` scope on OC_PERSONAL_TOKEN
|
||||
(granted 2026-11-13; before that this answered `The Personal Token is not
|
||||
allowed for operations in scope "orders"`).
|
||||
"""
|
||||
if not OC_PERSONAL_TOKEN:
|
||||
raise RuntimeError("OC_PERSONAL_TOKEN not set; cannot fetch orders")
|
||||
query = (
|
||||
'{ collective(slug: "%s") { orders(limit: %d, offset: %d) { totalCount nodes { '
|
||||
"id status createdAt processedAt frequency amount { value currency } "
|
||||
"platformTipAmount { value } totalAmount { value } "
|
||||
"tier { name slug } fromAccount { slug name type } } } } }"
|
||||
)
|
||||
out: list[dict] = []
|
||||
async with httpx.AsyncClient(timeout=30.0) as client:
|
||||
for page in range(max(1, max_pages)):
|
||||
r = await client.post(
|
||||
OC_GRAPHQL_URL,
|
||||
headers={"Content-Type": "application/json",
|
||||
"Authorization": f"Bearer {OC_PERSONAL_TOKEN}",
|
||||
"Api-Version": "2022-01-01",
|
||||
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64)"},
|
||||
json={"query": query % (OC_COLLECTIVE_SLUG, per_page, page * per_page)},
|
||||
)
|
||||
if r.status_code != 200:
|
||||
raise RuntimeError(f"OC order fetch failed: HTTP {r.status_code}")
|
||||
body = r.json()
|
||||
if body.get("errors"):
|
||||
raise RuntimeError(f"OC order fetch errors: {str(body['errors'])[:200]}")
|
||||
conn_ = ((body.get("data") or {}).get("collective") or {}).get("orders") or {}
|
||||
nodes = conn_.get("nodes") or []
|
||||
out.extend(nodes)
|
||||
if len(nodes) < per_page or len(out) >= (conn_.get("totalCount") or 0):
|
||||
break
|
||||
return out
|
||||
|
||||
|
||||
def _parse_ts(s) -> datetime | None:
|
||||
"""OC sends ISO-8601 (…Z); our ledger rows are 'YYYY-MM-DD HH:MM:SS' (UTC)."""
|
||||
if not s:
|
||||
return None
|
||||
t = str(s).strip()
|
||||
if t.endswith("Z"):
|
||||
t = t[:-1] + "+00:00"
|
||||
if "T" not in t:
|
||||
t = t.replace(" ", "T", 1)
|
||||
try:
|
||||
d = datetime.fromisoformat(t)
|
||||
except ValueError:
|
||||
return None
|
||||
return d if d.tzinfo else d.replace(tzinfo=timezone.utc)
|
||||
|
||||
|
||||
def credit_pack_accounted(order: dict, ledger_rows: list, held_rows: list) -> str | None:
|
||||
"""Evidence that this order is ALREADY in our books, else None.
|
||||
|
||||
The reference is NOT uniform across our own rows, and a bare
|
||||
`oc:<slug>:<order.id>` match therefore reports credited purchases as missing
|
||||
— it read 21 of them that way on 2026-11-13, all false:
|
||||
|
||||
* newer ledger rows `oc:nightrose:or_apShJowSoPPj1dlM3ExTH` (the v2 id)
|
||||
* older ledger rows `oc:benlo_0:a47byg9n-xozdp84r-…` (a hyphenated id)
|
||||
* held rows `oc:madrush:` (EMPTY suffix)
|
||||
|
||||
So match the exact reference when the row carries one, and otherwise demand a
|
||||
SECOND signal — same member, same tip-free amount, within a few days — before
|
||||
calling an order credited. A held row counts as accounted for: the money was
|
||||
recorded on the hold path, which is what non-member purchases are supposed to
|
||||
take.
|
||||
"""
|
||||
slug = ((order.get("fromAccount") or {}).get("slug") or "").strip()
|
||||
oid = (order.get("id") or "").strip()
|
||||
try:
|
||||
amount = float((order.get("amount") or {}).get("value") or 0)
|
||||
except (TypeError, ValueError):
|
||||
amount = 0.0
|
||||
when = _parse_ts(order.get("createdAt"))
|
||||
|
||||
if oid and slug:
|
||||
exact = f"oc:{slug}:{oid}"
|
||||
for r in ledger_rows:
|
||||
if (r.get("reference") or "") == exact:
|
||||
return f"ledger {exact}"
|
||||
for h in held_rows:
|
||||
if (h.get("oc_reference") or "") == exact:
|
||||
return f"held #{h.get('id')} {exact}"
|
||||
|
||||
for rows, what in ((ledger_rows, "ledger"), (held_rows, "held")):
|
||||
for r in rows:
|
||||
ref = (r.get("reference") or r.get("oc_reference") or "")
|
||||
email = (r.get("email") or "").lower()
|
||||
if slug and slug.lower() not in ref.lower() and slug.lower() not in email:
|
||||
continue
|
||||
try:
|
||||
amt = float(r.get("delta") if what == "ledger" else r.get("amount") or 0)
|
||||
except (TypeError, ValueError):
|
||||
continue
|
||||
if abs(amt - amount) > 0.005:
|
||||
continue
|
||||
rt = _parse_ts(r.get("created_at"))
|
||||
if when and rt and abs((rt - when).total_seconds()) <= 5 * 86400:
|
||||
return f"{what} {ref or '(no ref)'} — matched on member + amount + date"
|
||||
return None
|
||||
|
||||
|
||||
async def reconcile_credit_packs(apply: bool = False) -> dict:
|
||||
"""Credit PAID credit packs that never reached our books (issue #7).
|
||||
|
||||
`apply=False` (report) is the default and changes NOTHING; granting also
|
||||
requires CREDIT_RECONCILE == "on", so the pass can run on every sweep without
|
||||
moving money until it has been read once in production. Every grant goes
|
||||
through `add_credits` with `credit_pack_reference()` — the same key the
|
||||
webhook writes — never a hand-set balance, so a late OC delivery and this
|
||||
pass cannot both credit the same purchase.
|
||||
"""
|
||||
if CREDIT_RECONCILE == "off":
|
||||
return {"status": "skipped", "reason": "credit_reconcile_off"}
|
||||
try:
|
||||
orders = await fetch_collective_orders(max_pages=CREDIT_RECONCILE_MAX_PAGES)
|
||||
except Exception as e: # noqa: BLE001
|
||||
# Never "nothing missing": an unreadable order list has to read as UNKNOWN.
|
||||
logger.warning("Credit reconcile: order fetch failed; skipping (fail-safe): %s", e)
|
||||
return {"status": "skipped", "reason": "order_fetch_failed", "detail": str(e)[:200]}
|
||||
try:
|
||||
index = await fetch_member_index()
|
||||
except Exception as e: # noqa: BLE001
|
||||
logger.warning("Credit reconcile: member index fetch failed; skipping: %s", e)
|
||||
return {"status": "skipped", "reason": "member_index_failed", "detail": str(e)[:200]}
|
||||
|
||||
# Orders are matched on the slug from the LIVE OC list, never on the stored
|
||||
# `members.slug` column, which can hold a stale `guest-…` from provisioning.
|
||||
slug2email: dict[str, str] = {}
|
||||
for email, rec in index.items():
|
||||
s = (rec.get("slug") or "").strip()
|
||||
if s:
|
||||
slug2email.setdefault(s, email)
|
||||
|
||||
conn = get_db()
|
||||
# Positional rows: get_db() hands back a plain connection (no row_factory),
|
||||
# which is how the rest of this module reads it too.
|
||||
ledger_rows = [
|
||||
{"email": r[0], "delta": r[1], "reference": r[2], "created_at": r[3]}
|
||||
for r in conn.execute(
|
||||
"SELECT email, delta, reference, created_at FROM credit_ledger")]
|
||||
held_rows = [
|
||||
{"id": r[0], "slug": r[1], "email": r[2], "amount": r[3],
|
||||
"oc_reference": r[4], "status": r[5], "created_at": r[6]}
|
||||
for r in conn.execute(
|
||||
"SELECT id, slug, email, amount, oc_reference, status, created_at "
|
||||
"FROM held_credits")]
|
||||
active = {r[0].strip().lower() for r in conn.execute("SELECT email FROM members WHERE active = 1")}
|
||||
conn.close()
|
||||
|
||||
cutoff = datetime.now(timezone.utc) - timedelta(days=CREDIT_RECONCILE_DAYS)
|
||||
packs = 0
|
||||
unaccounted: list[dict] = []
|
||||
for o in orders:
|
||||
if (o.get("status") or "").upper() != "PAID":
|
||||
continue
|
||||
if (o.get("frequency") or "").upper() != "ONETIME":
|
||||
continue
|
||||
if "credit" not in (((o.get("tier") or {}).get("name") or "").lower()):
|
||||
continue
|
||||
when = _parse_ts(o.get("createdAt"))
|
||||
if when and when < cutoff:
|
||||
continue
|
||||
packs += 1
|
||||
if not credit_pack_accounted(o, ledger_rows, held_rows):
|
||||
unaccounted.append(o)
|
||||
|
||||
acting = bool(apply and CREDIT_RECONCILE == "on")
|
||||
candidates: list[dict] = []
|
||||
applied: list[dict] = []
|
||||
needs_hold: list[dict] = []
|
||||
for o in unaccounted:
|
||||
slug = ((o.get("fromAccount") or {}).get("slug") or "").strip()
|
||||
rec = {"slug": slug,
|
||||
"order_id": o.get("id"),
|
||||
"amount": float((o.get("amount") or {}).get("value") or 0),
|
||||
"created_at": str(o.get("createdAt") or "")[:19],
|
||||
"tier": (o.get("tier") or {}).get("name")}
|
||||
email = slug2email.get(slug, "")
|
||||
if not email or email.lower() not in active:
|
||||
# A purchase by someone who is not an active member keeps the
|
||||
# hold-and-email path. This pass must never mint credits for a buyer
|
||||
# who has no membership to spend them under.
|
||||
rec["reason"] = "no active member for this OC slug — hold path, not a grant"
|
||||
needs_hold.append(rec)
|
||||
continue
|
||||
rec["email"] = email
|
||||
ref = credit_pack_reference(slug, {"idV2": o.get("id")})
|
||||
rec["reference"] = ref
|
||||
candidates.append(rec)
|
||||
if not acting or credit_reference_exists(ref):
|
||||
continue
|
||||
new_balance = add_credits(email, rec["amount"],
|
||||
reason="credit pack purchase (Open Collective)",
|
||||
reference=ref)
|
||||
try:
|
||||
await sync_team_budget_from_balance(email)
|
||||
rec["result"] = f"granted (balance {new_balance})"
|
||||
except Exception as e: # noqa: BLE001
|
||||
# The ledger row is the record that matters; the budget push is
|
||||
# retried by the next sweep rather than failing the grant.
|
||||
logger.warning("Credit reconcile: budget sync failed for %s: %s", email, e)
|
||||
rec["result"] = f"granted (balance {new_balance}); budget sync failed: {str(e)[:80]}"
|
||||
applied.append(rec)
|
||||
logger.warning("Credit reconcile: GRANTED $%.2f to %s for order %s (was missing)",
|
||||
rec["amount"], email, rec["order_id"])
|
||||
|
||||
if candidates and not acting:
|
||||
logger.warning(
|
||||
"Credit reconcile: %d PAID credit pack(s) with no record in our books "
|
||||
"(report mode — nothing changed): %s", len(candidates),
|
||||
"; ".join(f"${c['amount']:.2f} {c['slug']} {c['created_at']}" for c in candidates))
|
||||
|
||||
return {
|
||||
"status": "ok",
|
||||
"dry_run": not acting,
|
||||
"window_days": CREDIT_RECONCILE_DAYS,
|
||||
"orders_read": len(orders),
|
||||
"packs_in_window": packs,
|
||||
"unaccounted": len(candidates) + len(needs_hold),
|
||||
"candidates": candidates,
|
||||
"needs_hold": needs_hold,
|
||||
"applied": applied,
|
||||
}
|
||||
|
||||
|
||||
async def reconcile_allowance_shortfalls(send: bool = False) -> dict:
|
||||
"""Credit members whose PAID cycle Open Collective's billing-day alignment truncated.
|
||||
|
||||
@@ -7860,9 +8116,33 @@ async def admin_sweep(request: Request):
|
||||
except Exception as exc: # noqa: BLE001
|
||||
result["shortfall_credits"] = {"error": str(exc)[:200]}
|
||||
|
||||
# Paid credit packs with no record in our books (issue #7). REPORT-ONLY
|
||||
# unless CREDIT_RECONCILE=on; wrapped so it can never break reconciliation.
|
||||
try:
|
||||
result["credit_reconcile"] = await reconcile_credit_packs(
|
||||
apply=(CREDIT_RECONCILE == "on"))
|
||||
except Exception as exc: # noqa: BLE001
|
||||
result["credit_reconcile"] = {"error": str(exc)[:200]}
|
||||
|
||||
return result
|
||||
|
||||
|
||||
@app.post("/admin/credit-pack-reconcile")
|
||||
@limiter.limit("20/minute")
|
||||
async def admin_credit_pack_reconcile(request: Request, apply: int = 0):
|
||||
"""Reconcile PAID credit packs against our books — DRY RUN by default.
|
||||
|
||||
Authenticated by the X-Admin-Token header, like the other /admin/* routes.
|
||||
`?apply=1` asks for the grants to be applied, and is honoured only when
|
||||
CREDIT_RECONCILE=on: the point of the default is that an operator can read
|
||||
this pass's output in production before any member's money is touched. Every
|
||||
grant uses the same per-order reference the webhook writes, so running it
|
||||
twice — or running it after a late delivery — credits a purchase once.
|
||||
"""
|
||||
_verify_admin_token(request)
|
||||
return await reconcile_credit_packs(apply=bool(apply))
|
||||
|
||||
|
||||
def _service_sync_pending() -> list[str]:
|
||||
"""Active members still waiting on their Loomio invitation.
|
||||
|
||||
|
||||
@@ -0,0 +1,315 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Regression test: credit-pack reconciliation (issue #7).
|
||||
|
||||
WHY THIS EXISTS
|
||||
Credits are granted on exactly ONE path — the OC `order.processed` webhook —
|
||||
and that path can fail AFTER the money has moved. On 2026-11-13 two purchases
|
||||
did: a $50 pack whose delivery 500'd (httpx's invisible 5-second default), and
|
||||
a $10 pack with no delivery in the retained log at all. Neither produced a
|
||||
ledger row, and nothing on a schedule noticed. The reconcile pass reads OC's
|
||||
OWN order list and compares it with our books.
|
||||
|
||||
The thing that makes this dangerous rather than merely absent is the matching:
|
||||
our own stored references are NOT uniform, so the obvious
|
||||
`oc:<slug>:<order.id>` test reported 21 already-credited purchases as missing
|
||||
on its first run. A pass that "fixes" those would double-credit members. This
|
||||
test pins down, case by case, what counts as accounted for:
|
||||
|
||||
* newer ledger rows carry the v2 id `oc:alice:or_…`
|
||||
* older ledger rows carry a HYPHENATED id `oc:bob:8k03reyd-…`
|
||||
* legacy rows carry an EMPTY suffix `oc:erin:`
|
||||
* non-member purchases live in `held_credits`, not the ledger, and are
|
||||
accounted for — they took the hold path on purpose
|
||||
|
||||
and that a purchase with NO such record is reported (dry run) or granted ONCE
|
||||
(apply), using the same reference the webhook writes, so the two paths are
|
||||
idempotent against each other in both directions.
|
||||
|
||||
Network is STUBBED; assertions are on our own recorded state.
|
||||
|
||||
Runnable: python3 tests/test_credit_pack_reconcile.py (from the repo root)
|
||||
"""
|
||||
import os
|
||||
import sys
|
||||
import asyncio
|
||||
import tempfile
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import httpx
|
||||
|
||||
os.environ["DB_PATH"] = os.path.join(tempfile.mkdtemp(), "fixture.db")
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
import app.main as m # noqa: E402
|
||||
|
||||
m.DB_PATH = os.environ["DB_PATH"]
|
||||
m.ADMIN_TOKEN = "test-admin-token"
|
||||
m.ADMIN_TOKEN_LEGACY = ""
|
||||
m.CREDIT_RECONCILE = "report" # the shipped default
|
||||
m.CREDIT_RECONCILE_DAYS = 30
|
||||
|
||||
NOW = datetime.now(timezone.utc)
|
||||
|
||||
|
||||
def iso(days_ago: float) -> str:
|
||||
return (NOW - timedelta(days=days_ago)).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
|
||||
|
||||
def stamp(days_ago: float) -> str:
|
||||
return (NOW - timedelta(days=days_ago)).strftime("%Y-%m-%d %H:%M:%S")
|
||||
|
||||
|
||||
def order(oid, slug, amount, days_ago=1.0, tier="Credit pack", status="PAID",
|
||||
frequency="ONETIME"):
|
||||
return {"id": oid, "status": status, "createdAt": iso(days_ago),
|
||||
"frequency": frequency, "amount": {"value": amount, "currency": "USD"},
|
||||
"platformTipAmount": {"value": 0}, "totalAmount": {"value": amount},
|
||||
"tier": {"name": tier, "slug": (tier or "").lower().replace(" ", "-")},
|
||||
"fromAccount": {"slug": slug, "name": slug.title(), "type": "INDIVIDUAL"}}
|
||||
|
||||
|
||||
ORDERS = [
|
||||
# accounted by the EXACT v2 reference (and the ledger amount deliberately
|
||||
# disagrees, to prove the exact match does not need the fuzzy second signal)
|
||||
order("or_AAA", "alice", 50, 2.0),
|
||||
# accounted only by the SECOND signal: older rows carry a hyphenated id
|
||||
order("or_BBB", "bob", 50, 3.0),
|
||||
# accounted only by the SECOND signal: a legacy EMPTY-suffix reference
|
||||
order("or_EEE", "erin", 5, 4.0),
|
||||
# accounted as HELD — a non-member purchase recorded on the hold path
|
||||
order("or_HHH", "holdguy", 10, 5.0),
|
||||
# GENUINELY UNACCOUNTED, and by an active member → the one real candidate
|
||||
order("or_CCC", "carol", 10, 0.5),
|
||||
# unaccounted, but the buyer is not an active member → hold path, not a grant
|
||||
order("or_DDD", "dave", 10, 0.7),
|
||||
# unaccounted and active, but outside the window → not this pass's business
|
||||
order("or_FFF", "frank", 10, 100.0),
|
||||
# not a credit pack / not paid / not one-time → never examined as a pack
|
||||
order("or_G1", "carol", 15, 1.0, tier="Membership"),
|
||||
order("or_G2", "carol", 10, 1.0, status="PENDING"),
|
||||
order("or_G3", "carol", 10, 1.0, status="REFUNDED"),
|
||||
order("or_G4", "carol", 10, 1.0, frequency="MONTHLY"),
|
||||
]
|
||||
|
||||
|
||||
class Boom(Exception):
|
||||
pass
|
||||
|
||||
|
||||
async def fake_orders(per_page=100, max_pages=3):
|
||||
if FAIL_ORDERS:
|
||||
raise Boom("OC said 500")
|
||||
return list(ORDERS)
|
||||
|
||||
|
||||
async def fake_index():
|
||||
if FAIL_INDEX:
|
||||
raise Boom("member index unavailable")
|
||||
return {e: {"slug": e.split("@")[0]} for e in
|
||||
("alice@x.org", "bob@x.org", "carol@x.org", "dave@x.org",
|
||||
"erin@x.org", "frank@x.org")}
|
||||
|
||||
|
||||
BUDGET_PUSHES = []
|
||||
|
||||
|
||||
async def fake_budget(email):
|
||||
BUDGET_PUSHES.append(email)
|
||||
return 25.0
|
||||
|
||||
|
||||
FAIL_ORDERS = False
|
||||
FAIL_INDEX = False
|
||||
|
||||
results = []
|
||||
|
||||
|
||||
def check(name, ok):
|
||||
results.append((name, bool(ok)))
|
||||
|
||||
|
||||
def ledger_rows():
|
||||
conn = m.get_db()
|
||||
conn.row_factory = __import__("sqlite3").Row
|
||||
rows = [dict(r) for r in conn.execute(
|
||||
"SELECT email, delta, reference, created_at FROM credit_ledger ORDER BY id")]
|
||||
conn.close()
|
||||
return rows
|
||||
|
||||
|
||||
def balance(email):
|
||||
return m.get_credit_balance(email)
|
||||
|
||||
|
||||
def seed():
|
||||
conn = m.get_db()
|
||||
conn.execute("DELETE FROM members")
|
||||
conn.execute("DELETE FROM credit_ledger")
|
||||
conn.execute("DELETE FROM held_credits")
|
||||
for email, slug, active in (("alice@x.org", "alice", 1), ("bob@x.org", "bob", 1),
|
||||
("carol@x.org", "carol", 1), ("dave@x.org", "dave", 0),
|
||||
("erin@x.org", "erin", 1), ("frank@x.org", "frank", 1)):
|
||||
conn.execute(
|
||||
"INSERT INTO members (email, slug, active, account_type, credit_balance) "
|
||||
"VALUES (?, ?, ?, 'member', 0)", (email, slug, active))
|
||||
# alice: the v2 form, with an amount that DISAGREES with the order ($48 ≠ $50)
|
||||
conn.execute("INSERT INTO credit_ledger (email, delta, reason, reference, created_at) "
|
||||
"VALUES ('alice@x.org', 48.0, 'credit pack', ?, ?)",
|
||||
("oc:alice:or_AAA", stamp(2.0)))
|
||||
# bob: the hyphenated legacy id for the SAME order (or_BBB)
|
||||
conn.execute("INSERT INTO credit_ledger (email, delta, reason, reference, created_at) "
|
||||
"VALUES ('bob@x.org', 50.0, 'credit pack', ?, ?)",
|
||||
("oc:bob:8k03reyd-5agmq5x4-4nkplbwo-z7j4nxv9", stamp(3.0)))
|
||||
# erin: the empty-suffix legacy form
|
||||
conn.execute("INSERT INTO credit_ledger (email, delta, reason, reference, created_at) "
|
||||
"VALUES ('erin@x.org', 5.0, 'credit pack', 'oc:erin:', ?)", (stamp(4.0),))
|
||||
# holdguy: a non-member purchase held, never a ledger row
|
||||
conn.execute("INSERT INTO held_credits (slug, email, name, amount, oc_reference, "
|
||||
"status, created_at) VALUES ('holdguy', 'hold@x.org', 'Hold Guy', 10.0, "
|
||||
"'oc:holdguy:', 'pending', ?)", (stamp(5.0),))
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
|
||||
def names(res):
|
||||
return sorted(c["slug"] for c in res["candidates"])
|
||||
|
||||
|
||||
async def main():
|
||||
global FAIL_ORDERS, FAIL_INDEX
|
||||
m.fetch_collective_orders = fake_orders
|
||||
m.fetch_member_index = fake_index
|
||||
m.sync_team_budget_from_balance = fake_budget
|
||||
|
||||
# ---------- dry run: report only, change nothing --------------------------
|
||||
seed()
|
||||
before = ledger_rows()
|
||||
res = await m.reconcile_credit_packs(apply=False)
|
||||
check(f"dry run reports ok (got {res.get('status')})", res.get("status") == "ok")
|
||||
check("dry run is marked dry_run", res.get("dry_run") is True)
|
||||
check(f"packs in window counted (got {res.get('packs_in_window')})",
|
||||
res.get("packs_in_window") == 6)
|
||||
check(f"only the unaccounted active-member pack is a candidate (got {names(res)})",
|
||||
names(res) == ["carol"])
|
||||
check(f"the inactive member's pack is a hold, not a grant (got {res['needs_hold']})",
|
||||
[c["slug"] for c in res["needs_hold"]] == ["dave"])
|
||||
check("dry run applied nothing", res.get("applied") == [])
|
||||
check("dry run wrote no ledger row", ledger_rows() == before)
|
||||
check(f"dry run left carol's balance alone (got {balance('carol@x.org')})",
|
||||
balance("carol@x.org") == 0.0)
|
||||
check("dry run pushed no budget", BUDGET_PUSHES == [])
|
||||
|
||||
# the three accounting shapes that a v2-only match would have called missing
|
||||
for oid, slug, why in (("or_AAA", "alice", "exact v2 reference"),
|
||||
("or_BBB", "bob", "hyphenated legacy id"),
|
||||
("or_EEE", "erin", "empty-suffix legacy reference"),
|
||||
("or_HHH", "holdguy", "held (non-member) purchase")):
|
||||
o = next(x for x in ORDERS if x["id"] == oid)
|
||||
ev = m.credit_pack_accounted(o, ledger_rows(), [
|
||||
dict(r) for r in
|
||||
conn_rows("SELECT id, slug, email, amount, oc_reference, status, created_at FROM held_credits")
|
||||
])
|
||||
check(f"accounted: {why} ({oid}) -> {ev}", ev is not None)
|
||||
check("a genuinely unaccounted order stays unaccounted",
|
||||
m.credit_pack_accounted(next(x for x in ORDERS if x["id"] == "or_CCC"),
|
||||
ledger_rows(), []) is None)
|
||||
|
||||
# ---------- a read that fails is UNKNOWN, never 'nothing missing' ---------
|
||||
FAIL_ORDERS = True
|
||||
res = await m.reconcile_credit_packs(apply=False)
|
||||
check(f"order fetch failure is skipped/order_fetch_failed (got {res})",
|
||||
res.get("status") == "skipped" and res.get("reason") == "order_fetch_failed")
|
||||
FAIL_ORDERS = False
|
||||
FAIL_INDEX = True
|
||||
res = await m.reconcile_credit_packs(apply=False)
|
||||
check(f"member index failure is skipped/member_index_failed (got {res})",
|
||||
res.get("status") == "skipped" and res.get("reason") == "member_index_failed")
|
||||
FAIL_INDEX = False
|
||||
|
||||
# ---------- apply=1 while the policy is still 'report' --------------------
|
||||
seed()
|
||||
res = await m.reconcile_credit_packs(apply=True)
|
||||
check("apply is IGNORED while CREDIT_RECONCILE != 'on'",
|
||||
res.get("dry_run") is True and res.get("applied") == [])
|
||||
check(f"and still nothing was credited (got {balance('carol@x.org')})",
|
||||
balance("carol@x.org") == 0.0)
|
||||
|
||||
# ---------- CREDIT_RECONCILE=on: grant exactly once ----------------------
|
||||
seed()
|
||||
m.CREDIT_RECONCILE = "on"
|
||||
res = await m.reconcile_credit_packs(apply=True)
|
||||
check(f"apply grants the one candidate (got {res.get('applied')})",
|
||||
[c["slug"] for c in res.get("applied", [])] == ["carol"])
|
||||
check(f"carol credited $10 (got {balance('carol@x.org')})",
|
||||
balance("carol@x.org") == 10.0)
|
||||
ref = f"oc:carol:or_CCC"
|
||||
check(f"the grant used the webhook's own reference ({ref})",
|
||||
any(r["reference"] == ref for r in ledger_rows()))
|
||||
check(f"exactly one ledger row was added (got {len(ledger_rows())})",
|
||||
len(ledger_rows()) == 4)
|
||||
check("the budget was pushed once", BUDGET_PUSHES == ["carol@x.org"])
|
||||
check("the inactive member was NOT credited",
|
||||
balance("dave@x.org") == 0.0)
|
||||
check("the out-of-window member was NOT credited",
|
||||
balance("frank@x.org") == 0.0)
|
||||
|
||||
# the webhook and this pass must dedupe each other, whichever runs first
|
||||
check("the webhook would now answer already_credited for this order",
|
||||
m.credit_reference_exists(m.credit_pack_reference("carol", {"idV2": "or_CCC"})))
|
||||
|
||||
res = await m.reconcile_credit_packs(apply=True)
|
||||
check(f"a second apply changes nothing (got {res.get('applied')})",
|
||||
res.get("applied") == [] and res.get("candidates") == [])
|
||||
check(f"carol's balance unchanged by the re-run (got {balance('carol@x.org')})",
|
||||
balance("carol@x.org") == 10.0)
|
||||
check(f"still one row for the order (got {len(ledger_rows())})",
|
||||
len(ledger_rows()) == 4)
|
||||
|
||||
# ---------- the endpoint -------------------------------------------------
|
||||
# Re-seed: the direct apply above already credited carol, so the endpoint
|
||||
# would have no candidate and the check would pass for the wrong reason.
|
||||
seed()
|
||||
m.CREDIT_RECONCILE = "report"
|
||||
|
||||
async def endpoint_checks():
|
||||
transport = httpx.ASGITransport(app=m.app)
|
||||
async with httpx.AsyncClient(transport=transport, base_url="http://t") as c:
|
||||
r = await c.post("/admin/credit-pack-reconcile")
|
||||
check(f"endpoint needs the admin token (got {r.status_code})",
|
||||
r.status_code in (401, 403))
|
||||
r = await c.post("/admin/credit-pack-reconcile",
|
||||
headers={"X-Admin-Token": "test-admin-token"})
|
||||
body = r.json()
|
||||
check(f"endpoint defaults to a DRY RUN (got {body})",
|
||||
r.status_code == 200 and body.get("dry_run") is True)
|
||||
r = await c.post("/admin/credit-pack-reconcile?apply=1",
|
||||
headers={"X-Admin-Token": "test-admin-token"})
|
||||
body = r.json()
|
||||
check("endpoint ?apply=1 is still a dry run while the policy is 'report'",
|
||||
body.get("dry_run") is True and body.get("applied") == [])
|
||||
m.CREDIT_RECONCILE = "on"
|
||||
r = await c.post("/admin/credit-pack-reconcile?apply=1",
|
||||
headers={"X-Admin-Token": "test-admin-token"})
|
||||
body = r.json()
|
||||
check(f"endpoint ?apply=1 grants once the policy is 'on' (got {body})",
|
||||
body.get("dry_run") is False
|
||||
and [x["slug"] for x in body.get("applied", [])] == ["carol"])
|
||||
m.CREDIT_RECONCILE = "report"
|
||||
|
||||
await endpoint_checks()
|
||||
|
||||
|
||||
def conn_rows(sql):
|
||||
conn = m.get_db()
|
||||
conn.row_factory = __import__("sqlite3").Row
|
||||
rows = conn.execute(sql).fetchall()
|
||||
conn.close()
|
||||
return rows
|
||||
|
||||
|
||||
asyncio.run(main())
|
||||
|
||||
fails = sum(not ok for _, ok in results)
|
||||
for name, ok in results:
|
||||
print(f" [{'PASS' if ok else 'FAIL'}] {name}")
|
||||
print(f"\n{len(results) - fails}/{len(results)} passed")
|
||||
sys.exit(1 if fails else 0)
|
||||
Reference in new issue
Block a user