Backstop: reconcile PAID credit packs against the ledger (money received must never go uncredited) #7

Open
opened 2026-11-13 20:04:46 +00:00 by inference-bot · 2 comments
Owner

The gap

Credits are granted on exactly one path: the Open Collective webhook. If that
delivery fails, the co-op holds the member's money and the member has no
credits, and nothing on a schedule looks for that.

On 2026-11-13 an order.processed delivery for a $50 credit pack returned 500
(httpx.ReadTimeout out of fetch_members, because fetch_all_member_nodes
used httpx's 5-second default). No credits were written, OC's re-delivery never
arrived, and the purchase sat uncredited until it was noticed in the app log and
granted by hand from the verified OC order. The handler/timeout fix is in
f45e4ef; this issue is the backstop that makes the failure recoverable without
a human reading logs.

What already reconciles, and what does not

  • The membership sweep reconciles MEMBERSHIP: provisions missed signups,
    deactivates lapsed members, syncs Loomio + Matrix. It never looks at payments.
  • held_credits covers the non-member case (purchase held, buyer emailed).
    A member's failed grant has no record at all.

Proposal

A sweep step that finds credit-pack payments with no ledger row and grants them:

  1. List the collective's recent orders.
  2. Keep one-time PAID orders on the Credit pack tier within a bounded window
    (propose 30 days, and state the window in the code and the log line — an
    unbounded window would re-examine ancient orders on every run).
  3. For each, build the reference exactly as the webhook does
    (credit_pack_reference(slug, {"idV2": ...}) -> oc:<slug>:<idV2>) and skip
    it if credit_reference_exists().
  4. Otherwise grant the tip-free amount via add_credits() and
    sync_team_budget_from_balance(), and log at INFO with the order id.

Idempotent in both directions by construction: a later webhook delivery of the
same order is a no-op, and a sweep after a successful webhook is a no-op, because
both key on the same per-order reference.

Blocker to decide: the OC token cannot read orders

collective(slug:…){ orders } and account(slug:…){ orders } both answer
The Personal Token is not allowed for operations in scope "orders" with the
portal's OC_PERSONAL_TOKEN. Two ways forward:

  • Preferred — give the token the orders scope in Open Collective (operator
    step, a field in the token's settings), then the collective-level order list is
    one query.
  • Fallback — the PUBLIC GraphQL API does answer account(slug:…){ orders }
    (that is how the 2026-11-13 purchase was verified), so the same check is
    possible per member: one query each, ~131 active members, run after the member
    fetch in the sweep. Chattier, no permission change, and it would also cover
    members whose OC slug differs from the portal's stored slug.

Acceptance criteria

  • An order that is PAID on the Credit pack tier with no matching ledger row is
    credited by the next sweep, with the tip-free amount (never totalAmount).
  • Re-running the sweep changes nothing (no second ledger row, no balance move).
  • A webhook delivered after the sweep is credited exactly once, and vice versa.
  • The sweep logs what it examined and what it granted — "checked N orders in the
    window, granted M" — so a thin run is visible as thin, and a run that could not
    read OC reports FAILED rather than "nothing to do".
  • Nothing here may grant credits to a non-member purchase: those keep the
    existing hold-and-email path.

Related

  • The webhook watcher (~/.hermes/scripts/portal_webhook_watch.py) now reports a
    5xx as OUR error and says a member may be out credits, instead of blaming the
    webhook URL. It is still only as good as a human acting on it.
## The gap Credits are granted on exactly one path: the Open Collective webhook. If that delivery fails, the co-op holds the member's money and the member has no credits, and **nothing on a schedule looks for that**. On 2026-11-13 an `order.processed` delivery for a $50 credit pack returned 500 (`httpx.ReadTimeout` out of `fetch_members`, because `fetch_all_member_nodes` used httpx's 5-second default). No credits were written, OC's re-delivery never arrived, and the purchase sat uncredited until it was noticed in the app log and granted by hand from the verified OC order. The handler/timeout fix is in `f45e4ef`; this issue is the backstop that makes the failure recoverable without a human reading logs. ## What already reconciles, and what does not * The membership sweep reconciles MEMBERSHIP: provisions missed signups, deactivates lapsed members, syncs Loomio + Matrix. It never looks at payments. * `held_credits` covers the non-member case (purchase held, buyer emailed). A member's failed grant has no record at all. ## Proposal A sweep step that finds credit-pack payments with no ledger row and grants them: 1. List the collective's recent orders. 2. Keep one-time `PAID` orders on the Credit pack tier within a bounded window (propose 30 days, and state the window in the code and the log line — an unbounded window would re-examine ancient orders on every run). 3. For each, build the reference exactly as the webhook does (`credit_pack_reference(slug, {"idV2": ...})` -> `oc:<slug>:<idV2>`) and skip it if `credit_reference_exists()`. 4. Otherwise grant the tip-free amount via `add_credits()` and `sync_team_budget_from_balance()`, and log at INFO with the order id. Idempotent in both directions by construction: a later webhook delivery of the same order is a no-op, and a sweep after a successful webhook is a no-op, because both key on the same per-order reference. ## Blocker to decide: the OC token cannot read orders `collective(slug:…){ orders }` and `account(slug:…){ orders }` both answer `The Personal Token is not allowed for operations in scope "orders"` with the portal's `OC_PERSONAL_TOKEN`. Two ways forward: * **Preferred** — give the token the `orders` scope in Open Collective (operator step, a field in the token's settings), then the collective-level order list is one query. * **Fallback** — the PUBLIC GraphQL API does answer `account(slug:…){ orders }` (that is how the 2026-11-13 purchase was verified), so the same check is possible per member: one query each, ~131 active members, run after the member fetch in the sweep. Chattier, no permission change, and it would also cover members whose OC slug differs from the portal's stored slug. ## Acceptance criteria * An order that is PAID on the Credit pack tier with no matching ledger row is credited by the next sweep, with the tip-free amount (never `totalAmount`). * Re-running the sweep changes nothing (no second ledger row, no balance move). * A webhook delivered after the sweep is credited exactly once, and vice versa. * The sweep logs what it examined and what it granted — "checked N orders in the window, granted M" — so a thin run is visible as thin, and a run that could not read OC reports FAILED rather than "nothing to do". * Nothing here may grant credits to a non-member purchase: those keep the existing hold-and-email path. ## Related * The webhook watcher (`~/.hermes/scripts/portal_webhook_watch.py`) now reports a 5xx as OUR error and says a member may be out credits, instead of blaming the webhook URL. It is still only as good as a human acting on it.
Author
Owner

Token scope: granted 2026-11-13, and the order list works

collective(slug:"inference-cooperative"){ orders(limit, offset) } now answers with
OC_PERSONAL_TOKEN — totalCount: 150, 100 per page, all the fields a reconcile
needs (id, status, createdAt, frequency, amount, platformTipAmount,
totalAmount, tier{name slug}, fromAccount{slug name type}). No new code path
needed for that half.

TRAP — the reference key is NOT uniform across the rows already in our DBs

A first pass that matched oc:<slug>:<order.id> found 21 "uncredited" paid
credit packs
. Almost all were false: the historical rows carry a DIFFERENT id.

  • newer credit_ledger rows: oc:nightrose:or_apShJowSoPPj1dlM3ExTH (the v2 or_… id)
  • older credit_ledger rows: oc:mattwelke:8k03reyd-5agmq5x4-4nkplbwo-z7j4nxv9
    (a hyphenated id that is neither Order.id nor Order.legacyId)
  • held_credits.oc_reference rows exist with an empty suffix (oc:madrush:)
    as well as the hyphenated form

So a reconcile keyed only on the v2 id would re-grant already-credited
purchases
. Any implementation must treat "no or_… match" as unknown, not as
uncredited, and require a second signal — the same member slug with the same
tip-free amount in the same month, matched against BOTH credit_ledger and
held_credits (a held pack is accounted for, not missing).

Result of the careful pass — one genuine gap

22 paid credit packs in the most recent 100 orders; 21 accounted for (19 by
member+amount+period, 2 as held_credits with status: granted). One is not:

  • 2026-11-13 08:32:57, sebastian-beschke, $10.00 credit pack
    ($11.50 with the buyer's $1.50 tip), order or_CgpwMD9zWyZbfOAOTA5yc,
    status: PAID, frequency: ONETIME. No credit_ledger row, no held_credits
    row, and the member's credit_balance is 0.0.

Whether the delivery arrived and failed, or never arrived, is not recoverable:
the portal's retained log has whole hours missing (06:00–13:00 and 15:00 are
absent), so it cannot be read as "no webhook at 08:32". OC's own webhook
delivery history would answer it, but this version's Webhook type exposes no
url/eventTypes/lastDelivery field, so that is a UI question, not an API one.

What this changes here

  1. The backstop's first real job: reconciling is not theoretical — it found a
    second unreconciled purchase on the same day as the incident the issue was
    filed for.
  2. The span of the window matters: this one is 11 hours old and the log that
    would have shown it is already gone, which is the argument for the window
    being measured in DAYS off OC's own data rather than in "whatever the log
    still holds".
  3. Run the reconcile in a dry-run mode first and print, per candidate, the exact
    evidence (order id, slug, amount, date) — a bare "no match" is what produced
    the 21-item false alarm above.
## Token scope: granted 2026-11-13, and the order list works `collective(slug:"inference-cooperative"){ orders(limit, offset) }` now answers with `OC_PERSONAL_TOKEN` — `totalCount: 150`, 100 per page, all the fields a reconcile needs (`id`, `status`, `createdAt`, `frequency`, `amount`, `platformTipAmount`, `totalAmount`, `tier{name slug}`, `fromAccount{slug name type}`). No new code path needed for that half. ## TRAP — the reference key is NOT uniform across the rows already in our DBs A first pass that matched `oc:<slug>:<order.id>` found **21 "uncredited" paid credit packs**. Almost all were false: the historical rows carry a DIFFERENT id. * newer `credit_ledger` rows: `oc:nightrose:or_apShJowSoPPj1dlM3ExTH` (the v2 `or_…` id) * older `credit_ledger` rows: `oc:mattwelke:8k03reyd-5agmq5x4-4nkplbwo-z7j4nxv9` (a hyphenated id that is neither `Order.id` nor `Order.legacyId`) * `held_credits.oc_reference` rows exist with an **empty suffix** (`oc:madrush:`) as well as the hyphenated form So a reconcile keyed only on the v2 id would **re-grant already-credited purchases**. Any implementation must treat "no `or_…` match" as *unknown*, not as *uncredited*, and require a second signal — the same member slug with the same tip-free amount in the same month, matched against BOTH `credit_ledger` and `held_credits` (a held pack is accounted for, not missing). ## Result of the careful pass — one genuine gap 22 paid credit packs in the most recent 100 orders; 21 accounted for (19 by member+amount+period, 2 as `held_credits` with `status: granted`). One is not: * **2026-11-13 08:32:57, `sebastian-beschke`, $10.00 credit pack** (`$11.50` with the buyer's $1.50 tip), order `or_CgpwMD9zWyZbfOAOTA5yc`, `status: PAID`, `frequency: ONETIME`. No `credit_ledger` row, no `held_credits` row, and the member's `credit_balance` is `0.0`. Whether the delivery arrived and failed, or never arrived, is not recoverable: the portal's retained log has whole hours missing (06:00–13:00 and 15:00 are absent), so it cannot be read as "no webhook at 08:32". OC's own webhook delivery history would answer it, but this version's `Webhook` type exposes no `url`/`eventTypes`/`lastDelivery` field, so that is a UI question, not an API one. ## What this changes here 1. The backstop's first real job: reconciling is not theoretical — it found a second unreconciled purchase on the same day as the incident the issue was filed for. 2. The **span of the window matters**: this one is 11 hours old and the log that would have shown it is already gone, which is the argument for the window being measured in DAYS off OC's own data rather than in "whatever the log still holds". 3. Run the reconcile in a dry-run mode first and print, per candidate, the exact evidence (order id, slug, amount, date) — a bare "no match" is what produced the 21-item false alarm above.
Author
Owner

Built and deployed — report-first, and the production dry run is clean

reconcile_credit_packs() now runs inside /admin/sweep and on demand at
POST /admin/credit-pack-reconcile (X-Admin-Token). CREDIT_RECONCILE
defaults to report — it logs findings and changes nothing — and grants only
at on. Every grant uses credit_pack_reference(), the same per-order key the
webhook writes, so a late delivery and this pass cannot both credit one purchase.
36 checks in tests/test_credit_pack_reconcile.py; all 18 suites green.

Production dry run, 30-day window: orders_read: 150, packs_in_window: 23, unaccounted: 0. Nothing to grant — the two hand grants made on 2026-11-13
($50 and $10) both match on their exact references, which is the result that
proves the matcher works on real data and not just on fixtures.

Widening the window past 30 days: three candidates, all false — and why

Setting the window to OC's whole order history (150 orders, 2026-09-06 →
2026-11-13) reported three unaccounted $5 packs, all ntnsndr, all on
2026-09-28 (03:51, 04:00, 04:11). They are NOT gaps: the ledger shows they were
credited by hand that morning, when the tier matcher was broken at arrival time:

+20.00  "credit pack purchases x2 (Open Collective) - manual application, tier matcher was broken at arrival time"
 +5.75  "credit pack purchase (Open Collective)"   reference oc:ntnsndr:
-10.75  "correction: prior grants included OC platform tips; true contributions were 3 x $5.00 credit packs"
        = exactly $15.00, i.e. three $5.00 packs

credit_pack_accounted() cannot see this, and must not: inferring "accounted
for" from a nearby amount would hide genuine gaps. But a reviewer could not tell
the two cases apart either. So each candidate now carries its nearby
ledger/held rows (7-day window, capped at 5) and the report-mode log line prints
them — the manual grant explains itself.

Consequence for the window setting

With CREDIT_RECONCILE_DAYS set anywhere that reaches 2026-09-28 and
apply on, three already-granted packs would be credited a second time. Two
consequences for whoever turns this on:

  1. Keep the window inside the range where per-order references exist (the 30-day
    default is clean), or
  2. read every candidate's nearby_book_rows before applying — which is now
    possible, and is what the dry run is for.

A stronger rule, if anyone wants it later: treat a candidate whose member has an
UNREFERENCED manual row inside the window as "needs human" rather than"gran-
table". Not implemented — it would need a policy decision about manual grants,
not just a matching rule.

## Built and deployed — report-first, and the production dry run is clean `reconcile_credit_packs()` now runs inside `/admin/sweep` and on demand at `POST /admin/credit-pack-reconcile` (`X-Admin-Token`). `CREDIT_RECONCILE` defaults to **`report`** — it logs findings and changes nothing — and grants only at `on`. Every grant uses `credit_pack_reference()`, the same per-order key the webhook writes, so a late delivery and this pass cannot both credit one purchase. 36 checks in `tests/test_credit_pack_reconcile.py`; all 18 suites green. **Production dry run, 30-day window: `orders_read: 150, packs_in_window: 23, unaccounted: 0`.** Nothing to grant — the two hand grants made on 2026-11-13 ($50 and $10) both match on their exact references, which is the result that proves the matcher works on real data and not just on fixtures. ## Widening the window past 30 days: three candidates, all false — and why Setting the window to OC's whole order history (150 orders, 2026-09-06 → 2026-11-13) reported three unaccounted $5 packs, all `ntnsndr`, all on 2026-09-28 (03:51, 04:00, 04:11). They are NOT gaps: the ledger shows they were credited by hand that morning, when the tier matcher was broken at arrival time: ``` +20.00 "credit pack purchases x2 (Open Collective) - manual application, tier matcher was broken at arrival time" +5.75 "credit pack purchase (Open Collective)" reference oc:ntnsndr: -10.75 "correction: prior grants included OC platform tips; true contributions were 3 x $5.00 credit packs" = exactly $15.00, i.e. three $5.00 packs ``` `credit_pack_accounted()` cannot see this, and must not: inferring "accounted for" from a nearby amount would hide genuine gaps. But a reviewer could not tell the two cases apart either. So each candidate now carries its nearby ledger/held rows (7-day window, capped at 5) and the report-mode log line prints them — the manual grant explains itself. ## Consequence for the window setting With `CREDIT_RECONCILE_DAYS` set anywhere that reaches 2026-09-28 **and** `apply` on, three already-granted packs would be credited a second time. Two consequences for whoever turns this on: 1. Keep the window inside the range where per-order references exist (the 30-day default is clean), or 2. read every candidate's `nearby_book_rows` before applying — which is now possible, and is what the dry run is for. A stronger rule, if anyone wants it later: treat a candidate whose member has an UNREFERENCED manual row inside the window as "needs human" rather than"gran- table". Not implemented — it would need a policy decision about manual grants, not just a matching rule.
Sign in to join this conversation.
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: code/member-portal#7