Credits are granted on exactly one path: the Open Collective webhook. If that
delivery fails, the co-op holds the member's money and the member has no
credits, and nothing on a schedule looks for that.
On 2026-11-13 an order.processed delivery for a $50 credit pack returned 500
(httpx.ReadTimeout out of fetch_members, because fetch_all_member_nodes
used httpx's 5-second default). No credits were written, OC's re-delivery never
arrived, and the purchase sat uncredited until it was noticed in the app log and
granted by hand from the verified OC order. The handler/timeout fix is in f45e4ef; this issue is the backstop that makes the failure recoverable without
a human reading logs.
What already reconciles, and what does not
The membership sweep reconciles MEMBERSHIP: provisions missed signups,
deactivates lapsed members, syncs Loomio + Matrix. It never looks at payments.
held_credits covers the non-member case (purchase held, buyer emailed).
A member's failed grant has no record at all.
Proposal
A sweep step that finds credit-pack payments with no ledger row and grants them:
List the collective's recent orders.
Keep one-time PAID orders on the Credit pack tier within a bounded window
(propose 30 days, and state the window in the code and the log line — an
unbounded window would re-examine ancient orders on every run).
For each, build the reference exactly as the webhook does
(credit_pack_reference(slug, {"idV2": ...}) -> oc:<slug>:<idV2>) and skip
it if credit_reference_exists().
Otherwise grant the tip-free amount via add_credits() and sync_team_budget_from_balance(), and log at INFO with the order id.
Idempotent in both directions by construction: a later webhook delivery of the
same order is a no-op, and a sweep after a successful webhook is a no-op, because
both key on the same per-order reference.
Blocker to decide: the OC token cannot read orders
collective(slug:…){ orders } and account(slug:…){ orders } both answer The Personal Token is not allowed for operations in scope "orders" with the
portal's OC_PERSONAL_TOKEN. Two ways forward:
Preferred — give the token the orders scope in Open Collective (operator
step, a field in the token's settings), then the collective-level order list is
one query.
Fallback — the PUBLIC GraphQL API does answer account(slug:…){ orders }
(that is how the 2026-11-13 purchase was verified), so the same check is
possible per member: one query each, ~131 active members, run after the member
fetch in the sweep. Chattier, no permission change, and it would also cover
members whose OC slug differs from the portal's stored slug.
Acceptance criteria
An order that is PAID on the Credit pack tier with no matching ledger row is
credited by the next sweep, with the tip-free amount (never totalAmount).
Re-running the sweep changes nothing (no second ledger row, no balance move).
A webhook delivered after the sweep is credited exactly once, and vice versa.
The sweep logs what it examined and what it granted — "checked N orders in the
window, granted M" — so a thin run is visible as thin, and a run that could not
read OC reports FAILED rather than "nothing to do".
Nothing here may grant credits to a non-member purchase: those keep the
existing hold-and-email path.
Related
The webhook watcher (~/.hermes/scripts/portal_webhook_watch.py) now reports a
5xx as OUR error and says a member may be out credits, instead of blaming the
webhook URL. It is still only as good as a human acting on it.
## The gap
Credits are granted on exactly one path: the Open Collective webhook. If that
delivery fails, the co-op holds the member's money and the member has no
credits, and **nothing on a schedule looks for that**.
On 2026-11-13 an `order.processed` delivery for a $50 credit pack returned 500
(`httpx.ReadTimeout` out of `fetch_members`, because `fetch_all_member_nodes`
used httpx's 5-second default). No credits were written, OC's re-delivery never
arrived, and the purchase sat uncredited until it was noticed in the app log and
granted by hand from the verified OC order. The handler/timeout fix is in
`f45e4ef`; this issue is the backstop that makes the failure recoverable without
a human reading logs.
## What already reconciles, and what does not
* The membership sweep reconciles MEMBERSHIP: provisions missed signups,
deactivates lapsed members, syncs Loomio + Matrix. It never looks at payments.
* `held_credits` covers the non-member case (purchase held, buyer emailed).
A member's failed grant has no record at all.
## Proposal
A sweep step that finds credit-pack payments with no ledger row and grants them:
1. List the collective's recent orders.
2. Keep one-time `PAID` orders on the Credit pack tier within a bounded window
(propose 30 days, and state the window in the code and the log line — an
unbounded window would re-examine ancient orders on every run).
3. For each, build the reference exactly as the webhook does
(`credit_pack_reference(slug, {"idV2": ...})` -> `oc:<slug>:<idV2>`) and skip
it if `credit_reference_exists()`.
4. Otherwise grant the tip-free amount via `add_credits()` and
`sync_team_budget_from_balance()`, and log at INFO with the order id.
Idempotent in both directions by construction: a later webhook delivery of the
same order is a no-op, and a sweep after a successful webhook is a no-op, because
both key on the same per-order reference.
## Blocker to decide: the OC token cannot read orders
`collective(slug:…){ orders }` and `account(slug:…){ orders }` both answer
`The Personal Token is not allowed for operations in scope "orders"` with the
portal's `OC_PERSONAL_TOKEN`. Two ways forward:
* **Preferred** — give the token the `orders` scope in Open Collective (operator
step, a field in the token's settings), then the collective-level order list is
one query.
* **Fallback** — the PUBLIC GraphQL API does answer `account(slug:…){ orders }`
(that is how the 2026-11-13 purchase was verified), so the same check is
possible per member: one query each, ~131 active members, run after the member
fetch in the sweep. Chattier, no permission change, and it would also cover
members whose OC slug differs from the portal's stored slug.
## Acceptance criteria
* An order that is PAID on the Credit pack tier with no matching ledger row is
credited by the next sweep, with the tip-free amount (never `totalAmount`).
* Re-running the sweep changes nothing (no second ledger row, no balance move).
* A webhook delivered after the sweep is credited exactly once, and vice versa.
* The sweep logs what it examined and what it granted — "checked N orders in the
window, granted M" — so a thin run is visible as thin, and a run that could not
read OC reports FAILED rather than "nothing to do".
* Nothing here may grant credits to a non-member purchase: those keep the
existing hold-and-email path.
## Related
* The webhook watcher (`~/.hermes/scripts/portal_webhook_watch.py`) now reports a
5xx as OUR error and says a member may be out credits, instead of blaming the
webhook URL. It is still only as good as a human acting on it.
Token scope: granted 2026-11-13, and the order list works
collective(slug:"inference-cooperative"){ orders(limit, offset) } now answers with OC_PERSONAL_TOKEN — totalCount: 150, 100 per page, all the fields a reconcile
needs (id, status, createdAt, frequency, amount, platformTipAmount, totalAmount, tier{name slug}, fromAccount{slug name type}). No new code path
needed for that half.
TRAP — the reference key is NOT uniform across the rows already in our DBs
A first pass that matched oc:<slug>:<order.id> found 21 "uncredited" paid
credit packs. Almost all were false: the historical rows carry a DIFFERENT id.
newer credit_ledger rows: oc:nightrose:or_apShJowSoPPj1dlM3ExTH (the v2 or_… id)
older credit_ledger rows: oc:mattwelke:8k03reyd-5agmq5x4-4nkplbwo-z7j4nxv9
(a hyphenated id that is neither Order.id nor Order.legacyId)
held_credits.oc_reference rows exist with an empty suffix (oc:madrush:)
as well as the hyphenated form
So a reconcile keyed only on the v2 id would re-grant already-credited
purchases. Any implementation must treat "no or_… match" as unknown, not as uncredited, and require a second signal — the same member slug with the same
tip-free amount in the same month, matched against BOTH credit_ledger and held_credits (a held pack is accounted for, not missing).
Result of the careful pass — one genuine gap
22 paid credit packs in the most recent 100 orders; 21 accounted for (19 by
member+amount+period, 2 as held_credits with status: granted). One is not:
2026-11-13 08:32:57, sebastian-beschke, $10.00 credit pack
($11.50 with the buyer's $1.50 tip), order or_CgpwMD9zWyZbfOAOTA5yc, status: PAID, frequency: ONETIME. No credit_ledger row, no held_credits
row, and the member's credit_balance is 0.0.
Whether the delivery arrived and failed, or never arrived, is not recoverable:
the portal's retained log has whole hours missing (06:00–13:00 and 15:00 are
absent), so it cannot be read as "no webhook at 08:32". OC's own webhook
delivery history would answer it, but this version's Webhook type exposes no url/eventTypes/lastDelivery field, so that is a UI question, not an API one.
What this changes here
The backstop's first real job: reconciling is not theoretical — it found a
second unreconciled purchase on the same day as the incident the issue was
filed for.
The span of the window matters: this one is 11 hours old and the log that
would have shown it is already gone, which is the argument for the window
being measured in DAYS off OC's own data rather than in "whatever the log
still holds".
Run the reconcile in a dry-run mode first and print, per candidate, the exact
evidence (order id, slug, amount, date) — a bare "no match" is what produced
the 21-item false alarm above.
## Token scope: granted 2026-11-13, and the order list works
`collective(slug:"inference-cooperative"){ orders(limit, offset) }` now answers with
`OC_PERSONAL_TOKEN` — `totalCount: 150`, 100 per page, all the fields a reconcile
needs (`id`, `status`, `createdAt`, `frequency`, `amount`, `platformTipAmount`,
`totalAmount`, `tier{name slug}`, `fromAccount{slug name type}`). No new code path
needed for that half.
## TRAP — the reference key is NOT uniform across the rows already in our DBs
A first pass that matched `oc:<slug>:<order.id>` found **21 "uncredited" paid
credit packs**. Almost all were false: the historical rows carry a DIFFERENT id.
* newer `credit_ledger` rows: `oc:nightrose:or_apShJowSoPPj1dlM3ExTH` (the v2 `or_…` id)
* older `credit_ledger` rows: `oc:mattwelke:8k03reyd-5agmq5x4-4nkplbwo-z7j4nxv9`
(a hyphenated id that is neither `Order.id` nor `Order.legacyId`)
* `held_credits.oc_reference` rows exist with an **empty suffix** (`oc:madrush:`)
as well as the hyphenated form
So a reconcile keyed only on the v2 id would **re-grant already-credited
purchases**. Any implementation must treat "no `or_…` match" as *unknown*, not as
*uncredited*, and require a second signal — the same member slug with the same
tip-free amount in the same month, matched against BOTH `credit_ledger` and
`held_credits` (a held pack is accounted for, not missing).
## Result of the careful pass — one genuine gap
22 paid credit packs in the most recent 100 orders; 21 accounted for (19 by
member+amount+period, 2 as `held_credits` with `status: granted`). One is not:
* **2026-11-13 08:32:57, `sebastian-beschke`, $10.00 credit pack**
(`$11.50` with the buyer's $1.50 tip), order `or_CgpwMD9zWyZbfOAOTA5yc`,
`status: PAID`, `frequency: ONETIME`. No `credit_ledger` row, no `held_credits`
row, and the member's `credit_balance` is `0.0`.
Whether the delivery arrived and failed, or never arrived, is not recoverable:
the portal's retained log has whole hours missing (06:00–13:00 and 15:00 are
absent), so it cannot be read as "no webhook at 08:32". OC's own webhook
delivery history would answer it, but this version's `Webhook` type exposes no
`url`/`eventTypes`/`lastDelivery` field, so that is a UI question, not an API one.
## What this changes here
1. The backstop's first real job: reconciling is not theoretical — it found a
second unreconciled purchase on the same day as the incident the issue was
filed for.
2. The **span of the window matters**: this one is 11 hours old and the log that
would have shown it is already gone, which is the argument for the window
being measured in DAYS off OC's own data rather than in "whatever the log
still holds".
3. Run the reconcile in a dry-run mode first and print, per candidate, the exact
evidence (order id, slug, amount, date) — a bare "no match" is what produced
the 21-item false alarm above.
Built and deployed — report-first, and the production dry run is clean
reconcile_credit_packs() now runs inside /admin/sweep and on demand at POST /admin/credit-pack-reconcile (X-Admin-Token). CREDIT_RECONCILE
defaults to report — it logs findings and changes nothing — and grants only
at on. Every grant uses credit_pack_reference(), the same per-order key the
webhook writes, so a late delivery and this pass cannot both credit one purchase.
36 checks in tests/test_credit_pack_reconcile.py; all 18 suites green.
Production dry run, 30-day window: orders_read: 150, packs_in_window: 23, unaccounted: 0. Nothing to grant — the two hand grants made on 2026-11-13
($50 and $10) both match on their exact references, which is the result that
proves the matcher works on real data and not just on fixtures.
Widening the window past 30 days: three candidates, all false — and why
Setting the window to OC's whole order history (150 orders, 2026-09-06 →
2026-11-13) reported three unaccounted $5 packs, all ntnsndr, all on
2026-09-28 (03:51, 04:00, 04:11). They are NOT gaps: the ledger shows they were
credited by hand that morning, when the tier matcher was broken at arrival time:
+20.00 "credit pack purchases x2 (Open Collective) - manual application, tier matcher was broken at arrival time"
+5.75 "credit pack purchase (Open Collective)" reference oc:ntnsndr:
-10.75 "correction: prior grants included OC platform tips; true contributions were 3 x $5.00 credit packs"
= exactly $15.00, i.e. three $5.00 packs
credit_pack_accounted() cannot see this, and must not: inferring "accounted
for" from a nearby amount would hide genuine gaps. But a reviewer could not tell
the two cases apart either. So each candidate now carries its nearby
ledger/held rows (7-day window, capped at 5) and the report-mode log line prints
them — the manual grant explains itself.
Consequence for the window setting
With CREDIT_RECONCILE_DAYS set anywhere that reaches 2026-09-28 and apply on, three already-granted packs would be credited a second time. Two
consequences for whoever turns this on:
Keep the window inside the range where per-order references exist (the 30-day
default is clean), or
read every candidate's nearby_book_rows before applying — which is now
possible, and is what the dry run is for.
A stronger rule, if anyone wants it later: treat a candidate whose member has an
UNREFERENCED manual row inside the window as "needs human" rather than"gran-
table". Not implemented — it would need a policy decision about manual grants,
not just a matching rule.
## Built and deployed — report-first, and the production dry run is clean
`reconcile_credit_packs()` now runs inside `/admin/sweep` and on demand at
`POST /admin/credit-pack-reconcile` (`X-Admin-Token`). `CREDIT_RECONCILE`
defaults to **`report`** — it logs findings and changes nothing — and grants only
at `on`. Every grant uses `credit_pack_reference()`, the same per-order key the
webhook writes, so a late delivery and this pass cannot both credit one purchase.
36 checks in `tests/test_credit_pack_reconcile.py`; all 18 suites green.
**Production dry run, 30-day window: `orders_read: 150, packs_in_window: 23,
unaccounted: 0`.** Nothing to grant — the two hand grants made on 2026-11-13
($50 and $10) both match on their exact references, which is the result that
proves the matcher works on real data and not just on fixtures.
## Widening the window past 30 days: three candidates, all false — and why
Setting the window to OC's whole order history (150 orders, 2026-09-06 →
2026-11-13) reported three unaccounted $5 packs, all `ntnsndr`, all on
2026-09-28 (03:51, 04:00, 04:11). They are NOT gaps: the ledger shows they were
credited by hand that morning, when the tier matcher was broken at arrival time:
```
+20.00 "credit pack purchases x2 (Open Collective) - manual application, tier matcher was broken at arrival time"
+5.75 "credit pack purchase (Open Collective)" reference oc:ntnsndr:
-10.75 "correction: prior grants included OC platform tips; true contributions were 3 x $5.00 credit packs"
= exactly $15.00, i.e. three $5.00 packs
```
`credit_pack_accounted()` cannot see this, and must not: inferring "accounted
for" from a nearby amount would hide genuine gaps. But a reviewer could not tell
the two cases apart either. So each candidate now carries its nearby
ledger/held rows (7-day window, capped at 5) and the report-mode log line prints
them — the manual grant explains itself.
## Consequence for the window setting
With `CREDIT_RECONCILE_DAYS` set anywhere that reaches 2026-09-28 **and**
`apply` on, three already-granted packs would be credited a second time. Two
consequences for whoever turns this on:
1. Keep the window inside the range where per-order references exist (the 30-day
default is clean), or
2. read every candidate's `nearby_book_rows` before applying — which is now
possible, and is what the dry run is for.
A stronger rule, if anyone wants it later: treat a candidate whose member has an
UNREFERENCED manual row inside the window as "needs human" rather than"gran-
table". Not implemented — it would need a policy decision about manual grants,
not just a matching rule.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
The gap
Credits are granted on exactly one path: the Open Collective webhook. If that
delivery fails, the co-op holds the member's money and the member has no
credits, and nothing on a schedule looks for that.
On 2026-11-13 an
order.processeddelivery for a $50 credit pack returned 500(
httpx.ReadTimeoutout offetch_members, becausefetch_all_member_nodesused httpx's 5-second default). No credits were written, OC's re-delivery never
arrived, and the purchase sat uncredited until it was noticed in the app log and
granted by hand from the verified OC order. The handler/timeout fix is in
f45e4ef; this issue is the backstop that makes the failure recoverable withouta human reading logs.
What already reconciles, and what does not
deactivates lapsed members, syncs Loomio + Matrix. It never looks at payments.
held_creditscovers the non-member case (purchase held, buyer emailed).A member's failed grant has no record at all.
Proposal
A sweep step that finds credit-pack payments with no ledger row and grants them:
PAIDorders on the Credit pack tier within a bounded window(propose 30 days, and state the window in the code and the log line — an
unbounded window would re-examine ancient orders on every run).
(
credit_pack_reference(slug, {"idV2": ...})->oc:<slug>:<idV2>) and skipit if
credit_reference_exists().add_credits()andsync_team_budget_from_balance(), and log at INFO with the order id.Idempotent in both directions by construction: a later webhook delivery of the
same order is a no-op, and a sweep after a successful webhook is a no-op, because
both key on the same per-order reference.
Blocker to decide: the OC token cannot read orders
collective(slug:…){ orders }andaccount(slug:…){ orders }both answerThe Personal Token is not allowed for operations in scope "orders"with theportal's
OC_PERSONAL_TOKEN. Two ways forward:ordersscope in Open Collective (operatorstep, a field in the token's settings), then the collective-level order list is
one query.
account(slug:…){ orders }(that is how the 2026-11-13 purchase was verified), so the same check is
possible per member: one query each, ~131 active members, run after the member
fetch in the sweep. Chattier, no permission change, and it would also cover
members whose OC slug differs from the portal's stored slug.
Acceptance criteria
credited by the next sweep, with the tip-free amount (never
totalAmount).window, granted M" — so a thin run is visible as thin, and a run that could not
read OC reports FAILED rather than "nothing to do".
existing hold-and-email path.
Related
~/.hermes/scripts/portal_webhook_watch.py) now reports a5xx as OUR error and says a member may be out credits, instead of blaming the
webhook URL. It is still only as good as a human acting on it.
Token scope: granted 2026-11-13, and the order list works
collective(slug:"inference-cooperative"){ orders(limit, offset) }now answers withOC_PERSONAL_TOKEN—totalCount: 150, 100 per page, all the fields a reconcileneeds (
id,status,createdAt,frequency,amount,platformTipAmount,totalAmount,tier{name slug},fromAccount{slug name type}). No new code pathneeded for that half.
TRAP — the reference key is NOT uniform across the rows already in our DBs
A first pass that matched
oc:<slug>:<order.id>found 21 "uncredited" paidcredit packs. Almost all were false: the historical rows carry a DIFFERENT id.
credit_ledgerrows:oc:nightrose:or_apShJowSoPPj1dlM3ExTH(the v2or_…id)credit_ledgerrows:oc:mattwelke:8k03reyd-5agmq5x4-4nkplbwo-z7j4nxv9(a hyphenated id that is neither
Order.idnorOrder.legacyId)held_credits.oc_referencerows exist with an empty suffix (oc:madrush:)as well as the hyphenated form
So a reconcile keyed only on the v2 id would re-grant already-credited
purchases. Any implementation must treat "no
or_…match" as unknown, not asuncredited, and require a second signal — the same member slug with the same
tip-free amount in the same month, matched against BOTH
credit_ledgerandheld_credits(a held pack is accounted for, not missing).Result of the careful pass — one genuine gap
22 paid credit packs in the most recent 100 orders; 21 accounted for (19 by
member+amount+period, 2 as
held_creditswithstatus: granted). One is not:sebastian-beschke, $10.00 credit pack(
$11.50with the buyer's $1.50 tip), orderor_CgpwMD9zWyZbfOAOTA5yc,status: PAID,frequency: ONETIME. Nocredit_ledgerrow, noheld_creditsrow, and the member's
credit_balanceis0.0.Whether the delivery arrived and failed, or never arrived, is not recoverable:
the portal's retained log has whole hours missing (06:00–13:00 and 15:00 are
absent), so it cannot be read as "no webhook at 08:32". OC's own webhook
delivery history would answer it, but this version's
Webhooktype exposes nourl/eventTypes/lastDeliveryfield, so that is a UI question, not an API one.What this changes here
second unreconciled purchase on the same day as the incident the issue was
filed for.
would have shown it is already gone, which is the argument for the window
being measured in DAYS off OC's own data rather than in "whatever the log
still holds".
evidence (order id, slug, amount, date) — a bare "no match" is what produced
the 21-item false alarm above.
Built and deployed — report-first, and the production dry run is clean
reconcile_credit_packs()now runs inside/admin/sweepand on demand atPOST /admin/credit-pack-reconcile(X-Admin-Token).CREDIT_RECONCILEdefaults to
report— it logs findings and changes nothing — and grants onlyat
on. Every grant usescredit_pack_reference(), the same per-order key thewebhook writes, so a late delivery and this pass cannot both credit one purchase.
36 checks in
tests/test_credit_pack_reconcile.py; all 18 suites green.Production dry run, 30-day window:
orders_read: 150, packs_in_window: 23, unaccounted: 0. Nothing to grant — the two hand grants made on 2026-11-13($50 and $10) both match on their exact references, which is the result that
proves the matcher works on real data and not just on fixtures.
Widening the window past 30 days: three candidates, all false — and why
Setting the window to OC's whole order history (150 orders, 2026-09-06 →
2026-11-13) reported three unaccounted $5 packs, all
ntnsndr, all on2026-09-28 (03:51, 04:00, 04:11). They are NOT gaps: the ledger shows they were
credited by hand that morning, when the tier matcher was broken at arrival time:
credit_pack_accounted()cannot see this, and must not: inferring "accountedfor" from a nearby amount would hide genuine gaps. But a reviewer could not tell
the two cases apart either. So each candidate now carries its nearby
ledger/held rows (7-day window, capped at 5) and the report-mode log line prints
them — the manual grant explains itself.
Consequence for the window setting
With
CREDIT_RECONCILE_DAYSset anywhere that reaches 2026-09-28 andapplyon, three already-granted packs would be credited a second time. Twoconsequences for whoever turns this on:
default is clean), or
nearby_book_rowsbefore applying — which is nowpossible, and is what the dry run is for.
A stronger rule, if anyone wants it later: treat a candidate whose member has an
UNREFERENCED manual row inside the window as "needs human" rather than"gran-
table". Not implemented — it would need a policy decision about manual grants,
not just a matching rule.