webhook: stop a transient OC read failure from costing a member credits
On 2026-11-13 an `order.processed` delivery for a $50 credit pack returned 500. fetch_members() -> fetch_all_member_nodes() used a bare httpx.AsyncClient(), so OC was given httpx's DEFAULT 5-second timeout; the read timed out, httpx.ReadTimeout escaped the handler as an unhandled ASGI exception, and the purchase was never credited. The member paid and got nothing, and the log held a stack trace rather than anything naming the member or the order. Three changes, each aimed at a different failure: * fetch_all_member_nodes: an EXPLICIT 30s timeout, and one retry per page on a transport error (`_oc_graphql_post`). Still RAISES when it cannot get the full list — the sweep's fail-safes depend on that. * opencollective_webhook: the work moves to _process_opencollective_webhook and a thin wrapper answers 503 (retryable) with an ERROR line naming the exception if anything unexpected escapes. A 401 stays a 401; that status is the signal that the webhook URL and WEBHOOK_TOKEN disagree, and swallowing it would hide a misconfigured webhook. * tests/test_webhook_resilience.py proves it: 401 preserved, 503 (not 500) on a simulated ReadTimeout, credits granted exactly once per OC order, an identical re-delivery answering already_credited with no money moved, and the retry/raise behaviour of the transport helper.
This commit is contained in:
1 parent
5705d02792
commit
f45e4ef305
2 files changed
+262
-5
No files matched your search
+69
-5
@@ -1364,6 +1364,32 @@ def compute_max_budget(allowance: float, credit_balance: float, team_spend: floa
|
||||
}
|
||||
|
||||
|
||||
async def _oc_graphql_post(client: httpx.AsyncClient, json_body: dict,
|
||||
headers: dict | None = None, attempts: int = 2):
|
||||
"""POST to Open Collective's GraphQL API, retrying once on a transport error.
|
||||
|
||||
A read timeout here is load-correlated and transient — not a permission,
|
||||
query or scope problem — so one retry converts the common case into a
|
||||
success. Whatever still fails is RAISED: every caller in this module is
|
||||
built to fail loudly (the sweep must never deactivate members from a
|
||||
partial list), so swallowing it would be worse than the error.
|
||||
"""
|
||||
last: Exception | None = None
|
||||
for attempt in range(attempts):
|
||||
try:
|
||||
return await client.post(OC_GRAPHQL_URL, headers=headers or {}, json=json_body)
|
||||
except httpx.TransportError as e: # ReadTimeout, ConnectError, …
|
||||
last = e
|
||||
if attempt + 1 < attempts:
|
||||
logger.warning(
|
||||
"OC GraphQL %s (attempt %d/%d); retrying once",
|
||||
type(e).__name__, attempt + 1, attempts,
|
||||
)
|
||||
await asyncio.sleep(2.0)
|
||||
assert last is not None
|
||||
raise last
|
||||
|
||||
|
||||
async def fetch_all_member_nodes() -> list[dict]:
|
||||
"""Every member node of the collective, via OC's offset pagination.
|
||||
|
||||
@@ -1373,22 +1399,27 @@ async def fetch_all_member_nodes() -> list[dict]:
|
||||
user list. Raises on ANY page failure so callers can apply their own
|
||||
fail-safes; never returns a partial list (a partial member list would make
|
||||
the sweep deactivate everyone missing from it).
|
||||
|
||||
The timeout is set EXPLICITLY on purpose. httpx's default is 5 seconds, and
|
||||
on 2026-11-13 a page of this walk took longer than that: `httpx.ReadTimeout`
|
||||
escaped the OC webhook handler as an unhandled exception, so a member's $50
|
||||
credit-pack purchase returned 500 and her credits were not granted. A
|
||||
generous timeout plus one retry makes the transient case a success; the
|
||||
handler's own error net (see opencollective_webhook) covers the rest.
|
||||
"""
|
||||
if not OC_PERSONAL_TOKEN:
|
||||
raise RuntimeError("OC_PERSONAL_TOKEN not set; cannot fetch members")
|
||||
nodes: list[dict] = []
|
||||
offset = 0
|
||||
async with httpx.AsyncClient() as client:
|
||||
async with httpx.AsyncClient(timeout=30.0) as client:
|
||||
for _ in range(20): # hard cap: 20 pages × 100 = 2000 members
|
||||
query = (
|
||||
'{ collective(slug: "%s") { members(limit: 100, offset: %d) { '
|
||||
"totalCount nodes { role tier { name } account { name slug type ... on Individual { email } } } } } }"
|
||||
% (OC_COLLECTIVE_SLUG, offset)
|
||||
)
|
||||
r = await client.post(
|
||||
OC_GRAPHQL_URL,
|
||||
headers={"Personal-Token": OC_PERSONAL_TOKEN},
|
||||
json={"query": query},
|
||||
r = await _oc_graphql_post(
|
||||
client, {"query": query}, {"Personal-Token": OC_PERSONAL_TOKEN},
|
||||
)
|
||||
if r.status_code != 200:
|
||||
raise RuntimeError(f"OC member fetch failed: HTTP {r.status_code}")
|
||||
@@ -3711,6 +3742,39 @@ async def broker_revoke_api_key(email: str, name: str) -> None:
|
||||
async def opencollective_webhook(request: Request, token: str):
|
||||
"""Handle Open Collective membership events.
|
||||
|
||||
The work happens in _process_opencollective_webhook; this wrapper exists
|
||||
ONLY to stop an unexpected failure inside it from escaping as an unhandled
|
||||
ASGI exception. On 2026-11-13 an httpx.ReadTimeout raised by the OC member
|
||||
fetch did exactly that: the delivery returned 500, a member's $50
|
||||
credit-pack purchase was never credited, and the log recorded an anonymous
|
||||
server error rather than anything naming the member or the order.
|
||||
|
||||
Answering 503 says "temporary — send it again", which is both true and the
|
||||
only outcome that repairs itself; the ERROR line names the exception so the
|
||||
purchase can still be reconciled by hand if OC stops retrying. Auth
|
||||
failures are HTTPExceptions and pass straight through — a 401 must stay a
|
||||
401, because that is the signal that the webhook URL and WEBHOOK_TOKEN
|
||||
disagree.
|
||||
"""
|
||||
try:
|
||||
return await _process_opencollective_webhook(request, token)
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e: # noqa: BLE001
|
||||
logger.exception(
|
||||
"OC webhook: handler failed (%s: %s) — answering 503 so OC retries; "
|
||||
"if the payload was an order, confirm the member was credited",
|
||||
type(e).__name__, e,
|
||||
)
|
||||
return JSONResponse(
|
||||
{"status": "error", "retry": True, "detail": type(e).__name__},
|
||||
status_code=503,
|
||||
)
|
||||
|
||||
|
||||
async def _process_opencollective_webhook(request: Request, token: str):
|
||||
"""Handle Open Collective membership events (the implementation).
|
||||
|
||||
Authenticated by a secret token in the URL path (Open Collective's
|
||||
generic webhooks are not HMAC-signed, so a secret URL is the standard
|
||||
way to authenticate them).
|
||||
|
||||
Reference in new issue
Block a user