The alert for a failed Open Collective delivery lived in portal_webhook_watch.py on
the operator's machine, polling this app's log through the Cloudron API. That put a
money-critical signal outside the system, and on a log buffer with only hours of
memory — during the 2026-11-13 investigation whole hours (06:00-13:00, 15:00) were
simply absent, so 'no failed delivery in the log' could not be distinguished from
'no delivery at all'.
The portal handles every delivery, so the portal now knows: a new webhook_failures
table records every non-2xx (including the 401 that means the URL and WEBHOOK_TOKEN
disagree, whose event type is read from a body cached before the token check), and
webhook_failure_report() in the sweep emails info@ ONCE per new batch via a marker
in the new portal_state table, so a wrong token does not mail every sweep while the
rows stay queryable. The delivery path is unchanged: bookkeeping failures are
logged, never raised, and the response stays 401/503.
19 checks in test_webhook_resilience.py; all 18 suites green.
On 2026-11-13 an `order.processed` delivery for a $50 credit pack returned 500.
fetch_members() -> fetch_all_member_nodes() used a bare httpx.AsyncClient(), so OC
was given httpx's DEFAULT 5-second timeout; the read timed out, httpx.ReadTimeout
escaped the handler as an unhandled ASGI exception, and the purchase was never
credited. The member paid and got nothing, and the log held a stack trace rather
than anything naming the member or the order.
Three changes, each aimed at a different failure:
* fetch_all_member_nodes: an EXPLICIT 30s timeout, and one retry per page on a
transport error (`_oc_graphql_post`). Still RAISES when it cannot get the full
list — the sweep's fail-safes depend on that.
* opencollective_webhook: the work moves to _process_opencollective_webhook and a
thin wrapper answers 503 (retryable) with an ERROR line naming the exception if
anything unexpected escapes. A 401 stays a 401; that status is the signal that
the webhook URL and WEBHOOK_TOKEN disagree, and swallowing it would hide a
misconfigured webhook.
* tests/test_webhook_resilience.py proves it: 401 preserved, 503 (not 500) on a
simulated ReadTimeout, credits granted exactly once per OC order, an identical
re-delivery answering already_credited with no money moved, and the retry/raise
behaviour of the transport helper.