webhook failures: the portal records and reports its own, not a scraper on my host
ci/woodpecker/push/woodpecker Pipeline was successful

The alert for a failed Open Collective delivery lived in portal_webhook_watch.py on
the operator's machine, polling this app's log through the Cloudron API. That put a
money-critical signal outside the system, and on a log buffer with only hours of
memory — during the 2026-11-13 investigation whole hours (06:00-13:00, 15:00) were
simply absent, so 'no failed delivery in the log' could not be distinguished from
'no delivery at all'.

The portal handles every delivery, so the portal now knows: a new webhook_failures
table records every non-2xx (including the 401 that means the URL and WEBHOOK_TOKEN
disagree, whose event type is read from a body cached before the token check), and
webhook_failure_report() in the sweep emails info@ ONCE per new batch via a marker
in the new portal_state table, so a wrong token does not mail every sweep while the
rows stay queryable. The delivery path is unchanged: bookkeeping failures are
logged, never raised, and the response stays 401/503.

19 checks in test_webhook_resilience.py; all 18 suites green.
This commit is contained in:
inference-admin committed 2026-11-13 20:23:33 +00:00
1 parent 6c82d7d689
commit 2e40387245
2 files changed
+215 -1

No files matched your search

+41
View File
@@ -80,6 +80,13 @@ def check(name, ok):
results.append((name, ok))
SENT = []
def fake_send(to, subject, body, *a, **k):
SENT.append({"to": to, "subject": subject, "body": body})
# --- stubs ----------------------------------------------------------------- #
async def fetch_members_ok():
return [{"email": MEMBER_EMAIL, "name": "NightRose", "slug": MEMBER_SLUG,
@@ -129,6 +136,40 @@ async def main():
"(httpx.ReadTimeout was caught by the handler net)",
r.status_code != 500)
# --- 2b. The SYSTEM records the failures it refuses. ---
# The money-critical alert used to be a log scraper on the operator's own
# machine, reading a buffer with hours of memory — whole hours were missing
# from it during the 2026-11-13 investigation. The portal handles every
# delivery, so the portal is what must notice and say.
conn = m.get_db()
rows = [dict(zip(("id", "at", "status", "event_type", "detail"), r)) for r in
conn.execute("SELECT id, at, status, event_type, detail FROM webhook_failures "
"ORDER BY id")]
conn.close()
check(f"record: the refused 401 delivery is recorded by the portal (got {rows})",
any(r["status"] == 401 for r in rows))
check(f"record: the 503 is recorded with its event type (got {rows})",
any(r["status"] == 503 and r["event_type"] == "order.processed" for r in rows))
check("record: the 503 row names the exception that caused it",
any(r["status"] == 503 and "ReadTimeout" in (r["detail"] or "") for r in rows))
SENT.clear()
m.send_email = fake_send
rep = m.webhook_failure_report()
check(f"report: new failures are summarised (got {rep.get('new_failures')} rows)",
rep.get("status") == "reported" and rep.get("new_failures") == len(rows))
check(f"report: info@ is told, naming both kinds of failure (got {SENT})",
len(SENT) == 1 and SENT[0]["to"] == "info@inference.coop"
and "401" in SENT[0]["body"] and "503" in SENT[0]["body"]
and "WEBHOOK_TOKEN" in SENT[0]["body"])
rep2 = m.webhook_failure_report()
check(f"report: the SAME batch is not re-reported (got {rep2})",
rep2.get("new_failures") == 0 and len(SENT) == 1)
await c.post("/webhook/opencollective/wrong-token", json=PAYLOAD)
rep3 = m.webhook_failure_report()
check(f"report: a LATER failure is reported again (got {rep3.get('new_failures')})",
rep3.get("new_failures") == 1 and len(SENT) == 2)
# --- 3. MONEY: a resolvable purchase credits exactly once. ---
m.fetch_members = fetch_members_ok
r = await c.post(url, json=PAYLOAD)