webhook failures: the portal records and reports its own, not a scraper on my host
ci/woodpecker/push/woodpecker Pipeline was successful
ci/woodpecker/push/woodpecker Pipeline was successful
The alert for a failed Open Collective delivery lived in portal_webhook_watch.py on the operator's machine, polling this app's log through the Cloudron API. That put a money-critical signal outside the system, and on a log buffer with only hours of memory — during the 2026-11-13 investigation whole hours (06:00-13:00, 15:00) were simply absent, so 'no failed delivery in the log' could not be distinguished from 'no delivery at all'. The portal handles every delivery, so the portal now knows: a new webhook_failures table records every non-2xx (including the 401 that means the URL and WEBHOOK_TOKEN disagree, whose event type is read from a body cached before the token check), and webhook_failure_report() in the sweep emails info@ ONCE per new batch via a marker in the new portal_state table, so a wrong token does not mail every sweep while the rows stay queryable. The delivery path is unchanged: bookkeeping failures are logged, never raised, and the response stays 401/503. 19 checks in test_webhook_resilience.py; all 18 suites green.
This commit is contained in:
1 parent
6c82d7d689
commit
2e40387245
2 files changed
+215
-1
No files matched your search
@@ -80,6 +80,13 @@ def check(name, ok):
|
||||
results.append((name, ok))
|
||||
|
||||
|
||||
SENT = []
|
||||
|
||||
|
||||
def fake_send(to, subject, body, *a, **k):
|
||||
SENT.append({"to": to, "subject": subject, "body": body})
|
||||
|
||||
|
||||
# --- stubs ----------------------------------------------------------------- #
|
||||
async def fetch_members_ok():
|
||||
return [{"email": MEMBER_EMAIL, "name": "NightRose", "slug": MEMBER_SLUG,
|
||||
@@ -129,6 +136,40 @@ async def main():
|
||||
"(httpx.ReadTimeout was caught by the handler net)",
|
||||
r.status_code != 500)
|
||||
|
||||
# --- 2b. The SYSTEM records the failures it refuses. ---
|
||||
# The money-critical alert used to be a log scraper on the operator's own
|
||||
# machine, reading a buffer with hours of memory — whole hours were missing
|
||||
# from it during the 2026-11-13 investigation. The portal handles every
|
||||
# delivery, so the portal is what must notice and say.
|
||||
conn = m.get_db()
|
||||
rows = [dict(zip(("id", "at", "status", "event_type", "detail"), r)) for r in
|
||||
conn.execute("SELECT id, at, status, event_type, detail FROM webhook_failures "
|
||||
"ORDER BY id")]
|
||||
conn.close()
|
||||
check(f"record: the refused 401 delivery is recorded by the portal (got {rows})",
|
||||
any(r["status"] == 401 for r in rows))
|
||||
check(f"record: the 503 is recorded with its event type (got {rows})",
|
||||
any(r["status"] == 503 and r["event_type"] == "order.processed" for r in rows))
|
||||
check("record: the 503 row names the exception that caused it",
|
||||
any(r["status"] == 503 and "ReadTimeout" in (r["detail"] or "") for r in rows))
|
||||
|
||||
SENT.clear()
|
||||
m.send_email = fake_send
|
||||
rep = m.webhook_failure_report()
|
||||
check(f"report: new failures are summarised (got {rep.get('new_failures')} rows)",
|
||||
rep.get("status") == "reported" and rep.get("new_failures") == len(rows))
|
||||
check(f"report: info@ is told, naming both kinds of failure (got {SENT})",
|
||||
len(SENT) == 1 and SENT[0]["to"] == "info@inference.coop"
|
||||
and "401" in SENT[0]["body"] and "503" in SENT[0]["body"]
|
||||
and "WEBHOOK_TOKEN" in SENT[0]["body"])
|
||||
rep2 = m.webhook_failure_report()
|
||||
check(f"report: the SAME batch is not re-reported (got {rep2})",
|
||||
rep2.get("new_failures") == 0 and len(SENT) == 1)
|
||||
await c.post("/webhook/opencollective/wrong-token", json=PAYLOAD)
|
||||
rep3 = m.webhook_failure_report()
|
||||
check(f"report: a LATER failure is reported again (got {rep3.get('new_failures')})",
|
||||
rep3.get("new_failures") == 1 and len(SENT) == 2)
|
||||
|
||||
# --- 3. MONEY: a resolvable purchase credits exactly once. ---
|
||||
m.fetch_members = fetch_members_ok
|
||||
r = await c.post(url, json=PAYLOAD)
|
||||
|
||||
Reference in new issue
Block a user