The failure this exists for (2026-11-05): the portal's own all-time-spend aggregate
failed SOFT when its copy of the gateway's LiteLLM DB credential went stale — the
query returned nothing, every surface showed $0, and nothing looked wrong, because
the portal WAS the thing that was wrong. The guard ran as a daily script on the
bot's host.
It now runs in the portal, and keeps two genuinely independent reads: the portal's
SpendLogs aggregate over its own DB credential, and LiteLLM's own HTTP API
(/global/spend/keys) over the master key — a different transport AND a different
credential, which is the point, since the DB copy is what goes stale.
It adds the signature guard the host script could only infer from outside: an
EMPTY//usr/bin/bash portal aggregate while LiteLLM reports real spend is a FAILURE, named as
the stale-credential case; an API side that returns no keys is a failure too, not
agreement. A read that cannot be made is skipped — never a clean bill. Silent when
they agree; info@ once per failing episode, marker cleared on recovery.
POST /admin/spend-check runs it on demand. 23 checks; all 20 suites green.
The first production run reported a null-account node as an uncredited payer. The
cause was in my port, not the data: the members query selected id-less,
date-less nodes, so every null-account node fell into the date "" bucket, matched
no same-day order, and was reported as unattributable — while the host-side script
(correctly) said clean.
Two fixes: select id + createdAt in fetch_all_member_nodes, and never emit a
null-account finding when the node carries no date — there is nothing to attribute
it against, and guessing is what produced the phantom. 27 checks in the suite.
Three fixes to the 'email only when there is a problem' rule, found by testing it:
1. The marker recorded the CURRENT set, so a set that SHRANK looked changed — one
purchase granted dropped out, the remaining hold looked new, and it was mailed
about twice. What a human has been TOLD is now a growing SET of keys
(order:grant / order:hold, slug:class for backers): only keys not in it are
mailed, and the union is stored.
2. A notify=False run advanced the marker, so a verification look CONSUMED the
only alert a finding gets. The marker now advances only when the notice was
actually sent (a failed send retries next sweep), and a clean pass clears it so
a recurrence is news again.
3. The credit reconcile had the same shape of bug.
Tests updated to pin the new semantics: a smaller set re-mails nobody, a silent
look does not swallow the alert, a clean run clears the marker and a recurrence
speaks. 26 + 40 checks in those suites; all 19 suites green.
A contributor who pays and never gets an account is invisible to the sweep —
fetch_members() drops every member node with no email, and an OC Organization
never exposes one — so they wait until they ask (the mmoya case). The net for them
was a daily script on the bot's host driving `cloudron exec` from outside. The
portal holds both sides of that comparison, so it now owns it.
reconcile_unmapped_backers() runs in the sweep and reports three classes, because
each needs a different human action: an org account (link it), a VISIBLE email
(the sweep should have provisioned this — check the token/sweep), and a hidden
email (fetch it out-of-band). A node with no account object is attributed through
same-day current orders, and stays silent when every same-day payer is mapped.
MAIL POLICY (ntnsndr, 2026-11-13): only when there is a problem — and only when
the finding SET changes, fingerprinted in portal_state (slug+class only, never an
email), so a stuck finding does not mail three times a day while a fresh one
always speaks. A clean run clears the marker. A failed read returns skipped,
never a clean bill.
POST /admin/unmapped-check (X-Admin-Token, ?notify=0 to stay quiet). The
credit-pack reconcile now mails on the same rule. 25 checks; all 19 suites green.
The alert for a failed Open Collective delivery lived in portal_webhook_watch.py on
the operator's machine, polling this app's log through the Cloudron API. That put a
money-critical signal outside the system, and on a log buffer with only hours of
memory — during the 2026-11-13 investigation whole hours (06:00-13:00, 15:00) were
simply absent, so 'no failed delivery in the log' could not be distinguished from
'no delivery at all'.
The portal handles every delivery, so the portal now knows: a new webhook_failures
table records every non-2xx (including the 401 that means the URL and WEBHOOK_TOKEN
disagree, whose event type is read from a body cached before the token check), and
webhook_failure_report() in the sweep emails info@ ONCE per new batch via a marker
in the new portal_state table, so a wrong token does not mail every sweep while the
rows stay queryable. The delivery path is unchanged: bookkeeping failures are
logged, never raised, and the response stays 401/503.
19 checks in test_webhook_resilience.py; all 18 suites green.
Both routes are needed. A manual grant often carries no reference at all, so the
row's EMAIL is the only link to its member — and the slug and email local part
routinely differ ('henry-smith' vs 'h.smith@x.org'). The membership test was
slug-first and, when the slug matched neither the reference nor the email, dropped
the row outright: a hand-credited purchase therefore reads as a gap (ntnsndr's
2026-09-28 packs, where the ledger carries +20.00, +5.75 and a -10.75 correction
with no references). _row_belongs_to() accepts either route, in both the
accounted-for test and the nearby-rows context. 38 checks; all 18 suites green.
Widening the window to OC's whole order history found three $5 packs from
2026-09-28 reported as uncredited — ntnsndr's launch-day purchases, which WERE
credited by hand that morning: +20.00, +5.75, then a -10.75 correction ('true
contributions were 3 x $5.00'), netting exactly $15, with no per-order reference
on any row.
credit_pack_accounted() is right not to infer 'accounted for' from a nearby
amount — that would hide genuine gaps — but a reviewer could not tell the two
cases apart from the candidate alone. Each candidate now carries its nearby
ledger/held rows (7-day window, capped at 5) and the report-mode log line prints
them, so a manual grant explains itself. Nothing about the decision changes.
36 checks; all 18 suites green.
Credits have one path in: the OC order.processed webhook. On 2026-11-13 two paid
packs got nothing through it — a $50 whose delivery 500'd on httpx's 5s default,
and a $10 with no delivery in the retained log at all — and nothing on a
schedule noticed. This adds a pass that reads OC's own order list and compares it
with our books over a day-based window (not "whatever the log still holds": the
$10 gap was 11h old and the log had lost the whole morning).
Report-first: CREDIT_RECONCILE defaults to 'report' (log findings, change
nothing) and grants only when set to 'on'. Grants go through add_credits() with
credit_pack_reference(), the same per-order key the webhook writes, so a late
delivery and this pass are idempotent against each other in both directions. A
buyer with no active membership is reported, never credited — those keep the
hold-and-email path.
credit_pack_accounted() encodes what our own rows actually look like: a v2 id, a
hyphenated legacy id, an empty-suffix legacy reference, and held_credits rows. A
bare oc:<slug>:<order.id> match read 21 credited purchases as missing during
development, so the fallback demands a second signal (member + tip-free amount +
date) before calling an order credited.
POST /admin/credit-pack-reconcile (X-Admin-Token) runs it on demand, dry unless
?apply=1 and the policy is on. 33 new checks; all 18 suites green.
On 2026-11-13 an `order.processed` delivery for a $50 credit pack returned 500.
fetch_members() -> fetch_all_member_nodes() used a bare httpx.AsyncClient(), so OC
was given httpx's DEFAULT 5-second timeout; the read timed out, httpx.ReadTimeout
escaped the handler as an unhandled ASGI exception, and the purchase was never
credited. The member paid and got nothing, and the log held a stack trace rather
than anything naming the member or the order.
Three changes, each aimed at a different failure:
* fetch_all_member_nodes: an EXPLICIT 30s timeout, and one retry per page on a
transport error (`_oc_graphql_post`). Still RAISES when it cannot get the full
list — the sweep's fail-safes depend on that.
* opencollective_webhook: the work moves to _process_opencollective_webhook and a
thin wrapper answers 503 (retryable) with an ERROR line naming the exception if
anything unexpected escapes. A 401 stays a 401; that status is the signal that
the webhook URL and WEBHOOK_TOKEN disagree, and swallowing it would hide a
misconfigured webhook.
* tests/test_webhook_resilience.py proves it: 401 preserved, 503 (not 500) on a
simulated ReadTimeout, credits granted exactly once per OC order, an identical
re-delivery answering already_credited with no money moved, and the retry/raise
behaviour of the transport helper.
A new member could end up with a Loomio account and a Matrix account and see
NOTHING: no forum group, no Space. Three defects:
* sync_matrix_space() gated each member on the SPACE's membership, so an
account already joined/invited to the Space was skipped for EVERY room. A
member whose invite pass was interrupted (or who predates a room added to
MATRIX_INVITE_ROOMS later) never received that room — and a restricted child
room is invisible until you are invited to it. The gate is now per room, from
a single up-front membership snapshot; a room whose membership cannot be read
is left alone rather than invited into blindly.
* a 429 slept for retry_after_ms (minutes) INSIDE the pass, so a scheduled run
could outlive its own tick. A rate limit now ENDS the pass; the next run
resumes (every invite is idempotent).
* a member sitting in the Loomio group with no loomio_invited_at stamp reads as
'never invited' — the dedupe guard was only as good as the rows it had. The
sync now heals the record from the live roster.
And the activation window itself: provisioning skips an unactivated account by
design, and activation happens later with nobody watching, so the 3x/day sweep
was the pickup. New /admin/service-sync + a 5-minute scheduler task fires the
same idempotent syncs while any member still awaits an invite (one SELECT when
none do), and deliberately is NOT /admin/sweep — it never touches Open
Collective, allowance logic or member mail.
tests/test_service_invites.py proves the per-room gate, resumable rate limits,
the unreadable-room skip, the record heal and the endpoint's idle/run paths.