The alert for a failed Open Collective delivery lived in portal_webhook_watch.py on
the operator's machine, polling this app's log through the Cloudron API. That put a
money-critical signal outside the system, and on a log buffer with only hours of
memory — during the 2026-11-13 investigation whole hours (06:00-13:00, 15:00) were
simply absent, so 'no failed delivery in the log' could not be distinguished from
'no delivery at all'.
The portal handles every delivery, so the portal now knows: a new webhook_failures
table records every non-2xx (including the 401 that means the URL and WEBHOOK_TOKEN
disagree, whose event type is read from a body cached before the token check), and
webhook_failure_report() in the sweep emails info@ ONCE per new batch via a marker
in the new portal_state table, so a wrong token does not mail every sweep while the
rows stay queryable. The delivery path is unchanged: bookkeeping failures are
logged, never raised, and the response stays 401/503.
19 checks in test_webhook_resilience.py; all 18 suites green.
Both routes are needed. A manual grant often carries no reference at all, so the
row's EMAIL is the only link to its member — and the slug and email local part
routinely differ ('henry-smith' vs 'h.smith@x.org'). The membership test was
slug-first and, when the slug matched neither the reference nor the email, dropped
the row outright: a hand-credited purchase therefore reads as a gap (ntnsndr's
2026-09-28 packs, where the ledger carries +20.00, +5.75 and a -10.75 correction
with no references). _row_belongs_to() accepts either route, in both the
accounted-for test and the nearby-rows context. 38 checks; all 18 suites green.
Widening the window to OC's whole order history found three $5 packs from
2026-09-28 reported as uncredited — ntnsndr's launch-day purchases, which WERE
credited by hand that morning: +20.00, +5.75, then a -10.75 correction ('true
contributions were 3 x $5.00'), netting exactly $15, with no per-order reference
on any row.
credit_pack_accounted() is right not to infer 'accounted for' from a nearby
amount — that would hide genuine gaps — but a reviewer could not tell the two
cases apart from the candidate alone. Each candidate now carries its nearby
ledger/held rows (7-day window, capped at 5) and the report-mode log line prints
them, so a manual grant explains itself. Nothing about the decision changes.
36 checks; all 18 suites green.
Credits have one path in: the OC order.processed webhook. On 2026-11-13 two paid
packs got nothing through it — a $50 whose delivery 500'd on httpx's 5s default,
and a $10 with no delivery in the retained log at all — and nothing on a
schedule noticed. This adds a pass that reads OC's own order list and compares it
with our books over a day-based window (not "whatever the log still holds": the
$10 gap was 11h old and the log had lost the whole morning).
Report-first: CREDIT_RECONCILE defaults to 'report' (log findings, change
nothing) and grants only when set to 'on'. Grants go through add_credits() with
credit_pack_reference(), the same per-order key the webhook writes, so a late
delivery and this pass are idempotent against each other in both directions. A
buyer with no active membership is reported, never credited — those keep the
hold-and-email path.
credit_pack_accounted() encodes what our own rows actually look like: a v2 id, a
hyphenated legacy id, an empty-suffix legacy reference, and held_credits rows. A
bare oc:<slug>:<order.id> match read 21 credited purchases as missing during
development, so the fallback demands a second signal (member + tip-free amount +
date) before calling an order credited.
POST /admin/credit-pack-reconcile (X-Admin-Token) runs it on demand, dry unless
?apply=1 and the policy is on. 33 new checks; all 18 suites green.
On 2026-11-13 an `order.processed` delivery for a $50 credit pack returned 500.
fetch_members() -> fetch_all_member_nodes() used a bare httpx.AsyncClient(), so OC
was given httpx's DEFAULT 5-second timeout; the read timed out, httpx.ReadTimeout
escaped the handler as an unhandled ASGI exception, and the purchase was never
credited. The member paid and got nothing, and the log held a stack trace rather
than anything naming the member or the order.
Three changes, each aimed at a different failure:
* fetch_all_member_nodes: an EXPLICIT 30s timeout, and one retry per page on a
transport error (`_oc_graphql_post`). Still RAISES when it cannot get the full
list — the sweep's fail-safes depend on that.
* opencollective_webhook: the work moves to _process_opencollective_webhook and a
thin wrapper answers 503 (retryable) with an ERROR line naming the exception if
anything unexpected escapes. A 401 stays a 401; that status is the signal that
the webhook URL and WEBHOOK_TOKEN disagree, and swallowing it would hide a
misconfigured webhook.
* tests/test_webhook_resilience.py proves it: 401 preserved, 503 (not 500) on a
simulated ReadTimeout, credits granted exactly once per OC order, an identical
re-delivery answering already_credited with no money moved, and the retry/raise
behaviour of the transport helper.
A new member could end up with a Loomio account and a Matrix account and see
NOTHING: no forum group, no Space. Three defects:
* sync_matrix_space() gated each member on the SPACE's membership, so an
account already joined/invited to the Space was skipped for EVERY room. A
member whose invite pass was interrupted (or who predates a room added to
MATRIX_INVITE_ROOMS later) never received that room — and a restricted child
room is invisible until you are invited to it. The gate is now per room, from
a single up-front membership snapshot; a room whose membership cannot be read
is left alone rather than invited into blindly.
* a 429 slept for retry_after_ms (minutes) INSIDE the pass, so a scheduled run
could outlive its own tick. A rate limit now ENDS the pass; the next run
resumes (every invite is idempotent).
* a member sitting in the Loomio group with no loomio_invited_at stamp reads as
'never invited' — the dedupe guard was only as good as the rows it had. The
sync now heals the record from the live roster.
And the activation window itself: provisioning skips an unactivated account by
design, and activation happens later with nobody watching, so the 3x/day sweep
was the pickup. New /admin/service-sync + a 5-minute scheduler task fires the
same idempotent syncs while any member still awaits an invite (one SELECT when
none do), and deliberately is NOT /admin/sweep — it never touches Open
Collective, allowance logic or member mail.
tests/test_service_invites.py proves the per-room gate, resumable rate limits,
the unreadable-room skip, the record heal and the endpoint's idle/run paths.
The member dashboard's usage chart reads per-day spend + requests for the
signed-in member, bounded by the existing days param (0 = all time, the
unchanged default). Same source and day convention as the co-op daily series
(to_char(startTime)); interior zero days are explicit so chart spacing is
true; requests excludes failed attempts; spend sums every row. Additive and
backward compatible: every existing field untouched. Tested in
tests/test_broker_usage_daily.py (route table, SQL shape, fallback twin,
fail-soft degradation).
The 2026-11-12 hole was NOT a missed sample: the 03:15 run recorded
oc_balance_usd=1357.39, and the evening repair backfill's full-row
upsert of the then-current day — whose dict carries no balance key —
nulled it on conflict. Nothing ever revisited a day's balance, so the
hole was permanent.
Two fixes:
- upsert() can no longer erase a balance it did not measure: the
conflict clause COALESCEs, so a NULL (a failed OC read) keeps the
previously recorded value.
- the nightly snapshot heals any recent day (within a week, after the
series' first recorded balance) whose balance is missing: the value is
reconstructed from Open Collective's transaction ledger
(balance(cutoff) = current - net flow after the 03:15 sample moment,
ledger window clamped to the balance read), matching the daily sample
semantics exactly. It is written balance-column-only, so membership
and usage series are untouched. Days before 2026-11-08 stay NULL by
design. Fail-soft: an OC failure prints and retries next night.
The reconstruction was verified to the cent against OC's own ledger:
full-ledger signed sum == stats.balance (1470.44 on 2026-11-13), and
1470.44 - 113.05 (net flow 2026-11-12 03:15 -> 2026-11-13 03:15:58)
== 1357.39, exactly the sample the 03:15 run had recorded.
snapshot() fetched the full spend window (since = yesterday) but kept only
spend.get(TODAY): the day's first hours, written at 03:15 and never revisited.
Every metrics_daily row after the last backfill under-counted its day by the
hours after 03:15 (Nov 7 recorded $2.19/691 requests against a true
$29.06/3384). The completed day's aggregates were computed and thrown away.
Now each run completes the day that just ENDED — full-day spend, requests,
tokens and impact, written to the usage columns only (upsert_usage: membership
columns are point-in-time measurements and are never rewritten) — and then
records today's live partial as before.
backfill() gains --since YYYY-MM-DD: a targeted usage-only repair that cannot
touch the published membership curve.
Regression test: tests/test_metrics_completion.py (day completion, membership
preservation, since-mode repair boundaries, full-backfill parity).
Single derivation point (tested once, here): 'invited' while the Cloudron
invite is unaccepted, else the guests-ledger status (the record of record),
membership state only as a no-ledger-row fallback. The panel and dashboard
become pure consumers of `state` — their local invited/active derivation
goes away in their own repos next. Live-behaviour note: today's four live
guests render byte-identically; the change only shows for ended guests
(ledger 'expired' instead of the bare inactive flag), which is what the
panel already displayed via its own guest_status preference.
The one-per-30-days rule is a READ of the guests ledger, provisioning
takes seconds of external calls, and the ledger row that closes the
window is written LAST — so two PARALLEL direct calls (curl, bypassing
the dashboard JS) could both pass guest_request_guard before either row
exists and slip two guests past the rule. The member invite path now
runs guard+provision under _GUEST_INVITE_LOCK (single-process uvicorn),
making check-and-send atomic: the second caller re-runs the guard after
the first caller's ledger row exists and is refused 429.
tests/test_guest_concurrency.py proves both halves on the fixture DB:
unguarded, two parallel invites both provision (the test has teeth);
with the lock, exactly one provisions, one is refused 429, one ledger
row, one account, one email; sequential semantics unchanged.
A guest invitation provisions immediately, but the Cloudron account is not
ACCEPTED until the guest clicks the setup link (inviteAccepted=false, no
username) — until then they cannot log in, so no surface may read them as
active. guest_invite_pending() derives it from the same Cloudron signal the
overview's activated flag uses; unseen-record (users fetch failed) is never
pending, so the state falls back instead of asserting what the data does
not show.
/admin/overview: guest rows report state='invited' with an 'awaiting
setup' reason (invite date named when the ledger has it); /broker/guests
trail entries carry state='invited' the same way, status stays the ledger
record of record. Display/state semantics only — access rules, the $2 cap
and the expiry box are unchanged.
Tests: new test_guest_invited_state.py (derivation, overview flip,
fetch-failure fallback, member rows untouched, trail state);
test_guest_requests.py gains the trail invited/accepted assertions and a
constructible-but-loud fake client so the stubbed users fetch runs.
Operator decision 2026-11-08: the remaining protections (active member,
1 guest per inviter per rolling 30 days, one guest per recipient ever,
no existing portal row, MANUAL_MEMBERS refusal, the $2 cap and the
GUEST_DAYS box) are sufficient without the approval step.
* POST /broker/guests provisions through provision_guest_member at once —
no pending state; no guest_requests row is ever written.
* GUEST_AUTO_APPROVE flag removed (the live env never carried it).
* guest_request_guard keeps the member-path rules and drops the
trial-era pending-request checks; the 30-day window is counted from
the guests ledger.
* GET /admin/guest-requests + approve/reject removed; GET /admin/guest-ledger
serves the panel's duplicate-send guard.
* The member trail keeps legacy rejected requests as history only.
Built on the sibling $2-cap commit 6be8e2d.
Operator edits:
- The co-op is described only as a member-governed AI cooperative; the
renewable-energy claim is removed (not all providers are renewable).
- Membership pitch now states the pricing exactly: pay on a sliding
scale, $15/month allowance plus optional credit packs, plus a say in
how it's run.
The admin panel's /admin/provision-guest now takes ONE field (the guest's
email): no name is collected and none is derived from the email — the
guest chooses their own display name and username during Cloudron account
setup, and the invitation email greets neutrally when no name is known.
The guests ledger records the co-op's ADMIN account (ADMIN_INVITER_EMAIL,
default info@inference.coop) as the inviter. Admin invites provision
immediately (the operator IS the approver) and are not bound by the
member's one-per-30-days cap; recipient-side guards (one guest per
recipient ever, no existing portal row) still run. The member paths
(broker invite, request approval) keep every protection and likewise pass
no derived name.
POST /broker/guests (member broker identity — guests cannot reach it) creates
a guest REQUEST; GET /broker/guests returns the member's invitation allowance
(rolling one-per-30-days, counting pending + approved requests) and their own
guest trail. guest_request_guard enforces every existing protection plus the
member-side ones: recipient never an existing member, a prior guest, a pending
request, or a MANUAL_MEMBERS address; the inviter's window counts requests too.
Trial gate (GUEST_AUTO_APPROVE, default off): requests land PENDING in the new
guest_requests table; the operator approves or rejects them via
GET/POST /admin/guest-requests (X-Admin-Token). Approval fires the full
provision_guest_member pipeline (Cloudron user, $5 credit, GUEST_DAYS box,
invitation email naming the inviter, newsletter, ledger row); a guard refusal
at approve-time is recorded on the request and surfaced. The member sees
pending/not-approved status in their own trail — the recipient is contacted
only on approval. With the flag on, a member's request provisions
immediately, no code change needed.
38-check fixture suite (tests/test_guest_requests.py) proves the gate, every
refusal path, the approve/reject actions and auto-approve with the network
replaced by a raising guard.
Two defects found by booting the renamed app against a fixture of the
pre-rename database (the live portal's exact state):
1. members.gift_expires_at: the RENAME above uses the stale
snapshot, so the plain
below fired right after a successful rename and crashed boot with
'duplicate column name' — the portal would have failed its deploy
health check on the live database. Make the two branches exclusive.
2. The ledger's own expiry column: ALTER TABLE guests RENAME moved the
table but kept its legacy gift_expires_at column, which the provision
INSERT (writing guest_expires_at) would fail on at first use. Rename
the column with the table.
Also moves the credit-ledger dedupe references ('gift:<email>' /
'gift-return:<email>') to their guest prefixes at boot — returns first,
since 'gift-return:…' also matches 'gift:%' — so a re-grant after the
rename cannot double-credit.
Verified: migration harness 10/10 on a legacy-state fixture (renames,
value conversion, ref migration, idempotent second boot, readback
helpers); full portal suite 9/9 files green.
Pure rename, no behaviour change, matching the renamed panel and dashboard
builds. Includes one-time idempotent migrations for existing databases
(gifts->guests table, gift_expires_at->guest_expires_at column,
account_type 'gift'->'guest' values, gift_lapse_notices->guest_lapse_notices)
and four transitional alias routes (admin/provision-gift, admin/gift-convert,
broker/gift-status, broker/cancel-gift) so pre-rename dashboard/panel builds
keep working until they are redeployed.
POST /broker/cancel-gift (dashboard broker identity, gift accounts only,
idempotent) + GET /broker/gift-status for the ended-state view. Semantics
match the gift lapse: row deactivated, account kept for a clean resume,
gateway budget zeroed, newsletter off — plus the return a lapse does not
do: unspent credit back to the co-op as a NEGATIVE credit_ledger row whose
reason names the cancellation and the original gift reference, deduped on
its own reference. Gifts ledger records inviter → recipient → 'cancelled
by recipient'. The user leaves the guests group for GUESTS_ENDED_GROUP_ID
(dashboard-only visibility for the ended state); the reaching-limits
notice is pre-marked so a self-cancelled guest is never told their
credits were spent. Tests: tests/test_gift_cancel.py (19 fixture checks,
network-guarded).
- gift_welcome_email becomes gift_invitation_email: the ONE email a gift
sends (the invitation and the guest welcome merged, not two mails at
once). Subject 'You've been invited to the Inference Cooperative'; names
the inviter (member slug), the $5 credits, the setup link, the
Become-a-member button on BECOME_MEMBER_URL. Uses the shared
PINE_LOGO_SVG instead of an inline copy.
- reaching-limits email (gift_limits_email, 'Your guest access has ended —
become a member to continue') wired into the sweep's gift deactivation:
sent once per guest, gated by a gift_lapse_notices sent-state row (same
pattern as mark_shortfall_notice_sent). Says the access ended, the
account and data are kept, and membership ($15/mo via Open Collective)
restores a full monthly allowance + community spaces.
- catch-up pass in reconcile_gift_lifecycles: guests deactivated before the
email shipped (or a failed send) get exactly one notice on a later sweep;
converted guests are excluded by account_type.
- sweep summary now reports gift_lapse_notices.
- test_gift_lifecycle: invitation-subject assertion; send_email stub
returns True (the bool is the gate's contract); new once-only + catch-up
+ healthy-untouched checks. 43/43.
/admin/overview now returns gift (guest) accounts in a separate guests[]
list instead of inside members[], and excludes them from totals.members,
totals.active and the member spend totals. Guests get their own totals
breakout (guests, guests_active, guest_spend, guest_all_time_spend) so
their spend stays visible without ever reading as member activity — the
same reasoning as the operator (admins) and agent categories.
Builds on the gift feature's account_type marker: a guest is any row with
account_type='gift'. The panel reads guests[] and ignores gift rows in
its member count either way.
Operator feature: a member gives a friend access with $5 of credits, no OC
membership. Cloudron user lands in the guests group (chat + dashboard only),
LiteLLM team capped at GIFT_BUDGET with NO monthly reset, portal row marked
account_type='gift' with gift_expires_at set, gift welcome email, newsletter
subscription, and a gifts ledger recording inviter -> recipient (one gift per
inviter per 30 days, one per recipient ever).
Sweep safety: reconcile_memberships never deactivates a gift for lacking an
OC contribution; reconcile_gift_lifecycles lapses gifts by their own rules
(credits spent out, or the GIFT_DAYS time box) with member-deactivation
semantics. Guests are excluded from Loomio/Matrix invite lists, the reinvite
path, budget notices, the allowance reconciliation and the metrics member
counts.
Conversion: a gift whose OC identity appears under its own email converts
automatically (webhook + sweep + /admin/provision all funnel through
provision_member): group guests->members, cap lifted to allowance + carried
credits, anchor day kept, congrats email, full member onboarding.
/admin/gift-convert links a membership paid under a different OC email
(members.oc_email, matched by the sweep and webhook).
Fixture test: tests/test_gift_lifecycle.py (37 checks, network replaced by
a raising guard).