The first live run FAILED with api $0.00 over 306 keys against the portal's $935.35.
The disagreement was real, the verdict was right, and the cause was mine: LiteLLM's
/global/spend/keys returns each key's all-time spend as `total_spend`, not `spend`.
A checker that trusts its own parse then reports "the portal is wrong" — the exact
mirror of the failure this check exists to catch, which is why the guard now names
both possibilities when the independent side reads zero against real portal spend.
Parser accepts either field name; a regression test exercises the real function
against a stubbed httpx (the module attribute is stubbed for the other checks, so
the real one is captured at import). 24 checks; all 20 suites green.
The failure this exists for (2026-11-05): the portal's own all-time-spend aggregate
failed SOFT when its copy of the gateway's LiteLLM DB credential went stale — the
query returned nothing, every surface showed $0, and nothing looked wrong, because
the portal WAS the thing that was wrong. The guard ran as a daily script on the
bot's host.
It now runs in the portal, and keeps two genuinely independent reads: the portal's
SpendLogs aggregate over its own DB credential, and LiteLLM's own HTTP API
(/global/spend/keys) over the master key — a different transport AND a different
credential, which is the point, since the DB copy is what goes stale.
It adds the signature guard the host script could only infer from outside: an
EMPTY//usr/bin/bash portal aggregate while LiteLLM reports real spend is a FAILURE, named as
the stale-credential case; an API side that returns no keys is a failure too, not
agreement. A read that cannot be made is skipped — never a clean bill. Silent when
they agree; info@ once per failing episode, marker cleared on recovery.
POST /admin/spend-check runs it on demand. 23 checks; all 20 suites green.
The first production run reported a null-account node as an uncredited payer. The
cause was in my port, not the data: the members query selected id-less,
date-less nodes, so every null-account node fell into the date "" bucket, matched
no same-day order, and was reported as unattributable — while the host-side script
(correctly) said clean.
Two fixes: select id + createdAt in fetch_all_member_nodes, and never emit a
null-account finding when the node carries no date — there is nothing to attribute
it against, and guessing is what produced the phantom. 27 checks in the suite.
Three fixes to the 'email only when there is a problem' rule, found by testing it:
1. The marker recorded the CURRENT set, so a set that SHRANK looked changed — one
purchase granted dropped out, the remaining hold looked new, and it was mailed
about twice. What a human has been TOLD is now a growing SET of keys
(order:grant / order:hold, slug:class for backers): only keys not in it are
mailed, and the union is stored.
2. A notify=False run advanced the marker, so a verification look CONSUMED the
only alert a finding gets. The marker now advances only when the notice was
actually sent (a failed send retries next sweep), and a clean pass clears it so
a recurrence is news again.
3. The credit reconcile had the same shape of bug.
Tests updated to pin the new semantics: a smaller set re-mails nobody, a silent
look does not swallow the alert, a clean run clears the marker and a recurrence
speaks. 26 + 40 checks in those suites; all 19 suites green.
A contributor who pays and never gets an account is invisible to the sweep —
fetch_members() drops every member node with no email, and an OC Organization
never exposes one — so they wait until they ask (the mmoya case). The net for them
was a daily script on the bot's host driving `cloudron exec` from outside. The
portal holds both sides of that comparison, so it now owns it.
reconcile_unmapped_backers() runs in the sweep and reports three classes, because
each needs a different human action: an org account (link it), a VISIBLE email
(the sweep should have provisioned this — check the token/sweep), and a hidden
email (fetch it out-of-band). A node with no account object is attributed through
same-day current orders, and stays silent when every same-day payer is mapped.
MAIL POLICY (ntnsndr, 2026-11-13): only when there is a problem — and only when
the finding SET changes, fingerprinted in portal_state (slug+class only, never an
email), so a stuck finding does not mail three times a day while a fresh one
always speaks. A clean run clears the marker. A failed read returns skipped,
never a clean bill.
POST /admin/unmapped-check (X-Admin-Token, ?notify=0 to stay quiet). The
credit-pack reconcile now mails on the same rule. 25 checks; all 19 suites green.
The alert for a failed Open Collective delivery lived in portal_webhook_watch.py on
the operator's machine, polling this app's log through the Cloudron API. That put a
money-critical signal outside the system, and on a log buffer with only hours of
memory — during the 2026-11-13 investigation whole hours (06:00-13:00, 15:00) were
simply absent, so 'no failed delivery in the log' could not be distinguished from
'no delivery at all'.
The portal handles every delivery, so the portal now knows: a new webhook_failures
table records every non-2xx (including the 401 that means the URL and WEBHOOK_TOKEN
disagree, whose event type is read from a body cached before the token check), and
webhook_failure_report() in the sweep emails info@ ONCE per new batch via a marker
in the new portal_state table, so a wrong token does not mail every sweep while the
rows stay queryable. The delivery path is unchanged: bookkeeping failures are
logged, never raised, and the response stays 401/503.
19 checks in test_webhook_resilience.py; all 18 suites green.
Both routes are needed. A manual grant often carries no reference at all, so the
row's EMAIL is the only link to its member — and the slug and email local part
routinely differ ('henry-smith' vs 'h.smith@x.org'). The membership test was
slug-first and, when the slug matched neither the reference nor the email, dropped
the row outright: a hand-credited purchase therefore reads as a gap (ntnsndr's
2026-09-28 packs, where the ledger carries +20.00, +5.75 and a -10.75 correction
with no references). _row_belongs_to() accepts either route, in both the
accounted-for test and the nearby-rows context. 38 checks; all 18 suites green.
Widening the window to OC's whole order history found three $5 packs from
2026-09-28 reported as uncredited — ntnsndr's launch-day purchases, which WERE
credited by hand that morning: +20.00, +5.75, then a -10.75 correction ('true
contributions were 3 x $5.00'), netting exactly $15, with no per-order reference
on any row.
credit_pack_accounted() is right not to infer 'accounted for' from a nearby
amount — that would hide genuine gaps — but a reviewer could not tell the two
cases apart from the candidate alone. Each candidate now carries its nearby
ledger/held rows (7-day window, capped at 5) and the report-mode log line prints
them, so a manual grant explains itself. Nothing about the decision changes.
36 checks; all 18 suites green.
Credits have one path in: the OC order.processed webhook. On 2026-11-13 two paid
packs got nothing through it — a $50 whose delivery 500'd on httpx's 5s default,
and a $10 with no delivery in the retained log at all — and nothing on a
schedule noticed. This adds a pass that reads OC's own order list and compares it
with our books over a day-based window (not "whatever the log still holds": the
$10 gap was 11h old and the log had lost the whole morning).
Report-first: CREDIT_RECONCILE defaults to 'report' (log findings, change
nothing) and grants only when set to 'on'. Grants go through add_credits() with
credit_pack_reference(), the same per-order key the webhook writes, so a late
delivery and this pass are idempotent against each other in both directions. A
buyer with no active membership is reported, never credited — those keep the
hold-and-email path.
credit_pack_accounted() encodes what our own rows actually look like: a v2 id, a
hyphenated legacy id, an empty-suffix legacy reference, and held_credits rows. A
bare oc:<slug>:<order.id> match read 21 credited purchases as missing during
development, so the fallback demands a second signal (member + tip-free amount +
date) before calling an order credited.
POST /admin/credit-pack-reconcile (X-Admin-Token) runs it on demand, dry unless
?apply=1 and the policy is on. 33 new checks; all 18 suites green.
On 2026-11-13 an `order.processed` delivery for a $50 credit pack returned 500.
fetch_members() -> fetch_all_member_nodes() used a bare httpx.AsyncClient(), so OC
was given httpx's DEFAULT 5-second timeout; the read timed out, httpx.ReadTimeout
escaped the handler as an unhandled ASGI exception, and the purchase was never
credited. The member paid and got nothing, and the log held a stack trace rather
than anything naming the member or the order.
Three changes, each aimed at a different failure:
* fetch_all_member_nodes: an EXPLICIT 30s timeout, and one retry per page on a
transport error (`_oc_graphql_post`). Still RAISES when it cannot get the full
list — the sweep's fail-safes depend on that.
* opencollective_webhook: the work moves to _process_opencollective_webhook and a
thin wrapper answers 503 (retryable) with an ERROR line naming the exception if
anything unexpected escapes. A 401 stays a 401; that status is the signal that
the webhook URL and WEBHOOK_TOKEN disagree, and swallowing it would hide a
misconfigured webhook.
* tests/test_webhook_resilience.py proves it: 401 preserved, 503 (not 500) on a
simulated ReadTimeout, credits granted exactly once per OC order, an identical
re-delivery answering already_credited with no money moved, and the retry/raise
behaviour of the transport helper.
A new member could end up with a Loomio account and a Matrix account and see
NOTHING: no forum group, no Space. Three defects:
* sync_matrix_space() gated each member on the SPACE's membership, so an
account already joined/invited to the Space was skipped for EVERY room. A
member whose invite pass was interrupted (or who predates a room added to
MATRIX_INVITE_ROOMS later) never received that room — and a restricted child
room is invisible until you are invited to it. The gate is now per room, from
a single up-front membership snapshot; a room whose membership cannot be read
is left alone rather than invited into blindly.
* a 429 slept for retry_after_ms (minutes) INSIDE the pass, so a scheduled run
could outlive its own tick. A rate limit now ENDS the pass; the next run
resumes (every invite is idempotent).
* a member sitting in the Loomio group with no loomio_invited_at stamp reads as
'never invited' — the dedupe guard was only as good as the rows it had. The
sync now heals the record from the live roster.
And the activation window itself: provisioning skips an unactivated account by
design, and activation happens later with nobody watching, so the 3x/day sweep
was the pickup. New /admin/service-sync + a 5-minute scheduler task fires the
same idempotent syncs while any member still awaits an invite (one SELECT when
none do), and deliberately is NOT /admin/sweep — it never touches Open
Collective, allowance logic or member mail.
tests/test_service_invites.py proves the per-room gate, resumable rate limits,
the unreadable-room skip, the record heal and the endpoint's idle/run paths.
The member dashboard's usage chart reads per-day spend + requests for the
signed-in member, bounded by the existing days param (0 = all time, the
unchanged default). Same source and day convention as the co-op daily series
(to_char(startTime)); interior zero days are explicit so chart spacing is
true; requests excludes failed attempts; spend sums every row. Additive and
backward compatible: every existing field untouched. Tested in
tests/test_broker_usage_daily.py (route table, SQL shape, fallback twin,
fail-soft degradation).
The 2026-11-12 hole was NOT a missed sample: the 03:15 run recorded
oc_balance_usd=1357.39, and the evening repair backfill's full-row
upsert of the then-current day — whose dict carries no balance key —
nulled it on conflict. Nothing ever revisited a day's balance, so the
hole was permanent.
Two fixes:
- upsert() can no longer erase a balance it did not measure: the
conflict clause COALESCEs, so a NULL (a failed OC read) keeps the
previously recorded value.
- the nightly snapshot heals any recent day (within a week, after the
series' first recorded balance) whose balance is missing: the value is
reconstructed from Open Collective's transaction ledger
(balance(cutoff) = current - net flow after the 03:15 sample moment,
ledger window clamped to the balance read), matching the daily sample
semantics exactly. It is written balance-column-only, so membership
and usage series are untouched. Days before 2026-11-08 stay NULL by
design. Fail-soft: an OC failure prints and retries next night.
The reconstruction was verified to the cent against OC's own ledger:
full-ledger signed sum == stats.balance (1470.44 on 2026-11-13), and
1470.44 - 113.05 (net flow 2026-11-12 03:15 -> 2026-11-13 03:15:58)
== 1357.39, exactly the sample the 03:15 run had recorded.
snapshot() fetched the full spend window (since = yesterday) but kept only
spend.get(TODAY): the day's first hours, written at 03:15 and never revisited.
Every metrics_daily row after the last backfill under-counted its day by the
hours after 03:15 (Nov 7 recorded $2.19/691 requests against a true
$29.06/3384). The completed day's aggregates were computed and thrown away.
Now each run completes the day that just ENDED — full-day spend, requests,
tokens and impact, written to the usage columns only (upsert_usage: membership
columns are point-in-time measurements and are never rewritten) — and then
records today's live partial as before.
backfill() gains --since YYYY-MM-DD: a targeted usage-only repair that cannot
touch the published membership curve.
Regression test: tests/test_metrics_completion.py (day completion, membership
preservation, since-mode repair boundaries, full-backfill parity).
Single derivation point (tested once, here): 'invited' while the Cloudron
invite is unaccepted, else the guests-ledger status (the record of record),
membership state only as a no-ledger-row fallback. The panel and dashboard
become pure consumers of `state` — their local invited/active derivation
goes away in their own repos next. Live-behaviour note: today's four live
guests render byte-identically; the change only shows for ended guests
(ledger 'expired' instead of the bare inactive flag), which is what the
panel already displayed via its own guest_status preference.
The one-per-30-days rule is a READ of the guests ledger, provisioning
takes seconds of external calls, and the ledger row that closes the
window is written LAST — so two PARALLEL direct calls (curl, bypassing
the dashboard JS) could both pass guest_request_guard before either row
exists and slip two guests past the rule. The member invite path now
runs guard+provision under _GUEST_INVITE_LOCK (single-process uvicorn),
making check-and-send atomic: the second caller re-runs the guard after
the first caller's ledger row exists and is refused 429.
tests/test_guest_concurrency.py proves both halves on the fixture DB:
unguarded, two parallel invites both provision (the test has teeth);
with the lock, exactly one provisions, one is refused 429, one ledger
row, one account, one email; sequential semantics unchanged.
A guest invitation provisions immediately, but the Cloudron account is not
ACCEPTED until the guest clicks the setup link (inviteAccepted=false, no
username) — until then they cannot log in, so no surface may read them as
active. guest_invite_pending() derives it from the same Cloudron signal the
overview's activated flag uses; unseen-record (users fetch failed) is never
pending, so the state falls back instead of asserting what the data does
not show.
/admin/overview: guest rows report state='invited' with an 'awaiting
setup' reason (invite date named when the ledger has it); /broker/guests
trail entries carry state='invited' the same way, status stays the ledger
record of record. Display/state semantics only — access rules, the $2 cap
and the expiry box are unchanged.
Tests: new test_guest_invited_state.py (derivation, overview flip,
fetch-failure fallback, member rows untouched, trail state);
test_guest_requests.py gains the trail invited/accepted assertions and a
constructible-but-loud fake client so the stubbed users fetch runs.
Operator decision 2026-11-08: the remaining protections (active member,
1 guest per inviter per rolling 30 days, one guest per recipient ever,
no existing portal row, MANUAL_MEMBERS refusal, the $2 cap and the
GUEST_DAYS box) are sufficient without the approval step.
* POST /broker/guests provisions through provision_guest_member at once —
no pending state; no guest_requests row is ever written.
* GUEST_AUTO_APPROVE flag removed (the live env never carried it).
* guest_request_guard keeps the member-path rules and drops the
trial-era pending-request checks; the 30-day window is counted from
the guests ledger.
* GET /admin/guest-requests + approve/reject removed; GET /admin/guest-ledger
serves the panel's duplicate-send guard.
* The member trail keeps legacy rejected requests as history only.
Built on the sibling $2-cap commit 6be8e2d.
Operator edits:
- The co-op is described only as a member-governed AI cooperative; the
renewable-energy claim is removed (not all providers are renewable).
- Membership pitch now states the pricing exactly: pay on a sliding
scale, $15/month allowance plus optional credit packs, plus a say in
how it's run.
The admin panel's /admin/provision-guest now takes ONE field (the guest's
email): no name is collected and none is derived from the email — the
guest chooses their own display name and username during Cloudron account
setup, and the invitation email greets neutrally when no name is known.
The guests ledger records the co-op's ADMIN account (ADMIN_INVITER_EMAIL,
default info@inference.coop) as the inviter. Admin invites provision
immediately (the operator IS the approver) and are not bound by the
member's one-per-30-days cap; recipient-side guards (one guest per
recipient ever, no existing portal row) still run. The member paths
(broker invite, request approval) keep every protection and likewise pass
no derived name.
POST /broker/guests (member broker identity — guests cannot reach it) creates
a guest REQUEST; GET /broker/guests returns the member's invitation allowance
(rolling one-per-30-days, counting pending + approved requests) and their own
guest trail. guest_request_guard enforces every existing protection plus the
member-side ones: recipient never an existing member, a prior guest, a pending
request, or a MANUAL_MEMBERS address; the inviter's window counts requests too.
Trial gate (GUEST_AUTO_APPROVE, default off): requests land PENDING in the new
guest_requests table; the operator approves or rejects them via
GET/POST /admin/guest-requests (X-Admin-Token). Approval fires the full
provision_guest_member pipeline (Cloudron user, $5 credit, GUEST_DAYS box,
invitation email naming the inviter, newsletter, ledger row); a guard refusal
at approve-time is recorded on the request and surfaced. The member sees
pending/not-approved status in their own trail — the recipient is contacted
only on approval. With the flag on, a member's request provisions
immediately, no code change needed.
38-check fixture suite (tests/test_guest_requests.py) proves the gate, every
refusal path, the approve/reject actions and auto-approve with the network
replaced by a raising guard.
Two defects found by booting the renamed app against a fixture of the
pre-rename database (the live portal's exact state):
1. members.gift_expires_at: the RENAME above uses the stale
snapshot, so the plain
below fired right after a successful rename and crashed boot with
'duplicate column name' — the portal would have failed its deploy
health check on the live database. Make the two branches exclusive.
2. The ledger's own expiry column: ALTER TABLE guests RENAME moved the
table but kept its legacy gift_expires_at column, which the provision
INSERT (writing guest_expires_at) would fail on at first use. Rename
the column with the table.
Also moves the credit-ledger dedupe references ('gift:<email>' /
'gift-return:<email>') to their guest prefixes at boot — returns first,
since 'gift-return:…' also matches 'gift:%' — so a re-grant after the
rename cannot double-credit.
Verified: migration harness 10/10 on a legacy-state fixture (renames,
value conversion, ref migration, idempotent second boot, readback
helpers); full portal suite 9/9 files green.
Pure rename, no behaviour change, matching the renamed panel and dashboard
builds. Includes one-time idempotent migrations for existing databases
(gifts->guests table, gift_expires_at->guest_expires_at column,
account_type 'gift'->'guest' values, gift_lapse_notices->guest_lapse_notices)
and four transitional alias routes (admin/provision-gift, admin/gift-convert,
broker/gift-status, broker/cancel-gift) so pre-rename dashboard/panel builds
keep working until they are redeployed.