Commit Graph
54 Commits
Author SHA1 Message Date
inference-admin af4135ac36 manifest: pin to 20261117-85e8e62
ci/woodpecker/push/woodpecker Pipeline was successful
2026-11-17 04:57:18 +00:00
inference-admin 85e8e62e5b portal: membership-status reports the computed allowance, not members.balance (often NULL)
ci/woodpecker/push/woodpecker Pipeline was canceled
2026-11-17 04:56:58 +00:00
inference-admin 265fae993f manifest: pin to 20261117-27b0ea5
ci/woodpecker/push/woodpecker Pipeline was successful
2026-11-17 04:56:13 +00:00
inference-admin 27b0ea564a portal: answer for a lapsed membership instead of leaving the dashboard blank
ci/woodpecker/push/woodpecker Pipeline was canceled
New read-only /broker/membership-status, authenticated by the broker secret with a
verifier that admits a member row in ANY state (usage and keys still require active).
Returns active/state, the latest allowance_log decision with its evidence, the OC
minimum contribution (MIN_CONTRIBUTION, copy-only — OC stays the authority), credits,
allowance and the contribute URL.

MIN_CONTRIBUTION is deliberately NOT wired into deactivation: a number we half-know
should not cost anyone access.
2026-11-17 04:55:34 +00:00
inference-admin 6041c0cdb2 manifest: pin to 20261117-e9d0fd2
ci/woodpecker/push/woodpecker Pipeline was successful
2026-11-17 04:52:08 +00:00
inference-admin e9d0fd259a portal: settle held credits in the sweep, and report the ones nobody can be credited for
ci/woodpecker/push/woodpecker Pipeline was canceled
A credit pack bought while the buyer's membership was inactive is held — correctly,
we must never mint credits for a non-member. The gap was what happened next: the hold
sat until a human noticed (mmoya's $10 waited from 01:45 until they wrote in).

Every sweep now grants any hold that resolves to an ACTIVE member (by email first, then
slug), reusing grant_held_credits() so held:<id> keeps it idempotent; holds with no
active member older than HELD_ALERT_HOURS are emailed to info@. Tests cover grant/hold/
no-double-apply.
2026-11-17 04:51:40 +00:00
inference-admin 34547075cf manifest: pin to 20261113-bed7f03
ci/woodpecker/push/woodpecker Pipeline was successful
2026-11-13 22:46:23 +00:00
inference-admin bed7f03063 rate limits: one bucket per ROUTE, not one per proxy address
ci/woodpecker/push/woodpecker Pipeline was canceled
Every request arrives through Cloudrons
2026-11-13 22:45:48 +00:00
inference-admin 6f6261eb1f manifest: pin to 20261113-0a6939f
ci/woodpecker/push/woodpecker Pipeline was successful
2026-11-13 22:33:35 +00:00
inference-admin 0a6939fe07 retire the Open WebUI integration
ci/woodpecker/push/woodpecker Pipeline was canceled
The app is stopped (`com.openwebui.cloudronapp`, installed|stopped), so every call
to it failed with "Temporary failure in name resolution" — a stopped container has
no DNS entry. Four of the last four provisions logged exactly that; no new member
ever reached the group.

Its "members group" was never a Cloudron group: the id pointed at a group inside
Open WebUI's own database, which is why it appears nowhere in Cloudron's UI.

Removed: owui_add_to_members_group, both call sites (provision_member and guest
convert), the OWUI_* constants, and the signed-JWT identity path in inject_key.
That last one is the part that matters beyond tidiness: it accepted a
shared-secret-signed token as proof of identity for an integration nobody runs.
inject_key keeps its one live path (LibreChat: X-User-Email + X-Portal-Secret,
still verified).

First attempt at this cut too much — a greedy splice off the guest-convert call
took listmonk_sync and the held-credit grant with it, and test_guest_lifecycle.py
caught it before anything shipped. Redone with exact anchors.
2026-11-13 22:33:10 +00:00
inference-admin 5c494cb261 manifest: pin to 20261113-b1022eb
ci/woodpecker/push/woodpecker Pipeline was successful
2026-11-13 22:28:15 +00:00
inference-admin b1022eb02a oc-incognito: take the slug from the member's own row
ci/woodpecker/push/woodpecker Pipeline was canceled
Setting the source was not enough, and verification is what caught it: mmoya still
produced no allowance decision at all.

The reconciler gets a member's OC slug from the member INDEX — which is keyed by
email, and DROPS anyone whose profile hides its email. So `oc_slug` came back
empty, the member fell into "not in the live Open Collective member list — cannot
verify payment", and was counted as skipped. An oc-incognito member would have
been "not manual" and still not reconciled: a nicer label over the same hole.

For source oc-incognito the slug now comes from the member's own row, which is
the same key fetch_member_orders() groups orders by — the whole reason matching
can work for these people. The test pins the fallback, not just the source.
2026-11-13 22:27:55 +00:00
inference-admin 66a770ae10 manifest: pin to 20261113-64d1ea5
ci/woodpecker/push/woodpecker Pipeline was successful
2026-11-13 22:25:07 +00:00
inference-admin 64d1ea5093 members: an explicit SOURCE, so 'oc-incognito' stops being 'manual'
ci/woodpecker/push/woodpecker Pipeline was canceled
mmoya@mmoya.org sat in MANUAL_MEMBERS because OC hides the address on their
profile (incognito-5ad60182, email null since 2023), so email matching can never
find them. But "manual" means hand-provisioned, and the sweep SKIPS manual
members entirely: a paying subscriber had no lapse logic, no dues tracking and no
order attribution. Two different facts were sharing one word.

members.source now carries which one it is:
  oc            — OC-backed, matched by email (the default; NULL means this)
  oc-incognito  — OC-backed, profile hides the email, matched by the stored OC
                  SLUG — which the reconciliation already uses, since
                  fetch_member_orders() groups orders by the payer's slug and
                  members.slug holds exactly that for these people
  manual        — hand-provisioned; no contribution expected, nothing reconciles,
                  nobody lapses

So an oc-incognito member needs no exemption from reconciliation — they need to
stop being skipped, and the existing slug path does the rest.

Every decision that keyed on the raw MANUAL_MEMBERS list now keys on the source:
the reconciliation skip, the sweep's deactivation guard, and the shortfall-credit
exemption. An explicit source beats the list, so a member can be moved without
editing the env — and the env list can shrink toward deprecation. An
unrecognised or unreadable source falls back to 'oc', the direction that keeps
reconciling someone rather than quietly exempting them.

anchor_day_for_member also resolves by slug now: an incognito member is absent
from the member index (it drops anyone without an email), and falling back to
"today" would silently move their renewal day.

tests/test_oc_incognito.py pins all of it, including that the column is optional
(NULL = not stated) so this ships without a data migration.
2026-11-13 22:24:46 +00:00
inference-admin a98db6e36e manifest: pin to 20261113-2cde368
ci/woodpecker/push/woodpecker Pipeline was successful
2026-11-13 22:14:59 +00:00
inference-admin 2cde368c1e resolve payer: authenticate the lookup — without the token OC hides the email
ci/woodpecker/push/woodpecker Pipeline was canceled
Second live-only bug in the same helper. Sent unauthenticated, OC answers the
public view of an account: name comes back, email does not. A hand query with the
Personal-Token header returned conor.canaday@gmail.com for the very slug the
helper reported as email=None — so the fix would have resolved nobody, silently,
while looking like it worked.

The test now pins the header (the stub records what it was called with), because
the failure mode here is an absence, and an absence is exactly what a stub
forgets to check.
2026-11-13 22:14:36 +00:00
inference-admin 0458e8e5a5 manifest: pin to 20261113-9c10e62
ci/woodpecker/push/woodpecker Pipeline was successful
2026-11-13 22:13:50 +00:00
inference-admin 9c10e627fc resolve payer: read the response's JSON (the stub hid this)
ci/woodpecker/push/woodpecker Pipeline was canceled
_oc_graphql_post returns the RESPONSE — it is a retrying wrapper around
client.post — so fetch_account_by_slug has to call .json() on it. My unit test
stubbed it as a dict, so the test passed and the first live call in the container
died with "Response object has no attribute get".

The test now stubs the real shape (a response wrapper with .json()). Folding it
in here rather than hiding it: a stub that does not match the real return shape
certifies nothing.
2026-11-13 22:13:22 +00:00
inference-admin cbf5a6870b manifest: pin to 20261113-5a73806
ci/woodpecker/push/woodpecker Pipeline was successful
2026-11-13 22:12:24 +00:00
inference-admin 5a7380686c webhook: a first-time contributor's order must not vanish in silence
ci/woodpecker/push/woodpecker Pipeline was canceled
On 2026-11-13 21:59:43 Open Collective delivered order.processed for a live
$5/month membership (Conor W, conor.canaday@gmail.com, order
or_eU8gRkrPCvFsaYHrgiCfz). The portal answered 200 and did nothing.

The membership branch resolves the payer by SLUG against the collective's
members list:

    members = await fetch_members()
    member = next((m for m in members if m["slug"] == slug), None)

and a first-time contributor's account is not in that list yet — OC reports the
payer as guest-…/incognito-…. So `member` was None, the branch fell through, and
because the delivery had SUCCEEDED there was no error to record either: no
account, no welcome email, no credit, no log line. The sweep's unmapped-backers
check found it hours later, which is the wrong place to find it.

Now an unknown payer is resolved from the order's account slug
(`Individual.email`, which OC exposes for an ordinary guest checkout) and handed
to the same provisioning path a known member gets. When even that yields no
email — an incognito profile, hidden on purpose — the webhook logs at ERROR that
nobody was provisioned or credited, instead of shrugging.

The defect worth naming is the SHAPE: an `if` with no `else` on a money path,
where doing nothing leaves no trace. tests/test_resolve_payer.py pins the
resolution cases (guest with email, incognito, unknown account, read failure, no
token) and the fact that the branch can no longer fall through quietly.
2026-11-13 22:12:02 +00:00
inference-admin 27929987b2 manifest: pin to 20261113-241ac80
ci/woodpecker/push/woodpecker Pipeline was successful
2026-11-13 21:53:53 +00:00
inference-admin 241ac800c6 panel state: a reminder is history, not a state — renewals clear grace, grace expires
ci/woodpecker/push/woodpecker Pipeline was canceled
mark@ofosos.org showed as in grace while his contribution was live. The sweep's
own trail says it plainly: reminded 2026-11-04 for the cycle that looked unpaid,
grace elapsed 2026-11-11, and `renewed` logged at 19:02 on 2026-11-13 (last
charge 2026-11-11, next reset 2026-11-26). The panel said grace anyway, because
member_membership_state() asked only "is there a row in payment_reminders?".

That table is a HISTORY — kept so the sweep does not re-remind inside a cycle. It
is not a state, and treating it as one broke two things: a reminder never
expired (grace is PAYMENT_GRACE_DAYS long), and a renewal never cleared it.

Now the LATEST reconciliation decision decides:
  renewed / would_renew            -> active (they are paying)
  reminder inside the window       -> in_grace
  reminder past the window         -> awaiting_contribution, saying grace elapsed
  would_deactivate / deactivated   -> awaiting_contribution
Ordered by decided_at, not id: the same member had id 2296 dated 2026-11-11 next
to id 2286 dated 2026-11-13, so "latest by id" reads the wrong decision.

Reads still fail safe: an unreadable DB reports active, never a payment problem.

tests/test_membership_state.py pins all of it, including the ordering trap.
2026-11-13 21:53:27 +00:00
inference-admin b1f5a30272 manifest: pin to 20261113-2a77b0c
ci/woodpecker/push/woodpecker Pipeline was successful
2026-11-13 21:44:46 +00:00
inference-admin 2a77b0c988 spend: ONE all-time figure, computed once, for every surface that shows it
ci/woodpecker/push/woodpecker Pipeline was canceled
The panel's card said $941.37 and the member dashboard's said $900.51 for the
same claim. The panel used its own per-team query; the dashboard summed the
chart series IN THE BROWSER, which disagrees with the server whenever a row is
null, capped or stale.

coop_spend_totals() is now the definition: one GROUP BY over SpendLogs joined to
the team table, so
  spend_all_time      = every row (members, operators, agent teams, sandboxes) —
                        the same universe litellm_spend_by_day() sums, so the
                        cards and the money chart agree by construction
  own_all_time_spend  = the part that is the co-op's own (non-member teams, plus
                        rows with no team)
and member_all_time_spend = the rest. Exposed through /admin/overview (the
panel's boot payload), the money payload, and /broker/metrics (the dashboard's).
_coop_spend_totals_safe() returns {} rather than a zero: a failed computation
must never read as "the co-op has spent nothing".

tests/test_coop_spend_totals.py pins the arithmetic (total == member + own ==
every row) and the no-fake-zero rule.
2026-11-13 21:44:13 +00:00
inference-admin e15a01c338 manifest: pin to member-portal:20261113-9af0604
ci/woodpecker/push/woodpecker Pipeline was successful
2026-11-13 21:02:57 +00:00
inference-admin 9af0604f47 portal: /broker/money-series, so members chart the co-op's money beside its other charts
ci/woodpecker/push/woodpecker Pipeline was successful
The member dashboard could not show the money chart because the broker exposed no
money series — the same payload /admin/money-series serves the panel is aggregate
and carries no member identities (cumulative revenue, cumulative token spend, the
OC balance), so it goes through the broker for the same reason /broker/metrics
does. One chart definition, drawn identically for members and stewards.
2026-11-13 21:01:38 +00:00
inference-admin 0d5ffede05 README: the spend check compares day by day over the shared window
ci/woodpecker/push/woodpecker Pipeline was successful
2026-11-13 20:48:02 +00:00
inference-admin b30e2c8080 manifest: pin to 20261113-c92d897
ci/woodpecker/push/woodpecker Pipeline was successful
2026-11-13 20:47:23 +00:00
inference-admin c92d8979eb spend check: compare like for like (same window, same kind of number)
ci/woodpecker/push/woodpecker Pipeline was canceled
The second live run FAILED again, correctly and informatively: portal $935.52
all-time vs LiteLLM $827.89 — and the $107.63 'drift' was the WINDOW. LiteLLM's
per-day view reaches back a fixed number of days (31 on 2026-11-13); our table
holds 69. Over the days both sides cover the two agree to $0.07, with no day
differing by more than $0.022 (the API rounds to cents).

So the check now compares DAY BY DAY over the shared window, which localises drift
instead of hiding it in a sum, with a per-day tolerance ($0.10) plus a window-total
tolerance ($5). It reads /global/spend/teams rather than /global/spend/keys: the
per-key figure follows each key's own budget cycle, so summing it is not an all-time
number at all (and /global/spend/report, the all-time one, is Enterprise-gated).

Degenerate reads are judged BEFORE the window overlap, because in the failure this
check exists for the portal's aggregate comes back EMPTY — there would be no shared
days to compare, and the overlap logic would report 'cannot compare' instead of the
stale-credential signature. All failure reporting goes through one helper that mails
info@ once per episode and advances the marker only after a notice is sent.

23 checks; all 20 suites green.
2026-11-13 20:46:50 +00:00
inference-admin c13880cfab manifest: pin to 20261113-c0ef046
ci/woodpecker/push/woodpecker Pipeline was successful
2026-11-13 20:44:16 +00:00
inference-admin c0ef046b0d spend check: read the field LiteLLM actually sends (total_spend), and fail loudly either way
ci/woodpecker/push/woodpecker Pipeline was canceled
The first live run FAILED with api $0.00 over 306 keys against the portal's $935.35.
The disagreement was real, the verdict was right, and the cause was mine: LiteLLM's
/global/spend/keys returns each key's all-time spend as `total_spend`, not `spend`.
A checker that trusts its own parse then reports "the portal is wrong" — the exact
mirror of the failure this check exists to catch, which is why the guard now names
both possibilities when the independent side reads zero against real portal spend.

Parser accepts either field name; a regression test exercises the real function
against a stubbed httpx (the module attribute is stubbed for the other checks, so
the real one is captured at import). 24 checks; all 20 suites green.
2026-11-13 20:43:41 +00:00
inference-admin 1c1fced5ba manifest: pin to 20261113-e2eaa7a
ci/woodpecker/push/woodpecker Pipeline was successful
2026-11-13 20:42:11 +00:00
inference-admin dac8f2c9ea README: the spend-consistency step
ci/woodpecker/push/woodpecker Pipeline was canceled
2026-11-13 20:41:58 +00:00
inference-admin e2eaa7ae40 spend consistency: the check moves into the portal's sweep
ci/woodpecker/push/woodpecker Pipeline was canceled
The failure this exists for (2026-11-05): the portal's own all-time-spend aggregate
failed SOFT when its copy of the gateway's LiteLLM DB credential went stale — the
query returned nothing, every surface showed $0, and nothing looked wrong, because
the portal WAS the thing that was wrong. The guard ran as a daily script on the
bot's host.

It now runs in the portal, and keeps two genuinely independent reads: the portal's
SpendLogs aggregate over its own DB credential, and LiteLLM's own HTTP API
(/global/spend/keys) over the master key — a different transport AND a different
credential, which is the point, since the DB copy is what goes stale.

It adds the signature guard the host script could only infer from outside: an
EMPTY//usr/bin/bash portal aggregate while LiteLLM reports real spend is a FAILURE, named as
the stale-credential case; an API side that returns no keys is a failure too, not
agreement. A read that cannot be made is skipped — never a clean bill. Silent when
they agree; info@ once per failing episode, marker cleared on recovery.
POST /admin/spend-check runs it on demand. 23 checks; all 20 suites green.
2026-11-13 20:41:27 +00:00
inference-admin bb0a02dcd9 manifest: pin to 20261113-bf1e5f0
ci/woodpecker/push/woodpecker Pipeline was successful
2026-11-13 20:36:39 +00:00
inference-admin 06e7ae0ea5 README: the unmapped-backer step and the notice policy
ci/woodpecker/push/woodpecker Pipeline was successful
2026-11-13 20:35:42 +00:00
inference-admin bf1e5f013b unmapped backers: stop the phantom finding (the members query had no createdAt)
ci/woodpecker/push/woodpecker Pipeline was canceled
The first production run reported a null-account node as an uncredited payer. The
cause was in my port, not the data: the members query selected id-less,
date-less nodes, so every null-account node fell into the date "" bucket, matched
no same-day order, and was reported as unattributable — while the host-side script
(correctly) said clean.

Two fixes: select id + createdAt in fetch_all_member_nodes, and never emit a
null-account finding when the node carries no date — there is nothing to attribute
it against, and guessing is what produced the phantom. 27 checks in the suite.
2026-11-13 20:35:28 +00:00
inference-admin 13b6131652 notices: tell a human about what is NEW, and only when a notice actually went out
ci/woodpecker/push/woodpecker Pipeline was successful
Three fixes to the 'email only when there is a problem' rule, found by testing it:

1. The marker recorded the CURRENT set, so a set that SHRANK looked changed — one
   purchase granted dropped out, the remaining hold looked new, and it was mailed
   about twice. What a human has been TOLD is now a growing SET of keys
   (order:grant / order:hold, slug:class for backers): only keys not in it are
   mailed, and the union is stored.
2. A notify=False run advanced the marker, so a verification look CONSUMED the
   only alert a finding gets. The marker now advances only when the notice was
   actually sent (a failed send retries next sweep), and a clean pass clears it so
   a recurrence is news again.
3. The credit reconcile had the same shape of bug.

Tests updated to pin the new semantics: a smaller set re-mails nobody, a silent
look does not swallow the alert, a clean run clears the marker and a recurrence
speaks. 26 + 40 checks in those suites; all 19 suites green.
2026-11-13 20:34:24 +00:00
inference-admin f258a22aa3 manifest: pin to 20261113-eaaabf3
ci/woodpecker/push/woodpecker Pipeline was successful
2026-11-13 20:31:22 +00:00
inference-admin eaaabf3763 unmapped backers: the paid-but-unprovisioned net moves into the portal
ci/woodpecker/push/woodpecker Pipeline was canceled
A contributor who pays and never gets an account is invisible to the sweep —
fetch_members() drops every member node with no email, and an OC Organization
never exposes one — so they wait until they ask (the mmoya case). The net for them
was a daily script on the bot's host driving `cloudron exec` from outside. The
portal holds both sides of that comparison, so it now owns it.

reconcile_unmapped_backers() runs in the sweep and reports three classes, because
each needs a different human action: an org account (link it), a VISIBLE email
(the sweep should have provisioned this — check the token/sweep), and a hidden
email (fetch it out-of-band). A node with no account object is attributed through
same-day current orders, and stays silent when every same-day payer is mapped.

MAIL POLICY (ntnsndr, 2026-11-13): only when there is a problem — and only when
the finding SET changes, fingerprinted in portal_state (slug+class only, never an
email), so a stuck finding does not mail three times a day while a fresh one
always speaks. A clean run clears the marker. A failed read returns skipped,
never a clean bill.

POST /admin/unmapped-check (X-Admin-Token, ?notify=0 to stay quiet). The
credit-pack reconcile now mails on the same rule. 25 checks; all 19 suites green.
2026-11-13 20:30:46 +00:00
inference-admin d117da3dc6 README: the sweep's webhook-failure report, and why the scraper is gone
ci/woodpecker/push/woodpecker Pipeline was successful
2026-11-13 20:25:51 +00:00
inference-admin 9f7934121f manifest: pin to 20261113-2e40387
ci/woodpecker/push/woodpecker Pipeline was canceled
2026-11-13 20:25:18 +00:00
inference-admin 2e40387245 webhook failures: the portal records and reports its own, not a scraper on my host
ci/woodpecker/push/woodpecker Pipeline was successful
The alert for a failed Open Collective delivery lived in portal_webhook_watch.py on
the operator's machine, polling this app's log through the Cloudron API. That put a
money-critical signal outside the system, and on a log buffer with only hours of
memory — during the 2026-11-13 investigation whole hours (06:00-13:00, 15:00) were
simply absent, so 'no failed delivery in the log' could not be distinguished from
'no delivery at all'.

The portal handles every delivery, so the portal now knows: a new webhook_failures
table records every non-2xx (including the 401 that means the URL and WEBHOOK_TOKEN
disagree, whose event type is read from a body cached before the token check), and
webhook_failure_report() in the sweep emails info@ ONCE per new batch via a marker
in the new portal_state table, so a wrong token does not mail every sweep while the
rows stay queryable. The delivery path is unchanged: bookkeeping failures are
logged, never raised, and the response stays 401/503.

19 checks in test_webhook_resilience.py; all 18 suites green.
2026-11-13 20:23:33 +00:00
inference-admin 6c82d7d689 manifest: pin to 20261113-c174fa2
ci/woodpecker/push/woodpecker Pipeline was successful
2026-11-13 20:21:22 +00:00
inference-admin c174fa2ffe reconcile: link a row to its member by slug OR email
ci/woodpecker/push/woodpecker Pipeline was successful
Both routes are needed. A manual grant often carries no reference at all, so the
row's EMAIL is the only link to its member — and the slug and email local part
routinely differ ('henry-smith' vs 'h.smith@x.org'). The membership test was
slug-first and, when the slug matched neither the reference nor the email, dropped
the row outright: a hand-credited purchase therefore reads as a gap (ntnsndr's
2026-09-28 packs, where the ledger carries +20.00, +5.75 and a -10.75 correction
with no references). _row_belongs_to() accepts either route, in both the
accounted-for test and the nearby-rows context. 38 checks; all 18 suites green.
2026-11-13 20:20:40 +00:00
inference-admin 711371f9d7 manifest: pin to 20261113-f2b6153
ci/woodpecker/push/woodpecker Pipeline was successful
2026-11-13 20:19:17 +00:00
inference-admin f2b6153aa6 reconcile: show a candidate's nearby ledger rows, so a manual grant isn't read as a gap
ci/woodpecker/push/woodpecker Pipeline was canceled
Widening the window to OC's whole order history found three $5 packs from
2026-09-28 reported as uncredited — ntnsndr's launch-day purchases, which WERE
credited by hand that morning: +20.00, +5.75, then a -10.75 correction ('true
contributions were 3 x $5.00'), netting exactly $15, with no per-order reference
on any row.

credit_pack_accounted() is right not to infer 'accounted for' from a nearby
amount — that would hide genuine gaps — but a reviewer could not tell the two
cases apart from the candidate alone. Each candidate now carries its nearby
ledger/held rows (7-day window, capped at 5) and the report-mode log line prints
them, so a manual grant explains itself. Nothing about the decision changes.

36 checks; all 18 suites green.
2026-11-13 20:18:44 +00:00
inference-admin d69c518b47 manifest: pin to 20261113-5868ca8
ci/woodpecker/push/woodpecker Pipeline was successful
2026-11-13 20:16:34 +00:00
inference-admin 5868ca82e8 reconcile paid credit packs against the ledger, report-first (issue #7)
ci/woodpecker/push/woodpecker Pipeline was canceled
Credits have one path in: the OC order.processed webhook. On 2026-11-13 two paid
packs got nothing through it — a $50 whose delivery 500'd on httpx's 5s default,
and a $10 with no delivery in the retained log at all — and nothing on a
schedule noticed. This adds a pass that reads OC's own order list and compares it
with our books over a day-based window (not "whatever the log still holds": the
$10 gap was 11h old and the log had lost the whole morning).

Report-first: CREDIT_RECONCILE defaults to 'report' (log findings, change
nothing) and grants only when set to 'on'. Grants go through add_credits() with
credit_pack_reference(), the same per-order key the webhook writes, so a late
delivery and this pass are idempotent against each other in both directions. A
buyer with no active membership is reported, never credited — those keep the
hold-and-email path.

credit_pack_accounted() encodes what our own rows actually look like: a v2 id, a
hyphenated legacy id, an empty-suffix legacy reference, and held_credits rows. A
bare oc:<slug>:<order.id> match read 21 credited purchases as missing during
development, so the fallback demands a second signal (member + tip-free amount +
date) before calling an order credited.

POST /admin/credit-pack-reconcile (X-Admin-Token) runs it on demand, dry unless
?apply=1 and the policy is on. 33 new checks; all 18 suites green.
2026-11-13 20:16:24 +00:00
inference-admin 32e744d9ee README: document the webhook error net, the OC timeouts, and the missing reconciliation
ci/woodpecker/push/woodpecker Pipeline was successful
2026-11-13 20:05:23 +00:00
inference-admin 9c53e5e727 manifest: pin to 20261113-f45e4ef
ci/woodpecker/push/woodpecker Pipeline was canceled
2026-11-13 20:04:53 +00:00
inference-admin f45e4ef305 webhook: stop a transient OC read failure from costing a member credits
On 2026-11-13 an `order.processed` delivery for a $50 credit pack returned 500.
fetch_members() -> fetch_all_member_nodes() used a bare httpx.AsyncClient(), so OC
was given httpx's DEFAULT 5-second timeout; the read timed out, httpx.ReadTimeout
escaped the handler as an unhandled ASGI exception, and the purchase was never
credited. The member paid and got nothing, and the log held a stack trace rather
than anything naming the member or the order.

Three changes, each aimed at a different failure:

* fetch_all_member_nodes: an EXPLICIT 30s timeout, and one retry per page on a
  transport error (`_oc_graphql_post`). Still RAISES when it cannot get the full
  list — the sweep's fail-safes depend on that.
* opencollective_webhook: the work moves to _process_opencollective_webhook and a
  thin wrapper answers 503 (retryable) with an ERROR line naming the exception if
  anything unexpected escapes. A 401 stays a 401; that status is the signal that
  the webhook URL and WEBHOOK_TOKEN disagree, and swallowing it would hide a
  misconfigured webhook.
* tests/test_webhook_resilience.py proves it: 401 preserved, 503 (not 500) on a
  simulated ReadTimeout, credits granted exactly once per OC order, an identical
  re-delivery answering already_credited with no money moved, and the retry/raise
  behaviour of the transport helper.
2026-11-13 20:04:09 +00:00
inference-admin 5705d02792 manifest: pin to 20261113-0e825e6
ci/woodpecker/push/woodpecker Pipeline was successful
2026-11-13 17:38:35 +00:00
inference-admin 0e825e6d61 service invites: per-room Matrix gate, resumable rate limits, Loomio record heal, 5-min sync tick
A new member could end up with a Loomio account and a Matrix account and see
NOTHING: no forum group, no Space. Three defects:

* sync_matrix_space() gated each member on the SPACE's membership, so an
  account already joined/invited to the Space was skipped for EVERY room. A
  member whose invite pass was interrupted (or who predates a room added to
  MATRIX_INVITE_ROOMS later) never received that room — and a restricted child
  room is invisible until you are invited to it. The gate is now per room, from
  a single up-front membership snapshot; a room whose membership cannot be read
  is left alone rather than invited into blindly.
* a 429 slept for retry_after_ms (minutes) INSIDE the pass, so a scheduled run
  could outlive its own tick. A rate limit now ENDS the pass; the next run
  resumes (every invite is idempotent).
* a member sitting in the Loomio group with no loomio_invited_at stamp reads as
  'never invited' — the dedupe guard was only as good as the rows it had. The
  sync now heals the record from the live roster.

And the activation window itself: provisioning skips an unactivated account by
design, and activation happens later with nobody watching, so the 3x/day sweep
was the pickup. New /admin/service-sync + a 5-minute scheduler task fires the
same idempotent syncs while any member still awaits an invite (one SELECT when
none do), and deliberately is NOT /admin/sweep — it never touches Open
Collective, allowance logic or member mail.

tests/test_service_invites.py proves the per-room gate, resumable rate limits,
the unreadable-room skip, the record heal and the endpoint's idle/run paths.
2026-11-13 17:37:57 +00:00