Commit Graph
195 Commits
Author SHA1 Message Date
inference-admin c92d8979eb spend check: compare like for like (same window, same kind of number)
ci/woodpecker/push/woodpecker Pipeline was canceled
The second live run FAILED again, correctly and informatively: portal $935.52
all-time vs LiteLLM $827.89 — and the $107.63 'drift' was the WINDOW. LiteLLM's
per-day view reaches back a fixed number of days (31 on 2026-11-13); our table
holds 69. Over the days both sides cover the two agree to $0.07, with no day
differing by more than $0.022 (the API rounds to cents).

So the check now compares DAY BY DAY over the shared window, which localises drift
instead of hiding it in a sum, with a per-day tolerance ($0.10) plus a window-total
tolerance ($5). It reads /global/spend/teams rather than /global/spend/keys: the
per-key figure follows each key's own budget cycle, so summing it is not an all-time
number at all (and /global/spend/report, the all-time one, is Enterprise-gated).

Degenerate reads are judged BEFORE the window overlap, because in the failure this
check exists for the portal's aggregate comes back EMPTY — there would be no shared
days to compare, and the overlap logic would report 'cannot compare' instead of the
stale-credential signature. All failure reporting goes through one helper that mails
info@ once per episode and advances the marker only after a notice is sent.

23 checks; all 20 suites green.
2026-11-13 20:46:50 +00:00
inference-admin c0ef046b0d spend check: read the field LiteLLM actually sends (total_spend), and fail loudly either way
ci/woodpecker/push/woodpecker Pipeline was canceled
The first live run FAILED with api $0.00 over 306 keys against the portal's $935.35.
The disagreement was real, the verdict was right, and the cause was mine: LiteLLM's
/global/spend/keys returns each key's all-time spend as `total_spend`, not `spend`.
A checker that trusts its own parse then reports "the portal is wrong" — the exact
mirror of the failure this check exists to catch, which is why the guard now names
both possibilities when the independent side reads zero against real portal spend.

Parser accepts either field name; a regression test exercises the real function
against a stubbed httpx (the module attribute is stubbed for the other checks, so
the real one is captured at import). 24 checks; all 20 suites green.
2026-11-13 20:43:41 +00:00
inference-admin e2eaa7ae40 spend consistency: the check moves into the portal's sweep
ci/woodpecker/push/woodpecker Pipeline was canceled
The failure this exists for (2026-11-05): the portal's own all-time-spend aggregate
failed SOFT when its copy of the gateway's LiteLLM DB credential went stale — the
query returned nothing, every surface showed $0, and nothing looked wrong, because
the portal WAS the thing that was wrong. The guard ran as a daily script on the
bot's host.

It now runs in the portal, and keeps two genuinely independent reads: the portal's
SpendLogs aggregate over its own DB credential, and LiteLLM's own HTTP API
(/global/spend/keys) over the master key — a different transport AND a different
credential, which is the point, since the DB copy is what goes stale.

It adds the signature guard the host script could only infer from outside: an
EMPTY//usr/bin/bash portal aggregate while LiteLLM reports real spend is a FAILURE, named as
the stale-credential case; an API side that returns no keys is a failure too, not
agreement. A read that cannot be made is skipped — never a clean bill. Silent when
they agree; info@ once per failing episode, marker cleared on recovery.
POST /admin/spend-check runs it on demand. 23 checks; all 20 suites green.
2026-11-13 20:41:27 +00:00
inference-admin bf1e5f013b unmapped backers: stop the phantom finding (the members query had no createdAt)
ci/woodpecker/push/woodpecker Pipeline was canceled
The first production run reported a null-account node as an uncredited payer. The
cause was in my port, not the data: the members query selected id-less,
date-less nodes, so every null-account node fell into the date "" bucket, matched
no same-day order, and was reported as unattributable — while the host-side script
(correctly) said clean.

Two fixes: select id + createdAt in fetch_all_member_nodes, and never emit a
null-account finding when the node carries no date — there is nothing to attribute
it against, and guessing is what produced the phantom. 27 checks in the suite.
2026-11-13 20:35:28 +00:00
inference-admin 13b6131652 notices: tell a human about what is NEW, and only when a notice actually went out
ci/woodpecker/push/woodpecker Pipeline was successful
Three fixes to the 'email only when there is a problem' rule, found by testing it:

1. The marker recorded the CURRENT set, so a set that SHRANK looked changed — one
   purchase granted dropped out, the remaining hold looked new, and it was mailed
   about twice. What a human has been TOLD is now a growing SET of keys
   (order:grant / order:hold, slug:class for backers): only keys not in it are
   mailed, and the union is stored.
2. A notify=False run advanced the marker, so a verification look CONSUMED the
   only alert a finding gets. The marker now advances only when the notice was
   actually sent (a failed send retries next sweep), and a clean pass clears it so
   a recurrence is news again.
3. The credit reconcile had the same shape of bug.

Tests updated to pin the new semantics: a smaller set re-mails nobody, a silent
look does not swallow the alert, a clean run clears the marker and a recurrence
speaks. 26 + 40 checks in those suites; all 19 suites green.
2026-11-13 20:34:24 +00:00
inference-admin eaaabf3763 unmapped backers: the paid-but-unprovisioned net moves into the portal
ci/woodpecker/push/woodpecker Pipeline was canceled
A contributor who pays and never gets an account is invisible to the sweep —
fetch_members() drops every member node with no email, and an OC Organization
never exposes one — so they wait until they ask (the mmoya case). The net for them
was a daily script on the bot's host driving `cloudron exec` from outside. The
portal holds both sides of that comparison, so it now owns it.

reconcile_unmapped_backers() runs in the sweep and reports three classes, because
each needs a different human action: an org account (link it), a VISIBLE email
(the sweep should have provisioned this — check the token/sweep), and a hidden
email (fetch it out-of-band). A node with no account object is attributed through
same-day current orders, and stays silent when every same-day payer is mapped.

MAIL POLICY (ntnsndr, 2026-11-13): only when there is a problem — and only when
the finding SET changes, fingerprinted in portal_state (slug+class only, never an
email), so a stuck finding does not mail three times a day while a fresh one
always speaks. A clean run clears the marker. A failed read returns skipped,
never a clean bill.

POST /admin/unmapped-check (X-Admin-Token, ?notify=0 to stay quiet). The
credit-pack reconcile now mails on the same rule. 25 checks; all 19 suites green.
2026-11-13 20:30:46 +00:00
inference-admin 2e40387245 webhook failures: the portal records and reports its own, not a scraper on my host
ci/woodpecker/push/woodpecker Pipeline was successful
The alert for a failed Open Collective delivery lived in portal_webhook_watch.py on
the operator's machine, polling this app's log through the Cloudron API. That put a
money-critical signal outside the system, and on a log buffer with only hours of
memory — during the 2026-11-13 investigation whole hours (06:00-13:00, 15:00) were
simply absent, so 'no failed delivery in the log' could not be distinguished from
'no delivery at all'.

The portal handles every delivery, so the portal now knows: a new webhook_failures
table records every non-2xx (including the 401 that means the URL and WEBHOOK_TOKEN
disagree, whose event type is read from a body cached before the token check), and
webhook_failure_report() in the sweep emails info@ ONCE per new batch via a marker
in the new portal_state table, so a wrong token does not mail every sweep while the
rows stay queryable. The delivery path is unchanged: bookkeeping failures are
logged, never raised, and the response stays 401/503.

19 checks in test_webhook_resilience.py; all 18 suites green.
2026-11-13 20:23:33 +00:00
inference-admin c174fa2ffe reconcile: link a row to its member by slug OR email
ci/woodpecker/push/woodpecker Pipeline was successful
Both routes are needed. A manual grant often carries no reference at all, so the
row's EMAIL is the only link to its member — and the slug and email local part
routinely differ ('henry-smith' vs 'h.smith@x.org'). The membership test was
slug-first and, when the slug matched neither the reference nor the email, dropped
the row outright: a hand-credited purchase therefore reads as a gap (ntnsndr's
2026-09-28 packs, where the ledger carries +20.00, +5.75 and a -10.75 correction
with no references). _row_belongs_to() accepts either route, in both the
accounted-for test and the nearby-rows context. 38 checks; all 18 suites green.
2026-11-13 20:20:40 +00:00
inference-admin f2b6153aa6 reconcile: show a candidate's nearby ledger rows, so a manual grant isn't read as a gap
ci/woodpecker/push/woodpecker Pipeline was canceled
Widening the window to OC's whole order history found three $5 packs from
2026-09-28 reported as uncredited — ntnsndr's launch-day purchases, which WERE
credited by hand that morning: +20.00, +5.75, then a -10.75 correction ('true
contributions were 3 x $5.00'), netting exactly $15, with no per-order reference
on any row.

credit_pack_accounted() is right not to infer 'accounted for' from a nearby
amount — that would hide genuine gaps — but a reviewer could not tell the two
cases apart from the candidate alone. Each candidate now carries its nearby
ledger/held rows (7-day window, capped at 5) and the report-mode log line prints
them, so a manual grant explains itself. Nothing about the decision changes.

36 checks; all 18 suites green.
2026-11-13 20:18:44 +00:00
inference-admin 5868ca82e8 reconcile paid credit packs against the ledger, report-first (issue #7)
ci/woodpecker/push/woodpecker Pipeline was canceled
Credits have one path in: the OC order.processed webhook. On 2026-11-13 two paid
packs got nothing through it — a $50 whose delivery 500'd on httpx's 5s default,
and a $10 with no delivery in the retained log at all — and nothing on a
schedule noticed. This adds a pass that reads OC's own order list and compares it
with our books over a day-based window (not "whatever the log still holds": the
$10 gap was 11h old and the log had lost the whole morning).

Report-first: CREDIT_RECONCILE defaults to 'report' (log findings, change
nothing) and grants only when set to 'on'. Grants go through add_credits() with
credit_pack_reference(), the same per-order key the webhook writes, so a late
delivery and this pass are idempotent against each other in both directions. A
buyer with no active membership is reported, never credited — those keep the
hold-and-email path.

credit_pack_accounted() encodes what our own rows actually look like: a v2 id, a
hyphenated legacy id, an empty-suffix legacy reference, and held_credits rows. A
bare oc:<slug>:<order.id> match read 21 credited purchases as missing during
development, so the fallback demands a second signal (member + tip-free amount +
date) before calling an order credited.

POST /admin/credit-pack-reconcile (X-Admin-Token) runs it on demand, dry unless
?apply=1 and the policy is on. 33 new checks; all 18 suites green.
2026-11-13 20:16:24 +00:00
inference-admin f45e4ef305 webhook: stop a transient OC read failure from costing a member credits
On 2026-11-13 an `order.processed` delivery for a $50 credit pack returned 500.
fetch_members() -> fetch_all_member_nodes() used a bare httpx.AsyncClient(), so OC
was given httpx's DEFAULT 5-second timeout; the read timed out, httpx.ReadTimeout
escaped the handler as an unhandled ASGI exception, and the purchase was never
credited. The member paid and got nothing, and the log held a stack trace rather
than anything naming the member or the order.

Three changes, each aimed at a different failure:

* fetch_all_member_nodes: an EXPLICIT 30s timeout, and one retry per page on a
  transport error (`_oc_graphql_post`). Still RAISES when it cannot get the full
  list — the sweep's fail-safes depend on that.
* opencollective_webhook: the work moves to _process_opencollective_webhook and a
  thin wrapper answers 503 (retryable) with an ERROR line naming the exception if
  anything unexpected escapes. A 401 stays a 401; that status is the signal that
  the webhook URL and WEBHOOK_TOKEN disagree, and swallowing it would hide a
  misconfigured webhook.
* tests/test_webhook_resilience.py proves it: 401 preserved, 503 (not 500) on a
  simulated ReadTimeout, credits granted exactly once per OC order, an identical
  re-delivery answering already_credited with no money moved, and the retry/raise
  behaviour of the transport helper.
2026-11-13 20:04:09 +00:00
inference-admin 0e825e6d61 service invites: per-room Matrix gate, resumable rate limits, Loomio record heal, 5-min sync tick
A new member could end up with a Loomio account and a Matrix account and see
NOTHING: no forum group, no Space. Three defects:

* sync_matrix_space() gated each member on the SPACE's membership, so an
  account already joined/invited to the Space was skipped for EVERY room. A
  member whose invite pass was interrupted (or who predates a room added to
  MATRIX_INVITE_ROOMS later) never received that room — and a restricted child
  room is invisible until you are invited to it. The gate is now per room, from
  a single up-front membership snapshot; a room whose membership cannot be read
  is left alone rather than invited into blindly.
* a 429 slept for retry_after_ms (minutes) INSIDE the pass, so a scheduled run
  could outlive its own tick. A rate limit now ENDS the pass; the next run
  resumes (every invite is idempotent).
* a member sitting in the Loomio group with no loomio_invited_at stamp reads as
  'never invited' — the dedupe guard was only as good as the rows it had. The
  sync now heals the record from the live roster.

And the activation window itself: provisioning skips an unactivated account by
design, and activation happens later with nobody watching, so the 3x/day sweep
was the pickup. New /admin/service-sync + a 5-minute scheduler task fires the
same idempotent syncs while any member still awaits an invite (one SELECT when
none do), and deliberately is NOT /admin/sweep — it never touches Open
Collective, allowance logic or member mail.

tests/test_service_invites.py proves the per-room gate, resumable rate limits,
the unreadable-room skip, the record heal and the endpoint's idle/run paths.
2026-11-13 17:37:57 +00:00
inference-bot 3c64b150db broker: per-member daily usage series in /broker/usage (daily: day/spend/requests)
ci/woodpecker/push/woodpecker Pipeline was successful
The member dashboard's usage chart reads per-day spend + requests for the
signed-in member, bounded by the existing days param (0 = all time, the
unchanged default). Same source and day convention as the co-op daily series
(to_char(startTime)); interior zero days are explicit so chart spacing is
true; requests excludes failed attempts; spend sums every row. Additive and
backward compatible: every existing field untouched. Tested in
tests/test_broker_usage_daily.py (route table, SQL shape, fallback twin,
fail-soft degradation).
2026-11-13 16:11:16 +00:00
inference-bot 58214be0db Guest state: the overview payload carries the definitive display state
Single derivation point (tested once, here): 'invited' while the Cloudron
invite is unaccepted, else the guests-ledger status (the record of record),
membership state only as a no-ledger-row fallback. The panel and dashboard
become pure consumers of `state` — their local invited/active derivation
goes away in their own repos next. Live-behaviour note: today's four live
guests render byte-identically; the change only shows for ended guests
(ledger 'expired' instead of the bare inactive flag), which is what the
panel already displayed via its own guest_status preference.
2026-11-11 05:16:13 +00:00
inference-bot 2954fe609f Guest invite: serialize member invites — parallel direct calls can't slip two guests
The one-per-30-days rule is a READ of the guests ledger, provisioning
takes seconds of external calls, and the ledger row that closes the
window is written LAST — so two PARALLEL direct calls (curl, bypassing
the dashboard JS) could both pass guest_request_guard before either row
exists and slip two guests past the rule. The member invite path now
runs guard+provision under _GUEST_INVITE_LOCK (single-process uvicorn),
making check-and-send atomic: the second caller re-runs the guard after
the first caller's ledger row exists and is refused 429.

tests/test_guest_concurrency.py proves both halves on the fixture DB:
unguarded, two parallel invites both provision (the test has teeth);
with the lock, exactly one provisions, one is refused 429, one ledger
row, one account, one email; sequential semantics unchanged.
2026-11-08 20:53:54 +00:00
Inferencebot 88ca8bfe18 Guest state: provisioned-but-not-set-up reads 'invited', never 'active'
A guest invitation provisions immediately, but the Cloudron account is not
ACCEPTED until the guest clicks the setup link (inviteAccepted=false, no
username) — until then they cannot log in, so no surface may read them as
active. guest_invite_pending() derives it from the same Cloudron signal the
overview's activated flag uses; unseen-record (users fetch failed) is never
pending, so the state falls back instead of asserting what the data does
not show.

/admin/overview: guest rows report state='invited' with an 'awaiting
setup' reason (invite date named when the ledger has it); /broker/guests
trail entries carry state='invited' the same way, status stays the ledger
record of record. Display/state semantics only — access rules, the $2 cap
and the expiry box are unchanged.

Tests: new test_guest_invited_state.py (derivation, overview flip,
fetch-failure fallback, member rows untouched, trail state);
test_guest_requests.py gains the trail invited/accepted assertions and a
constructible-but-loud fake client so the stubbed users fetch runs.
2026-11-08 20:40:23 +00:00
inference-bot 285aa66525 Guest invites provision immediately: remove the member approval gate
Operator decision 2026-11-08: the remaining protections (active member,
1 guest per inviter per rolling 30 days, one guest per recipient ever,
no existing portal row, MANUAL_MEMBERS refusal, the $2 cap and the
GUEST_DAYS box) are sufficient without the approval step.

* POST /broker/guests provisions through provision_guest_member at once —
  no pending state; no guest_requests row is ever written.
* GUEST_AUTO_APPROVE flag removed (the live env never carried it).
* guest_request_guard keeps the member-path rules and drops the
  trial-era pending-request checks; the 30-day window is counted from
  the guests ledger.
* GET /admin/guest-requests + approve/reject removed; GET /admin/guest-ledger
  serves the panel's duplicate-send guard.
* The member trail keeps legacy rejected requests as history only.

Built on the sibling $2-cap commit 6be8e2d.
2026-11-08 20:20:30 +00:00
inference-bot 6be8e2d632 Guest cap $5 -> $2: GUEST_BUDGET default, emails, docs, tests 2026-11-08 20:12:47 +00:00
Inferencebot 6f5e479900 Guest invitation email: drop the renewable-energy claim, state sliding-scale pricing
Operator edits:
- The co-op is described only as a member-governed AI cooperative; the
  renewable-energy claim is removed (not all providers are renewable).
- Membership pitch now states the pricing exactly: pay on a sliding
  scale, $15/month allowance plus optional credit packs, plus a say in
  how it's run.
2026-11-08 05:13:52 +00:00
inference-bot ed50031663 Admin guest invites: admin inviter, no member cap, no name guessing
The admin panel's /admin/provision-guest now takes ONE field (the guest's
email): no name is collected and none is derived from the email — the
guest chooses their own display name and username during Cloudron account
setup, and the invitation email greets neutrally when no name is known.
The guests ledger records the co-op's ADMIN account (ADMIN_INVITER_EMAIL,
default info@inference.coop) as the inviter. Admin invites provision
immediately (the operator IS the approver) and are not bound by the
member's one-per-30-days cap; recipient-side guards (one guest per
recipient ever, no existing portal row) still run. The member paths
(broker invite, request approval) keep every protection and likewise pass
no derived name.
2026-11-08 02:55:10 +00:00
inference-bot 0293d66c1e Member self-serve guest invitations with a trial approval gate
POST /broker/guests (member broker identity — guests cannot reach it) creates
a guest REQUEST; GET /broker/guests returns the member's invitation allowance
(rolling one-per-30-days, counting pending + approved requests) and their own
guest trail. guest_request_guard enforces every existing protection plus the
member-side ones: recipient never an existing member, a prior guest, a pending
request, or a MANUAL_MEMBERS address; the inviter's window counts requests too.

Trial gate (GUEST_AUTO_APPROVE, default off): requests land PENDING in the new
guest_requests table; the operator approves or rejects them via
GET/POST /admin/guest-requests (X-Admin-Token). Approval fires the full
provision_guest_member pipeline (Cloudron user, $5 credit, GUEST_DAYS box,
invitation email naming the inviter, newsletter, ledger row); a guard refusal
at approve-time is recorded on the request and surfaced. The member sees
pending/not-approved status in their own trail — the recipient is contacted
only on approval. With the flag on, a member's request provisions
immediately, no code change needed.

38-check fixture suite (tests/test_guest_requests.py) proves the gate, every
refusal path, the approve/reject actions and auto-approve with the network
replaced by a raising guard.
2026-11-08 02:42:56 +00:00
inference-bot 93184b6011 Fix the gift→guest rename migration for legacy databases
Two defects found by booting the renamed app against a fixture of the
pre-rename database (the live portal's exact state):

1. members.gift_expires_at: the RENAME above uses the stale
   snapshot, so the plain
   below fired right after a successful rename and crashed boot with
   'duplicate column name' — the portal would have failed its deploy
   health check on the live database. Make the two branches exclusive.

2. The ledger's own expiry column: ALTER TABLE guests RENAME moved the
   table but kept its legacy gift_expires_at column, which the provision
   INSERT (writing guest_expires_at) would fail on at first use. Rename
   the column with the table.

Also moves the credit-ledger dedupe references ('gift:<email>' /
'gift-return:<email>') to their guest prefixes at boot — returns first,
since 'gift-return:…' also matches 'gift:%' — so a re-grant after the
rename cannot double-credit.

Verified: migration harness 10/10 on a legacy-state fixture (renames,
value conversion, ref migration, idempotent second boot, readback
helpers); full portal suite 9/9 files green.
2026-11-08 02:42:53 +00:00
inference-bot 27f9cc06b2 Rename gift to guest throughout: routes, ledger table, account_type values, env names, emails, tests
Pure rename, no behaviour change, matching the renamed panel and dashboard
builds. Includes one-time idempotent migrations for existing databases
(gifts->guests table, gift_expires_at->guest_expires_at column,
account_type 'gift'->'guest' values, gift_lapse_notices->guest_lapse_notices)
and four transitional alias routes (admin/provision-gift, admin/gift-convert,
broker/gift-status, broker/cancel-gift) so pre-rename dashboard/panel builds
keep working until they are redeployed.
2026-11-08 02:41:05 +00:00
inference-bot 40fb75a34b gift cancel: self-service cancellation with credit return and full audit trail
POST /broker/cancel-gift (dashboard broker identity, gift accounts only,
idempotent) + GET /broker/gift-status for the ended-state view. Semantics
match the gift lapse: row deactivated, account kept for a clean resume,
gateway budget zeroed, newsletter off — plus the return a lapse does not
do: unspent credit back to the co-op as a NEGATIVE credit_ledger row whose
reason names the cancellation and the original gift reference, deduped on
its own reference. Gifts ledger records inviter → recipient → 'cancelled
by recipient'. The user leaves the guests group for GUESTS_ENDED_GROUP_ID
(dashboard-only visibility for the ended state); the reaching-limits
notice is pre-marked so a self-cancelled guest is never told their
credits were spent. Tests: tests/test_gift_cancel.py (19 fixture checks,
network-guarded).
2026-11-08 02:28:45 +00:00
inference-bot 0b0bdaab6f gift emails: invitation at gift time + once-only reaching-limits notice
- gift_welcome_email becomes gift_invitation_email: the ONE email a gift
  sends (the invitation and the guest welcome merged, not two mails at
  once). Subject 'You've been invited to the Inference Cooperative'; names
  the inviter (member slug), the $5 credits, the setup link, the
  Become-a-member button on BECOME_MEMBER_URL. Uses the shared
  PINE_LOGO_SVG instead of an inline copy.
- reaching-limits email (gift_limits_email, 'Your guest access has ended —
  become a member to continue') wired into the sweep's gift deactivation:
  sent once per guest, gated by a gift_lapse_notices sent-state row (same
  pattern as mark_shortfall_notice_sent). Says the access ended, the
  account and data are kept, and membership ($15/mo via Open Collective)
  restores a full monthly allowance + community spaces.
- catch-up pass in reconcile_gift_lifecycles: guests deactivated before the
  email shipped (or a failed send) get exactly one notice on a later sweep;
  converted guests are excluded by account_type.
- sweep summary now reports gift_lapse_notices.
- test_gift_lifecycle: invitation-subject assertion; send_email stub
  returns True (the bool is the gate's contract); new once-only + catch-up
  + healthy-untouched checks. 43/43.
2026-11-08 02:22:59 +00:00
inference-bot c6eb2b44e8 overview: guests are their own list, excluded from member counts
/admin/overview now returns gift (guest) accounts in a separate guests[]
list instead of inside members[], and excludes them from totals.members,
totals.active and the member spend totals. Guests get their own totals
breakout (guests, guests_active, guest_spend, guest_all_time_spend) so
their spend stays visible without ever reading as member activity — the
same reasoning as the operator (admins) and agent categories.

Builds on the gift feature's account_type marker: a guest is any row with
account_type='gift'. The panel reads guests[] and ignores gift rows in
its member count either way.
2026-11-08 02:09:28 +00:00
inference-bot 6d9074eefe gift accounts: $5 guest passes with time box, inviter ledger, auto member conversion
Operator feature: a member gives a friend access with $5 of credits, no OC
membership. Cloudron user lands in the guests group (chat + dashboard only),
LiteLLM team capped at GIFT_BUDGET with NO monthly reset, portal row marked
account_type='gift' with gift_expires_at set, gift welcome email, newsletter
subscription, and a gifts ledger recording inviter -> recipient (one gift per
inviter per 30 days, one per recipient ever).

Sweep safety: reconcile_memberships never deactivates a gift for lacking an
OC contribution; reconcile_gift_lifecycles lapses gifts by their own rules
(credits spent out, or the GIFT_DAYS time box) with member-deactivation
semantics. Guests are excluded from Loomio/Matrix invite lists, the reinvite
path, budget notices, the allowance reconciliation and the metrics member
counts.

Conversion: a gift whose OC identity appears under its own email converts
automatically (webhook + sweep + /admin/provision all funnel through
provision_member): group guests->members, cap lifted to allowance + carried
credits, anchor day kept, congrats email, full member onboarding.
/admin/gift-convert links a membership paid under a different OC email
(members.oc_email, matched by the sweep and webhook).

Fixture test: tests/test_gift_lifecycle.py (37 checks, network replaced by
a raising guard).
2026-11-08 02:05:37 +00:00
inference-bot 2d2d191e11 money-series: drop the caveat field from the payload
The operator removed all caveats from the finance chart; the balance-history
note is gone from the API payload too so it cannot reappear anywhere.
balance_history_start, the series, totals, sources and errors are unchanged.
2026-11-08 01:55:23 +00:00
inference-bot dcce9fcb35 money series: /admin/money-series (revenue/spend/OC balance) + daily OC balance snapshot
- /admin/money-series (X-Admin-Token, 30/min): three series for the admin
  panel's Money chart, hourly in-memory cache with stale-on-error fallback.
  revenue = OC contribution transactions (CREDIT/CONTRIBUTION/non-refund,
  monthly + one-time, org-backed included), cumulative from 2026-09-06;
  spend = one GROUP BY day pass over the full LiteLLM SpendLogs (no team
  filter, same definition as metrics_snapshot.spend_by_day); balance = OC
  current balance + daily snapshots.
- OC exposes only the CURRENT balance — no historical series exists — so
  metrics_snapshot.py now writes oc_balance_usd into metrics_daily daily
  (from 2026-11-08 onward; balance history cannot be backfilled). The
  endpoint seeds an existing row's balance with the live value without
  touching the snapshot's other columns.
- Aggregate totals only: no member emails or names selected or returned.
2026-11-08 01:50:17 +00:00
inference-bot ed66d73327 activation hook: fire Loomio+Matrix syncs at first broker login
New members were skipped by the syncs until their Cloudron account was
activated (deliberately - never invite a half-created account), but nothing
fired the syncs at the moment they DID activate: the 3x/day sweep was the
only pickup, so a newly-activated member could wait hours for forum/space
access (limdingwen waited ~5h on 2026-11-06).

Hook the broker identity path (_verify_broker_secret): after an active
member resolves, if loomio_invited_at is still null, confirm their Cloudron
account is activated (inviteAccepted, fail-CLOSED - an unreadable user list
never fires) and run the same idempotent syncs the sweep runs, as a
background task off the response. Established members pay one indexed
SELECT; concurrent first-logins collapse into one run via an inflight flag;
each service is independently best-effort so a Loomio outage cannot block
the Matrix invite or the response.

Tests: tests/test_first_login_sync.py (18 checks, fixture db + stubbed
network). Also fix test_allowance_shortfall, whose expected exclusion
reference still asserted the pre-a88ca86 placeholder.

Sweep, provisioning and deactivation paths are untouched.
2026-11-06 19:51:12 +00:00
inference-bot a88ca860d4 fix: restore the real OC tx reference in SHORTFALL_REMEDIATED (repo must match the deployed image)
The deployed portal image (20261105-ca3b8db) was built from a working tree
carrying the real transaction id; commit 22c0167 over-redacted it to a
placeholder that matches nothing, so the next build would lose the exclusion
that prevents double-crediting the manually-remediated 2026-11-04 shortfall
charge. Sync HEAD to the deployed bytes (md5-identical to the running
container). The tx id alone is the H2-approved form: it identifies the OC
transaction, carries no member email.
2026-11-05 18:55:15 +00:00
inference-bot 00f60f1374 ops: bound allowance_log growth; correct stale README claims (audit L4+L5)
allowance_log: the sweep and every report-mode run used to append a row
per member per run (~500/day measured). The insert is now an upsert per
(member, cycle, decision) — a re-derived decision updates the row's
evidence and timestamp; a decision CHANGE still gets its own row. No
index or migration; the latest-decision readers are unaffected; history
is not rewritten.

README: scheduler runs 3x/day (0 3,11,19 UTC), not nightly-3am; the
sweep's auth is the ADMIN_TOKEN X-Admin-Token header, not WEBHOOK_TOKEN;
the retired free/member/supporter tier budgets replaced by the actual
single MEMBER_BUDGET model; the unread OPENCOLLECTIVE_WEBHOOK_SECRET env
row removed.
2026-11-05 16:26:16 +00:00
inference-bot ce1bc7bee6 security: log every key-feed fetch (audit M2)
/admin/librechat/keys deliberately serves every member's chat key on one
sync token (the LibreChat sandbox seeds from it; a per-member fetch would
require changing the sandbox's identity model — decision recorded in
ops-docs maintenance/librechat-key-feed.md). Every fetch now logs a
count-only WARNING (no key material; the email redactor strips addresses)
with caller + user-agent, and the daily digest watches the rate against
the sandbox's ~10-min poll cadence.
2026-11-05 16:21:51 +00:00
inference-bot d63f702ec3 privacy: redact member emails from portal logs (audit M1)
The _RedactWebhookPath filter now also strips email-shaped strings from
every log record — messages, %-format args (recursing into tuple/list/
dict args), and any exception text already attached to the record — the
same retention discipline the webhook token got in ops/security#2.
_send_email recipients, listmonk failures, Loomio invite lists and
provisioning/credit lines carry member addresses into container logs,
which persist on the host and ride the backups. MXIDs, counts, slugs and
amounts stay readable; the daily digest's own sanitisation is unchanged.
Filter keeps its never-break-logging guarantee (all redaction inside
try/except; non-string args untouched).

Live pre-state measured: 99 email-bearing lines in the current app log.
2026-11-05 16:17:21 +00:00
inference-bot b9561a49d2 correctness: paginate Cloudron user fetches; atomic held-credit grant (audit M5+M6)
M6: a single-page ?per_page=100 fetch silently truncates past 100 users —
the box is at 99. New shared walkers (async + sync) page through
/api/v1/users like the Matrix paths, capped at 50 pages. All six single-
page call sites now walk: duplicate-email check (provisioning 409s),
reinvite loop, Loomio activation filter, admin overview, member-email
resolution, local MXID resolution. Failure behavior per site is preserved
(page-1 failure fails loudly; later-page failure keeps what was read).

M5: provisioning-time held-credit grant moved into grant_held_credits():
per hold, the status flip, balance move and ledger row commit in ONE
transaction (the old code committed the flip before add_credits, so an
exception between them left hold=granted and the member uncredited).
Mirrors admin_held_grant, keyed on held:<hold_id> so re-runs are no-ops;
mixed-case emails normalized. Tests in tests/test_portal_hardening.py.
2026-11-05 16:06:52 +00:00
inference-bot 22c0167978 privacy: de-PII the shortfall exclusion and policy comment (audit H2)
SHORTFALL_REMEDIATED is keyed on the OC charge reference alone (the same
idempotence key the shortfall path uses); the member's identity and the
timeline of the 2026-11-04 manual allowance reset move to the operations
docs and stay in the ledger. The policy comment that named a member's
join/charge dates is generalized. No behavioral change: the exclusion
matches the same charge it always matched.
2026-11-05 15:59:07 +00:00
inference-bot e6ab177a05 credits: make the credit-pack webhook grant idempotent (audit H1)
- reference built per-order from the webhook's order.idV2 (the id the
  amount lookup resolves) instead of the absent data.id that produced
  the legacy empty-suffix 'oc:<slug>:' format; legacy ledger rows are
  neither re-keyed nor rejected (a new-format reference can never match
  one)
- credit_reference_exists() gates add_credits; a re-delivered order
  returns 'already_credited' with the standing balance
- hold path: one hold per purchase via a pre-check plus a UNIQUE index
  on held_credits.oc_reference (index creation logs CRITICAL and
  continues if the live table ever carried duplicates — audited: none)
- get_order_amount sends the webhook-supplied order id as a GraphQL
  variable instead of interpolating it into the query (audit L9)
- regression test: replay credits exactly once; legacy rows never
  block a first grant; unique-index semantics incl. NULLs
2026-11-05 15:54:29 +00:00
inference-bot 6337bcfc24 portal: don't double the oc: prefix in the held-credit grant reason 2026-11-05 03:48:01 +00:00
inference-bot e103e6779b portal: add admin endpoint to grant a held credit pack to a member
Held credit packs (purchases by non-members) could previously only be
refunded or auto-granted when the SAME email/slug later became a member.
A purchase made on another account's behalf therefore had no admin path
and forced a hand-edit of money state.

Add POST /admin/held-credits/{hold_id}/grant with body {email}: applies
the hold to a specified member account in one transaction (balance +
ledger + hold -> granted), idempotent on reference held:<hold_id>.
2026-11-05 03:45:14 +00:00
inference-bot db64dac235 allowances: credit the shortfall when OC's billing day truncates a paid cycle
Open Collective aligns recurring charges onto the collective's common billing
day, so a mid-month joiner is charged again before a full month is up: they pay
for a full month and receive less than one. Credit the missing ALLOWANCE (the
flat MEMBER_BUDGET pro-rated for the days lost, unrounded) as non-expiring
credits.

- derived from the collective's real per-order charge history, on every sweep;
  any consecutive pair on the same order < a full month apart is truncated
- idempotent on the OC charge reference (dedupe on the charge, not the member)
- skips and logs any cycle whose dates cannot be determined
- ledger reason names the cycle; the notice is staged and defaults OFF
- [redacted member reference]'s remediated charge is excluded by ref

Tests: tests/test_allowance_shortfall.py (pure derivation).
2026-11-04 20:01:10 +00:00
inference-bot f7bbec8a0a loomio: enable by default; idempotent add (never re-invite) + targeted remove
The guard direction from 2026-11-04 is corrected: the co-op DOES use Loomio and
wants every new member added. Removing members/notifications is not the goal —
not duplicating them is. The feature is switched back ON and the sync is made
idempotent instead.

Enablement
  * LOOMIO_SYNC_ENABLED now defaults to ENABLED (kill switch: set =0 to pause).
    Restoring LOOMIO_API_KEY and deploying is all that is needed again.

Non-duplicating ADD (the real fix), two layers:
  * (a) live pre-check against Loomio group membership (b3 index, ordered and
    paginated stably). An address already in the group is skipped SILENTLY —
    it is never placed in an invite payload, so no API call about it can fire
    a notification or email.
  * (b) members.loomio_invited_at: an address we have handed over before is
    never sent again, even if it is not (yet) visible in the live roster.
  * FAIL CLOSED if the live roster cannot be read.
  * remove_absent is never sent.

Removal reconciliation preserved (deactivation must still propagate):
  * A member in the group who is no longer active is removed via a TARGETED
    b3 DELETE by membership id — it names only the member being removed, so it
    cannot re-invite or notify anyone staying. Operator allowlist
    (LOOMIO_ALWAYS_KEEP) is never removed. 404 (already gone) counts as
    success, so a re-run neither errors nor churns.
  * The keep-set is computed from the UNFILTERED active list: an active member
    whose Cloudron account is not yet activated is not removed (only their
    invite is deferred).

Tests: tests/test_loomio_guard.py (26/26) — add/dedupe, silent skip, record
guard, fail-closed, remove-once, allowlist-never-removed, targeted removal
(no present member in any payload), double-run no-op for both phases, 404
tolerance, kill switch.
2026-11-04 19:25:45 +00:00
inference-bot 120a24678d loomio: never invite an existing member (off by default + positive exclusion)
Incident 2026-11-04: a live run of sync_loomio_memberships() emailed a Loomio
invite notification to essentially the WHOLE membership (notification 744:
~83 in-app recipients, ~79 emails 'sent' 18:53-18:56 UTC). The sync handed
Loomio the full active-member list every run with no record of who had been
invited, relying on Loomio's add-if-absent semantics — which produced a
whole-group notification.

Guards now make a repeat impossible:
  * LOOMIO_SYNC_ENABLED (off by default) — restoring LOOMIO_API_KEY alone no
    longer re-arms the integration.
  * members.loomio_invited_at — positive exclusion: an address is sent only if
    it is neither in Loomio's current member list NOR ever handed over, and
    each is recorded once. (The new column starts NULL for the ~89 members
    already in Loomio, so the current-member check is what closes that gap.)
  * FAIL CLOSED — no invite if Loomio's member list cannot be read.
  * remove_absent is dropped: no evict-then-re-add (re-invite) path.

Tests: tests/test_loomio_guard.py (10/10).
2026-11-04 19:09:33 +00:00
inference-bot 8c85c912e8 allowances: add a would_deactivate PREVIEW (deactivation stays gated off)
Report mode (and a send run while ALLOWANCE_DEACTIVATE is off) now reports a
member whose reminder is past the grace window as would_deactivate, with the
reminder date, grace length and deadline in the evidence. The real action is
unchanged and still needs BOTH ALLOWANCE_POLICY=on AND ALLOWANCE_DEACTIVATE=on.

The lapse decision is extracted into a pure function (allowance_lapse_decision)
so it is unit-testable against a synthetic reminder time:
tests/test_allowance_lapse.py (would_deactivate past grace; in_grace inside).
2026-11-04 18:46:54 +00:00
inference-bot f1945aaf53 admin overview: expose derived membership state (active / in_grace / awaiting_contribution / inactive)
The admin panel needs to show a member's membership state without inventing
policy of its own. Compute it in the portal (the source of truth) from what the
allowance reconciliation already knows — the payment_reminders send record and
the latest allowance_log decision — and carry it on each /admin/overview row as
state + state_reason. Read-only; no side effects.
2026-11-04 18:30:42 +00:00
inference-bot a8200fdf54 allowances: correct the dormant-yearly note (collective order list is complete; a member's account.orders spans collectives) 2026-11-04 17:31:11 +00:00
inference-bot a984a443e1 allowances: recognise YEARLY contributions so a yearly payer isn't read as unpaid
_payment_verdict treated MONTHLY as the only recurring frequency, so a live
ACTIVE YEARLY contribution — one charge, current for twelve cycles — fell
through to "no recurring (monthly) contribution" and would have been reminded
and (with deactivation on) deactivated for eleven of those months.

Make the verdict frequency-aware: only a MONTHLY or YEARLY order can cover an
allowance cycle; an ACTIVE YEARLY covers while its window (last charge + 1yr,
or nextChargeDate) includes now; PROCESSING stays paid-pending; a ONE-TIME
payment is credit, never an allowance period and never a mask. A failed
monthly charge still takes the remind path.

Dormant on today's data: inference-cooperative has 0 yearly orders (104 orders:
88 MONTHLY, 16 ONETIME), verified per member ACCOUNT because a member's own
yearly order does not appear in the collective-level order list. This changes
no current member's decision — the report is identical before and after.

Adds tests/test_payment_verdict.py (yearly branch + monthly/processing/onetime/
error/cancelled cases), runnable in the app container.
2026-11-04 17:29:47 +00:00
inference-bot ced8c4f082 email: set an explicit Message-ID on outbound mail so a send is traceable 2026-11-04 05:19:43 +00:00
inference-bot 5237e6352b allowances: add mode=test to /admin/allowance to verify a reminder really sends 2026-11-04 05:18:39 +00:00
inference-bot a8bdb70e08 allowances: fix decision-label mapping (would_renew not would_renewed) 2026-11-04 05:17:54 +00:00
inference-bot d878c32d68 allowances: restore fetch_member_orders/_payment_verdict/payment_reminder_email dropped in the slug fix 2026-11-04 05:17:12 +00:00