service invites: per-room Matrix gate, resumable rate limits, Loomio record heal, 5-min sync tick
A new member could end up with a Loomio account and a Matrix account and see NOTHING: no forum group, no Space. Three defects: * sync_matrix_space() gated each member on the SPACE's membership, so an account already joined/invited to the Space was skipped for EVERY room. A member whose invite pass was interrupted (or who predates a room added to MATRIX_INVITE_ROOMS later) never received that room — and a restricted child room is invisible until you are invited to it. The gate is now per room, from a single up-front membership snapshot; a room whose membership cannot be read is left alone rather than invited into blindly. * a 429 slept for retry_after_ms (minutes) INSIDE the pass, so a scheduled run could outlive its own tick. A rate limit now ENDS the pass; the next run resumes (every invite is idempotent). * a member sitting in the Loomio group with no loomio_invited_at stamp reads as 'never invited' — the dedupe guard was only as good as the rows it had. The sync now heals the record from the live roster. And the activation window itself: provisioning skips an unactivated account by design, and activation happens later with nobody watching, so the 3x/day sweep was the pickup. New /admin/service-sync + a 5-minute scheduler task fires the same idempotent syncs while any member still awaits an invite (one SELECT when none do), and deliberately is NOT /admin/sweep — it never touches Open Collective, allowance logic or member mail. tests/test_service_invites.py proves the per-room gate, resumable rate limits, the unreadable-room skip, the record heal and the endpoint's idle/run paths.
This commit is contained in:
1 parent
de0f5c13c4
commit
0e825e6d61
6 files changed
+596
-49
No files matched your search
@@ -0,0 +1,338 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Regression test: the service-invite sync (per-room invites + frequent tick).
|
||||
|
||||
WHY THIS EXISTS
|
||||
A new member got a Loomio account and a Matrix account but could see NOTHING:
|
||||
no forum group, no Space. Two separate defects produced that, and a third
|
||||
made it take hours.
|
||||
|
||||
1. PER-ROOM GATE (the visibility bug). sync_matrix_space() gated each member
|
||||
on the SPACE's membership: anyone already joined/invited to the Space was
|
||||
skipped for EVERY room. So a member whose invite pass was interrupted, or
|
||||
who predates a room added to MATRIX_INVITE_ROOMS later, never received
|
||||
that room's invite — and a restricted child room is invisible until you
|
||||
are invited to it.
|
||||
2. RATE LIMITS ENDED NOTHING. A 429 slept for retry_after_ms (minutes) inside
|
||||
the pass, so a scheduled run could outlive its own tick. A rate limit must
|
||||
END the pass; the next run resumes (every invite is idempotent).
|
||||
3. THE DEDUPE RECORD DRIFTED. A member sitting in the Loomio group with no
|
||||
members.loomio_invited_at stamp (added by an earlier sync generation, or
|
||||
by hand) reads as "never invited" — the guard is only as good as the rows
|
||||
it happens to have. The sync now heals the record from the live roster.
|
||||
4. NOTHING WATCHED ACTIVATION. Provisioning skips an unactivated account by
|
||||
design, and activation happens later with nobody watching, so the 3x/day
|
||||
sweep was the pickup. /admin/service-sync is the frequent tick that closes
|
||||
that window; it must be a no-op (one SELECT) in the steady state.
|
||||
|
||||
Network is STUBBED throughout; assertions are on what we SENT and on our own
|
||||
recorded state.
|
||||
|
||||
Runnable: python3 tests/test_service_invites.py (from the repo root)
|
||||
"""
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import asyncio
|
||||
|
||||
import httpx
|
||||
|
||||
os.environ["DB_PATH"] = os.path.join(tempfile.mkdtemp(), "fixture.db")
|
||||
os.environ["MATRIX_BOT_TOKEN"] = "test-bot-token"
|
||||
os.environ["MATRIX_SPACE_ID"] = "!space:inference.coop"
|
||||
os.environ["MATRIX_HOMESERVER"] = "https://matrix.inference.coop"
|
||||
os.environ["MATRIX_INVITE_ROOMS"] = "!space:inference.coop,!general:inference.coop,!design:inference.coop"
|
||||
os.environ["LOOMIO_API_KEY"] = "test-user-key"
|
||||
os.environ["LOOMIO_B3_KEY"] = "test-server-key"
|
||||
os.environ["LOOMIO_GROUP_ID"] = "1"
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
import app.main as m # noqa: E402
|
||||
|
||||
# The app module and this test share the SAME httpx module object, so stubbing
|
||||
# the client below replaces it process-wide. Keep a handle on the real class for
|
||||
# the ASGI call at the end.
|
||||
REAL_ASYNC_CLIENT = httpx.AsyncClient
|
||||
|
||||
m.DB_PATH = os.environ["DB_PATH"]
|
||||
m.MATRIX_BOT_TOKEN = "test-bot-token"
|
||||
m.MATRIX_SPACE_ID = "!space:inference.coop"
|
||||
m.MATRIX_HOMESERVER = "https://matrix.inference.coop"
|
||||
m.MATRIX_INVITE_ROOMS = [
|
||||
"!space:inference.coop", "!general:inference.coop", "!design:inference.coop"
|
||||
]
|
||||
m.MATRIX_EXTRA_ROOMS = []
|
||||
m.LOOMIO_ALWAYS_KEEP = []
|
||||
m.ADMIN_TOKEN = "test-admin-token"
|
||||
m.ADMIN_TOKEN_LEGACY = ""
|
||||
m.WEBHOOK_TOKEN = ""
|
||||
|
||||
ROOMS = ["!space:inference.coop", "!general:inference.coop", "!design:inference.coop"]
|
||||
USERS = {"a@x.org": "amember", "b@x.org": "bmember", "c@x.org": "cmember"}
|
||||
|
||||
ROOM_JOINED = {} # room -> set(mxid)
|
||||
ROOM_INVITED = {} # room -> set(mxid)
|
||||
UNREADABLE = set() # rooms whose joined_members 500s
|
||||
invites_sent = [] # (room, mxid)
|
||||
INVITE_429_FOR = set() # mxids whose invite comes back 429
|
||||
POSTS = [] # Loomio POST payloads
|
||||
|
||||
ROSTER = {} # Loomio roster: email -> membership id
|
||||
|
||||
|
||||
class FakeResp:
|
||||
def __init__(self, status=200, payload=None, text=None):
|
||||
self.status_code = status
|
||||
self._payload = payload or {}
|
||||
self.text = text if text is not None else str(payload or {})
|
||||
|
||||
def raise_for_status(self):
|
||||
if self.status_code >= 400:
|
||||
raise RuntimeError(f"HTTP {self.status_code}")
|
||||
|
||||
def json(self):
|
||||
return self._payload
|
||||
|
||||
|
||||
class FakeClient:
|
||||
def __init__(self, *a, **k):
|
||||
pass
|
||||
|
||||
async def __aenter__(self):
|
||||
return self
|
||||
|
||||
async def __aexit__(self, *a):
|
||||
return False
|
||||
|
||||
async def get(self, url, headers=None, params=None):
|
||||
if "/api/v1/users" in url:
|
||||
return FakeResp(200, {"users": [
|
||||
{"email": e, "username": u, "inviteAccepted": True}
|
||||
for e, u in USERS.items()
|
||||
]})
|
||||
for room in ROOMS:
|
||||
if f"/rooms/{room}/joined_members" in url:
|
||||
if room in UNREADABLE:
|
||||
return FakeResp(500, {"error": "boom"})
|
||||
return FakeResp(200, {"joined": {m: {} for m in ROOM_JOINED.get(room, set())}})
|
||||
if f"/rooms/{room}/members" in url:
|
||||
return FakeResp(200, {"chunk": [
|
||||
{"state_key": mx} for mx in ROOM_INVITED.get(room, set())
|
||||
]})
|
||||
if "/api/b3/memberships" in url:
|
||||
rows = [{"id": i, "user_email": e} for e, i in ROSTER.items()]
|
||||
offset = (params or {}).get("offset", 0)
|
||||
limit = (params or {}).get("limit", 50)
|
||||
return FakeResp(200, {"memberships": rows[offset:offset + limit]})
|
||||
return FakeResp(404, {})
|
||||
|
||||
async def post(self, url, headers=None, json=None):
|
||||
if "/invite" in url:
|
||||
mxid = (json or {}).get("user_id")
|
||||
if mxid in INVITE_429_FOR:
|
||||
return FakeResp(429, {"retry_after_ms": 300000},
|
||||
text='{"errcode":"M_LIMIT_EXCEEDED"}')
|
||||
for room in ROOMS:
|
||||
if f"/rooms/{room}/invite" in url:
|
||||
invites_sent.append((room, mxid))
|
||||
ROOM_INVITED.setdefault(room, set()).add(mxid)
|
||||
return FakeResp(200, {})
|
||||
return FakeResp(200, {})
|
||||
if "/api/b2/memberships" in url:
|
||||
POSTS.append(json or {})
|
||||
return FakeResp(200, {"added_emails": (json or {}).get("emails", [])})
|
||||
return FakeResp(200, {})
|
||||
|
||||
async def put(self, *a, **k):
|
||||
return FakeResp(200, {})
|
||||
|
||||
async def delete(self, *a, **k):
|
||||
return FakeResp(200, {})
|
||||
|
||||
async def patch(self, *a, **k):
|
||||
return FakeResp(200, {})
|
||||
|
||||
|
||||
results = []
|
||||
|
||||
|
||||
def check(name, ok):
|
||||
results.append((name, ok))
|
||||
|
||||
|
||||
def reset():
|
||||
global invites_sent
|
||||
invites_sent = []
|
||||
POSTS.clear()
|
||||
|
||||
|
||||
async def main():
|
||||
global ROOM_JOINED, ROOM_INVITED, UNREADABLE, INVITE_429_FOR, ROSTER
|
||||
|
||||
m.get_db().close() # create + migrate the fixture schema
|
||||
conn = m.get_db()
|
||||
for email in USERS:
|
||||
conn.execute("INSERT OR REPLACE INTO members (email, active) VALUES (?, 1)", (email,))
|
||||
conn.execute("INSERT OR REPLACE INTO members (email, active, account_type) VALUES (?, 1, 'guest')",
|
||||
("guest@x.org",))
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
m.httpx.AsyncClient = FakeClient
|
||||
|
||||
async def _active():
|
||||
return list(USERS.keys())
|
||||
m.get_active_member_emails = _active
|
||||
|
||||
# --- 1. PER-ROOM GATE -------------------------------------------------
|
||||
# a@ is already in every room. b@ has the Space + General but is MISSING
|
||||
# Design — the exact state an interrupted pass leaves behind.
|
||||
ROOM_JOINED = {"!space:inference.coop": {"@amember:inference.coop"},
|
||||
"!general:inference.coop": {"@amember:inference.coop"},
|
||||
"!design:inference.coop": {"@amember:inference.coop"}}
|
||||
ROOM_INVITED = {"!space:inference.coop": {"@bmember:inference.coop"},
|
||||
"!general:inference.coop": {"@bmember:inference.coop"},
|
||||
"!design:inference.coop": set()}
|
||||
USERS["c@x.org"] = "cmember"
|
||||
ROOM_JOINED["!space:inference.coop"].add("@cmember:inference.coop")
|
||||
ROOM_INVITED["!space:inference.coop"].add("@cmember:inference.coop")
|
||||
ROOM_JOINED["!general:inference.coop"].add("@cmember:inference.coop")
|
||||
ROOM_INVITED["!general:inference.coop"].add("@cmember:inference.coop")
|
||||
ROOM_JOINED["!design:inference.coop"].add("@cmember:inference.coop")
|
||||
ROOM_INVITED["!design:inference.coop"].add("@cmember:inference.coop")
|
||||
reset()
|
||||
r = await m.sync_matrix_space()
|
||||
check("per-room: only the MISSING room is invited, by room id",
|
||||
invites_sent == [("!design:inference.coop", "@bmember:inference.coop")])
|
||||
check("per-room: an account already in every room is not re-invited",
|
||||
not any(mx == "@amember:inference.coop" for _, mx in invites_sent))
|
||||
check("per-room: two members can be pending on different rooms",
|
||||
not any(mx == "@cmember:inference.coop" for _, mx in invites_sent))
|
||||
check("return shape: the result carries invited + rate_limited",
|
||||
set(r) == {"invited", "rate_limited"} and r["rate_limited"] is False)
|
||||
|
||||
# --- 2. IDEMPOTENCE: a second pass sends nothing ----------------------
|
||||
reset()
|
||||
await m.sync_matrix_space()
|
||||
check("idempotent: a second pass sends no invites", invites_sent == [])
|
||||
|
||||
# --- 3. RATE LIMIT ENDS THE PASS (no sleeping inside a scheduled tick) -
|
||||
# d@ is missing all three rooms; the FIRST invite 429s.
|
||||
USERS["d@x.org"] = "dmember"
|
||||
m.get_db().close()
|
||||
conn = m.get_db()
|
||||
conn.execute("INSERT OR REPLACE INTO members (email, active) VALUES ('d@x.org', 1)")
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
async def active_with_d():
|
||||
return ["d@x.org", "a@x.org"]
|
||||
m.get_active_member_emails = active_with_d
|
||||
INVITE_429_FOR = {"@dmember:inference.coop"}
|
||||
reset()
|
||||
r = await m.sync_matrix_space()
|
||||
check("rate limit: the pass reports it was rate limited", r["rate_limited"] is True)
|
||||
check("rate limit: it does NOT retry in a loop inside the pass",
|
||||
invites_sent == [])
|
||||
INVITE_429_FOR = set()
|
||||
m.get_active_member_emails = _active
|
||||
|
||||
# A resumed pass DOES send the invites the limited one could not.
|
||||
reset()
|
||||
r = await m.sync_matrix_space()
|
||||
check("rate limit: the next pass resumes and invites the missed rooms",
|
||||
len(invites_sent) == 3 and {room for room, _ in invites_sent} == set(ROOMS))
|
||||
|
||||
# --- 4. A ROOM WE CANNOT READ IS LEFT ALONE ---------------------------
|
||||
# e@ is only missing Design, and Design's membership cannot be read.
|
||||
USERS["e@x.org"] = "emember"
|
||||
conn = m.get_db()
|
||||
conn.execute("INSERT OR REPLACE INTO members (email, active) VALUES ('e@x.org', 1)")
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
async def active_with_e():
|
||||
return ["e@x.org"]
|
||||
m.get_active_member_emails = active_with_e
|
||||
ROOM_JOINED = {r_: set() for r_ in ROOMS}
|
||||
ROOM_INVITED = {"!space:inference.coop": {"@emember:inference.coop"},
|
||||
"!general:inference.coop": {"@emember:inference.coop"},
|
||||
"!design:inference.coop": set()}
|
||||
UNREADABLE = {"!design:inference.coop"}
|
||||
reset()
|
||||
await m.sync_matrix_space()
|
||||
check("unreadable room: never invited into a room whose members we could not read",
|
||||
invites_sent == [])
|
||||
UNREADABLE = set()
|
||||
m.get_active_member_emails = _active
|
||||
|
||||
# --- 5. LOOMIO RECORD SELF-HEAL --------------------------------------
|
||||
# b@ is sitting in the group with NO stamp: the guard would call them
|
||||
# "never invited". The sync must heal the record from the roster, and must
|
||||
# not send their address anywhere while doing it.
|
||||
conn = m.get_db()
|
||||
conn.execute("UPDATE members SET loomio_invited_at = NULL")
|
||||
conn.commit()
|
||||
conn.close()
|
||||
ROSTER = {"b@x.org": 55}
|
||||
reset()
|
||||
await m.sync_loomio_memberships()
|
||||
sent = sorted((POSTS[0].get("emails") if POSTS else []) or [])
|
||||
check("heal: a present member with no stamp is stamped as invited",
|
||||
m.get_loomio_invited_at("b@x.org") is not None)
|
||||
check("heal: healing never re-sends a member already in the group",
|
||||
"b@x.org" not in sent)
|
||||
check("heal: an unstamped member who is NOT present is still invited",
|
||||
sent == ["a@x.org", "c@x.org", "d@x.org", "e@x.org"])
|
||||
|
||||
# --- 6. THE FREQUENT TICK'S WAKE-UP CONDITION -------------------------
|
||||
conn = m.get_db()
|
||||
conn.execute("UPDATE members SET loomio_invited_at = NULL WHERE email = 'e@x.org'")
|
||||
conn.commit()
|
||||
conn.close()
|
||||
pending = m._service_sync_pending()
|
||||
check("pending: a member still awaiting their invite is counted", pending == ["e@x.org"])
|
||||
check("pending: an invited member is NOT counted", "a@x.org" not in pending)
|
||||
check("pending: guests are never counted", "guest@x.org" not in pending)
|
||||
m.mark_loomio_invited("e@x.org")
|
||||
check("pending: empty once every member is stamped", m._service_sync_pending() == [])
|
||||
|
||||
|
||||
asyncio.run(main())
|
||||
|
||||
# --- 7. THE ENDPOINT (real ASGI call, so the wiring is exercised) ----------
|
||||
|
||||
|
||||
async def endpoint_checks():
|
||||
transport = httpx.ASGITransport(app=m.app)
|
||||
|
||||
async with REAL_ASYNC_CLIENT(transport=transport, base_url="http://test") as c:
|
||||
r = await c.post("/admin/service-sync")
|
||||
check(f"endpoint: rejects a caller with no admin token (got {r.status_code})",
|
||||
r.status_code == 403)
|
||||
r = await c.post("/admin/service-sync", headers={"X-Admin-Token": "wrong"})
|
||||
check(f"endpoint: rejects a wrong admin token (got {r.status_code})",
|
||||
r.status_code == 403)
|
||||
r = await c.post("/admin/service-sync", headers={"X-Admin-Token": "test-admin-token"})
|
||||
check(f"endpoint: accepts the admin token (got {r.status_code})", r.status_code == 200)
|
||||
body = r.json()
|
||||
check(f"endpoint: is IDLE (one SELECT) when nobody is awaiting an invite (got {body})",
|
||||
body.get("status") == "idle")
|
||||
# Make one member await an invite again and prove the tick RUNS.
|
||||
conn = m.get_db()
|
||||
conn.execute("UPDATE members SET loomio_invited_at = NULL WHERE email = 'a@x.org'")
|
||||
conn.commit()
|
||||
conn.close()
|
||||
r = await c.post("/admin/service-sync", headers={"X-Admin-Token": "test-admin-token"})
|
||||
body = r.json()
|
||||
check(f"endpoint: RUNS while a member is awaiting invites (got {body})",
|
||||
body.get("status") == "ran" and body.get("pending") == 1)
|
||||
check(f"endpoint: runs BOTH syncs (got {body})",
|
||||
body.get("loomio") == "ok" and "matrix" in body)
|
||||
|
||||
|
||||
asyncio.run(endpoint_checks())
|
||||
|
||||
fails = sum(not ok for _, ok in results)
|
||||
for name, ok in results:
|
||||
print(f" [{'PASS' if ok else 'FAIL'}] {name}")
|
||||
print(f"\n{len(results) - fails}/{len(results)} passed")
|
||||
sys.exit(1 if fails else 0)
|
||||
Reference in new issue
Block a user