membership: record the allowance renewal and lapse policy (2026-11-04)

Renewal asks Open Collective for a live recurring contribution on the sweep's schedule, anchored to the member's join day rather than the payment date (OC aligns charges onto a common billing day). One reminder, then a grace window, then deactivate by removing group access while keeping the account. Member-facing summary: co-op/support#44.
2026-11-04 05:20:45 +00:00
4 changed files with 57 additions and 237 deletions
@@ -24,6 +24,7 @@ Read it on the web here: <https://git.inference.coop/co-op/ops-docs>
 | Path | Contents |
 |---|---|
 | `operator-token-runbook.md` | The Cloudron token model (agent vs portal, admin vs webhook), how to rotate each without breaking the stack |
 | `membership-allowances.md` | How a member's monthly allowance renews, how a lapse is handled, and the join-day anchoring rule |
 | `maintenance/chat-task-models.md` | The small pinned model used for chat titles/tags/suggestions |
 | `maintenance/librechat-sandbox.md` | Operating the LibreChat evaluation instance (config, overlay image, key sync, pitfalls) |
 | `incidents/` | Post-incident write-ups, one file per incident |
@@ -0,0 +1,54 @@
 # Membership allowance renewal and lapse
 Operator-facing record of the decision taken **2026-11-04** on how a member's
 monthly allowance renews, and how a lapse is handled. The member-facing summary
 is `co-op/support#44`; the code lives in `co-op/member-portal`.
 ## The rule
 - **Renewal is decided by asking Open Collective, on our own schedule.** A
  scheduled reconciliation asks whether the member has a *live recurring
  contribution* (order status `ACTIVE` or `PROCESSING` — a payment in flight is
  not a missed payment). Renewal is a decision of the sweep, not a side effect of
  webhook delivery, so a lost webhook costs a delayed renewal, never a silently
  skipped month.
 - **Anchor to the member's join day, never the payment date.** Open Collective
  aligns a whole collective's charges onto one common billing day, so anchoring
  to a charge lets an early charge move the boundary and skip a cycle. The
  member's join day is stored (`members.anchor_day`) and the reset is pinned to
  its next occurrence.
 - **Grace before deactivation.** One reminder email per cycle; deactivate only
  after `PAYMENT_GRACE_DAYS` unpaid.
 - **Deactivate by removing group access and stopping renewal, keeping the
  account**, so a returning payer resumes cleanly at the next cycle.
 - **Log every decision** (`allowance_log`) with the Open Collective evidence —
  renewed / reminded / deactivated, plus the payment date or its absence — so a
  dispute is answered from a record, not reconstructed.
 ## Scope: monthly only
 Verified **2026-11-04** that there are **no annual contributors**: 104 orders on
 Open Collective, 87 MONTHLY and 17 ONETIME, zero annual, and no order has a
 charge gap over 45 days. Only the monthly case is implemented. One-time
 contributions are not an allowance cycle — they top up the non-expiring credit
 balance and stay in the existing credits model.
 ## Operating it
 - `ALLOWANCE_POLICY` = `report` (default) | `on`. `report` computes every
  decision and changes nothing; `on` performs renewals and sends the reminder.
 - `ALLOWANCE_DEACTIVATE` = `off` (default) | `on`. Gated separately so renewals
  and reminders can run before anyone loses access.
 - `PAYMENT_GRACE_DAYS` (default `7`).
 - Manual check: `POST /admin/allowance?mode=report` (`X-Admin-Token`) is a no-op
  report; `mode=test&to=<addr>` emails one sample reminder; `mode=send` requires
  `ALLOWANCE_POLICY=on`.
 ## The bug that prompted it
 A member who joined on the 15th was charged on the 2nd (Open Collective's common
 billing day). The reset rule re-anchored his allowance to the new payment date
 and pushed his renewal from the 14th to the 2nd of the next month — skipping the
 reset he was owed while he had already paid. The fix is both changes above:
 anchor to the join day, and decide renewal on a schedule rather than on the
 payment.
@@ -3,85 +3,6 @@
 How the co-op's Cloudron credentials are organized, and how to rotate them
 without breaking the stack.
 ## Credential rotation policy
 **Cadence: 12 months for every credential.** One interval, no exceptions for now.
 Twelve months is a starting point, not a principle — revisit it once the token
 list is stable (co-op/support#35).
 **Rotate on a trigger immediately, off the calendar, when any of these happens:**
 - **Offboarding** — anyone who had access to the credential leaves or changes
  role, or the device/account it lived on is lost.
 - **Suspected exposure** — the value was seen somewhere it should not be: a log,
  a chat, a commit, a screenshot.
 - **After any incident, or any diagnostic session on the agent host in which a
  secret could have been read.** Reading a live secret to debug with it is itself
  the trigger; it does not have to be a confirmed breach.
 - **After any restore** — the credential travels in the backup, so a restored
  host re-materialises whatever values the snapshot held.
 **The GM controls all tokens for now** (operator, 2026-11-04), so this record
 needs **no owner column**: every row is rotated by the GM. Provider keys are
 minted on the provider account, but the value held in the co-op's environment is
 still the GM's to rotate.
 ### Last rotated
 One row per credential, or per credential group — a group rotates together, so it
 dates together. A blank cell means **no rotation has been recorded**, not "not
 rotated". Dates are UTC, `YYYY-MM-DD`.
 The reminder reads this exact table: `scripts/rotation_due.py` in this repository
 (deployed on the agent host at `~/.hermes/scripts/rotation_due.py`) prints a line
 only when a due date is within 30 days, and prints nothing otherwise. Keep the
 table machine-legible — one `YYYY-MM-DD` date per row.
 | Credential or group | Last rotated (UTC) |
 |---|---|
 | Open Collective token (`OC_PERSONAL_TOKEN`) | 2026-10-27 |
 | `LITELLM_MASTER_KEY` / `GW_MASTER_KEY` (one value, four holders) | |
 | Cloudron API tokens — portal + agent (both on `inference-bot`) | |
 | `REGISTRY_BUILD_TOKEN` (on `inference-admin`; moves to CI, co-op/support#30) | |
 | Gitea token (`inference-bot`) | |
 | `WEBHOOK_TOKEN` (portal env + Open Collective webhook URL) | |
 | `ADMIN_TOKEN` (portal + admin panel + agent host) | |
 | `BROKER_SECRET` and `PORTAL_SECRET` | |
 | Matrix tokens — portal `MATRIX_BOT_TOKEN` + agent `MATRIX_ADMIN_TOKEN` (one account, `@inference-admin`) | |
 | Matrix token — steward agent `@inferencebot` | |
 | Loomio API keys (`LOOMIO_API_KEY`, `LOOMIO_B3_KEY`) | |
 | Listmonk API (`LISTMONK_API_USER`, `LISTMONK_API_TOKEN`) | |
 | Provider keys — Tinfoil, GreenPT, Regolo, and the GreenPT fallback | |
 | Restic repository passphrase; Infomaniak Swiss Backup S3 keys | |
 ### Accounts behind the credentials
 A credential belongs to an **account**, which is what a rotation is actually
 performed against — ask the provider, do not read the variable name. Two
 credentials on one account are a single hazard: rotating one and not the other is
 the silent break.
 | Credential | Account | Basis |
 |---|---|---|
 | `LITELLM_MASTER_KEY` / `GW_MASTER_KEY` | self-issued by the gateway | one literal, four holders |
 | Cloudron API tokens — portal and agent | `inference-bot` | verified (co-op/support#35) |
 | `REGISTRY_BUILD_TOKEN` | `inference-admin` | verified (co-op/support#35) |
 | `MATRIX_BOT_TOKEN` (portal) + `MATRIX_ADMIN_TOKEN` (agent) | `@inference-admin:inference.coop` — two tokens, one account | verified, homeserver `whoami` 2026-11-04 |
 | Steward agent's `MATRIX_ACCESS_TOKEN` | `@inferencebot:inference.coop` | verified, homeserver `whoami` 2026-11-04 |
 | `OC_PERSONAL_TOKEN` | `info@inference.coop` | from the 2026-10-27 rotation record (inference) |
 | `TINFOIL_API_KEY` | `info@inference.coop` | operator-confirmed |
 | `GREENPT_API_KEY` (gateway) and `GREENPT_FALLBACK_API_KEY` (agent) | the GreenPT account — the **same value** in both | verified: the "fallback" isolates nothing |
 | `REGOLO_API_KEY` | Regolo account | not yet identified |
 | Gitea token | `inference-bot` | inferred, not verified |
 | `BROKER_SECRET`, `PORTAL_SECRET`, `WEBHOOK_TOKEN`, `ADMIN_TOKEN`, `LOOMIO_*`, `LISTMONK_*` | self-issued / service account | — |
 | Restic passphrase; Infomaniak Swiss Backup S3 keys | self-issued; the Infomaniak account | — |
 The two couplings to watch: **one account with several credentials** (the
 `@inference-admin` Matrix pair, the two Cloudron tokens, the four holders of the
 master key) and **one value with several holders** (the GreenPT key, which is the
 only such case found). Full account map, including the hash comparison behind the
 GreenPT finding, is held with the security working notes.
 ## The two-token split
 There are **two separate** Cloudron API tokens, deliberately decoupled so
@@ -281,7 +202,7 @@ row that names the same credential: several are held by more than one app.
 | `WEBHOOK_TOKEN` | Secret in the Open Collective webhook URL | Portal app env, and the webhook URL on Open Collective | GM | Rotating means re-creating the webhook on OC as well |
 | `LITELLM_MASTER_KEY` | Create gateway teams and keys for members | Portal app env (copy of the gateway's) | GM | Three holders: portal, gateway, steward agent. Rotate all together |
 | `LITELLM_DB_*` | Read spend logs from the gateway database | Portal app env | Cloudron (addon) | Postgres credentials for the gateway's database |
-| `MATRIX_BOT_TOKEN` | `@inference-admin:inference.coop` admin account: invites, space membership | Portal app env | GM | The account has no password. Paired with the steward agent's `MATRIX_ADMIN_TOKEN`: same account, separate token on a separate device (verified by homeserver `whoami`, 2026-11-04). Revoke or rotate both together |
+| `MATRIX_BOT_TOKEN` | `@inference-bot:inference.coop` admin account: invites, space membership | Portal app env | GM | The account has no password. Paired with the steward agent's `MATRIX_LOCAL_BOT_TOKEN`: same account, separate token on a separate device (verified). Revoke or rotate both together |
 | `LOOMIO_API_KEY`, `LOOMIO_B3_KEY` | Forum (Loomio) membership sync | Portal app env | GM | |
 | `LISTMONK_API_USER`, `LISTMONK_API_TOKEN` | Newsletter list sync | Portal app env | GM | |
 | `OWUI_API_KEY`, `OWUI_JWT_SECRET` | Open WebUI knowledge refresh and SSO | Portal app env | GM | Chat moved to LibreChat in October; confirm whether still needed, and retire if not |
@@ -315,7 +236,7 @@ granularity for the public half. Highest privilege first.
 | Credential | Purpose | Where it lives | Who rotates | Notes |
 |---|---|---|---|---|
 | Gateway master key (`GW_MASTER_KEY`) | Admin over every gateway team, key and spend record | Steward agent `.env` | GM | **Third holder**, with the portal and the gateway. The row most in need of coordinated rotation |
-| `MATRIX_ADMIN_TOKEN` | The `@inference-admin:inference.coop` automation account: room administration in every co-op room | Steward agent `.env` | GM | Same account as the portal's `MATRIX_BOT_TOKEN`, but a separate token on a separate device (verified by homeserver `whoami`, 2026-11-04). Revoking one does not revoke the other; revoke or rotate both together |
+| `MATRIX_LOCAL_BOT_TOKEN` | The `@inference-bot:inference.coop` automation account: room administration in every co-op room | Steward agent `.env` | GM | Same account as the portal's `MATRIX_BOT_TOKEN`, but a separate token on a separate device (verified). Revoking one does not revoke the other; revoke or rotate both together |
 | Cloudron agent token | Cloudron API | Steward agent `.env` | GM | See "The two-token split" above |
 | `REGISTRY_BUILD_TOKEN` (and its username) | Image build and push for the custom apps | Steward agent `.env` | GM | Moves to CI (co-op/support#30); the agent stops holding it then |
 | Gitea token | Commits, issues, PRs as `inference-bot` | Steward agent `.env` | GM | |
@@ -1,156 +0,0 @@
 #!/usr/bin/env python3
 """Rotation-due watchdog for the co-op credential rotation policy.
 Reads the `last rotated` table from the operator token runbook and prints a
 reminder ONLY when a credential is due -- i.e. within 30 days of its due date
 (last rotation + 12 months), or already past it. Prints NOTHING when nothing is
 due, so it can drive a watchdog cron where empty output means "no message".
 Deterministic: the output depends only on the table and the current date. It
 prints no timestamps of its own, so when nothing is due it produces no output at
 all and stays silent across runs.
 Table source (first that applies):
  1. --file PATH
  2. $ROTATION_RUNBOOK
  3. the runbook fetched from Gitea's main branch (authoritative, always current)
  4. the local clone ~/repos/ops-docs/operator-token-runbook.md
 Exit codes: 0 ok (including "nothing due", no output), 2 could not read the
 runbook or found no dated rows -- a loud failure so a watcher is not silently
 blind.
 """
 import argparse
 import calendar
 import datetime as dt
 import os
 import re
 import sys
 import urllib.request
 RUNBOOK_URL = "https://git.inference.coop/co-op/ops-docs/raw/branch/main/operator-token-runbook.md"
 LOCAL_FALLBACK = os.path.expanduser("~/repos/ops-docs/operator-token-runbook.md")
 DEFAULT_WITHIN_DAYS = 30
 SECTION_RE = re.compile(r"^#{1,6}\s+.*last\s+rotated.*$", re.IGNORECASE)
 HEADING_RE = re.compile(r"^#{1,6}\s+")
 SEP_RE = re.compile(r"[\s|:\-]+")
 DATE_RE = re.compile(r"\b(\d{4})-(\d{2})-(\d{2})\b")
 def fetch_text(args):
    if args.file:
        try:
            with open(args.file, encoding="utf-8") as fh:
                return fh.read()
        except OSError as exc:
            print(f"rotation_due: could not read --file {args.file} ({exc})", file=sys.stderr)
            raise SystemExit(2)
    env = os.environ.get("ROTATION_RUNBOOK")
    if env and os.path.exists(env):
        try:
            with open(env, encoding="utf-8") as fh:
                return fh.read()
        except OSError as exc:
            print(f"rotation_due: could not read $ROTATION_RUNBOOK {env} ({exc})", file=sys.stderr)
            raise SystemExit(2)
    try:
        req = urllib.request.Request(RUNBOOK_URL, headers={"User-Agent": "rotation_due/1.0"})
        with urllib.request.urlopen(req, timeout=20) as resp:
            return resp.read().decode("utf-8")
    except Exception as exc:  # noqa: BLE001 - fall back, then fail loudly
        if os.path.exists(LOCAL_FALLBACK):
            with open(LOCAL_FALLBACK, encoding="utf-8") as fh:
                return fh.read()
        print(
            f"rotation_due: could not read the runbook from {RUNBOOK_URL} ({exc}) "
            f"and no local copy at {LOCAL_FALLBACK}",
            file=sys.stderr,
        )
        raise SystemExit(2)
 def extract_section(text):
    """Return the lines of the first 'last rotated' section."""
    out, inside = [], False
    for line in text.splitlines():
        if SECTION_RE.match(line):
            inside = True
            continue
        if inside and HEADING_RE.match(line):
            break
        if inside:
            out.append(line)
    return out
 def parse_rows(section_lines):
    """Yield (name, last_rotated_date) for table rows that carry a date."""
    for line in section_lines:
        if not line.lstrip().startswith("|"):
            continue
        stripped = line.strip().strip("|")
        if not stripped or SEP_RE.fullmatch(line.strip()):
            continue
        cells = [c.strip() for c in stripped.split("|")]
        if not cells:
            continue
        match = DATE_RE.search(line)
        if not match:
            continue
        year, month, day = (int(g) for g in match.groups())
        try:
            last = dt.date(year, month, day)
        except ValueError:
            continue
        yield cells[0], last
 def add_months(date, months):
    month_index = date.month - 1 + months
    year = date.year + month_index // 12
    month = month_index % 12 + 1
    day = min(date.day, calendar.monthrange(year, month)[1])
    return dt.date(year, month, day)
 def main():
    ap = argparse.ArgumentParser(description="Remind when a credential rotation is due.")
    ap.add_argument("--file", help="read the runbook from this path instead of fetching it")
    ap.add_argument("--within-days", type=int, default=DEFAULT_WITHIN_DAYS,
                    help="days before the due date to start reminding (default 30)")
    ap.add_argument("--as-of", help="evaluate as of this date (YYYY-MM-DD); default today UTC")
    args = ap.parse_args()
    section = extract_section(fetch_text(args))
    if not section:
        print("rotation_due: no 'last rotated' section found in the runbook", file=sys.stderr)
        return 2
    parsed = list(parse_rows(section))
    if not parsed:
        print("rotation_due: the 'last rotated' section has no rows carrying a date", file=sys.stderr)
        return 2
    as_of = dt.date.fromisoformat(args.as_of) if args.as_of else dt.datetime.now(dt.timezone.utc).date()
    due = []
    for name, last in parsed:
        due_date = add_months(last, 12)
        days = (due_date - as_of).days
        if days <= args.within_days:
            due.append((days, name, due_date))
    if not due:
        return 0  # nothing due -> no output at all
    due.sort()
    for days, name, due_date in due:
        if days < 0:
            print(f"credential rotation OVERDUE: {name} -- was due {due_date.isoformat()} ({-days} days ago)")
        else:
            print(f"credential rotation due soon: {name} -- due {due_date.isoformat()} (in {days} days)")
    return 0
 if __name__ == "__main__":
    sys.exit(main())