docs: scheduled tasks in portal/gateway (self-maintaining apps) + operator verification steps

infrastructure.md

@@ -62,6 +62,30 @@ domain, port 443), so members can also join from any homeserver they already
 use. The `@inference-bot:inference.coop` admin account (used by the portal
 for invites) has no password — token lives in the portal env.
 
+## How the apps run themselves
+
+The portal and gateway are self-maintaining: their recurring housekeeping is
+built into the apps as scheduled tasks, not run from any member's or operator's
+personal machine.
+
+- **Member portal** — two scheduled jobs (Cloudron scheduler):
+  - *Membership sweep* (nightly, 3am) — reconciles members against the live
+    Open Collective list: provisions new members who signed up while a webhook
+    was missed, deactivates lapsed members, and syncs Loomio + Matrix
+    membership. A missed webhook self-heals within 24 hours.
+  - *Docs knowledge refresh* (nightly, 3:30am) — pulls this documentation from
+    the public Git repo and updates the shared knowledge base behind the chat's
+    models, so answers grounded in the docs stay current. Changes to the docs
+    appear in the chat within a day, no manual step.
+- **Gateway** — an *audio keep-alive* (every 10 minutes) sends a tiny text-to-
+  speech ping to the private enclave so voice mode responds quickly even after
+  long quiet periods, without compromising the end-to-end encryption model.
+
+Scheduled tasks are declared in each app's `CloudronManifest.json` (see the
+`code/` repos) and survive redeploys. If a task fails, it shows up in the
+app's logs on the admin panel; the portal also exposes a manual trigger for
+the membership sweep.
+
 ## How membership is wired
 
 Membership flows through the member portal, which is the single source of truth

operator-token-runbook.md

@@ -39,6 +39,28 @@ copy, the portal's Cloudron calls start returning `401` — which breaks
 provisioning (new members can't be created) and the member dashboard (its
 usage/keys data is resolved through the portal's Cloudron calls).
 
+## Scheduled tasks (scheduler addon)
+
+The portal and gateway run their own housekeeping via the Cloudron scheduler
+addon, declared in each app's `CloudronManifest.json` under `addons.scheduler`:
+
+- **portal**: `membership_sweep` (nightly 3am, `/app/code/sweep.sh` →
+  `POST /admin/sweep` with the `WEBHOOK_TOKEN` header) and `rag_refresh`
+  (nightly 3:30am, `/app/code/rag_refresh.py` — pulls docs from the Gitea raw
+  API and refreshes the chat knowledge base; needs `OWUI_API_KEY` +
+  `DOCS_KNOWLEDGE_ID` in the app env).
+- **gateway**: `audio_keepalive` (every 10 minutes, `/app/code/audio_keepalive.py`
+  — pings the Tinfoil audio models locally to avoid the ~24s TEE cold-start;
+  reads the master key from `/app/data/.master_key`, which `start.sh` writes —
+  the env var `LITELLM_MASTER_KEY` is only exported into the litellm process).
+
+**After any manifest change, verify the scheduler survived the deploy:** the
+build log must show `setupAddons: setting up addon scheduler with options
+{...}` listing your task(s), and each script must succeed when run manually
+inside the container (`cloudron exec --app <id> -- <command>`). The scheduler
+has historically gone missing from the manifest during edits — a deploy without
+the setupAddons line means nightly reconciliation is NOT running.
+
 ## Custom apps have no env-var UI
 
 The member portal is a *custom* app (packaged by us), not an App Store app.