From fed58a34379d4e6e887d42effcb5cca6bf4ca67b Mon Sep 17 00:00:00 2001 From: inference-bot Date: Mon, 12 Oct 2026 19:32:32 -0600 Subject: [PATCH] docs: scheduled tasks in portal/gateway (self-maintaining apps) + operator verification steps --- infrastructure.md | 24 ++++++++++++++++++++++++ operator-token-runbook.md | 22 ++++++++++++++++++++++ 2 files changed, 46 insertions(+) diff --git a/infrastructure.md b/infrastructure.md index 79346ac..b731433 100644 --- a/infrastructure.md +++ b/infrastructure.md @@ -62,6 +62,30 @@ domain, port 443), so members can also join from any homeserver they already use. The `@inference-bot:inference.coop` admin account (used by the portal for invites) has no password — token lives in the portal env. +## How the apps run themselves + +The portal and gateway are self-maintaining: their recurring housekeeping is +built into the apps as scheduled tasks, not run from any member's or operator's +personal machine. + +- **Member portal** — two scheduled jobs (Cloudron scheduler): + - *Membership sweep* (nightly, 3am) — reconciles members against the live + Open Collective list: provisions new members who signed up while a webhook + was missed, deactivates lapsed members, and syncs Loomio + Matrix + membership. A missed webhook self-heals within 24 hours. + - *Docs knowledge refresh* (nightly, 3:30am) — pulls this documentation from + the public Git repo and updates the shared knowledge base behind the chat's + models, so answers grounded in the docs stay current. Changes to the docs + appear in the chat within a day, no manual step. +- **Gateway** — an *audio keep-alive* (every 10 minutes) sends a tiny text-to- + speech ping to the private enclave so voice mode responds quickly even after + long quiet periods, without compromising the end-to-end encryption model. + +Scheduled tasks are declared in each app's `CloudronManifest.json` (see the +`code/` repos) and survive redeploys. If a task fails, it shows up in the +app's logs on the admin panel; the portal also exposes a manual trigger for +the membership sweep. + ## How membership is wired Membership flows through the member portal, which is the single source of truth diff --git a/operator-token-runbook.md b/operator-token-runbook.md index efe050d..cf29527 100644 --- a/operator-token-runbook.md +++ b/operator-token-runbook.md @@ -39,6 +39,28 @@ copy, the portal's Cloudron calls start returning `401` — which breaks provisioning (new members can't be created) and the member dashboard (its usage/keys data is resolved through the portal's Cloudron calls). +## Scheduled tasks (scheduler addon) + +The portal and gateway run their own housekeeping via the Cloudron scheduler +addon, declared in each app's `CloudronManifest.json` under `addons.scheduler`: + +- **portal**: `membership_sweep` (nightly 3am, `/app/code/sweep.sh` → + `POST /admin/sweep` with the `WEBHOOK_TOKEN` header) and `rag_refresh` + (nightly 3:30am, `/app/code/rag_refresh.py` — pulls docs from the Gitea raw + API and refreshes the chat knowledge base; needs `OWUI_API_KEY` + + `DOCS_KNOWLEDGE_ID` in the app env). +- **gateway**: `audio_keepalive` (every 10 minutes, `/app/code/audio_keepalive.py` + — pings the Tinfoil audio models locally to avoid the ~24s TEE cold-start; + reads the master key from `/app/data/.master_key`, which `start.sh` writes — + the env var `LITELLM_MASTER_KEY` is only exported into the litellm process). + +**After any manifest change, verify the scheduler survived the deploy:** the +build log must show `setupAddons: setting up addon scheduler with options +{...}` listing your task(s), and each script must succeed when run manually +inside the container (`cloudron exec --app -- `). The scheduler +has historically gone missing from the manifest during edits — a deploy without +the setupAddons line means nightly reconciliation is NOT running. + ## Custom apps have no env-var UI The member portal is a *custom* app (packaged by us), not an App Store app.