diff --git a/operator-token-runbook.md b/operator-token-runbook.md index efe050d..cf29527 100644 --- a/operator-token-runbook.md +++ b/operator-token-runbook.md @@ -39,6 +39,28 @@ copy, the portal's Cloudron calls start returning `401` — which breaks provisioning (new members can't be created) and the member dashboard (its usage/keys data is resolved through the portal's Cloudron calls). +## Scheduled tasks (scheduler addon) + +The portal and gateway run their own housekeeping via the Cloudron scheduler +addon, declared in each app's `CloudronManifest.json` under `addons.scheduler`: + +- **portal**: `membership_sweep` (nightly 3am, `/app/code/sweep.sh` → + `POST /admin/sweep` with the `WEBHOOK_TOKEN` header) and `rag_refresh` + (nightly 3:30am, `/app/code/rag_refresh.py` — pulls docs from the Gitea raw + API and refreshes the chat knowledge base; needs `OWUI_API_KEY` + + `DOCS_KNOWLEDGE_ID` in the app env). +- **gateway**: `audio_keepalive` (every 10 minutes, `/app/code/audio_keepalive.py` + — pings the Tinfoil audio models locally to avoid the ~24s TEE cold-start; + reads the master key from `/app/data/.master_key`, which `start.sh` writes — + the env var `LITELLM_MASTER_KEY` is only exported into the litellm process). + +**After any manifest change, verify the scheduler survived the deploy:** the +build log must show `setupAddons: setting up addon scheduler with options +{...}` listing your task(s), and each script must succeed when run manually +inside the container (`cloudron exec --app -- `). The scheduler +has historically gone missing from the manifest during edits — a deploy without +the setupAddons line means nightly reconciliation is NOT running. + ## Custom apps have no env-var UI The member portal is a *custom* app (packaged by us), not an App Store app.