docs/operator-token-runbook.md

Operator runbook — Cloudron tokens

How the co-op's Cloudron credentials are organized, and how to rotate them without breaking the stack.

The two-token split

There are two separate Cloudron API tokens, deliberately decoupled so rotating one doesn't break the other:

Token	Purpose	Where it lives
Agent token	Inferencebot's own Cloudron access	The agent's private .env (gitignored)
Portal token	The member portal's Cloudron access (create users, set groups, invite links)	Cloudron's env store, under the portal app's CLOUDRON_TOKEN

The portal needs only a handful of Cloudron capabilities (user/group management, invite links). It is the most exposed app (public webhook + broker endpoints), so it must not hold the agent's token — a portal compromise shouldn't reach the agent's wider Cloudron access.

Rotating a token

Portal token

1. In my.inference.coop → Settings → API Tokens, create a "Read and Write" token.
2. Set it as the portal's CLOUDRON_TOKEN env var (custom apps have no env-var UI — use the Cloudron CLI or the API; see the next section).
3. Verify: the portal can reach GET /api/v1/users with the new token (HTTP 200).

Agent token

1. Create a fresh token in Settings → API Tokens.
2. Put it in the agent's .env as CLOUDRON_TOKEN.
3. Verify against /api/v1/cloudron/status.

The trap this split exists to prevent: the portal keeps its own copy of a Cloudron token. If the agent's token is rotated without also updating the portal copy, the portal's Cloudron calls start returning 401 — which breaks provisioning (new members can't be created) and the member dashboard (its usage/keys data is resolved through the portal's Cloudron calls).

Scheduled tasks (scheduler addon)

The portal and gateway run their own housekeeping via the Cloudron scheduler addon, declared in each app's CloudronManifest.json under addons.scheduler:

• portal: membership_sweep (nightly 3am, /app/code/sweep.sh → POST /admin/sweep with the WEBHOOK_TOKEN header) and rag_refresh (nightly 3:30am, /app/code/rag_refresh.py — pulls docs from the Gitea raw API and refreshes the chat knowledge base; needs OWUI_API_KEY + DOCS_KNOWLEDGE_ID in the app env).
• gateway: audio_keepalive (every 10 minutes, /app/code/audio_keepalive.py — pings the Tinfoil audio models locally to avoid the ~24s TEE cold-start; reads the master key from /app/data/.master_key, which start.sh writes — the env var LITELLM_MASTER_KEY is only exported into the litellm process).

After any manifest change, verify the scheduler survived the deploy: the build log must show setupAddons: setting up addon scheduler with options {...} listing your task(s), and each script must succeed when run manually inside the container (cloudron exec --app <id> -- <command>). The scheduler has historically gone missing from the manifest during edits — a deploy without the setupAddons line means nightly reconciliation is NOT running.

Custom apps have no env-var UI

The member portal is a custom app (packaged by us), not an App Store app. Custom apps do not expose environment variables in the Cloudron dashboard. Env vars for custom apps are set via:

• Cloudron CLI (on your workstation, not the server): cloudron env set --app portal.inference.coop CLOUDRON_TOKEN=...
• Cloudron API: POST /apps/:appId/configure/env

configure/env is a full replace

POST /apps/:appId/configure/env with body {"env": {KEY: value, ...}} replaces the entire environment — it does not merge. Sending a partial map (e.g. a single test variable) wipes every other variable.

• To change one variable, read the current env first (GET /api/v1/apps/:id → .env), merge in memory, and POST the full map back.
• Never probe the endpoint with a throwaway body against a live app.
• If the env is wiped, it is recoverable from a Cloudron app backup: GET /api/v1/apps/:appId/backups → newest entry → GET .../backups/:backupId/download (a .tar.gz containing config.json, whose .env field holds the original map).

Reference

• Cloudron CLI: npm install -g cloudron (install on your workstation, not the server), then cloudron login my.inference.coop.
• Cloudron API docs: docs.cloudron.io/api/.

Open Collective: the bot token needs the email scope (Sept 2026)

The portal identifies members by email, read via the bot's OC personal token (OC_PERSONAL_TOKEN) as collective.members.nodes[].account.email with the ... on Individual { email } inline fragment.

The email scope is load-bearing. The bot's personal token must be minted with the email scope. Without it, the GraphQL email field returns null for every member — including the bot's own me { email } — even though:

• the bot still authenticates and reports isAdmin: true, and
• the OC admin CSV export still shows emails (it's a separate, non-GraphQL path; its contributorEmail column is not a GraphQL field).

Symptoms when the scope is missing: fetch_members() filters out empty emails and returns [], so the webhook can't provision anyone by email and the nightly sweep goes into its empty-list fail-safe (skips, doesn't deactivate). A new member's webhook fires cleanly but no "Sent email to …" log line follows, and no Cloudron user is created. (This is easy to misread as a "guest with no email" problem — it is not; verify the scope first.)

Diagnosis: run me { email } with the bot token. null here while isAdmin: true → the token is missing the email scope (or the scope was revoked). Fix: re-add the email scope to the token on Open Collective (Settings → Developers → Personal Tokens). No token rotation, no re-granting of admin — just the scope.