docs/operator-token-runbook.md

Operator runbook — Cloudron tokens

How the co-op's Cloudron credentials are organized, and how to rotate them without breaking the stack.

The two-token split

There are two separate Cloudron API tokens, deliberately decoupled so rotating one doesn't break the other:

Token	Purpose	Where it lives
Agent token	Inferencebot's own Cloudron access	The agent's private .env (gitignored)
Portal token	The member portal's Cloudron access (create users, set groups, invite links)	Cloudron's env store, under the portal app's CLOUDRON_TOKEN

The portal needs only a handful of Cloudron capabilities (user/group management, invite links). It is the most exposed app (public webhook + broker endpoints), so it must not hold the agent's token — a portal compromise shouldn't reach the agent's wider Cloudron access.

Rotating a token

Portal token

1. In my.inference.coop → Settings → API Tokens, create a "Read and Write" token.
2. Set it as the portal's CLOUDRON_TOKEN env var (custom apps have no env-var UI — use the Cloudron CLI or the API; see the next section).
3. Verify: the portal can reach GET /api/v1/users with the new token (HTTP 200).

Agent token

1. Create a fresh token in Settings → API Tokens.
2. Put it in the agent's .env as CLOUDRON_TOKEN.
3. Verify against /api/v1/cloudron/status.

The trap this split exists to prevent: the portal keeps its own copy of a Cloudron token. If the agent's token is rotated without also updating the portal copy, the portal's Cloudron calls start returning 401 — which breaks provisioning (new members can't be created) and the member dashboard (its usage/keys data is resolved through the portal's Cloudron calls).

Chat image: spaCy model for document parsing (EPUB/DOCX)

Open WebUI's document parser (unstructured) lazily downloads the spaCy model en_core_web_sm on first EPUB/DOCX parse — but the app directory is read-only under Cloudron, so that install always fails ("Errno 30 Read-only file system"). Fixed 2026-10-14 (issue #20) by pre-installing the model into persistent storage:

• /app/data/spacy_model/ holds the unpacked wheel (SHA-verified en_core_web_sm-3.8.0-py3-none-any.whl from the pinned URL in unstructured/nlp/tokenize.py).
• App env has PYTHONPATH=/app/data/spacy_model so import en_core_web_sm resolves and the lazy download never triggers.

Restore caveat: a backup restore from before 2026-10-14 will NOT contain the model dir — re-create it (download the pinned wheel, unzip into /app/data/spacy_model/, verify spacy.load("en_core_web_sm") works in the container). The PYTHONPATH env survives restores (it's in app config).

Scheduled tasks (scheduler addon)

The portal and gateway run their own housekeeping via the Cloudron scheduler addon, declared in each app's CloudronManifest.json under addons.scheduler:

• portal: membership_sweep (nightly 3am, /app/code/sweep.sh → POST /admin/sweep with the WEBHOOK_TOKEN header) and rag_refresh (nightly 3:30am, /app/code/rag_refresh.py — pulls docs from the Gitea raw API and refreshes the chat knowledge base; needs OWUI_API_KEY + DOCS_KNOWLEDGE_ID in the app env).
• gateway: audio_keepalive (every 10 minutes, /app/code/audio_keepalive.py — pings the Tinfoil audio models locally to avoid the ~24s TEE cold-start; reads the master key from /app/data/.master_key, which start.sh writes — the env var LITELLM_MASTER_KEY is only exported into the litellm process).

After any manifest change, verify the scheduler survived the deploy: the build log must show setupAddons: setting up addon scheduler with options {...} listing your task(s), and each script must succeed when run manually inside the container (cloudron exec --app <id> -- <command>). The scheduler has historically gone missing from the manifest during edits — a deploy without the setupAddons line means nightly reconciliation is NOT running.

Custom apps have no env-var UI

The member portal is a custom app (packaged by us), not an App Store app. Custom apps do not expose environment variables in the Cloudron dashboard. Env vars for custom apps are set via:

• Cloudron CLI (on your workstation, not the server): cloudron env set --app portal.inference.coop CLOUDRON_TOKEN=...
• Cloudron API: POST /apps/:appId/configure/env

configure/env is a full replace

POST /apps/:appId/configure/env with body {"env": {KEY: value, ...}} replaces the entire environment — it does not merge. Sending a partial map (e.g. a single test variable) wipes every other variable.

• To change one variable, read the current env first (GET /api/v1/apps/:id → .env), merge in memory, and POST the full map back.
• Never probe the endpoint with a throwaway body against a live app.
• If the env is wiped, it is recoverable from a Cloudron app backup: GET /api/v1/apps/:appId/backups → newest entry → GET .../backups/:backupId/download (a .tar.gz containing config.json, whose .env field holds the original map).

Reference

• Cloudron CLI: npm install -g cloudron (install on your workstation, not the server), then cloudron login my.inference.coop.
• Cloudron API docs: docs.cloudron.io/api/.

Open Collective: the bot token needs the email scope (Sept 2026)

The portal identifies members by email, read via the bot's OC personal token (OC_PERSONAL_TOKEN) as collective.members.nodes[].account.email with the ... on Individual { email } inline fragment.

The email scope is load-bearing. The bot's personal token must be minted with the email scope. Without it, the GraphQL email field returns null for every member — including the bot's own me { email } — even though:

• the bot still authenticates and reports isAdmin: true, and
• the OC admin CSV export still shows emails (it's a separate, non-GraphQL path; its contributorEmail column is not a GraphQL field).

Symptoms when the scope is missing: fetch_members() filters out empty emails and returns [], so the webhook can't provision anyone by email and the nightly sweep goes into its empty-list fail-safe (skips, doesn't deactivate). A new member's webhook fires cleanly but no "Sent email to …" log line follows, and no Cloudron user is created. (This is easy to misread as a "guest with no email" problem — it is not; verify the scope first.)

Diagnosis: run me { email } with the bot token. null here while isAdmin: true → the token is missing the email scope (or the scope was revoked). Fix: re-add the email scope to the token on Open Collective (Settings → Developers → Personal Tokens). No token rotation, no re-granting of admin — just the scope.

Credential inventory (draft, 2026-10-26)

Every credential the agents and the member-facing apps hold, compiled from the public code and docs, corrected against what the agent holds (PR #7 review). No secret values appear here, only names and roles. "Where it lives" and "Who rotates" are filled in where the docs say; blank cells are for the operator to complete. When rotating anything, check every row that names the same credential: several are held by more than one app.

Member portal (portal.inference.coop)

Credential	Purpose	Where it lives	Who rotates	Notes
CLOUDRON_TOKEN (portal token)	Create users and groups, invite links	Portal app env	GM	See "The two-token split" above
OC_PERSONAL_TOKEN	Read members and orders from Open Collective GraphQL	Portal app env	The Open Collective account that minted it	Needs the email scope (see below). Currently a personal token; a co-op-owned OC account would make it a co-op credential
WEBHOOK_TOKEN	Secret in the Open Collective webhook URL	Portal app env, and the webhook URL on Open Collective	GM	Rotating means re-creating the webhook on OC as well
LITELLM_MASTER_KEY	Create gateway teams and keys for members	Portal app env (copy of the gateway's)	GM	Three holders: portal, gateway, steward agent. Rotate all together
LITELLM_DB_*	Read spend logs from the gateway database	Portal app env	Cloudron (addon)	Postgres credentials for the gateway's database
MATRIX_BOT_TOKEN	@inference-bot:inference.coop admin account: invites, space membership	Portal app env	GM	The account has no password; the token is the only credential
LOOMIO_API_KEY, LOOMIO_B3_KEY	Forum (Loomio) membership sync	Portal app env	GM
LISTMONK_API_USER, LISTMONK_API_TOKEN	Newsletter list sync	Portal app env	GM
OWUI_API_KEY, OWUI_JWT_SECRET	Open WebUI knowledge refresh and SSO	Portal app env	GM	Chat moved to LibreChat in October; confirm whether still needed, and retire if not
BROKER_SECRET	Shared secret between the portal and the member dashboard	Portal and dashboard app env	GM	Held by both; rotate together
PORTAL_SECRET	Session signing	Portal app env	GM
CLOUDRON_MAIL_SMTP_*	Outbound mail	Cloudron mail addon	Cloudron	Managed by the platform

Gateway (gateway.inference.coop)

Credential	Purpose	Where it lives	Who rotates	Notes
LITELLM_MASTER_KEY	Gateway admin	/app/data/.master_key, exported into the litellm process by start.sh	GM	Also held by the portal (above), the steward agent (below) and the audio keep-alive job
Salt key (SALT_KEY_FILE)	Encrypts provider keys stored in the gateway database	/app/data	GM	Losing it means re-entering every provider key
TINFOIL_API_KEY	Provider account	Gateway app env	Provider account owner	Billing lives at the provider
GREENPT_API_KEY	Provider account	Gateway app env	Provider account owner	Billing lives at the provider
REGOLO_API_KEY	Provider account	Gateway app env	Provider account owner	Billing lives at the provider
CLOUDRON_POSTGRESQL_*, CLOUDRON_REDIS_*	Database and cache	Cloudron addons	Cloudron	Managed by the platform

Chat (LibreChat, chat.inference.coop)

Credential	Purpose	Where it lives	Who rotates	Notes
LibreChat encryption keys and app secrets	App secrets; the per-member key sync runs in the app's own Cloudron scheduler	App data directory	GM	Not held by the agent

Agents (on the agent host)

Exact file paths are deliberately not listed; "agent .env" is the right granularity for the public half. Highest privilege first.

Credential	Purpose	Where it lives	Who rotates	Notes
Gateway master key (GW_MASTER_KEY)	Admin over every gateway team, key and spend record	Steward agent .env	GM	Third holder, with the portal and the gateway. The row most in need of coordinated rotation
MATRIX_LOCAL_BOT_TOKEN	The @inference-bot:inference.coop automation account: room administration in every co-op room	Steward agent .env	GM	Same account as the portal's MATRIX_BOT_TOKEN; confirm whether it is the same token
Cloudron agent token	Cloudron API	Steward agent .env	GM	See "The two-token split" above
REGISTRY_BUILD_TOKEN (and its username)	Image build and push for the custom apps	Steward agent .env	GM	Moves to CI (co-op/support#30); the agent stops holding it then
Gitea token	Commits, issues, PRs as inference-bot	Steward agent .env	GM
Matrix access token (@inferencebot:inference.coop)	Reading and posting in rooms	Agent host, with its E2EE key store	GM	The off switch in Working with the agent revokes this second
Steward agent's gateway key (api:info@inference.coop:inferencebots)	The steward agent's own inference	Steward agent .env	GM	Today: empty model allowlist and no per-key budget; draws on the co-op account team's budget. To be scoped like the per-agent keys below. The off switch revokes this first
Per-agent gateway keys (infrastructure-bot, later agents)	That agent's inference	That agent's .env	GM	Model allowlist and monthly budget enforced at the gateway
GREENPT_FALLBACK_API_KEY	Direct-to-provider fallback inference during a gateway outage (live since 2026-10-26)	Steward agent .env	Provider account owner	Rotation-coupled to the gateway's GREENPT_API_KEY: rotate one without the other and the fallback silently breaks. A dedicated key is the cleaner end state
HERMES_CUSTOM_API_GREENPT_AI_API_KEY	Provider key, still present	Steward agent .env	Provider account owner	Confirm whether still needed; retire if not
Restic repository passphrase	Nightly encrypted backup of the agent's state	Agent host and the co-op vault	GM
Infomaniak Swiss Backup S3 keys (agent)	Backup target	Agent host	Infomaniak account owner

Accounts (not tokens, but the things tokens come from)

Account	Purpose	Owner today	Notes
Cloudron admin (my.inference.coop)	Everything on the main server
Hostinger	Hosts the main Cloudron server	Personal account (co-op/support#2)	Moving to Infomaniak
Infomaniak	Agent VPS, Swiss Backup S3 for Cloudron and the agent		S3 keys for Cloudron backups rotated after setup (co-op/support#12)
Open Collective (collective admins)	Dues, webhooks, the personal token above
Provider accounts (Tinfoil, GreenPT, Regolo)	Billing and API keys
Domain registrar and DNS for inference.coop
Gitea, Matrix, Loomio, Listmonk admin accounts	Service administration		Cloudron SSO where applicable

Keeping this current

Add a row when a new integration lands, and strike one when a service is retired. A credential held by two apps is the thing most likely to break on rotation; the "Notes" column marks each one.