ops-docs/operator-token-runbook.md

# Operator runbook — Cloudron tokens

How the co-op's Cloudron credentials are organized, and how to rotate them
without breaking the stack.

## Credential rotation policy

**Cadence: 12 months for every credential.** One interval, no exceptions for now.
Twelve months is a starting point, not a principle — revisit it once the token
list is stable (co-op/support#35).

**Rotate on a trigger immediately, off the calendar, when any of these happens:**

- **Offboarding** — anyone who had access to the credential leaves or changes
  role, or the device/account it lived on is lost.
- **Suspected exposure** — the value was seen somewhere it should not be: a log,
  a chat, a commit, a screenshot.
- **After any incident, or any diagnostic session on the agent host in which a
  secret could have been read.** Reading a live secret to debug with it is itself
  the trigger; it does not have to be a confirmed breach.
- **After any restore** — the credential travels in the backup, so a restored
  host re-materialises whatever values the snapshot held.

**The GM controls all tokens for now** (operator, 2026-11-04), so this record
needs **no owner column**: every row is rotated by the GM. Provider keys are
minted on the provider account, but the value held in the co-op's environment is
still the GM's to rotate.

### Last rotated

One row per credential, or per credential group — a group rotates together, so it
dates together. A blank cell means **no rotation has been recorded**, not "not
rotated". Dates are UTC, `YYYY-MM-DD`.

The reminder reads this exact table: `scripts/rotation_due.py` in this repository
(deployed on the agent host at `~/.hermes/scripts/rotation_due.py`) prints a line
only when a due date is within 30 days, and prints nothing otherwise. Keep the
table machine-legible — one `YYYY-MM-DD` date per row.

| Credential or group | Last rotated (UTC) |
|---|---|
| Open Collective token (`OC_PERSONAL_TOKEN`) | 2026-10-27 |
| `LITELLM_MASTER_KEY` / `GW_MASTER_KEY` (one value, four holders) | |
| Cloudron API tokens — portal + agent (both on `inference-bot`) | |
| `REGISTRY_BUILD_TOKEN` (on `inference-admin`; moves to CI, co-op/support#30) | |
| Gitea token (`inference-bot`) | |
| `WEBHOOK_TOKEN` (portal env + Open Collective webhook URL) | |
| `ADMIN_TOKEN` (portal + admin panel + agent host) | |
| `BROKER_SECRET` and `PORTAL_SECRET` | |
| Matrix tokens — portal `MATRIX_BOT_TOKEN` + agent `MATRIX_ADMIN_TOKEN` (one account, `@inference-admin`) | |
| Matrix token — steward agent `@inferencebot` | |
| Loomio API keys (`LOOMIO_API_KEY`, `LOOMIO_B3_KEY`) | |
| Listmonk API (`LISTMONK_API_USER`, `LISTMONK_API_TOKEN`) | |
| Provider keys — Tinfoil, GreenPT, Regolo, and the GreenPT fallback | |
| Restic repository passphrase; Infomaniak Swiss Backup S3 keys | |

### Accounts behind the credentials

A credential belongs to an **account**, which is what a rotation is actually
performed against — ask the provider, do not read the variable name. Two
credentials on one account are a single hazard: rotating one and not the other is
the silent break.

| Credential | Account | Basis |
|---|---|---|
| `LITELLM_MASTER_KEY` / `GW_MASTER_KEY` | self-issued by the gateway | one literal, four holders |
| Cloudron API tokens — portal and agent | `inference-bot` | verified (co-op/support#35) |
| `REGISTRY_BUILD_TOKEN` | `inference-admin` | verified (co-op/support#35) |
| `MATRIX_BOT_TOKEN` (portal) + `MATRIX_ADMIN_TOKEN` (agent) | `@inference-admin:inference.coop` — two tokens, one account | verified, homeserver `whoami` 2026-11-04 |
| Steward agent's `MATRIX_ACCESS_TOKEN` | `@inferencebot:inference.coop` | verified, homeserver `whoami` 2026-11-04 |
| `OC_PERSONAL_TOKEN` | `info@inference.coop` | from the 2026-10-27 rotation record (inference) |
| `TINFOIL_API_KEY` | `info@inference.coop` | operator-confirmed |
| `GREENPT_API_KEY` (gateway) and `GREENPT_FALLBACK_API_KEY` (agent) | the GreenPT account — the **same value** in both | verified: the "fallback" isolates nothing |
| `REGOLO_API_KEY` | Regolo account | not yet identified |
| Gitea token | `inference-bot` | inferred, not verified |
| `BROKER_SECRET`, `PORTAL_SECRET`, `WEBHOOK_TOKEN`, `ADMIN_TOKEN`, `LOOMIO_*`, `LISTMONK_*` | self-issued / service account | — |
| Restic passphrase; Infomaniak Swiss Backup S3 keys | self-issued; the Infomaniak account | — |

The two couplings to watch: **one account with several credentials** (the
`@inference-admin` Matrix pair, the two Cloudron tokens, the four holders of the
master key) and **one value with several holders** (the GreenPT key, which is the
only such case found). Full account map, including the hash comparison behind the
GreenPT finding, is held with the security working notes.

## The two-token split

There are **two separate** Cloudron API tokens, deliberately decoupled so
rotating one doesn't break the other:

| Token | Purpose | Where it lives |
|-------|---------|----------------|
| **Agent token** | Inferencebot's own Cloudron access | The agent's private `.env` (gitignored) |
| **Portal token** | The member portal's Cloudron access (create users, set groups, invite links) | Cloudron's env store, under the portal app's `CLOUDRON_TOKEN` |

The portal needs only a handful of Cloudron capabilities (user/group management,
invite links). It is the most exposed app (public webhook + broker endpoints),
so it must **not** hold the agent's token — a portal compromise shouldn't reach
the agent's wider Cloudron access.

## Rotating a token

### Portal token

1. In `my.inference.coop` → **Settings → API Tokens**, create a **"Read and Write"** token.
2. Set it as the portal's `CLOUDRON_TOKEN` env var (custom apps have no env-var
   UI — use the Cloudron CLI or the API; see the next section).
3. Verify: the portal can reach `GET /api/v1/users` with the new token (HTTP 200).

### Agent token

1. Create a fresh token in `Settings → API Tokens`.
2. Put it in the agent's `.env` as `CLOUDRON_TOKEN`.
3. Verify against `/api/v1/cloudron/status`.

**The trap this split exists to prevent:** the portal keeps its *own copy* of a
Cloudron token. If the agent's token is rotated without also updating the portal
copy, the portal's Cloudron calls start returning `401` — which breaks
provisioning (new members can't be created) and the member dashboard (its
usage/keys data is resolved through the portal's Cloudron calls).

## Admin token vs webhook token (portal)

Two more, separate from the Cloudron tokens above:

| Token | Purpose | Where it lives |
|-------|---------|----------------|
| **`ADMIN_TOKEN`** | Authenticates `/admin/*`; sent ONLY as an `X-Admin-Token` header | Portal env; the **admin panel's own env has a separate copy**; `~/.hermes/.env` on the agent host |
| **`WEBHOOK_TOKEN`** | Authenticates Open Collective's inbound webhook | Portal env, and inside the webhook URL configured on the collective |

They were once one value, so anything that learned the webhook URL — OC admins,
OC's logs, our access logs, a Referer — got portal admin. Split 2026-10-27; the
cutover completed 2026-10-28 (`ADMIN_TOKEN_LEGACY=0`), verified on the wire:
`ADMIN_TOKEN` → HTTP 200, `WEBHOOK_TOKEN` → 403, no header → 403.

- **`WEBHOOK_TOKEN` is not retired and cannot be.** OC's generic webhooks cannot
  send headers, so a secret in the URL is the standard authentication. Its only
  power now is authenticating that one inbound route, which is the intended end
  state.
- **Rotating `ADMIN_TOKEN` touches THREE places** — the portal env (shared by the
  app and the scheduler addon), the admin panel's separate copy, and the agent's
  `~/.hermes/.env`. Miss one and that caller 403s silently.
- **`WEBHOOK_TOKEN` was rotated on 2026-10-27**, as a precaution after a test
  handled the URL. Rotating it *again* means updating BOTH the portal env and the
  webhook URL configured on the collective, which needs Open Collective admin
  access — the same blocker as moving `OC_PERSONAL_TOKEN` (below) to a
  co-op-owned account.
- **Whether the collective's webhook URL matches the current token is still
  unconfirmed** (as of 2026-10-28). The route compares the URL token against the
  env var, so a mismatch means OC's deliveries 401 and new-member provisioning
  waits for the nightly sweep — delayed, not lost.
- **To confirm it directly**, compare by hash rather than by reading the value
  out: take the URL's last path segment in the collective's webhook settings and
  run `printf %s '<token>' | sha256sum` LOCALLY, then compare its first 12 hex
  characters against the same prefix of the portal's `WEBHOOK_TOKEN`. A hash
  prefix of a live secret does not belong in this file or in chat.
- **The monitor already covers the failure case.** The `portal webhook delivery
  watch` cron job alerts on ANY failed delivery (401/404 = OC is posting to a URL
  the portal rejects) and stays silent on the happy path, which is why silence in
  its output means "nothing to report" and not "blind". Verified 2026-10-28 by
  posting a benign event with the current token: the portal answered 200 and the
  job reported it with `--verbose`. Run it with `--verbose` to see accepted
  deliveries; run it plainly to see only problems.
- **No caller may fall back from `ADMIN_TOKEN` to `WEBHOOK_TOKEN`.** The legacy
  token 403s, so a fallback turns "unset" into a confusing mid-run failure
  instead of a clear abort.

## Chat image: spaCy model for document parsing (EPUB/DOCX)

Open WebUI's document parser (`unstructured`) lazily downloads the spaCy model
`en_core_web_sm` on first EPUB/DOCX parse — but the app directory is read-only
under Cloudron, so that install always fails ("Errno 30 Read-only file
system"). Fixed 2026-10-14 (issue #20) by pre-installing the model into
persistent storage:

- `/app/data/spacy_model/` holds the unpacked wheel
  (SHA-verified `en_core_web_sm-3.8.0-py3-none-any.whl` from the pinned URL in
  `unstructured/nlp/tokenize.py`).
- App env has `PYTHONPATH=/app/data/spacy_model` so `import en_core_web_sm`
  resolves and the lazy download never triggers.

**Restore caveat:** a backup restore from before 2026-10-14 will NOT contain
the model dir — re-create it (download the pinned wheel, unzip into
`/app/data/spacy_model/`, verify `spacy.load("en_core_web_sm")` works in the
container). The `PYTHONPATH` env survives restores (it's in app config).

## Scheduled tasks (scheduler addon)

The portal and gateway run their own housekeeping via the Cloudron scheduler
addon, declared in each app's `CloudronManifest.json` under `addons.scheduler`:

- **portal**: `membership_sweep` (three times daily — `0 3,11,19 * * *` UTC per
  the live manifest; earlier notes said "nightly at 3am" and are stale. Runs
  `/app/code/sweep.sh` → `POST /admin/sweep` with the **`ADMIN_TOKEN`** header — NOT `WEBHOOK_TOKEN`,
  which returns 403 on `/admin/*` since the 2026-10-28 cutover), `rag_refresh`
  (nightly 3:30am, `/app/code/rag_refresh.py` — pulls docs from the Gitea raw
  API and refreshes the chat knowledge base; needs `OWUI_API_KEY` +
  `DOCS_KNOWLEDGE_ID` in the app env), and `metrics_snapshot` (nightly 3:15am,
  `/app/code/metrics_snapshot.py` — writes the day's members / usage / energy /
  carbon row into the portal's `metrics_daily` table; `--backfill` rebuilds
  history and is idempotent).
- **gateway**: `audio_keepalive` (every 10 minutes, `/app/code/audio_keepalive.py`
  — pings the Tinfoil audio models locally to avoid the ~24s TEE cold-start;
  reads the master key from `/app/data/.master_key`, which `start.sh` writes —
  the env var `LITELLM_MASTER_KEY` is only exported into the litellm process).

**After any manifest change, verify the scheduler survived the deploy:** the
build log must show `setupAddons: setting up addon scheduler with options
{...}` listing your task(s), and each script must succeed when run manually
inside the container (`cloudron exec --app <id> -- <command>`). The scheduler
has historically gone missing from the manifest during edits — a deploy without
the setupAddons line means nightly reconciliation is NOT running.

## Custom apps have no env-var UI

The member portal is a *custom* app (packaged by us), not an App Store app.
Custom apps do **not** expose environment variables in the Cloudron dashboard.
Env vars for custom apps are set via:

- **Cloudron CLI** (on your workstation, *not* the server): `cloudron env set --app portal.inference.coop CLOUDRON_TOKEN=...`
- **Cloudron API**: `POST /apps/:appId/configure/env`

## `configure/env` is a full replace

`POST /apps/:appId/configure/env` with body `{"env": {KEY: value, ...}}`
**replaces the entire environment** — it does not merge. Sending a partial map
(e.g. a single test variable) wipes every other variable.

- To change **one** variable, read the current env first (`GET /api/v1/apps/:id`
  → `.env`), merge in memory, and POST the full map back.
- Never probe the endpoint with a throwaway body against a live app.
- If the env is wiped, it is recoverable from a Cloudron app backup:
  `GET /api/v1/apps/:appId/backups` → newest entry → `GET .../backups/:backupId/download`
  (a `.tar.gz` containing `config.json`, whose `.env` field holds the original map).

## Reference

- Cloudron CLI: `npm install -g cloudron` (install on your workstation, not the server), then `cloudron login my.inference.coop`.
- Cloudron API docs: `docs.cloudron.io/api/`.

## Open Collective: the bot token needs the `email` scope (Sept 2026)

The portal identifies members by email, read via a **personal** OC token
(`OC_PERSONAL_TOKEN`) — currently Nathan's own rather than a co-op-owned
account's (open item F5); the fix is a co-op account plus a token minted there as `collective.members.nodes[].account.email` with the
`... on Individual { email }` inline fragment.

**The `email` scope is load-bearing.** The bot's personal token must be minted
with the `email` scope. Without it, the GraphQL `email` field returns `null` for
*every* member — including the bot's own `me { email }` — even though:

- the bot still authenticates and reports `isAdmin: true`, and
- the OC **admin CSV export still shows emails** (it's a separate, non-GraphQL
  path; its `contributorEmail` column is not a GraphQL field).

**Symptoms when the scope is missing:** `fetch_members()` filters out empty
emails and returns `[]`, so the webhook can't provision anyone by email and the
nightly sweep goes into its empty-list fail-safe (skips, doesn't deactivate).
A new member's webhook fires cleanly but no "Sent email to …" log line follows,
and no Cloudron user is created. (This is easy to misread as a "guest with no
email" problem — it is not; verify the scope first.)

**Diagnosis:** run `me { email }` with the bot token. `null` here while
`isAdmin: true` → the token is missing the `email` scope (or the scope was
revoked). Fix: re-add the `email` scope to the token on Open Collective
(Settings → Developers → Personal Tokens). No token rotation, no re-granting of
admin — just the scope.

## Credential inventory (draft, 2026-10-26)

Every credential the agents and the member-facing apps hold, compiled from the
public code and docs, corrected against what the agent holds (PR #7 review). **No secret values appear here, only names and roles.**
"Where it lives" and "Who rotates" are filled in where the docs say; blank
cells are for the operator to complete. When rotating anything, check every
row that names the same credential: several are held by more than one app.

### Member portal (`portal.inference.coop`)

| Credential | Purpose | Where it lives | Who rotates | Notes |
|---|---|---|---|---|
| `CLOUDRON_TOKEN` (portal token) | Create users and groups, invite links | Portal app env | GM | See "The two-token split" above |
| `OC_PERSONAL_TOKEN` | Read the member list from Open Collective GraphQL | Portal app env, and the vault | GM, as `info@inference.coop` | Co-op-owned since 2026-10-27: minted by `info@inference.coop`, an Admin of the collective (OC returns member emails only to admins). Scopes `collective`, `individual`, `email`, read-only; the credit-pack order lookup is an anonymous query and needs no token. No expiry set; if one is added, record it here. Rotate on suspected exposure or when the account's admin role changes. The previous personal token has been revoked (confirmed by the operator, 2026-10-27) |
| `WEBHOOK_TOKEN` | Secret in the Open Collective webhook URL | Portal app env, and the webhook URL on Open Collective | GM | Rotating means re-creating the webhook on OC as well |
| `LITELLM_MASTER_KEY` | Create gateway teams and keys for members | Portal app env (copy of the gateway's) | GM | Three holders: portal, gateway, steward agent. Rotate all together |
| `LITELLM_DB_*` | Read spend logs from the gateway database | Portal app env | Cloudron (addon) | Postgres credentials for the gateway's database |
| `MATRIX_BOT_TOKEN` | `@inference-admin:inference.coop` admin account: invites, space membership | Portal app env | GM | The account has no password. Paired with the steward agent's `MATRIX_ADMIN_TOKEN`: same account, separate token on a separate device (verified by homeserver `whoami`, 2026-11-04). Revoke or rotate both together |
| `LOOMIO_API_KEY`, `LOOMIO_B3_KEY` | Forum (Loomio) membership sync | Portal app env | GM | |
| `LISTMONK_API_USER`, `LISTMONK_API_TOKEN` | Newsletter list sync | Portal app env | GM | |
| `OWUI_API_KEY`, `OWUI_JWT_SECRET` | Open WebUI knowledge refresh and SSO | Portal app env | GM | Chat moved to LibreChat in October; confirm whether still needed, and retire if not |
| `BROKER_SECRET` | Shared secret between the portal and the member dashboard | Portal and dashboard app env | GM | Held by both; rotate together |
| `PORTAL_SECRET` | Session signing | Portal app env | GM | |
| `CLOUDRON_MAIL_SMTP_*` | Outbound mail | Cloudron mail addon | Cloudron | Managed by the platform |

### Gateway (`gateway.inference.coop`)

| Credential | Purpose | Where it lives | Who rotates | Notes |
|---|---|---|---|---|
| `LITELLM_MASTER_KEY` | Gateway admin | `/app/data/.master_key`, exported into the litellm process by `start.sh` | GM | Also held by the portal (above), the steward agent (below) and the audio keep-alive job |
| Salt key (`SALT_KEY_FILE`) | Encrypts provider keys stored in the gateway database | `/app/data` | GM | Losing it means re-entering every provider key |
| `TINFOIL_API_KEY` | Provider account | Gateway app env | Provider account owner | Billing lives at the provider |
| `GREENPT_API_KEY` | Provider account | Gateway app env | Provider account owner | Billing lives at the provider |
| `REGOLO_API_KEY` | Provider account | Gateway app env | Provider account owner | Billing lives at the provider |
| `CLOUDRON_POSTGRESQL_*`, `CLOUDRON_REDIS_*` | Database and cache | Cloudron addons | Cloudron | Managed by the platform |

### Chat (LibreChat, `chat.inference.coop`)

| Credential | Purpose | Where it lives | Who rotates | Notes |
|---|---|---|---|---|
| LibreChat encryption keys and app secrets | App secrets; the per-member key sync runs in the app's own Cloudron scheduler | App data directory | GM | Not held by the agent |
| `RAG_OPENAI_API_KEY` (gateway key alias `rag-service-embeddings`) | LibreChat RAG service — embeddings for chat document upload/parse | LibreChat app `/app/data/env` (mode 600), as `RAG_OPENAI_API_KEY` + `RAG_OPENAI_BASEURL` | Gateway master-key holder (`POST /key/regenerate`) | One model only: `tinfoil/nomic-embed-text` (TEE, 768-dim); key budget cap $2 / 30d; on the co-op `info@inference.coop` team, so member keys cannot call it. Live 2026-10-27 (the service had not started since the Cloudron move). Key value recorded in no repository; rotate via the gateway, not a dashboard |

### Agents (on the agent host)

Exact file paths are deliberately not listed; "agent `.env`" is the right
granularity for the public half. Highest privilege first.

| Credential | Purpose | Where it lives | Who rotates | Notes |
|---|---|---|---|---|
| Gateway master key (`GW_MASTER_KEY`) | Admin over every gateway team, key and spend record | Steward agent `.env` | GM | **Third holder**, with the portal and the gateway. The row most in need of coordinated rotation |
| `MATRIX_ADMIN_TOKEN` | The `@inference-admin:inference.coop` automation account: room administration in every co-op room | Steward agent `.env` | GM | Same account as the portal's `MATRIX_BOT_TOKEN`, but a separate token on a separate device (verified by homeserver `whoami`, 2026-11-04). Revoking one does not revoke the other; revoke or rotate both together |
| Cloudron agent token | Cloudron API | Steward agent `.env` | GM | See "The two-token split" above |
| `REGISTRY_BUILD_TOKEN` (and its username) | Image build and push for the custom apps | Steward agent `.env` | GM | Moves to CI (co-op/support#30); the agent stops holding it then |
| Gitea token | Commits, issues, PRs as `inference-bot` | Steward agent `.env` | GM | |
| Matrix access token (`@inferencebot:inference.coop`) | Reading and posting in rooms | Agent host, with its E2EE key store | GM | The off switch in [Working with the agent](working-with-the-agent.md) revokes this second |
| Steward agent's gateway key (`api:info@inference.coop:inferencebots`) | The steward agent's own inference | Steward agent `.env` | GM | Today: empty model allowlist and no per-key budget; draws on the co-op account team's budget. To be scoped like the per-agent keys below. The off switch revokes this first |
| Per-agent gateway keys (`infrastructure-bot`, later agents) | That agent's inference | That agent's `.env` | GM | Model allowlist and monthly budget enforced at the gateway |
| `GREENPT_FALLBACK_API_KEY` | Direct-to-provider fallback inference during a gateway outage (live since 2026-10-26) | Steward agent `.env` | Provider account owner | Rotation-coupled to the gateway's `GREENPT_API_KEY`: rotate one without the other and the fallback silently breaks. A dedicated key is the cleaner end state |
| `HERMES_CUSTOM_API_GREENPT_AI_API_KEY` | Provider key, still present | Steward agent `.env` | Provider account owner | Confirm whether still needed; retire if not |
| Restic repository passphrase | Nightly encrypted backup of the agent's state | Agent host and the co-op vault | GM | |
| Infomaniak Swiss Backup S3 keys (agent) | Backup target | Agent host | Infomaniak account owner | |

### Accounts (not tokens, but the things tokens come from)

| Account | Purpose | Owner today | Notes |
|---|---|---|---|
| Cloudron admin (`my.inference.coop`) | Everything on the main server | | |
| Hostinger | Hosts the main Cloudron server | Personal account (co-op/support#2) | Moving to Infomaniak |
| Infomaniak | Agent VPS, Swiss Backup S3 for Cloudron and the agent | | S3 keys for Cloudron backups rotated after setup (co-op/support#12) |
| Open Collective (collective admins) | Dues, webhooks, the personal token above | | |
| Provider accounts (Tinfoil, GreenPT, Regolo) | Billing and API keys | | |
| Domain registrar and DNS for `inference.coop` | | | |
| Gitea, Matrix, Loomio, Listmonk admin accounts | Service administration | | Cloudron SSO where applicable |

### Keeping this current

Add a row when a new integration lands, and strike one when a service is
retired. A credential held by two apps is the thing most likely to break on
rotation; the "Notes" column marks each one.