Add operator token runbook (two-token split, configure/env gotcha)

infrastructure.md

@@ -136,3 +136,9 @@ architecturally private versus what's a policy commitment — is in the
   plaintext in the portal database (functionally necessary), and the gateway
   must stay publicly reachable for API access (its admin UI and docs are
   disabled; only the `/v1/*` API is exposed).
+
+## For operators
+
+Maintaining the co-op's infrastructure — rotating Cloudron tokens, editing the
+portal's environment, and troubleshooting the provisioning pipeline — is
+documented in the [operator runbook](operator-token-runbook.md).

operator-token-runbook.md

@@ -0,0 +1,67 @@
+# Operator runbook — Cloudron tokens
+
+How the co-op's Cloudron credentials are organized, and how to rotate them
+without breaking the stack.
+
+## The two-token split
+
+There are **two separate** Cloudron API tokens, deliberately decoupled so
+rotating one doesn't break the other:
+
+| Token | Purpose | Where it lives |
+|-------|---------|----------------|
+| **Agent token** | Inferencebot's own Cloudron access | The agent's private `.env` (gitignored) |
+| **Portal token** | The member portal's Cloudron access (create users, set groups, invite links) | Cloudron's env store, under the portal app's `CLOUDRON_TOKEN` |
+
+The portal needs only a handful of Cloudron capabilities (user/group management,
+invite links). It is the most exposed app (public webhook + broker endpoints),
+so it must **not** hold the agent's token — a portal compromise shouldn't reach
+the agent's wider Cloudron access.
+
+## Rotating a token
+
+### Portal token
+
+1. In `my.inference.coop` → **Settings → API Tokens**, create a **"Read and Write"** token.
+2. Set it as the portal's `CLOUDRON_TOKEN` env var (custom apps have no env-var
+   UI — use the Cloudron CLI or the API; see the next section).
+3. Verify: the portal can reach `GET /api/v1/users` with the new token (HTTP 200).
+
+### Agent token
+
+1. Create a fresh token in `Settings → API Tokens`.
+2. Put it in the agent's `.env` as `CLOUDRON_TOKEN`.
+3. Verify against `/api/v1/cloudron/status`.
+
+**The trap this split exists to prevent:** the portal keeps its *own copy* of a
+Cloudron token. If the agent's token is rotated without also updating the portal
+copy, the portal's Cloudron calls start returning `401` — which breaks
+provisioning (new members can't be created) and the member dashboard (its
+usage/keys data is resolved through the portal's Cloudron calls).
+
+## Custom apps have no env-var UI
+
+The member portal is a *custom* app (packaged by us), not an App Store app.
+Custom apps do **not** expose environment variables in the Cloudron dashboard.
+Env vars for custom apps are set via:
+
+- **Cloudron CLI** (on your workstation, *not* the server): `cloudron env set --app portal.inference.coop CLOUDRON_TOKEN=...`
+- **Cloudron API**: `POST /apps/:appId/configure/env`
+
+## `configure/env` is a full replace
+
+`POST /apps/:appId/configure/env` with body `{"env": {KEY: value, ...}}`
+**replaces the entire environment** — it does not merge. Sending a partial map
+(e.g. a single test variable) wipes every other variable.
+
+- To change **one** variable, read the current env first (`GET /api/v1/apps/:id`
+  → `.env`), merge in memory, and POST the full map back.
+- Never probe the endpoint with a throwaway body against a live app.
+- If the env is wiped, it is recoverable from a Cloudron app backup:
+  `GET /api/v1/apps/:appId/backups` → newest entry → `GET .../backups/:backupId/download`
+  (a `.tar.gz` containing `config.json`, whose `.env` field holds the original map).
+
+## Reference
+
+- Cloudron CLI: `npm install -g cloudron` (install on your workstation, not the server), then `cloudron login my.inference.coop`.
+- Cloudron API docs: `docs.cloudron.io/api/`.