operator-token-runbook: add the credential rotation policy and reminder

Records the 12-month cadence and the trigger rules agreed in co-op/support#35,
a last-rotated table (first entry: the Open Collective token, rotated
2026-10-27; other rows blank because nothing is recorded), and the account map
behind the credentials. Adds scripts/rotation_due.py, which reads the table and
prints only when a credential is within 30 days of its due date.
This commit is contained in:
inference-bot committed 2026-11-04 14:36:57 +00:00
1 parent 63d14156ec
commit 43f8f8c3fe
2 files changed
+237 -2

No files matched your search

+81 -2
View File
@@ -3,6 +3,85 @@
How the co-op's Cloudron credentials are organized, and how to rotate them
without breaking the stack.
## Credential rotation policy
**Cadence: 12 months for every credential.** One interval, no exceptions for now.
Twelve months is a starting point, not a principle — revisit it once the token
list is stable (co-op/support#35).
**Rotate on a trigger immediately, off the calendar, when any of these happens:**
- **Offboarding** — anyone who had access to the credential leaves or changes
role, or the device/account it lived on is lost.
- **Suspected exposure** — the value was seen somewhere it should not be: a log,
a chat, a commit, a screenshot.
- **After any incident, or any diagnostic session on the agent host in which a
secret could have been read.** Reading a live secret to debug with it is itself
the trigger; it does not have to be a confirmed breach.
- **After any restore** — the credential travels in the backup, so a restored
host re-materialises whatever values the snapshot held.
**The GM controls all tokens for now** (operator, 2026-11-04), so this record
needs **no owner column**: every row is rotated by the GM. Provider keys are
minted on the provider account, but the value held in the co-op's environment is
still the GM's to rotate.
### Last rotated
One row per credential, or per credential group — a group rotates together, so it
dates together. A blank cell means **no rotation has been recorded**, not "not
rotated". Dates are UTC, `YYYY-MM-DD`.
The reminder reads this exact table: `scripts/rotation_due.py` in this repository
(deployed on the agent host at `~/.hermes/scripts/rotation_due.py`) prints a line
only when a due date is within 30 days, and prints nothing otherwise. Keep the
table machine-legible — one `YYYY-MM-DD` date per row.
| Credential or group | Last rotated (UTC) |
|---|---|
| Open Collective token (`OC_PERSONAL_TOKEN`) | 2026-10-27 |
| `LITELLM_MASTER_KEY` / `GW_MASTER_KEY` (one value, four holders) | |
| Cloudron API tokens — portal + agent (both on `inference-bot`) | |
| `REGISTRY_BUILD_TOKEN` (on `inference-admin`; moves to CI, co-op/support#30) | |
| Gitea token (`inference-bot`) | |
| `WEBHOOK_TOKEN` (portal env + Open Collective webhook URL) | |
| `ADMIN_TOKEN` (portal + admin panel + agent host) | |
| `BROKER_SECRET` and `PORTAL_SECRET` | |
| Matrix tokens — portal `MATRIX_BOT_TOKEN` + agent `MATRIX_ADMIN_TOKEN` (one account, `@inference-admin`) | |
| Matrix token — steward agent `@inferencebot` | |
| Loomio API keys (`LOOMIO_API_KEY`, `LOOMIO_B3_KEY`) | |
| Listmonk API (`LISTMONK_API_USER`, `LISTMONK_API_TOKEN`) | |
| Provider keys — Tinfoil, GreenPT, Regolo, and the GreenPT fallback | |
| Restic repository passphrase; Infomaniak Swiss Backup S3 keys | |
### Accounts behind the credentials
A credential belongs to an **account**, which is what a rotation is actually
performed against — ask the provider, do not read the variable name. Two
credentials on one account are a single hazard: rotating one and not the other is
the silent break.
| Credential | Account | Basis |
|---|---|---|
| `LITELLM_MASTER_KEY` / `GW_MASTER_KEY` | self-issued by the gateway | one literal, four holders |
| Cloudron API tokens — portal and agent | `inference-bot` | verified (co-op/support#35) |
| `REGISTRY_BUILD_TOKEN` | `inference-admin` | verified (co-op/support#35) |
| `MATRIX_BOT_TOKEN` (portal) + `MATRIX_ADMIN_TOKEN` (agent) | `@inference-admin:inference.coop` — two tokens, one account | verified, homeserver `whoami` 2026-11-04 |
| Steward agent's `MATRIX_ACCESS_TOKEN` | `@inferencebot:inference.coop` | verified, homeserver `whoami` 2026-11-04 |
| `OC_PERSONAL_TOKEN` | `info@inference.coop` | from the 2026-10-27 rotation record (inference) |
| `TINFOIL_API_KEY` | `info@inference.coop` | operator-confirmed |
| `GREENPT_API_KEY` (gateway) and `GREENPT_FALLBACK_API_KEY` (agent) | the GreenPT account — the **same value** in both | verified: the "fallback" isolates nothing |
| `REGOLO_API_KEY` | Regolo account | not yet identified |
| Gitea token | `inference-bot` | inferred, not verified |
| `BROKER_SECRET`, `PORTAL_SECRET`, `WEBHOOK_TOKEN`, `ADMIN_TOKEN`, `LOOMIO_*`, `LISTMONK_*` | self-issued / service account | — |
| Restic passphrase; Infomaniak Swiss Backup S3 keys | self-issued; the Infomaniak account | — |
The two couplings to watch: **one account with several credentials** (the
`@inference-admin` Matrix pair, the two Cloudron tokens, the four holders of the
master key) and **one value with several holders** (the GreenPT key, which is the
only such case found). Full account map, including the hash comparison behind the
GreenPT finding, is held with the security working notes.
## The two-token split
There are **two separate** Cloudron API tokens, deliberately decoupled so
@@ -202,7 +281,7 @@ row that names the same credential: several are held by more than one app.
| `WEBHOOK_TOKEN` | Secret in the Open Collective webhook URL | Portal app env, and the webhook URL on Open Collective | GM | Rotating means re-creating the webhook on OC as well |
| `LITELLM_MASTER_KEY` | Create gateway teams and keys for members | Portal app env (copy of the gateway's) | GM | Three holders: portal, gateway, steward agent. Rotate all together |
| `LITELLM_DB_*` | Read spend logs from the gateway database | Portal app env | Cloudron (addon) | Postgres credentials for the gateway's database |
| `MATRIX_BOT_TOKEN` | `@inference-bot:inference.coop` admin account: invites, space membership | Portal app env | GM | The account has no password. Paired with the steward agent's `MATRIX_LOCAL_BOT_TOKEN`: same account, separate token on a separate device (verified). Revoke or rotate both together |
| `MATRIX_BOT_TOKEN` | `@inference-admin:inference.coop` admin account: invites, space membership | Portal app env | GM | The account has no password. Paired with the steward agent's `MATRIX_ADMIN_TOKEN`: same account, separate token on a separate device (verified by homeserver `whoami`, 2026-11-04). Revoke or rotate both together |
| `LOOMIO_API_KEY`, `LOOMIO_B3_KEY` | Forum (Loomio) membership sync | Portal app env | GM | |
| `LISTMONK_API_USER`, `LISTMONK_API_TOKEN` | Newsletter list sync | Portal app env | GM | |
| `OWUI_API_KEY`, `OWUI_JWT_SECRET` | Open WebUI knowledge refresh and SSO | Portal app env | GM | Chat moved to LibreChat in October; confirm whether still needed, and retire if not |
@@ -236,7 +315,7 @@ granularity for the public half. Highest privilege first.
| Credential | Purpose | Where it lives | Who rotates | Notes |
|---|---|---|---|---|
| Gateway master key (`GW_MASTER_KEY`) | Admin over every gateway team, key and spend record | Steward agent `.env` | GM | **Third holder**, with the portal and the gateway. The row most in need of coordinated rotation |
| `MATRIX_LOCAL_BOT_TOKEN` | The `@inference-bot:inference.coop` automation account: room administration in every co-op room | Steward agent `.env` | GM | Same account as the portal's `MATRIX_BOT_TOKEN`, but a separate token on a separate device (verified). Revoking one does not revoke the other; revoke or rotate both together |
| `MATRIX_ADMIN_TOKEN` | The `@inference-admin:inference.coop` automation account: room administration in every co-op room | Steward agent `.env` | GM | Same account as the portal's `MATRIX_BOT_TOKEN`, but a separate token on a separate device (verified by homeserver `whoami`, 2026-11-04). Revoking one does not revoke the other; revoke or rotate both together |
| Cloudron agent token | Cloudron API | Steward agent `.env` | GM | See "The two-token split" above |
| `REGISTRY_BUILD_TOKEN` (and its username) | Image build and push for the custom apps | Steward agent `.env` | GM | Moves to CI (co-op/support#30); the agent stops holding it then |
| Gitea token | Commits, issues, PRs as `inference-bot` | Steward agent `.env` | GM | |