From 5aa6098e5bcd21f7998ef2ef67ac5144d8f9264f Mon Sep 17 00:00:00 2001 From: Angel Marino Date: Mon, 26 Oct 2026 01:08:47 -0400 Subject: [PATCH 1/3] operator-token-runbook: add a credential inventory (every token the agent and apps hold, where it lives, who rotates) Drafted from the public code and docs, so names and roles only, never values. Blank cells are for the operator to fill in. Marks the credentials held by more than one app, which are the ones most likely to break on rotation. Offered on co-op/support#17. --- operator-token-runbook.md | 67 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+) diff --git a/operator-token-runbook.md b/operator-token-runbook.md index 6fabfc1..c047a78 100644 --- a/operator-token-runbook.md +++ b/operator-token-runbook.md @@ -133,3 +133,70 @@ email" problem — it is not; verify the scope first.) revoked). Fix: re-add the `email` scope to the token on Open Collective (Settings → Developers → Personal Tokens). No token rotation, no re-granting of admin — just the scope. + +## Credential inventory (draft, 2026-10-26) + +Every credential the agent and the member-facing apps hold, compiled from the +public code and docs. **No secret values appear here, only names and roles.** +"Where it lives" and "Who rotates" are filled in where the docs say; blank +cells are for the operator to complete. When rotating anything, check every +row that names the same credential: several are held by more than one app. + +### Member portal (`portal.inference.coop`) + +| Credential | Purpose | Where it lives | Who rotates | Notes | +|---|---|---|---|---| +| `CLOUDRON_TOKEN` (portal token) | Create users and groups, invite links | Portal app env | GM | See "The two-token split" above | +| `OC_PERSONAL_TOKEN` | Read members and orders from Open Collective GraphQL | Portal app env | The Open Collective account that minted it | Needs the `email` scope (see below). Currently a personal token; a co-op-owned OC account would make it a co-op credential | +| `WEBHOOK_TOKEN` | Secret in the Open Collective webhook URL | Portal app env, and the webhook URL on Open Collective | GM | Rotating means re-creating the webhook on OC as well | +| `LITELLM_MASTER_KEY` | Create gateway teams and keys for members | Portal app env (copy of the gateway's) | GM | Also held by the gateway; rotate both together | +| `LITELLM_DB_*` | Read spend logs from the gateway database | Portal app env | Cloudron (addon) | Postgres credentials for the gateway's database | +| `MATRIX_BOT_TOKEN` | `@inference-bot:inference.coop` admin account: invites, space membership | Portal app env | GM | The account has no password; the token is the only credential | +| `LOOMIO_API_KEY`, `LOOMIO_B3_KEY` | Forum (Loomio) membership sync | Portal app env | GM | | +| `LISTMONK_API_USER`, `LISTMONK_API_TOKEN` | Newsletter list sync | Portal app env | GM | | +| `OWUI_API_KEY`, `OWUI_JWT_SECRET` | Open WebUI knowledge refresh and SSO | Portal app env | GM | Chat moved to LibreChat in October; confirm whether still needed, and retire if not | +| `BROKER_SECRET` | Shared secret between the portal and the member dashboard | Portal and dashboard app env | GM | Held by both; rotate together | +| `PORTAL_SECRET` | Session signing | Portal app env | GM | | +| `CLOUDRON_MAIL_SMTP_*` | Outbound mail | Cloudron mail addon | Cloudron | Managed by the platform | + +### Gateway (`gateway.inference.coop`) + +| Credential | Purpose | Where it lives | Who rotates | Notes | +|---|---|---|---|---| +| `LITELLM_MASTER_KEY` | Gateway admin | `/app/data/.master_key`, exported into the litellm process by `start.sh` | GM | Also held by the portal (above) and the audio keep-alive job | +| Salt key (`SALT_KEY_FILE`) | Encrypts provider keys stored in the gateway database | `/app/data` | GM | Losing it means re-entering every provider key | +| `TINFOIL_API_KEY` | Provider account | Gateway app env | Provider account owner | Billing lives at the provider | +| `GREENPT_API_KEY` | Provider account | Gateway app env | Provider account owner | Billing lives at the provider | +| `REGOLO_API_KEY` | Provider account | Gateway app env | Provider account owner | Billing lives at the provider | +| `CLOUDRON_POSTGRESQL_*`, `CLOUDRON_REDIS_*` | Database and cache | Cloudron addons | Cloudron | Managed by the platform | + +### Agent (inferencebot, on the agent VPS) + +| Credential | Purpose | Where it lives | Who rotates | Notes | +|---|---|---|---|---| +| Cloudron agent token | Cloudron API | Agent `.env` | GM | See "The two-token split" above | +| Gitea token | Commits, issues, PRs as `inference-bot` | Agent `.env` | GM | | +| Matrix access token (`@inferencebot:inference.coop`) | Reading and posting in rooms | Agent host, with its E2EE key store | GM | The off switch in [Working with the agent](working-with-the-agent.md) revokes this second | +| Gateway virtual key | The agent's own inference | Agent `.env` | GM | Model allowlist and budget live on this key; the off switch revokes it first | +| Direct provider key (fallback) | Inference during a gateway outage | Agent `.env` | Provider account owner | Planned on co-op/support#17 | +| LibreChat admin credential | Per-member key sync job | Agent `.env` | GM | Inferred from the job list on #17 | +| Restic repository passphrase | Nightly encrypted backup of the agent's state | Agent host and the co-op vault | GM | Per #17 | +| Infomaniak Swiss Backup S3 keys (agent) | Backup target | Agent host | Infomaniak account owner | Per #17 | + +### Accounts (not tokens, but the things tokens come from) + +| Account | Purpose | Owner today | Notes | +|---|---|---|---| +| Cloudron admin (`my.inference.coop`) | Everything on the main server | | | +| Hostinger | Hosts the main Cloudron server | Personal account (co-op/support#2) | Moving to Infomaniak | +| Infomaniak | Agent VPS, Swiss Backup S3 for Cloudron and the agent | | S3 keys for Cloudron backups rotated after setup (co-op/support#12) | +| Open Collective (collective admins) | Dues, webhooks, the personal token above | | | +| Provider accounts (Tinfoil, GreenPT, Regolo) | Billing and API keys | | | +| Domain registrar and DNS for `inference.coop` | | | | +| Gitea, Matrix, Loomio, Listmonk admin accounts | Service administration | | Cloudron SSO where applicable | + +### Keeping this current + +Add a row when a new integration lands, and strike one when a service is +retired. A credential held by two apps is the thing most likely to break on +rotation; the "Notes" column marks each one. -- 2.53.0 From 11ea40485cf30ff81e6000c742d829931f8f8c74 Mon Sep 17 00:00:00 2001 From: Angel Marino Date: Mon, 26 Oct 2026 01:50:04 -0400 Subject: [PATCH 2/3] operator-token-runbook: apply the agent's review (master key as third holder, split gateway keys, fallback live, registry and Matrix admin tokens, LibreChat row moved) - The gateway master key is held by the steward agent too; marked as the three-holder row in all three places. - The steward agent's gateway key is unscoped today; per-agent scoped keys are a separate row. - Fallback inference is live, not planned; names the key and its rotation coupling to the gateway's GreenPT key. - Adds MATRIX_LOCAL_BOT_TOKEN, REGISTRY_BUILD_TOKEN (moving to CI, #30) and the leftover GreenPT key to confirm or retire. - The LibreChat credential moved to a Chat table; the agent no longer holds it. Exact file paths deliberately left out. --- operator-token-runbook.md | 37 +++++++++++++++++++++++++------------ 1 file changed, 25 insertions(+), 12 deletions(-) diff --git a/operator-token-runbook.md b/operator-token-runbook.md index c047a78..86496a6 100644 --- a/operator-token-runbook.md +++ b/operator-token-runbook.md @@ -136,8 +136,8 @@ admin — just the scope. ## Credential inventory (draft, 2026-10-26) -Every credential the agent and the member-facing apps hold, compiled from the -public code and docs. **No secret values appear here, only names and roles.** +Every credential the agents and the member-facing apps hold, compiled from the +public code and docs, corrected against what the agent holds (PR #7 review). **No secret values appear here, only names and roles.** "Where it lives" and "Who rotates" are filled in where the docs say; blank cells are for the operator to complete. When rotating anything, check every row that names the same credential: several are held by more than one app. @@ -149,7 +149,7 @@ row that names the same credential: several are held by more than one app. | `CLOUDRON_TOKEN` (portal token) | Create users and groups, invite links | Portal app env | GM | See "The two-token split" above | | `OC_PERSONAL_TOKEN` | Read members and orders from Open Collective GraphQL | Portal app env | The Open Collective account that minted it | Needs the `email` scope (see below). Currently a personal token; a co-op-owned OC account would make it a co-op credential | | `WEBHOOK_TOKEN` | Secret in the Open Collective webhook URL | Portal app env, and the webhook URL on Open Collective | GM | Rotating means re-creating the webhook on OC as well | -| `LITELLM_MASTER_KEY` | Create gateway teams and keys for members | Portal app env (copy of the gateway's) | GM | Also held by the gateway; rotate both together | +| `LITELLM_MASTER_KEY` | Create gateway teams and keys for members | Portal app env (copy of the gateway's) | GM | Three holders: portal, gateway, steward agent. Rotate all together | | `LITELLM_DB_*` | Read spend logs from the gateway database | Portal app env | Cloudron (addon) | Postgres credentials for the gateway's database | | `MATRIX_BOT_TOKEN` | `@inference-bot:inference.coop` admin account: invites, space membership | Portal app env | GM | The account has no password; the token is the only credential | | `LOOMIO_API_KEY`, `LOOMIO_B3_KEY` | Forum (Loomio) membership sync | Portal app env | GM | | @@ -163,25 +163,38 @@ row that names the same credential: several are held by more than one app. | Credential | Purpose | Where it lives | Who rotates | Notes | |---|---|---|---|---| -| `LITELLM_MASTER_KEY` | Gateway admin | `/app/data/.master_key`, exported into the litellm process by `start.sh` | GM | Also held by the portal (above) and the audio keep-alive job | +| `LITELLM_MASTER_KEY` | Gateway admin | `/app/data/.master_key`, exported into the litellm process by `start.sh` | GM | Also held by the portal (above), the steward agent (below) and the audio keep-alive job | | Salt key (`SALT_KEY_FILE`) | Encrypts provider keys stored in the gateway database | `/app/data` | GM | Losing it means re-entering every provider key | | `TINFOIL_API_KEY` | Provider account | Gateway app env | Provider account owner | Billing lives at the provider | | `GREENPT_API_KEY` | Provider account | Gateway app env | Provider account owner | Billing lives at the provider | | `REGOLO_API_KEY` | Provider account | Gateway app env | Provider account owner | Billing lives at the provider | | `CLOUDRON_POSTGRESQL_*`, `CLOUDRON_REDIS_*` | Database and cache | Cloudron addons | Cloudron | Managed by the platform | -### Agent (inferencebot, on the agent VPS) +### Chat (LibreChat, `chat.inference.coop`) | Credential | Purpose | Where it lives | Who rotates | Notes | |---|---|---|---|---| -| Cloudron agent token | Cloudron API | Agent `.env` | GM | See "The two-token split" above | -| Gitea token | Commits, issues, PRs as `inference-bot` | Agent `.env` | GM | | +| LibreChat encryption keys and app secrets | App secrets; the per-member key sync runs in the app's own Cloudron scheduler | App data directory | GM | Not held by the agent | + +### Agents (on the agent host) + +Exact file paths are deliberately not listed; "agent `.env`" is the right +granularity for the public half. Highest privilege first. + +| Credential | Purpose | Where it lives | Who rotates | Notes | +|---|---|---|---|---| +| Gateway master key (`GW_MASTER_KEY`) | Admin over every gateway team, key and spend record | Steward agent `.env` | GM | **Third holder**, with the portal and the gateway. The row most in need of coordinated rotation | +| `MATRIX_LOCAL_BOT_TOKEN` | The `@inference-bot:inference.coop` automation account: room administration in every co-op room | Steward agent `.env` | GM | Same account as the portal's `MATRIX_BOT_TOKEN`; confirm whether it is the same token | +| Cloudron agent token | Cloudron API | Steward agent `.env` | GM | See "The two-token split" above | +| `REGISTRY_BUILD_TOKEN` (and its username) | Image build and push for the custom apps | Steward agent `.env` | GM | Moves to CI (co-op/support#30); the agent stops holding it then | +| Gitea token | Commits, issues, PRs as `inference-bot` | Steward agent `.env` | GM | | | Matrix access token (`@inferencebot:inference.coop`) | Reading and posting in rooms | Agent host, with its E2EE key store | GM | The off switch in [Working with the agent](working-with-the-agent.md) revokes this second | -| Gateway virtual key | The agent's own inference | Agent `.env` | GM | Model allowlist and budget live on this key; the off switch revokes it first | -| Direct provider key (fallback) | Inference during a gateway outage | Agent `.env` | Provider account owner | Planned on co-op/support#17 | -| LibreChat admin credential | Per-member key sync job | Agent `.env` | GM | Inferred from the job list on #17 | -| Restic repository passphrase | Nightly encrypted backup of the agent's state | Agent host and the co-op vault | GM | Per #17 | -| Infomaniak Swiss Backup S3 keys (agent) | Backup target | Agent host | Infomaniak account owner | Per #17 | +| Steward agent's gateway key (`api:info@inference.coop:inferencebots`) | The steward agent's own inference | Steward agent `.env` | GM | Today: empty model allowlist and no per-key budget; draws on the co-op account team's budget. To be scoped like the per-agent keys below. The off switch revokes this first | +| Per-agent gateway keys (`infrastructure-bot`, later agents) | That agent's inference | That agent's `.env` | GM | Model allowlist and monthly budget enforced at the gateway | +| `GREENPT_FALLBACK_API_KEY` | Direct-to-provider fallback inference during a gateway outage (live since 2026-10-26) | Steward agent `.env` | Provider account owner | Rotation-coupled to the gateway's `GREENPT_API_KEY`: rotate one without the other and the fallback silently breaks. A dedicated key is the cleaner end state | +| `HERMES_CUSTOM_API_GREENPT_AI_API_KEY` | Provider key, still present | Steward agent `.env` | Provider account owner | Confirm whether still needed; retire if not | +| Restic repository passphrase | Nightly encrypted backup of the agent's state | Agent host and the co-op vault | GM | | +| Infomaniak Swiss Backup S3 keys (agent) | Backup target | Agent host | Infomaniak account owner | | ### Accounts (not tokens, but the things tokens come from) -- 2.53.0 From 3302ae35f4c32d395335824754d9a6c7abc4ae40 Mon Sep 17 00:00:00 2001 From: Angel Marino Date: Tue, 27 Oct 2026 01:37:07 -0400 Subject: [PATCH 3/3] operator-token-runbook: OC token is co-op-owned since 2026-10-27; Matrix bot tokens are a pair (review) Co-Authored-By: Claude Fable 5.1 --- operator-token-runbook.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/operator-token-runbook.md b/operator-token-runbook.md index 86496a6..821680f 100644 --- a/operator-token-runbook.md +++ b/operator-token-runbook.md @@ -147,11 +147,11 @@ row that names the same credential: several are held by more than one app. | Credential | Purpose | Where it lives | Who rotates | Notes | |---|---|---|---|---| | `CLOUDRON_TOKEN` (portal token) | Create users and groups, invite links | Portal app env | GM | See "The two-token split" above | -| `OC_PERSONAL_TOKEN` | Read members and orders from Open Collective GraphQL | Portal app env | The Open Collective account that minted it | Needs the `email` scope (see below). Currently a personal token; a co-op-owned OC account would make it a co-op credential | +| `OC_PERSONAL_TOKEN` | Read the member list from Open Collective GraphQL | Portal app env, and the vault | GM, as `info@inference.coop` | Co-op-owned since 2026-10-27: minted by `info@inference.coop`, an Admin of the collective (OC returns member emails only to admins). Scopes `collective`, `individual`, `email`, read-only; the credit-pack order lookup is an anonymous query and needs no token. No expiry set; if one is added, record it here. Rotate on suspected exposure or when the account's admin role changes. The previous personal token should be revoked | | `WEBHOOK_TOKEN` | Secret in the Open Collective webhook URL | Portal app env, and the webhook URL on Open Collective | GM | Rotating means re-creating the webhook on OC as well | | `LITELLM_MASTER_KEY` | Create gateway teams and keys for members | Portal app env (copy of the gateway's) | GM | Three holders: portal, gateway, steward agent. Rotate all together | | `LITELLM_DB_*` | Read spend logs from the gateway database | Portal app env | Cloudron (addon) | Postgres credentials for the gateway's database | -| `MATRIX_BOT_TOKEN` | `@inference-bot:inference.coop` admin account: invites, space membership | Portal app env | GM | The account has no password; the token is the only credential | +| `MATRIX_BOT_TOKEN` | `@inference-bot:inference.coop` admin account: invites, space membership | Portal app env | GM | The account has no password. Paired with the steward agent's `MATRIX_LOCAL_BOT_TOKEN`: same account, separate token on a separate device (verified). Revoke or rotate both together | | `LOOMIO_API_KEY`, `LOOMIO_B3_KEY` | Forum (Loomio) membership sync | Portal app env | GM | | | `LISTMONK_API_USER`, `LISTMONK_API_TOKEN` | Newsletter list sync | Portal app env | GM | | | `OWUI_API_KEY`, `OWUI_JWT_SECRET` | Open WebUI knowledge refresh and SSO | Portal app env | GM | Chat moved to LibreChat in October; confirm whether still needed, and retire if not | @@ -184,7 +184,7 @@ granularity for the public half. Highest privilege first. | Credential | Purpose | Where it lives | Who rotates | Notes | |---|---|---|---|---| | Gateway master key (`GW_MASTER_KEY`) | Admin over every gateway team, key and spend record | Steward agent `.env` | GM | **Third holder**, with the portal and the gateway. The row most in need of coordinated rotation | -| `MATRIX_LOCAL_BOT_TOKEN` | The `@inference-bot:inference.coop` automation account: room administration in every co-op room | Steward agent `.env` | GM | Same account as the portal's `MATRIX_BOT_TOKEN`; confirm whether it is the same token | +| `MATRIX_LOCAL_BOT_TOKEN` | The `@inference-bot:inference.coop` automation account: room administration in every co-op room | Steward agent `.env` | GM | Same account as the portal's `MATRIX_BOT_TOKEN`, but a separate token on a separate device (verified). Revoking one does not revoke the other; revoke or rotate both together | | Cloudron agent token | Cloudron API | Steward agent `.env` | GM | See "The two-token split" above | | `REGISTRY_BUILD_TOKEN` (and its username) | Image build and push for the custom apps | Steward agent `.env` | GM | Moves to CI (co-op/support#30); the agent stops holding it then | | Gitea token | Commits, issues, PRs as `inference-bot` | Steward agent `.env` | GM | | -- 2.53.0