co-op/docs
4
0 Fork 3
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

operator-token-runbook: add a credential inventory #7

Closed
solo.ist wants to merge 3 commits from solo.ist/docs:docs/credential-inventory into main
pull from: solo.ist/docs:docs/credential-inventory
merge into: co-op:main
Conversation 8 Commits 3 Files Changed 1
+80
solo.ist commented 2026-10-26 05:09:39 +00:00
Contributor
Copy link
Copy source

The inventory offered on co-op/support#17: every token the agent and the member-facing apps hold, with purpose, where it lives and who rotates it, plus the accounts they come from.

Drafted from the public code and docs, so it is names and roles only, never values. Blank cells are for the operator to fill in; the two things I could not see from outside are the rotation owners and the agent's exact file locations. The Notes column marks credentials held by more than one app, which are the ones most likely to break on rotation.

Refs co-op/support#17

The inventory offered on co-op/support#17: every token the agent and the member-facing apps hold, with purpose, where it lives and who rotates it, plus the accounts they come from. Drafted from the public code and docs, so it is names and roles only, never values. Blank cells are for the operator to fill in; the two things I could not see from outside are the rotation owners and the agent's exact file locations. The Notes column marks credentials held by more than one app, which are the ones most likely to break on rotation. Refs co-op/support#17
solo.ist added 1 commit 2026-10-26 05:09:39 +00:00
operator-token-runbook: add a credential inventory (every token the agent and apps hold, where it lives, who rotates) 5aa6098e5b 
Drafted from the public code and docs, so names and roles only, never
values. Blank cells are for the operator to fill in. Marks the credentials
held by more than one app, which are the ones most likely to break on
rotation. Offered on co-op/support#17.
inference-bot commented 2026-10-26 05:12:46 +00:00
Owner
Copy link
Copy source

Reviewed against what the agent actually holds. Names and roles only, no values — that is the right rule and this draft keeps it.

Corrections

  • Agent → "Gateway virtual key … model allowlist and budget live on this key." True for the new per-agent convention, not true for the steward bot today: its key (api:info@inference.coop:inferencebots) has an empty model allowlist and no per-key budget, drawing instead on the co-op account team's budget. Until that key is scoped like the new agent's, the two deserve separate rows.
  • Agent → "Direct provider key (fallback) … Planned on co-op/support#17." Now live: direct-to-provider fallback, wired and tested 2026-10-26 (and #17 is closed). Present tense.
  • Agent → "LibreChat admin credential | per-member key sync job | Agent .env." Stale: the per-member key sync now runs inside the LibreChat app's own Cloudron scheduler, and that app's encryption keys live in the app's data directory — not the agent's .env. Suggest deleting the row; the agent no longer holds it.

Missing rows (agent), highest privilege first — the ones rotation hygiene most needs:

  • The gateway master key (GW_MASTER_KEY). Admin over every team, key and spend record. Already listed above as held by the portal and the gateway; the agent is a third holder. This is the row that most belongs in the "held by more than one app" notes.
  • MATRIX_LOCAL_BOT_TOKEN — the room-admin automation account; it holds power in every co-op room.
  • REGISTRY_BUILD_TOKEN (plus its username) — image build/push for the custom apps. Intended to move to CI (#30), at which point the agent stops holding it.
  • GREENPT_FALLBACK_API_KEY — the fallback inference key; rotation-coupled to the gateway's provider key, so rotating one without the other silently breaks the fallback.
  • HERMES_CUSTOM_API_GREENPT_AI_API_KEY — a provider key still present; give it the same "confirm whether still needed, retire if not" treatment as the Open WebUI rows.

Keep out of the public half: exact file paths. "Agent .env" is the right granularity; precise locations belong in the operator's private copy.

Reviewed against what the agent actually holds. Names and roles only, no values — that is the right rule and this draft keeps it. **Corrections** - *Agent → "Gateway virtual key … model allowlist and budget live on this key."* True for the new per-agent convention, **not** true for the steward bot today: its key (`api:info@inference.coop:inferencebots`) has an **empty** model allowlist and **no** per-key budget, drawing instead on the co-op account team's budget. Until that key is scoped like the new agent's, the two deserve separate rows. - *Agent → "Direct provider key (fallback) … Planned on co-op/support#17."* Now live: direct-to-provider fallback, wired and tested 2026-10-26 (and #17 is closed). Present tense. - *Agent → "LibreChat admin credential | per-member key sync job | Agent `.env`."* Stale: the per-member key sync now runs inside the LibreChat app's own Cloudron scheduler, and that app's encryption keys live in the app's data directory — not the agent's `.env`. Suggest deleting the row; the agent no longer holds it. **Missing rows (agent), highest privilege first** — the ones rotation hygiene most needs: - **The gateway master key** (`GW_MASTER_KEY`). Admin over every team, key and spend record. Already listed above as held by the portal and the gateway; the agent is a **third** holder. This is the row that most belongs in the "held by more than one app" notes. - **`MATRIX_LOCAL_BOT_TOKEN`** — the room-admin automation account; it holds power in every co-op room. - **`REGISTRY_BUILD_TOKEN`** (plus its username) — image build/push for the custom apps. Intended to move to CI (#30), at which point the agent stops holding it. - **`GREENPT_FALLBACK_API_KEY`** — the fallback inference key; rotation-coupled to the gateway's provider key, so rotating one without the other silently breaks the fallback. - **`HERMES_CUSTOM_API_GREENPT_AI_API_KEY`** — a provider key still present; give it the same "confirm whether still needed, retire if not" treatment as the Open WebUI rows. **Keep out of the public half:** exact file paths. "Agent `.env`" is the right granularity; precise locations belong in the operator's private copy.
solo.ist added 1 commit 2026-10-26 05:50:19 +00:00
operator-token-runbook: apply the agent's review (master key as third holder, split gateway keys, fallback live, registry and Matrix admin tokens, LibreChat row moved) 11ea40485c 
- The gateway master key is held by the steward agent too; marked as the
  three-holder row in all three places.
- The steward agent's gateway key is unscoped today; per-agent scoped keys
  are a separate row.
- Fallback inference is live, not planned; names the key and its rotation
  coupling to the gateway's GreenPT key.
- Adds MATRIX_LOCAL_BOT_TOKEN, REGISTRY_BUILD_TOKEN (moving to CI, #30) and
  the leftover GreenPT key to confirm or retire.
- The LibreChat credential moved to a Chat table; the agent no longer
  holds it. Exact file paths deliberately left out.
solo.ist commented 2026-10-26 06:08:10 +00:00
Author
Contributor
Copy link
Copy source

Thanks, applied in 11ea404:

  • Master key marked as a three-holder credential in the portal, gateway and agent rows.
  • The steward key and the per-agent scoped keys are separate rows, with the steward key's current state (empty allowlist, no per-key budget) stated plainly.
  • Fallback is present tense, named, with the rotation coupling to the gateway's GreenPT key in the notes.
  • Added MATRIX_LOCAL_BOT_TOKEN, REGISTRY_BUILD_TOKEN (noting the move to CI in #30), and the leftover GreenPT key with the confirm-or-retire treatment.
  • LibreChat row moved to its own Chat table as not held by the agent; the agent table now says exact paths are deliberately omitted.

One question I couldn't answer from outside: is MATRIX_LOCAL_BOT_TOKEN the same token as the portal's MATRIX_BOT_TOKEN, or two tokens for the same account? The row says "confirm" for now.

Thanks, applied in 11ea404: - Master key marked as a three-holder credential in the portal, gateway and agent rows. - The steward key and the per-agent scoped keys are separate rows, with the steward key's current state (empty allowlist, no per-key budget) stated plainly. - Fallback is present tense, named, with the rotation coupling to the gateway's GreenPT key in the notes. - Added MATRIX_LOCAL_BOT_TOKEN, REGISTRY_BUILD_TOKEN (noting the move to CI in #30), and the leftover GreenPT key with the confirm-or-retire treatment. - LibreChat row moved to its own Chat table as not held by the agent; the agent table now says exact paths are deliberately omitted. One question I couldn't answer from outside: is MATRIX_LOCAL_BOT_TOKEN the same token as the portal's MATRIX_BOT_TOKEN, or two tokens for the same account? The row says "confirm" for now.
inference-bot commented 2026-10-26 14:58:23 +00:00
Owner
Copy link
Copy source

Answering the token question, and confirming the applied rows.

MATRIX_LOCAL_BOT_TOKEN and the portal's MATRIX_BOT_TOKEN are two different tokens for the same account. Both authenticate as @inference-bot:inference.coop, but they are distinct tokens on distinct devices — verified today by whoami from each side. So: same account, two independent credentials.

What that means for the row:

  • Revoking one does not revoke the other. A full cut-off of that account needs both, which is exactly the kind of detail this table exists to catch.
  • Rotating one alone leaves a live stale token. Rotate them together, or at least record that they are paired.
  • The blast radius is identical either way: both act as the room administrator in every co-op room.

Suggested wording: "Same account as the portal's token, but a separate token on a separate device (verified). Revoke or rotate both together." I would leave the device identifiers out of the public page — they are not secrets, but they are not useful here either.

The applied rows read correctly, and the two I care most about are right: the steward key's row states plainly that its allowlist is empty and it carries no per-key budget, and the gateway master key is marked a three-holder credential with the agent named. Not held by the agent is the right home for the chat credential.

Answering the token question, and confirming the applied rows. **`MATRIX_LOCAL_BOT_TOKEN` and the portal's `MATRIX_BOT_TOKEN` are two different tokens for the same account.** Both authenticate as `@inference-bot:inference.coop`, but they are distinct tokens on distinct devices — verified today by `whoami` from each side. So: same account, two independent credentials. What that means for the row: - **Revoking one does not revoke the other.** A full cut-off of that account needs both, which is exactly the kind of detail this table exists to catch. - **Rotating one alone leaves a live stale token.** Rotate them together, or at least record that they are paired. - The blast radius is identical either way: both act as the room administrator in every co-op room. Suggested wording: "Same account as the portal's token, but a separate token on a separate device (verified). Revoke or rotate both together." I would leave the device identifiers out of the public page — they are not secrets, but they are not useful here either. **The applied rows read correctly**, and the two I care most about are right: the steward key's row states plainly that its allowlist is empty and it carries no per-key budget, and the gateway master key is marked a three-holder credential with the agent named. Not held by the agent is the right home for the chat credential.
inference-bot commented 2026-10-27 04:57:24 +00:00
Owner
Copy link
Copy source

Adding the row this PR asks the operator for — the Open Collective credential, now co-op-owned. Names, roles and homes only; no values.

credential owner (account) home created scopes / authority rotation
OC_PERSONAL_TOKEN (member portal → OC GraphQL v2) info@inference.coop (Admin of the Inference Cooperative collective) portal env (OC_PERSONAL_TOKEN) + Vault 2026-10-27 personal token, scopes collective, individual, email — read-only member list; no 2FA/advanced privileges trigger: suspected exposure, or the account's admin role changing. No expiry set — if one is added, record the date here.

Why the owner and scope matter, in one line each:

  • Owner must be an Admin of the collective: OC returns member emails only to admins, and the portal's member query asks for exactly those.
  • Scopes are read-only on purpose: the portal makes precisely one authenticated OC call (the members query). The credit-pack order lookup is an anonymous public query and needs no token. Anything with write scopes (transactions, expenses, payouts, webhooks) is unused authority.

Verified 2026-10-27 immediately after the swap, read-only: members.totalCount = 68, 66 nodes carrying an individual email (the two without are organization-backed memberships, which OC never exposes an email for), no GraphQL errors. The previous token should be revoked now that this one is live.

Adding the row this PR asks the operator for — the Open Collective credential, now co-op-owned. Names, roles and homes only; no values. | credential | owner (account) | home | created | scopes / authority | rotation | |---|---|---|---|---|---| | `OC_PERSONAL_TOKEN` (member portal → OC GraphQL v2) | **info@inference.coop** (Admin of the Inference Cooperative collective) | portal env (`OC_PERSONAL_TOKEN`) + Vault | 2026-10-27 | personal token, scopes `collective`, `individual`, `email` — read-only member list; no 2FA/advanced privileges | trigger: suspected exposure, or the account's admin role changing. No expiry set — if one is added, record the date here. | Why the owner and scope matter, in one line each: - **Owner must be an Admin of the collective**: OC returns member *emails* only to admins, and the portal's member query asks for exactly those. - **Scopes are read-only on purpose**: the portal makes precisely one authenticated OC call (the members query). The credit-pack order lookup is an *anonymous* public query and needs no token. Anything with write scopes (`transactions`, `expenses`, `payouts`, `webhooks`) is unused authority. Verified 2026-10-27 immediately after the swap, read-only: `members.totalCount = 68`, 66 nodes carrying an individual email (the two without are organization-backed memberships, which OC never exposes an email for), no GraphQL errors. The previous token should be revoked now that this one is live.
solo.ist added 1 commit 2026-10-27 05:37:17 +00:00
operator-token-runbook: OC token is co-op-owned since 2026-10-27; Matrix bot tokens are a pair (review) 3302ae35f4 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
solo.ist commented 2026-10-27 05:41:35 +00:00
Author
Contributor
Copy link
Copy source

Both in, thanks:

  • The two Matrix rows now say it the way you put it: same account, separate token on a separate device, revoke or rotate both together. The portal's row carries the mirror note, so whichever side someone starts from, they find the other.
  • OC_PERSONAL_TOKEN is now a co-op credential in the table: minted by info@inference.coop on 2026-10-27, the three read-only scopes, the admin-role requirement, no expiry, and the rotation triggers. I left the device identifiers and the vault path out, per the rule at the top of the section.

@ntnsndr, that last one closes the loop on the continuity note I sent you. One action left on your side when you get a moment: revoke the old personal token now that the new one is live.

Both in, thanks: * The two Matrix rows now say it the way you put it: same account, separate token on a separate device, revoke or rotate both together. The portal's row carries the mirror note, so whichever side someone starts from, they find the other. * `OC_PERSONAL_TOKEN` is now a co-op credential in the table: minted by `info@inference.coop` on 2026-10-27, the three read-only scopes, the admin-role requirement, no expiry, and the rotation triggers. I left the device identifiers and the vault path out, per the rule at the top of the section. @ntnsndr, that last one closes the loop on the continuity note I sent you. One action left on your side when you get a moment: revoke the old personal token now that the new one is live.
ntnsndr commented 2026-10-27 14:01:44 +00:00
Contributor
Copy link
Copy source

@solo.ist I can confirm that the old token has already been revoked. Thanks for all your help with this!

@solo.ist I can confirm that the old token has already been revoked. Thanks for all your help with this!
inference-bot commented 2026-10-27 22:26:52 +00:00
Owner
Copy link
Copy source

RAG service credential added — 2026-10-27 (credential inventory entry)

Context: the LibreChat RAG service — document upload/parse for the chat — had never started. The supervisor entry refuses to launch it unless RAG_OPENAI_API_KEY (or OPENAI_API_KEY) is set, so supervisord sat at rag_api RUNNING for a gate loop while nothing listened on :8000. Members saw uploads fail; the app logged [parseText] RAG API health check failed. First failure in the logs: 2026-10-20T21:15Z — down since the Cloudron move, ~7 days.

LiteLLM key, alias rag-service-embeddings

  • Fingerprint (sha256 of the key, first 12): 72807e5fbf3f — the value is recorded nowhere in git, including this comment.
  • Scope: one model only — tinfoil/nomic-embed-text (/v1/embeddings, 768-dim, runs in TEE). Key budget cap $2 / 30d.
  • Team: cdfdf422-633f-4432-8893-cffee82a8325 — the co-op-operated info@inference.coop team, not a personal account.
  • Why that model: the gateway config scopes it to the admin/info@ team deliberately — "an operator resource for the co-op's own docs-RAG, not a member-facing model" — so member keys cannot call it, and no third-party embedding API is involved.
  • Lives in: /app/data/env (mode 600) inside the LibreChat app, as RAG_OPENAI_API_KEY plus RAG_OPENAI_BASEURL=https://gateway.inference.coop/v1, EMBEDDINGS_PROVIDER=openai, EMBEDDINGS_MODEL=tinfoil/nomic-embed-text, EMBEDDINGS_DIMENSIONS=768. Previous env kept at /app/data/env.bak-20261027T221836Z.
  • Rotates: anyone holding the gateway master key can POST /key/regenerate (or delete and recreate with the same scope); no dashboard steps, no personal token.
  • Verified: key → gateway returns a 768-dim vector (6 prompt tokens billed); RAG API returns {"status":"UP"}; its boot log shows Uvicorn running on http://0.0.0.0:8000 and Vector database startup DDL checks complete.

Note for this inventory: /app/data/env must NOT be mirrored into a repo (it holds credentials); only librechat.yaml is mirrored, and it contains env-var references and LibreChat's user_provided sentinel, never literal keys.

RAG service credential added — 2026-10-27 (credential inventory entry) Context: the LibreChat RAG service — document upload/parse for the chat — had never started. The supervisor entry refuses to launch it unless `RAG_OPENAI_API_KEY` (or `OPENAI_API_KEY`) is set, so supervisord sat at `rag_api RUNNING` for a gate loop while nothing listened on :8000. Members saw uploads fail; the app logged `[parseText] RAG API health check failed`. First failure in the logs: **2026-10-20T21:15Z** — down since the Cloudron move, ~7 days. **LiteLLM key, alias `rag-service-embeddings`** - Fingerprint (sha256 of the key, first 12): `72807e5fbf3f` — the value is recorded nowhere in git, including this comment. - Scope: **one model only** — `tinfoil/nomic-embed-text` (`/v1/embeddings`, 768-dim, runs in TEE). Key budget cap $2 / 30d. - Team: `cdfdf422-633f-4432-8893-cffee82a8325` — the co-op-operated `info@inference.coop` team, not a personal account. - Why that model: the gateway config scopes it to the admin/`info@` team deliberately — "an operator resource for the co-op's own docs-RAG, not a member-facing model" — so member keys cannot call it, and no third-party embedding API is involved. - Lives in: `/app/data/env` (mode 600) inside the LibreChat app, as `RAG_OPENAI_API_KEY` plus `RAG_OPENAI_BASEURL=https://gateway.inference.coop/v1`, `EMBEDDINGS_PROVIDER=openai`, `EMBEDDINGS_MODEL=tinfoil/nomic-embed-text`, `EMBEDDINGS_DIMENSIONS=768`. Previous env kept at `/app/data/env.bak-20261027T221836Z`. - Rotates: anyone holding the gateway master key can `POST /key/regenerate` (or delete and recreate with the same scope); no dashboard steps, no personal token. - Verified: key → gateway returns a 768-dim vector (6 prompt tokens billed); RAG API returns `{"status":"UP"}`; its boot log shows `Uvicorn running on http://0.0.0.0:8000` and `Vector database startup DDL checks complete`. Note for this inventory: `/app/data/env` must NOT be mirrored into a repo (it holds credentials); only `librechat.yaml` is mirrored, and it contains env-var references and LibreChat's `user_provided` sentinel, never literal keys.
inference-bot commented 2026-11-04 01:18:48 +00:00
Owner
Copy link
Copy source

Ported, and superseded by this. The runbook this PR patched was moved out of the published site into the git-only co-op/ops-docs repository (commit e0b9c0c), which is where a credential inventory belongs. Rather than rewrite @solo.ist's prose, their inventory is now committed in co-op/ops-docs with their authorship preserved:

Two points raised after the last review are reflected in what landed: the old OC_PERSONAL_TOKEN is recorded as already revoked (you confirmed on 2026-10-27), and the LibreChat RAG service key (RAG_OPENAI_API_KEY / rag-service-embeddings, added 2026-10-27) is now a row.

Closing this PR unmerged on purpose — nothing here should go to the published site. Thanks, @solo.ist. Refs co-op/support#31.

Ported, and superseded by this. The runbook this PR patched was moved out of the published site into the git-only **co-op/ops-docs** repository (commit e0b9c0c), which is where a credential inventory belongs. Rather than rewrite @solo.ist's prose, their inventory is now committed in co-op/ops-docs with their authorship preserved: - **co-op/ops-docs#1** — "operator-token-runbook: add a credential inventory (from co-op/docs#7, by solo.ist)" — merged as 63d1415. https://git.inference.coop/co-op/ops-docs/pulls/1 Two points raised after the last review are reflected in what landed: the old `OC_PERSONAL_TOKEN` is recorded as already revoked (you confirmed on 2026-10-27), and the LibreChat RAG service key (`RAG_OPENAI_API_KEY` / `rag-service-embeddings`, added 2026-10-27) is now a row. Closing this PR unmerged on purpose — nothing here should go to the published site. Thanks, @solo.ist. Refs co-op/support#31.
inference-bot closed this pull request 2026-11-04 01:18:48 +00:00

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
inference-admin (Admin) inference-bot (InferenceBot)
No Assignees
3 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: co-op/docs#7