Credential rotation: no schedule exists (inventory ready; needs cadence + owners) #35
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: co-op/support#35
Reference in new issue
Block a user
No credential has a rotation schedule today: nothing recorded says when anything is rotated, and nothing records when it was last rotated. The inventory (co-op/docs#7, refs #31) covers purpose, location and owner; it stops short of time.
Proposal. Four parts, in order of value:
Proposed cadence (names only, no values)
LITELLM_MASTER_KEY/GW_MASTER_KEY, Cloudron API token(s), Gitea token,OC_PERSONAL_TOKENBROKER_SECRET,PORTAL_SECRET,WEBHOOK_TOKEN,LOOMIO_API_KEY/LOOMIO_B3_KEY,LISTMONK_API_TOKEN,REGISTRY_BUILD_TOKEN12 months is a starting proposal, not a principle — say the word and I'll make it 6 for the account-level row, which is where the blast radius actually sits.
Rotation groups (rotate together, or something breaks silently)
LITELLM_MASTER_KEY— four holders: gateway (/app/data/.master_key), portal env, steward agent.env, audio keep-alive job.BROKER_SECRET— portal and dashboard app env.GREENPT_API_KEYand the agent'sGREENPT_FALLBACK_API_KEYare rotation-coupled; a dedicated fallback key is the cleaner end state.MATRIX_BOT_TOKEN/MATRIX_LOCAL_BOT_TOKEN— same account, different devices; per-device revocation, so rotating one does not revoke the other (verified 2026-10-26).WEBHOOK_TOKEN— rotating it means re-creating the Open Collective webhook, because the token is in its URL and OC cannot send headers.CLOUDRON_TOKEN— the portal token and the agent token are separate tokens on the same account; both are in play.Needed from the operator
OWUI_API_KEY/OWUI_JWT_SECRET(chat moved to LibreChat) andHERMES_CUSTOM_API_GREENPT_AI_API_KEY— confirm whether either is still needed.Refs co-op/docs#7, co-op/support#31
Recording the rotation that F5 needed, so the schedule has a first entry rather than starting from nothing.
Rotated 2026-10-27: the portal's Open Collective token is now owned by info@inference.coop (Admin on the collective) instead of an account whose ownership was ambiguous, with scopes limited to
collective,individual,email.totalCount = 68with 66 individual emails, which is the exact shape the portal needs. Measured before trusting, not assumed.Owners resolved (operator, 2026-11-04): the GM controls all tokens for now. No owner column is needed while that holds -- so the inventory can land without one, and the question this issue asks about ownership is answered.
The account layer was filled in the same pass by querying each provider rather than trusting environment variable names. Two findings change the rotation groups:
GREENPT_API_KEY(gateway) andGREENPT_FALLBACK_API_KEY(agent) hash to the same value, so the fallback isolates nothing. A genuinely separate key on the same account would fix that.inference-bot, as separate token values, so the deliberate two-token split holds -- but a rotation must update whichever holder keeps the old copy. The registry build token is the exception: it belongs toinference-admin.Also confirmed by direct query: the portal's Matrix bot token and the agent's admin token are two tokens on ONE account (
@inference-admin), and the LiteLLM master key is a single literal in four places.Still open here: the cadence. The proposal in the issue body stands (12 months, 6 for the account-level row). Trigger-based rotation needs no decision and can be adopted now.
Cadence decided (operator, 2026-11-04): 12 months for everything for now -- no tighter row for account-level credentials. Combined with the ownership answer above (the GM controls all tokens), both questions this issue opened are now answered.
What remains is writing the policy down where it can be seen, because a schedule nobody can read is not a schedule: the cadence and the trigger rules go into the operator token runbook in
co-op/ops-docs, with alast rotatedcolumn whose first entry is the Open Collective token rotated on 2026-10-27, plus a reminder in the ops digest so a due date surfaces on its own. I will do that once the credential inventory lands there (see #31).Adopted now, needing no cadence decision: rotate on offboarding anyone with access, on suspected exposure, after any incident or diagnostic session where a secret could have been read, and after any restore.
12 months is a starting point, not a principle -- revisit when the token list is stable.
The cadence and ownership questions were answered on 2026-11-04; what was missing was the record and the reminder. Both now exist.
The policy is written down in the operator token runbook: https://git.inference.coop/co-op/ops-docs/src/branch/main/operator-token-runbook.md
It now carries, in one place:
last rotatedtable. The only entry that is filled is the Open Collective token, rotated 2026-10-27; every other row is deliberately blank because nothing has been recorded. A blank means "nothing recorded", not "not rotated" — which is the honest state and the starting point.@inference-admin, not@inference-botas the earlier inventory said.The reminder exists as a small script that reads that table: https://git.inference.coop/co-op/ops-docs/src/branch/main/scripts/rotation_due.py
It prints a line only when a credential is within 30 days of its due date (last rotation + 12 months), and prints nothing at all otherwise — so it can drive a watchdog job where empty output means no message. It is deterministic (no timestamps of its own) and was tested in three states: silent against the current table, firing when a due date is brought inside the window, and failing loudly if it cannot read the runbook rather than going quiet.
What now exists that did not before: a written, reviewable rotation schedule with trigger rules and a last-rotated record, and an automatic reminder that will surface a due date on its own instead of relying on someone remembering. Closes the "no schedule exists" gap; the open item in the inventory's re-run checklist ("a rotation cadence is agreed and recorded") can now be checked off.
One remaining wiring step, deliberately not done here: attaching the reminder to a delivery target (a watchdog job or the ops digest). It is a standalone script now; it prints to stdout and is silent when idle.