Operator- and maintainer-facing documentation for the Inference Cooperative. This is the material that describes how the co-op's infrastructure is run — access tokens, scheduled tasks, maintenance procedures, outage write-ups, and the server-migration record — as opposed to the member-facing documentation published at https://docs.inference.coop/.

Anyone may read it; operators are the audience. Comment, or open an issue, where something looks wrong — that review is the point.

This repository is deliberately not a website

Unlike co-op/docs, nothing here is built or published by MkDocs. The repository is git-only: it is meant to be browsed, searched, and diffed in Gitea, or read from a clone. Every page carries operational detail (token rotation, deploy flows, failure modes) that belongs to operators rather than to members, so it is kept out of the public documentation site on purpose.

Read it on the web here: https://git.inference.coop/co-op/ops-docs

What is here

Path	Contents
`operator-token-runbook.md`	The Cloudron token model (agent vs portal, admin vs webhook), how to rotate each without breaking the stack
`app-inventory.md`	Every Cloudron app: id, fqdn, registry image path and pin (read from the running apps)
`agent-fleet.md`	The three agents (steward, infrastructure-bot, socialbot): Unix users, gateway units, teams, and the planned member-bot tier
`maintenance/chat-task-models.md`	The small pinned model used for chat titles/tags/suggestions
`maintenance/librechat-sandbox.md`	Operating the LibreChat evaluation instance (config, overlay image, key sync, pitfalls)
`maintenance/backups.md`	The two backup systems (agent-host restic + Cloudron app backup): schedules, retention, the cache split and the grouping fix
`maintenance/woodpecker-ci.md`	The self-hosted CI server and agent: build/pin, manual-only trigger, git-lfs, and its not-yet-green state
`maintenance/2026-11-08-litellm-1.108.0-librechat-1.4.8.md`	The Sunday-evening upgrade-window runbook (gateway → 1.108.0, chat overlay → appstore 1.4.8): candidates, order, exact commands, per-step verification, rollback, window estimate
`incidents/`	Post-incident write-ups, one file per incident
`migration/`	The Cloudron server-migration record: the plan, the DNS change sheet, and the captured pre-migration state
`provider-pricing/`	What each model provider charges, and when it changed (GreenPT, Tinfoil, Regolo, and the decision-model route)

The migration material

migration/ holds the record of moving Cloudron to its new server: the plan (cloudron-server-migration.md), the record-by-record cutover sheet (dns-change-sheet.md), and the captured pre-migration state the restore checker compares against (migration-before-state.json).

The migration is complete (2026-11-02) — this material is kept as the record of how it was planned and done, not as pending work. Current operational procedure lives in the co-op's cloudron-ops skill; the member-facing description lives in co-op/docs/infrastructure.md.

Read the DNS sheet with this in mind

Its record counts cannot be trusted until reconciled against the registrar's panel. The zone is wildcarded, so any hostname resolves — a name that answers proves nothing about whether a record exists. At least two listed records turned out not to exist, and the real record inventory is only visible in the registrar's own list.

Known open items — recorded, not hidden

These are genuinely unresolved; a tidy document that omits them would be wrong.

The old box 179.199.135.72 is still alive and internet-exposed (22/80/443/587/993 open), holding old data and config. Nothing in DNS points at it, so it serves no member traffic — but it needs an operator decision: shut it down or firewall it. Do not act on it without the operator's word. See migration/cloudron-server-migration.md (addendum).
CI is not green. The Woodpecker agent is healthy but the server answers its job-collection stream with HTTP 500, so no pipeline can run. See maintenance/woodpecker-ci.md.
The CI agent image source (/home/ubuntu/repos/woodpecker-agent) has no git remote — it exists only on the agent host.
Member mailbox contents after the migration are still unconfirmed (needs a real IMAP login).

No secrets here

Audited 2026-11-01, including the full commit history: no credentials, tokens, keys, member emails or account identifiers. Where a runbook needs a credential to exist (a backup passphrase, an S3 key), it records where the credential is held, never the credential.

Where operators look

Member-facing docs live in co-op/docs and are published at https://docs.inference.coop/.
The member-facing docs link here from their "For operators" section.

History

The operator runbook, maintenance pages and incident write-ups were moved here from co-op/docs, where they were previously published on the member-facing site. The migration material came from co-op/runbooks (now archived), and was brought across with its commit history intact via an unrelated-histories merge.