# Inference Cooperative — operator documentation

Operator- and maintainer-facing documentation for the Inference Cooperative.
This is the material that describes how the co-op's infrastructure is run —
access tokens, scheduled tasks, maintenance procedures, outage write-ups, and
the server-migration record — as opposed to the member-facing documentation
published at <https://docs.inference.coop/>.

Anyone may read it; operators are the audience. Comment, or open an issue,
where something looks wrong — that review is the point.

## This repository is deliberately not a website

Unlike `co-op/docs`, **nothing here is built or published by MkDocs**. The
repository is *git-only*: it is meant to be browsed, searched, and diffed in
Gitea, or read from a clone. Every page carries operational detail (token
rotation, deploy flows, failure modes) that belongs to operators rather than to
members, so it is kept out of the public documentation site on purpose.

Read it on the web here: <https://git.inference.coop/co-op/ops-docs>

## What is here

| Path | Contents |
|---|---|
| `operator-token-runbook.md` | The Cloudron token model (agent vs portal, admin vs webhook), how to rotate each without breaking the stack |
| `app-inventory.md` | Every Cloudron app: id, fqdn, registry image path and pin (read from the running apps) |
| `agent-fleet.md` | The three agents (steward, infrastructure-bot, socialbot): Unix users, gateway units, teams, and the planned member-bot tier |
| `maintenance/chat-task-models.md` | The small pinned model used for chat titles/tags/suggestions |
| `maintenance/librechat-sandbox.md` | Operating the LibreChat evaluation instance (config, overlay image, key sync, pitfalls) |
| `maintenance/backups.md` | The two backup systems (agent-host restic + Cloudron app backup): schedules, retention, the cache split and the grouping fix |
| `maintenance/woodpecker-ci.md` | The self-hosted CI server and agent: build/pin, manual-only trigger, git-lfs, and its not-yet-green state |
| `incidents/` | Post-incident write-ups, one file per incident |
| `migration/` | The Cloudron server-migration record: the plan, the DNS change sheet, and the captured pre-migration state |
| `provider-pricing/` | What each model provider charges, and when it changed (GreenPT, Tinfoil, Regolo, and the decision-model route) |

## The migration material

`migration/` holds the record of moving Cloudron to its new server: the plan
([`cloudron-server-migration.md`](migration/cloudron-server-migration.md)),
the record-by-record cutover sheet
([`dns-change-sheet.md`](migration/dns-change-sheet.md)), and the captured
pre-migration state the restore checker compares against
([`migration-before-state.json`](migration/migration-before-state.json)).

**The migration is complete (2026-11-02)** — this material is kept as the
record of how it was planned and done, not as pending work. Current
operational procedure lives in the co-op's `cloudron-ops` skill; the
member-facing description lives in `co-op/docs/infrastructure.md`.

### Read the DNS sheet with this in mind

Its record counts **cannot be trusted until reconciled against the registrar's
panel.** The zone is wildcarded, so *any* hostname resolves — a name that
answers proves nothing about whether a record exists. At least two listed
records turned out not to exist, and the real record inventory is only visible
in the registrar's own list.

## No secrets here

Audited 2026-11-01, including the full commit history: no credentials, tokens,
keys, member emails or account identifiers. Where a runbook needs a credential
to exist (a backup passphrase, an S3 key), it records **where the credential is
held**, never the credential.

## Where operators look

- **Member-facing docs** live in [`co-op/docs`](https://git.inference.coop/co-op/docs)
  and are published at <https://docs.inference.coop/>.
- The member-facing docs link here from their "For operators" section.

## History

The operator runbook, maintenance pages and incident write-ups were moved here
from `co-op/docs`, where they were previously published on the member-facing
site. The migration material came from `co-op/runbooks` (now archived), and was
brought across with its commit history intact via an unrelated-histories merge.