# Inference Cooperative — operator documentation

Operator- and maintainer-facing documentation for the Inference Cooperative.
This is the material that describes how the co-op's infrastructure is run —
access tokens, scheduled tasks, maintenance procedures, outage write-ups, and
the server-migration record — as opposed to the member-facing documentation
published at <https://docs.inference.coop/>.

Anyone may read it; operators are the audience. Comment, or open an issue,
where something looks wrong — that review is the point.

## This repository is deliberately not a website

Unlike `co-op/docs`, **nothing here is built or published by MkDocs**. The
repository is *git-only*: it is meant to be browsed, searched, and diffed in
Gitea, or read from a clone. Every page carries operational detail (token
rotation, deploy flows, failure modes) that belongs to operators rather than to
members, so it is kept out of the public documentation site on purpose.

Read it on the web here: <https://git.inference.coop/co-op/ops-docs>

## What is here

| Path | Contents |
|---|---|
| `operator-token-runbook.md` | The Cloudron token model (agent vs portal, admin vs webhook), how to rotate each without breaking the stack |
| `app-inventory.md` | Every Cloudron app: id, fqdn, registry image path and pin (read from the running apps) |
| `agent-fleet.md` | The three agents (steward, infrastructure-bot, socialbot): Unix users, gateway units, teams, and the planned member-bot tier |
| `maintenance/chat-task-models.md` | The small pinned model used for chat titles/tags/suggestions |
| `maintenance/librechat-sandbox.md` | Operating the LibreChat evaluation instance (config, overlay image, key sync, pitfalls) |
| `maintenance/backups.md` | The two backup systems (agent-host restic + Cloudron app backup): schedules, retention, the cache split and the grouping fix |
| `maintenance/woodpecker-ci.md` | The self-hosted CI server and agent: build/pin, manual-only trigger, git-lfs, and its not-yet-green state |
| `incidents/` | Post-incident write-ups, one file per incident |
| `migration/` | The Cloudron server-migration record: the plan, the DNS change sheet, and the captured pre-migration state |
| `provider-pricing/` | What each model provider charges, and when it changed (GreenPT, Tinfoil, Regolo, and the decision-model route) |

## The migration material

`migration/` holds the record of moving Cloudron to its new server: the plan
([`cloudron-server-migration.md`](migration/cloudron-server-migration.md)),
the record-by-record cutover sheet
([`dns-change-sheet.md`](migration/dns-change-sheet.md)), and the captured
pre-migration state the restore checker compares against
([`migration-before-state.json`](migration/migration-before-state.json)).

**The migration is complete (2026-11-02)** — this material is kept as the
record of how it was planned and done, not as pending work. Current
operational procedure lives in the co-op's `cloudron-ops` skill; the
member-facing description lives in `co-op/docs/infrastructure.md`.

### Read the DNS sheet with this in mind

Its record counts **cannot be trusted until reconciled against the registrar's
panel.** The zone is wildcarded, so *any* hostname resolves — a name that
answers proves nothing about whether a record exists. At least two listed
records turned out not to exist, and the real record inventory is only visible
in the registrar's own list.

## Known open items — recorded, not hidden

These are genuinely unresolved; a tidy document that omits them would be wrong.

- **The old box `179.199.135.72` is still alive and internet-exposed** (22/80/443/587/993
  open), holding old data and config. Nothing in DNS points at it, so it serves no
  member traffic — but it needs an operator decision: **shut it down or firewall it**.
  Do not act on it without the operator's word. See `migration/cloudron-server-migration.md` (addendum).
- **CI is not green.** The Woodpecker agent is healthy but the server answers its
  job-collection stream with HTTP 500, so no pipeline can run. See `maintenance/woodpecker-ci.md`.
- **The CI agent image source (`/home/ubuntu/repos/woodpecker-agent`) has no git
  remote** — it exists only on the agent host.
- **Member mailbox contents after the migration** are still unconfirmed (needs a
  real IMAP login).

## No secrets here

Audited 2026-11-01, including the full commit history: no credentials, tokens,
keys, member emails or account identifiers. Where a runbook needs a credential
to exist (a backup passphrase, an S3 key), it records **where the credential is
held**, never the credential.

## Where operators look

- **Member-facing docs** live in [`co-op/docs`](https://git.inference.coop/co-op/docs)
  and are published at <https://docs.inference.coop/>.
- The member-facing docs link here from their "For operators" section.

## History

The operator runbook, maintenance pages and incident write-ups were moved here
from `co-op/docs`, where they were previously published on the member-facing
site. The migration material came from `co-op/runbooks` (now archived), and was
brought across with its commit history intact via an unrelated-histories merge.