inference-bot
2eb88f55b4
Audit of the operator docs against live reality (2026-11-04): - app-inventory.md: all 19 Cloudron apps with id, fqdn and registry image pin. Corrects the portal path (code/member-portal, not co-op/member-portal) and the panel path (local/coop.inference.admin-panel, not code/admin-panel). - maintenance/backups.md: the two backup systems; the deliberate per-user restic cache split and the --group-by "" retention fix, with why. - maintenance/woodpecker-ci.md: CI server + agent, image build/pin, manual-only trigger, git-lfs requirement, and its current not-yet-green state (agent's job-collection long-poll gets HTTP 500 from the server). - provider-pricing/: index + GreenPT, Tinfoil (local EHBP proxy architecture) and the Metis /v1/systemone decision-model route; regolo.md updated to record the gateway prices were applied. - agent-fleet.md: steward, infrastructure-bot and socialbot (Unix users, gateway units, teams); member-bot tier 2 planned but not yet provisioned. - README: file list; operator-token-runbook: correct the membership_sweep cadence (3x daily, not nightly); migration doc: post-migration addendum (16->19 apps, DNS split resolved, old box still on with apps stopped).
78 lines
4.2 KiB
Markdown
78 lines
4.2 KiB
Markdown
# Inference Cooperative — operator documentation
|
|
|
|
Operator- and maintainer-facing documentation for the Inference Cooperative.
|
|
This is the material that describes how the co-op's infrastructure is run —
|
|
access tokens, scheduled tasks, maintenance procedures, outage write-ups, and
|
|
the server-migration record — as opposed to the member-facing documentation
|
|
published at <https://docs.inference.coop/>.
|
|
|
|
Anyone may read it; operators are the audience. Comment, or open an issue,
|
|
where something looks wrong — that review is the point.
|
|
|
|
## This repository is deliberately not a website
|
|
|
|
Unlike `co-op/docs`, **nothing here is built or published by MkDocs**. The
|
|
repository is *git-only*: it is meant to be browsed, searched, and diffed in
|
|
Gitea, or read from a clone. Every page carries operational detail (token
|
|
rotation, deploy flows, failure modes) that belongs to operators rather than to
|
|
members, so it is kept out of the public documentation site on purpose.
|
|
|
|
Read it on the web here: <https://git.inference.coop/co-op/ops-docs>
|
|
|
|
## What is here
|
|
|
|
| Path | Contents |
|
|
|---|---|
|
|
| `operator-token-runbook.md` | The Cloudron token model (agent vs portal, admin vs webhook), how to rotate each without breaking the stack |
|
|
| `app-inventory.md` | Every Cloudron app: id, fqdn, registry image path and pin (read from the running apps) |
|
|
| `agent-fleet.md` | The three agents (steward, infrastructure-bot, socialbot): Unix users, gateway units, teams, and the planned member-bot tier |
|
|
| `maintenance/chat-task-models.md` | The small pinned model used for chat titles/tags/suggestions |
|
|
| `maintenance/librechat-sandbox.md` | Operating the LibreChat evaluation instance (config, overlay image, key sync, pitfalls) |
|
|
| `maintenance/backups.md` | The two backup systems (agent-host restic + Cloudron app backup): schedules, retention, the cache split and the grouping fix |
|
|
| `maintenance/woodpecker-ci.md` | The self-hosted CI server and agent: build/pin, manual-only trigger, git-lfs, and its not-yet-green state |
|
|
| `incidents/` | Post-incident write-ups, one file per incident |
|
|
| `migration/` | The Cloudron server-migration record: the plan, the DNS change sheet, and the captured pre-migration state |
|
|
| `provider-pricing/` | What each model provider charges, and when it changed (GreenPT, Tinfoil, Regolo, and the decision-model route) |
|
|
|
|
## The migration material
|
|
|
|
`migration/` holds the record of moving Cloudron to its new server: the plan
|
|
([`cloudron-server-migration.md`](migration/cloudron-server-migration.md)),
|
|
the record-by-record cutover sheet
|
|
([`dns-change-sheet.md`](migration/dns-change-sheet.md)), and the captured
|
|
pre-migration state the restore checker compares against
|
|
([`migration-before-state.json`](migration/migration-before-state.json)).
|
|
|
|
**The migration is complete (2026-11-02)** — this material is kept as the
|
|
record of how it was planned and done, not as pending work. Current
|
|
operational procedure lives in the co-op's `cloudron-ops` skill; the
|
|
member-facing description lives in `co-op/docs/infrastructure.md`.
|
|
|
|
### Read the DNS sheet with this in mind
|
|
|
|
Its record counts **cannot be trusted until reconciled against the registrar's
|
|
panel.** The zone is wildcarded, so *any* hostname resolves — a name that
|
|
answers proves nothing about whether a record exists. At least two listed
|
|
records turned out not to exist, and the real record inventory is only visible
|
|
in the registrar's own list.
|
|
|
|
## No secrets here
|
|
|
|
Audited 2026-11-01, including the full commit history: no credentials, tokens,
|
|
keys, member emails or account identifiers. Where a runbook needs a credential
|
|
to exist (a backup passphrase, an S3 key), it records **where the credential is
|
|
held**, never the credential.
|
|
|
|
## Where operators look
|
|
|
|
- **Member-facing docs** live in [`co-op/docs`](https://git.inference.coop/co-op/docs)
|
|
and are published at <https://docs.inference.coop/>.
|
|
- The member-facing docs link here from their "For operators" section.
|
|
|
|
## History
|
|
|
|
The operator runbook, maintenance pages and incident write-ups were moved here
|
|
from `co-op/docs`, where they were previously published on the member-facing
|
|
site. The migration material came from `co-op/runbooks` (now archived), and was
|
|
brought across with its commit history intact via an unrelated-histories merge.
|