0 Fork 0

Files

T

inference-bot 2eb88f55b4 docs: infrastructure audit — app inventory, backups, CI, provider pricing, agent fleet

Audit of the operator docs against live reality (2026-11-04):
- app-inventory.md: all 19 Cloudron apps with id, fqdn and registry image pin.
  Corrects the portal path (code/member-portal, not co-op/member-portal) and the
  panel path (local/coop.inference.admin-panel, not code/admin-panel).
- maintenance/backups.md: the two backup systems; the deliberate per-user restic
  cache split and the --group-by "" retention fix, with why.
- maintenance/woodpecker-ci.md: CI server + agent, image build/pin, manual-only
  trigger, git-lfs requirement, and its current not-yet-green state (agent's
  job-collection long-poll gets HTTP 500 from the server).
- provider-pricing/: index + GreenPT, Tinfoil (local EHBP proxy architecture) and
  the Metis /v1/systemone decision-model route; regolo.md updated to record the
  gateway prices were applied.
- agent-fleet.md: steward, infrastructure-bot and socialbot (Unix users, gateway
  units, teams); member-bot tier 2 planned but not yet provisioned.
- README: file list; operator-token-runbook: correct the membership_sweep cadence
  (3x daily, not nightly); migration doc: post-migration addendum (16->19 apps,
  DNS split resolved, old box still on with apps stopped).

2026-11-04 14:46:12 +00:00

4.2 KiB

Raw Blame History

Inference Cooperative — operator documentation

Operator- and maintainer-facing documentation for the Inference Cooperative. This is the material that describes how the co-op's infrastructure is run — access tokens, scheduled tasks, maintenance procedures, outage write-ups, and the server-migration record — as opposed to the member-facing documentation published at https://docs.inference.coop/.

Anyone may read it; operators are the audience. Comment, or open an issue, where something looks wrong — that review is the point.

This repository is deliberately not a website

Unlike co-op/docs, nothing here is built or published by MkDocs. The repository is git-only: it is meant to be browsed, searched, and diffed in Gitea, or read from a clone. Every page carries operational detail (token rotation, deploy flows, failure modes) that belongs to operators rather than to members, so it is kept out of the public documentation site on purpose.

Read it on the web here: https://git.inference.coop/co-op/ops-docs

What is here

Path	Contents
`operator-token-runbook.md`	The Cloudron token model (agent vs portal, admin vs webhook), how to rotate each without breaking the stack
`app-inventory.md`	Every Cloudron app: id, fqdn, registry image path and pin (read from the running apps)
`agent-fleet.md`	The three agents (steward, infrastructure-bot, socialbot): Unix users, gateway units, teams, and the planned member-bot tier
`maintenance/chat-task-models.md`	The small pinned model used for chat titles/tags/suggestions
`maintenance/librechat-sandbox.md`	Operating the LibreChat evaluation instance (config, overlay image, key sync, pitfalls)
`maintenance/backups.md`	The two backup systems (agent-host restic + Cloudron app backup): schedules, retention, the cache split and the grouping fix
`maintenance/woodpecker-ci.md`	The self-hosted CI server and agent: build/pin, manual-only trigger, git-lfs, and its not-yet-green state
`incidents/`	Post-incident write-ups, one file per incident
`migration/`	The Cloudron server-migration record: the plan, the DNS change sheet, and the captured pre-migration state
`provider-pricing/`	What each model provider charges, and when it changed (GreenPT, Tinfoil, Regolo, and the decision-model route)

The migration material

migration/ holds the record of moving Cloudron to its new server: the plan (cloudron-server-migration.md), the record-by-record cutover sheet (dns-change-sheet.md), and the captured pre-migration state the restore checker compares against (migration-before-state.json).

The migration is complete (2026-11-02) — this material is kept as the record of how it was planned and done, not as pending work. Current operational procedure lives in the co-op's cloudron-ops skill; the member-facing description lives in co-op/docs/infrastructure.md.

Read the DNS sheet with this in mind

Its record counts cannot be trusted until reconciled against the registrar's panel. The zone is wildcarded, so any hostname resolves — a name that answers proves nothing about whether a record exists. At least two listed records turned out not to exist, and the real record inventory is only visible in the registrar's own list.

No secrets here

Audited 2026-11-01, including the full commit history: no credentials, tokens, keys, member emails or account identifiers. Where a runbook needs a credential to exist (a backup passphrase, an S3 key), it records where the credential is held, never the credential.

Where operators look

Member-facing docs live in co-op/docs and are published at https://docs.inference.coop/.
The member-facing docs link here from their "For operators" section.

History

The operator runbook, maintenance pages and incident write-ups were moved here from co-op/docs, where they were previously published on the member-facing site. The migration material came from co-op/runbooks (now archived), and was brought across with its commit history intact via an unrelated-histories merge.

4.2 KiB Raw Blame History