inference-bot
6f2a6d01ed
- Migration addendum: replace the 'DNS split' wording with the verified state — every hostname resolves to the new box (A+AAAA) per Cloudflare's public resolver; nothing points at the old box. - Reframe the old box: not a benign rollback spare but a decommissioned server still alive and internet-exposed (22/80/443/587/993 open), holding old data and config, serving no member traffic. UNRESOLVED — operator decision to shut it down or firewall it. Do not act on it without the operator's word. - README: add a 'Known open items' section listing the exposure, the CI failure, the unpushed CI-agent repo and the unconfirmed member mailbox contents.
5.1 KiB
Inference Cooperative — operator documentation
Operator- and maintainer-facing documentation for the Inference Cooperative. This is the material that describes how the co-op's infrastructure is run — access tokens, scheduled tasks, maintenance procedures, outage write-ups, and the server-migration record — as opposed to the member-facing documentation published at https://docs.inference.coop/.
Anyone may read it; operators are the audience. Comment, or open an issue, where something looks wrong — that review is the point.
This repository is deliberately not a website
Unlike co-op/docs, nothing here is built or published by MkDocs. The
repository is git-only: it is meant to be browsed, searched, and diffed in
Gitea, or read from a clone. Every page carries operational detail (token
rotation, deploy flows, failure modes) that belongs to operators rather than to
members, so it is kept out of the public documentation site on purpose.
Read it on the web here: https://git.inference.coop/co-op/ops-docs
What is here
| Path | Contents |
|---|---|
operator-token-runbook.md |
The Cloudron token model (agent vs portal, admin vs webhook), how to rotate each without breaking the stack |
app-inventory.md |
Every Cloudron app: id, fqdn, registry image path and pin (read from the running apps) |
agent-fleet.md |
The three agents (steward, infrastructure-bot, socialbot): Unix users, gateway units, teams, and the planned member-bot tier |
maintenance/chat-task-models.md |
The small pinned model used for chat titles/tags/suggestions |
maintenance/librechat-sandbox.md |
Operating the LibreChat evaluation instance (config, overlay image, key sync, pitfalls) |
maintenance/backups.md |
The two backup systems (agent-host restic + Cloudron app backup): schedules, retention, the cache split and the grouping fix |
maintenance/woodpecker-ci.md |
The self-hosted CI server and agent: build/pin, manual-only trigger, git-lfs, and its not-yet-green state |
incidents/ |
Post-incident write-ups, one file per incident |
migration/ |
The Cloudron server-migration record: the plan, the DNS change sheet, and the captured pre-migration state |
provider-pricing/ |
What each model provider charges, and when it changed (GreenPT, Tinfoil, Regolo, and the decision-model route) |
The migration material
migration/ holds the record of moving Cloudron to its new server: the plan
(cloudron-server-migration.md),
the record-by-record cutover sheet
(dns-change-sheet.md), and the captured
pre-migration state the restore checker compares against
(migration-before-state.json).
The migration is complete (2026-11-02) — this material is kept as the
record of how it was planned and done, not as pending work. Current
operational procedure lives in the co-op's cloudron-ops skill; the
member-facing description lives in co-op/docs/infrastructure.md.
Read the DNS sheet with this in mind
Its record counts cannot be trusted until reconciled against the registrar's panel. The zone is wildcarded, so any hostname resolves — a name that answers proves nothing about whether a record exists. At least two listed records turned out not to exist, and the real record inventory is only visible in the registrar's own list.
Known open items — recorded, not hidden
These are genuinely unresolved; a tidy document that omits them would be wrong.
- The old box
179.199.135.72is still alive and internet-exposed (22/80/443/587/993 open), holding old data and config. Nothing in DNS points at it, so it serves no member traffic — but it needs an operator decision: shut it down or firewall it. Do not act on it without the operator's word. Seemigration/cloudron-server-migration.md(addendum). - CI is not green. The Woodpecker agent is healthy but the server answers its
job-collection stream with HTTP 500, so no pipeline can run. See
maintenance/woodpecker-ci.md. - The CI agent image source (
/home/ubuntu/repos/woodpecker-agent) has no git remote — it exists only on the agent host. - Member mailbox contents after the migration are still unconfirmed (needs a real IMAP login).
No secrets here
Audited 2026-11-01, including the full commit history: no credentials, tokens, keys, member emails or account identifiers. Where a runbook needs a credential to exist (a backup passphrase, an S3 key), it records where the credential is held, never the credential.
Where operators look
- Member-facing docs live in
co-op/docsand are published at https://docs.inference.coop/. - The member-facing docs link here from their "For operators" section.
History
The operator runbook, maintenance pages and incident write-ups were moved here
from co-op/docs, where they were previously published on the member-facing
site. The migration material came from co-op/runbooks (now archived), and was
brought across with its commit history intact via an unrelated-histories merge.