inference-bot
6f2a6d01ed
- Migration addendum: replace the 'DNS split' wording with the verified state — every hostname resolves to the new box (A+AAAA) per Cloudflare's public resolver; nothing points at the old box. - Reframe the old box: not a benign rollback spare but a decommissioned server still alive and internet-exposed (22/80/443/587/993 open), holding old data and config, serving no member traffic. UNRESOLVED — operator decision to shut it down or firewall it. Do not act on it without the operator's word. - README: add a 'Known open items' section listing the exposure, the CI failure, the unpushed CI-agent repo and the unconfirmed member mailbox contents.
93 lines
5.1 KiB
Markdown
93 lines
5.1 KiB
Markdown
# Inference Cooperative — operator documentation
|
|
|
|
Operator- and maintainer-facing documentation for the Inference Cooperative.
|
|
This is the material that describes how the co-op's infrastructure is run —
|
|
access tokens, scheduled tasks, maintenance procedures, outage write-ups, and
|
|
the server-migration record — as opposed to the member-facing documentation
|
|
published at <https://docs.inference.coop/>.
|
|
|
|
Anyone may read it; operators are the audience. Comment, or open an issue,
|
|
where something looks wrong — that review is the point.
|
|
|
|
## This repository is deliberately not a website
|
|
|
|
Unlike `co-op/docs`, **nothing here is built or published by MkDocs**. The
|
|
repository is *git-only*: it is meant to be browsed, searched, and diffed in
|
|
Gitea, or read from a clone. Every page carries operational detail (token
|
|
rotation, deploy flows, failure modes) that belongs to operators rather than to
|
|
members, so it is kept out of the public documentation site on purpose.
|
|
|
|
Read it on the web here: <https://git.inference.coop/co-op/ops-docs>
|
|
|
|
## What is here
|
|
|
|
| Path | Contents |
|
|
|---|---|
|
|
| `operator-token-runbook.md` | The Cloudron token model (agent vs portal, admin vs webhook), how to rotate each without breaking the stack |
|
|
| `app-inventory.md` | Every Cloudron app: id, fqdn, registry image path and pin (read from the running apps) |
|
|
| `agent-fleet.md` | The three agents (steward, infrastructure-bot, socialbot): Unix users, gateway units, teams, and the planned member-bot tier |
|
|
| `maintenance/chat-task-models.md` | The small pinned model used for chat titles/tags/suggestions |
|
|
| `maintenance/librechat-sandbox.md` | Operating the LibreChat evaluation instance (config, overlay image, key sync, pitfalls) |
|
|
| `maintenance/backups.md` | The two backup systems (agent-host restic + Cloudron app backup): schedules, retention, the cache split and the grouping fix |
|
|
| `maintenance/woodpecker-ci.md` | The self-hosted CI server and agent: build/pin, manual-only trigger, git-lfs, and its not-yet-green state |
|
|
| `incidents/` | Post-incident write-ups, one file per incident |
|
|
| `migration/` | The Cloudron server-migration record: the plan, the DNS change sheet, and the captured pre-migration state |
|
|
| `provider-pricing/` | What each model provider charges, and when it changed (GreenPT, Tinfoil, Regolo, and the decision-model route) |
|
|
|
|
## The migration material
|
|
|
|
`migration/` holds the record of moving Cloudron to its new server: the plan
|
|
([`cloudron-server-migration.md`](migration/cloudron-server-migration.md)),
|
|
the record-by-record cutover sheet
|
|
([`dns-change-sheet.md`](migration/dns-change-sheet.md)), and the captured
|
|
pre-migration state the restore checker compares against
|
|
([`migration-before-state.json`](migration/migration-before-state.json)).
|
|
|
|
**The migration is complete (2026-11-02)** — this material is kept as the
|
|
record of how it was planned and done, not as pending work. Current
|
|
operational procedure lives in the co-op's `cloudron-ops` skill; the
|
|
member-facing description lives in `co-op/docs/infrastructure.md`.
|
|
|
|
### Read the DNS sheet with this in mind
|
|
|
|
Its record counts **cannot be trusted until reconciled against the registrar's
|
|
panel.** The zone is wildcarded, so *any* hostname resolves — a name that
|
|
answers proves nothing about whether a record exists. At least two listed
|
|
records turned out not to exist, and the real record inventory is only visible
|
|
in the registrar's own list.
|
|
|
|
## Known open items — recorded, not hidden
|
|
|
|
These are genuinely unresolved; a tidy document that omits them would be wrong.
|
|
|
|
- **The old box `179.199.135.72` is still alive and internet-exposed** (22/80/443/587/993
|
|
open), holding old data and config. Nothing in DNS points at it, so it serves no
|
|
member traffic — but it needs an operator decision: **shut it down or firewall it**.
|
|
Do not act on it without the operator's word. See `migration/cloudron-server-migration.md` (addendum).
|
|
- **CI is not green.** The Woodpecker agent is healthy but the server answers its
|
|
job-collection stream with HTTP 500, so no pipeline can run. See `maintenance/woodpecker-ci.md`.
|
|
- **The CI agent image source (`/home/ubuntu/repos/woodpecker-agent`) has no git
|
|
remote** — it exists only on the agent host.
|
|
- **Member mailbox contents after the migration** are still unconfirmed (needs a
|
|
real IMAP login).
|
|
|
|
## No secrets here
|
|
|
|
Audited 2026-11-01, including the full commit history: no credentials, tokens,
|
|
keys, member emails or account identifiers. Where a runbook needs a credential
|
|
to exist (a backup passphrase, an S3 key), it records **where the credential is
|
|
held**, never the credential.
|
|
|
|
## Where operators look
|
|
|
|
- **Member-facing docs** live in [`co-op/docs`](https://git.inference.coop/co-op/docs)
|
|
and are published at <https://docs.inference.coop/>.
|
|
- The member-facing docs link here from their "For operators" section.
|
|
|
|
## History
|
|
|
|
The operator runbook, maintenance pages and incident write-ups were moved here
|
|
from `co-op/docs`, where they were previously published on the member-facing
|
|
site. The migration material came from `co-op/runbooks` (now archived), and was
|
|
brought across with its commit history intact via an unrelated-histories merge.
|