diff --git a/README.md b/README.md index 47775d7..576249e 100644 --- a/README.md +++ b/README.md @@ -24,11 +24,15 @@ Read it on the web here: | Path | Contents | |---|---| | `operator-token-runbook.md` | The Cloudron token model (agent vs portal, admin vs webhook), how to rotate each without breaking the stack | +| `app-inventory.md` | Every Cloudron app: id, fqdn, registry image path and pin (read from the running apps) | +| `agent-fleet.md` | The three agents (steward, infrastructure-bot, socialbot): Unix users, gateway units, teams, and the planned member-bot tier | | `maintenance/chat-task-models.md` | The small pinned model used for chat titles/tags/suggestions | | `maintenance/librechat-sandbox.md` | Operating the LibreChat evaluation instance (config, overlay image, key sync, pitfalls) | +| `maintenance/backups.md` | The two backup systems (agent-host restic + Cloudron app backup): schedules, retention, the cache split and the grouping fix | +| `maintenance/woodpecker-ci.md` | The self-hosted CI server and agent: build/pin, manual-only trigger, git-lfs, and its not-yet-green state | | `incidents/` | Post-incident write-ups, one file per incident | | `migration/` | The Cloudron server-migration record: the plan, the DNS change sheet, and the captured pre-migration state | -| `provider-pricing/` | What our model providers charge, and when it changed | +| `provider-pricing/` | What each model provider charges, and when it changed (GreenPT, Tinfoil, Regolo, and the decision-model route) | ## The migration material diff --git a/agent-fleet.md b/agent-fleet.md new file mode 100644 index 0000000..c6c1200 --- /dev/null +++ b/agent-fleet.md @@ -0,0 +1,65 @@ +# The agent fleet + +The co-op runs **more than one agent**, each as its **own Unix user** on the agent +host (`ov-0f018e`, `179.237.103.165`) with its **own Hermes gateway unit** and its +**own LiteLLM team**. Any document that describes "the agent" as a single thing is +now wrong. + +## The three agents today + +| Agent | Unix user | systemd unit | Gateway team (budget) | Model | Presence | +|---|---|---|---|---|---| +| **Inferencebot** (steward) | `ubuntu` | `hermes-gateway.service` | *(co-op account team)* | per `~/.hermes/config.yaml` | Matrix `@inferencebot:inference.coop`; digests, newsletters, membership/key plumbing | +| **infrastructure-bot** | `infrabot` | `hermes-gateway-infrabot.service` | `agent:infrastructure-bot` ($5/30d) | `greenpt/glm-5.3-flash` | Matrix `#infrastructure` (answers only when mentioned); infrastructure scope | +| **socialbot** | `socialbot` | `hermes-gateway-socialbot.service` | `agent:socialbot` ($5/30d) | `greenpt/glm-5.3-flash` | Matrix `#social-drafts`; drafts the co-op's AT Protocol (Bluesky) account `inference.coop` | + +All three gateways run on the one host; each is a separate process, a separate +`.env`, and a separate set of rooms. There is **one LiteLLM gateway** (the Cloudron +app) that all three draw on — the agent host runs no gateway of its own. + +### Who is who (do not conflate them) + +- **Inferencebot** is the **steward** and the fleet administrator: full Cloudron + token, gateway master key, member-data plumbing. It is NOT a member-facing chat + persona. +- **infrastructure-bot** is a **lower-privileged** infrastructure agent. Per its + own SOUL it has **no sudo, no Cloudron admin token, and no member data**; when a + task needs more privilege it hands up to the steward rather than working around + it. Its gateway key has a small monthly budget and a flash-tier allowlist. +- **socialbot** has **no Cloudron, portal or member-data access** and no power over + Matrix rooms; its only writable surface is the `#social-drafts` room, and it + **never publishes an unapproved draft** — silence is its correct failure mode. + +## Member bots — TIER 2 (planned, not yet provisioned) + +The intended path for **member-facing** bots is a second tier: each member bot is a +**Hermes profile under the `infrabot` Unix user**, sharing **one** gateway process +(the infrabot one), with its own gateway team/key. This is tracked as **co-op/support#33**. + +**Status (2026-11-04): not yet provisioned.** `/home/infrabot/.hermes/profiles/` +holds only a `.deleted` directory with two probe files — there are **no active +member-bot profiles**. Record this honestly: the design exists, the instances do +not. + +## Why one Unix user per agent — and the limit of it + +**Profile isolation is organizational, not security.** Run one OS user (or a +rootless container) per agent from day one: **on a shared Unix user, sibling +Hermes profiles can read each other's `.env`.** So tier-2 member bots sharing the +`infrabot` user are isolated from *members* but **not from each other** — that +sharing is accepted for member bots, not extended to the privileged agents. + +## Backup coverage is automatic — but prove it + +The nightly restic run (see `maintenance/backups.md`) enumerates `/home/*` **as +root** and stages every agent home, so a new agent is covered without editing any +list. Coverage is still **unproven until that agent's files show up in a +snapshot**. + +## Gateway teams (member vs agent) + +From `GET https://gateway.inference.coop/team/list` (2026-11-04): 97 teams — one +`member:` team per member (mostly $15/30d), `member:info@inference.coop` +($135/30d, the co-op account), the two `agent:*` teams above, and +`sandbox:librechat-test` ($3). Agent teams carry a **small budget and a flash-tier +model allowlist** enforced at the gateway. diff --git a/app-inventory.md b/app-inventory.md new file mode 100644 index 0000000..85d373a --- /dev/null +++ b/app-inventory.md @@ -0,0 +1,82 @@ +# Cloudron app inventory + +The co-op's Cloudron server carries **19 apps**. Before 2026-11-02 it carried 16; +the three added since the migration are the docs site, the CI server and the CI +agent (marked **NEW** below). + +**Source of truth:** read it from the running apps, not from memory or a sibling +deploy. `GET https://my.inference.coop/api/v1/apps` returns every app with its +`id`, `fqdn`, install state and — inside `manifest` — `dockerImage`. The app +record's own top-level `image` field is **null**; the CLI's `list` output does +not carry the image at all. The registry paths are **per-app and not uniform** +(there are three different namespaces in use — see below), and a path copied from +a remembered deploy of a *different* app points the update at a repository that +does not hold it. + +*Verified against the live API 2026-11-04.* + +## The 19 apps + +| FQDN | App id | Label | Image (pin) | Install | +|---|---|---|---|---| +| `inference.coop` | `55f5f904-3233-44de-ad7b-5dc7c8533b75` | — | `cloudron/io.cloudron.surfer:202611021342220000` | store (Surfer — marketing site) | +| `chat.inference.coop` | `4023ccbf-d57a-4500-97df-14de9cf3516e` | AI Chat | `registry.inference.coop/co-op/librechat-custom:20261031-2a98892` | **custom** (LibreChat overlay) | +| `chat-legacy.inference.coop` | `50ae6c97-e851-4d17-b013-51ca7a4e5be0` | Legacy AI Chat | `cloudron/com.openwebui.cloudronapp:202611030420300000` | store (Open WebUI) | +| `portal.inference.coop` | `d4f05e4b-75c6-492e-9faf-7afb0712fc94` | — | `registry.inference.coop/code/member-portal:20261104-d076729` | **custom** | +| `dashboard.inference.coop` | `daf8750e-2683-4104-83ea-f77d1b4601e8` | Dashboard | `registry.inference.coop/code/member-dashboard:20261103-3453131` | **custom** | +| `panel.inference.coop` | `ae187071-7a43-4a6e-8a02-986609a66a1a` | Admin Panel | `local/coop.inference.admin-panel:0.1.0-1793634279942` | **custom** | +| `gateway.inference.coop` | `adababd4-2427-4bdd-9bf7-6bdad0302610` | LiteLLM | `registry.inference.coop/local/ai.coop.litellm:1.0.0-20261103-d6cfe78` | **custom** | +| `matrix.inference.coop` | `aa6f4597-6206-4509-b32e-c89750b2c1d8` | Matrix Chat | `cloudron/org.matrix.synapse:202611030719520000` | store (Synapse) | +| `git.inference.coop` | `c91c13aa-30ed-4af9-9599-7a87ab4f221d` | Code | `cloudron/io.gitea.cloudronapp:202611011719500000` | store (Gitea) | +| `registry.inference.coop` | `c257ba3c-6737-4375-ad51-ea19586bf3f0` | — | `cloudron/io.cloudron.container-registry:202610221511140000` | store (container registry) | +| `vault.inference.coop` | `cb838bcf-f2e2-4670-b117-dca04b0ee957` | — | `cloudron/com.github.bitwardenrs:202611020820270000` | store (Vaultwarden) | +| `forum.inference.coop` | `8b1a972c-08af-4e92-852c-d573c37b01ad` | Forum | `cloudron/org.loomio.cloudronapp:202611010321240000` | store (Loomio) | +| `newsletter.inference.coop` | `eed2217c-b165-430e-bc29-7ccfef9b11e4` | — | `cloudron/app.listmonk.cloudronapp:202610220221130000` | store (Listmonk) | +| `search.inference.coop` | `d7c73fff-ee42-4db0-98c0-4ce7a8ab35cc` | — | `cloudron/io.github.ascimoo.searx:202611012319290000` | store (SearXNG) | +| `cloud.inference.coop` | `fd261d67-b790-424f-9e82-f74883ee8585` | — | `cloudron/com.nextcloud.cloudronapp:202610211606270000` | store (Nextcloud) | +| `email.inference.coop` | `fbb256b9-ead3-41d8-bee0-d5ea87149436` | — | `cloudron/io.cloudron.mail:202610261602550000` | store (Cloudron mail UI) | +| `docs.inference.coop` | `85b7ce06-e2af-49a4-bd4f-a9c2e692e3ab` | Docs | `cloudron/io.cloudron.surfer:202611021342220000` | store (Surfer) — **NEW 2026-11-03** | +| `ci.inference.coop` | `4984f95e-0c0a-4af3-9ce1-4ca4f3e8483d` | Woodpecker | `cloudron/org.woodpecker_ci.cloudronapp:202610300219490000` | store (Woodpecker server) — **NEW 2026-11-03** | +| `ci-agent.inference.coop` | `5ac63873-cf2c-4ad9-acd6-81dc0e016c83` | — | `registry.inference.coop/co-op/woodpecker-agent:20261104-e070275` | **custom** (Woodpecker agent) — **NEW 2026-11-03** | + +(`my.inference.coop` is the Cloudron dashboard hostname — the platform itself, not +an app object — so it does not appear here.) + +## Registry namespaces — they are NOT uniform + +Five custom apps pull from four different namespaces. Deriving one path from +another app's pattern deploys the wrong (or a missing) image: + +| App | Registry path | +|---|---| +| chat (LibreChat overlay) | `registry.inference.coop/co-op/librechat-custom` | +| CI agent | `registry.inference.coop/co-op/woodpecker-agent` | +| member portal | `registry.inference.coop/code/member-portal` | +| member dashboard | `registry.inference.coop/code/member-dashboard` | +| gateway (LiteLLM) | `registry.inference.coop/local/ai.coop.litellm` | +| admin panel | `local/coop.inference.admin-panel` | + +**Correcting a stale memory:** the portal is `code/member-portal` (not +`co-op/member-portal`), and the admin panel is `local/coop.inference.admin-panel` +(not `code/admin-panel`). Both were cited wrongly in earlier notes; the values +above are read from the running apps. + +## Store vs custom, and what that decides + +Six apps are **custom** installs (packaged by us, `appStoreId == ""`): chat, +CI agent, portal, dashboard, panel, gateway. Custom apps get **no automatic +update** (Cloudron says "not available for custom apps") — their protection is the +`manifest.dockerImage` pin, which an update deploys, so a custom app must never be +pinned to a floating tag. The rest carry a real `appStoreId` and update through +the store. + +## Notes that are easy to get wrong + +- **`enableAutomaticUpdate` exists on every app object and means nothing for a + custom one.** Judge update behaviour by the install SOURCE (`appStoreId`), not + by that flag. +- **Public collectives reach `co-op/*` and `code/*` repos; the only private repos + are `ops/inferencebot` and `ops/security`.** +- **`info@inference.coop` is the co-op's gateway team account** (`member:info@inference.coop`, + budget $135/30d) and also the owner account of several provider keys — see the + token runbook. diff --git a/maintenance/backups.md b/maintenance/backups.md new file mode 100644 index 0000000..eab6661 --- /dev/null +++ b/maintenance/backups.md @@ -0,0 +1,137 @@ +# Backups + +There are **two independent backup systems**, on two different machines, with two +different targets and two different passphrases. Confusing them is the most likely +mistake — they do not back each other up, and neither one covers the other's host. + +| | Agent-host backup | Cloudron app backup | +|---|---|---| +| What | The agent host's state (all agent homes, `~/.hermes`, `~/.ssh`, `~/repos`, gateway unit files) | Every Cloudron app's `/app/data` + addon databases, plus platform config | +| Machine | Agent host `ov-0f018e` (`179.237.103.165`) | The Cloudron server `179.237.86.116` | +| Tool | `restic` (client-side encrypted, run nightly by a systemd timer) | Cloudron's built-in backup (dashboard-configured) | +| Target | Infomaniak **Swiss Backup** (S3-compatible) | Infomaniak **Swiss Backup** (S3-compatible, `s3-v4-compat`) | +| Schedule | 04:30 UTC + up to 5 min jitter (`agent-backup.timer`) | daily 23:00 UTC (`00 00 23 * * *`) | +| Retention | 7 daily / 4 weekly / 3 monthly, no grouping | 7 days (`keepWithinSecs: 604800`) | +| Config lives | `~/.hermes/scripts/backup.sh`, unit at `/etc/systemd/system/agent-backup.*` | Cloudron dashboard (no write API — see below) | +| Last verified | 2026-11-04 (13 snapshots in the repo) | 2026-11-02 (runs 2026-10-30 → 11-03 all `normal`, ~5.4 GB) | + +Nothing on the Cloudron **host OS** is backed up by either system (netplan, sshd, +`authorized_keys`, hand-edited `/etc`) — build the box, do not expect to restore +it. This was the migration's central caveat and it still holds. + +--- + +## 1. Agent-host backup (restic) + +`agent-backup.timer` → `agent-backup.service` → `~/.hermes/scripts/backup.sh`. + +- **The unit runs as `User=root` on purpose.** The agent homes under `/home/*` + are mode `0700`, so only root can enumerate them; `ubuntu` sees `EACCES` and a + non-root `[ -d .../.hermes ]` test reports **false for a home that exists** — + which would silently skip that agent. The script therefore discovers agents by + enumerating `/home/*` **as root** and stages each one; a new agent is covered + without editing any list. Coverage is still unproven until its files show up in + a snapshot. +- **It backs up:** `/home/ubuntu/.hermes` (config, `.env`, skills, cron, memory, + logs, state DBs), `/home/ubuntu/.ssh`, `/home/ubuntu/repos`, every other + agent's `/home//.hermes`, and `/etc/systemd/system/hermes-gateway*.service`. + It **excludes** rebuildable bulk (`tools/`, `hermes-agent/`, `installs/`, + `cache/`, `lsp/`, model caches) and the live SQLite files — consistent copies of + those are staged first with `sqlite3 .backup`. +- **Failure is visible, not silent.** On failure the script appends a note to + `~/.hermes/cron/updates_notes.jsonl`, which the daily stewarding digest reads. + Exit codes: 0 ok, 1 config, 2 backup, 3 prune. + +### Deliberate-but-surprising: the PER-USER restic cache split + +`RESTIC_CACHE_DIR` defaults to **`/var/cache/restic`** (root's run), while manual +runs by `ubuntu` use **`~/.cache/restic-manual`**. This is intentional — **do not +"tidy" it back into one shared `~/.cache/restic`.** + +A shared cache directory let each user create files the other could not read; +restic then **retried every snapshot with exponential backoff** (30s, 1m25s, …) +until the run crawled or timed out. The visible symptom (2026-11-03) was a prune +that "failed" **three nights running** — which looked exactly like retention being +broken, when it was a permissions/retry storm. Splitting the cache per user ended +it. The lock and log also live **under `$HOME/.hermes/`** (not `/tmp`): with +`fs.protected_regular`, a root run is denied a sticky-dir file owned by another +user (`EACCES`), and under `set -e` that killed the root run *silently*. + +### Deliberate-but-surprising: `--group-by ""` on retention + +Retention is: + +``` +restic forget --tag nightly --group-by "" \ + --keep-daily 7 --keep-weekly 4 --keep-monthly 3 --prune +``` + +**`--group-by ""` (no grouping) is required, not cosmetic.** restic's default is +`--group-by host,paths`, which splits the nightly snapshots into one group per +distinct **paths set**. The backup glob `/etc/systemd/system/hermes-gateway*.service` +grew from one unit to three when the infrabot and socialbot gateways were added +(2026-10-30), so the paths set changed mid-history: snapshots before 2026-10-30 +formed one group and snapshots after formed another. Each group then sat **under** +the keep-7-daily limit *on its own*, so **nothing was ever expired and the repo +could never shrink**. Empty grouping applies one policy across all nightly +snapshots. `--group-by host` would also collapse today's groups, but the box has +already migrated once and a hostname change would reintroduce the same split — no +grouping is immune to both path-set drift and a hostname change. + +**Current state (honest):** the repo still holds **13 snapshots** — the pre-fix +accumulation. The script on disk carries the fix, and a `--dry-run` (2026-11-04) +confirms it **keeps 8 and removes 5**. The next timer run (2026-11-05 04:31 UTC) +applies it. Until it runs, treat "13 snapshots" as expected, not as the fix +having failed. + +### Other mechanics + +- Weekly `restic check` on Sundays (repo metadata); full data check monthly. +- The passphrase (`RESTIC_PASSWORD`) lives in `~/.hermes/.env` **and** the co-op + Vault. A backup whose key exists only on the backed-up box is not a backup. +- S3 credentials come from `SWB_S3_ACCESS_KEY` / `SWB_S3_SECRET_KEY` (mapped to + `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` in the script). + +## 2. Cloudron app backup + +- Configured in the **dashboard** (*Settings → Backups*). There is **no write + API**: `GET /api/v1/backup_sites` works, but `PUT`/`POST` on it and + `POST /backups` all 404. Schedule and retention are dashboard-only. +- **Two sites exist:** the live one (`s3-v4-compat`, daily 23:00, 7-day + retention, id `03cfbdca-88fc-4ccc-97c3-7e1b58cf892d`) and an inactive + `filesystem` site with schedule `never`. Do not be alarmed by the second one. +- **A run reads `state=normal` even when it failed.** The failure's real error + text is in `GET /api/v1/tasks` (task id `backup_`, field `error.message`), + not in the run list. `/events` needs a `?since=` range and `/system/logs` does + not exist — a 404 there is not "no failures". +- `/backup_sites` carries `schedule` as a 6-field cron and `retention.keepWithinSecs`. + Check retention **before** claiming it needs changing. +- **Quota math:** a failure whose upload pushed the retained set over the + provider ceiling is the signature. The old box's `EntityTooLarge` failures came + from a 4.82 GB run twice daily against the ceiling; the ceiling was raised and + the schedule moved to once daily. Per-run size still grows (mostly registry + image blobs) — watch the number. +- **Two boxes on one S3 bucket interleave and prune each other's backups.** The + old box's schedule must stay **disabled** while it is kept for rollback. +- **Cross-check `siteId` per run:** runs on an old site id are pre-migration + leftovers, kept `state=normal` forever and never pruned by the current site's + retention. + +## Restoring + +- **Cloudron** restores apps per-app, from the **newest** box-backup entry: + `GET /apps/:id/backups` → pick by `creationTime` → `POST /apps/:id/restore` + with `{"backupId": ...}`. Restore apps **sequentially** (parallel restores + exhaust disk). `repair` recreates a container but restores **no data** — a + repaired app comes up empty. +- Full procedure, triage and the post-restore checker: the `cloudron-ops` skill + and `migration/cloudron-server-migration.md`. +- **Agent host:** `restic restore latest --target /` (or a specific snapshot) with + `RESTIC_PASSWORD` and the S3 keys in the environment, run as root. + +## The credential that carries the backups + +The agent-host `RESTIC_PASSWORD` and the Cloudron backup passphrase are **two +different secrets**, both held by the GM and recorded in the token runbook's +inventory. The Cloudron backup passphrase + S3 `secretAccessKey` are the +chicken-and-egg pair a bare-metal rebuild needs first — they must be held off-box. diff --git a/maintenance/woodpecker-ci.md b/maintenance/woodpecker-ci.md new file mode 100644 index 0000000..85f8f8f --- /dev/null +++ b/maintenance/woodpecker-ci.md @@ -0,0 +1,94 @@ +# Woodpecker CI (server + agent) + +Self-hosted CI for the co-op's repos. **As of 2026-11-04 the CI is installed but +NOT working end to end** — read the *Current state* section before trusting it for +anything. + +Two Cloudron apps, added 2026-11-03: + +| App | FQDN | Cloudron id | Image | +|---|---|---|---| +| Woodpecker **server** | `ci.inference.coop` | `4984f95e-0c0a-4af3-9ce1-4ca4f3e8483d` | `cloudron/org.woodpecker_ci.cloudronapp:202610300219490000` (store, v2.22.1 / upstream 3.20.1) | +| Woodpecker **agent** | `ci-agent.inference.coop` | `5ac63873-cf2c-4ad9-acd6-81dc0e016c83` | `registry.inference.coop/co-op/woodpecker-agent:20261104-e070275` (**custom**) | + +- The **server** is the stock Cloudron app. It uses the `mysql` addon for storage + and exposes `GRPC_PORT` 9000 to the agent. Its config lives in + `/app/data/env.sh` (`WOODPECKER_GITEA`, `WOODPECKER_ADMIN`, + `WOODPECKER_AGENT_SECRET`, `WOODPECKER_OPEN`, …). **No env vars are set through + the app's Cloudron env** (`env-keys` is empty) — everything is in that file. +- The **agent** is a custom app. Its config is set as Cloudron env (names only): + `WOODPECKER_SERVER` = `ci.inference.coop:9000`, `WOODPECKER_GRPC_SECURE=true`, + `WOODPECKER_BACKEND=local`, `WOODPECKER_MAX_WORKFLOWS=2`, + `WOODPECKER_AGENT_CONFIG_FILE=/app/data/agent.conf`, plus the shared + `WOODPECKER_AGENT_SECRET`. + +## Why the agent image is custom (and why the backend is `local`) + +Cloudron apps have **no Docker socket**, so the default `docker` backend cannot +run and the agent uses the **`local` backend**: each pipeline step executes as a +shell command *inside the agent container*, and a step's `image:` field selects +the **shell**, not a container image. Consequence: **every tool a pipeline step +needs must be baked into the agent image**, and the Cloudron rootfs is read-only +at run time, so nothing can install at run time. + +The image is `FROM woodpeckerci/woodpecker-agent:v3.20.1-alpine` (the **alpine** +variant — the default `scratch` image has no shell). It adds `bash`, `git`, +**`git-lfs`**, `tar`, `ca-certificates`, `python3`, `node`/`npm`, the `cloudron` +CLI, and a pinned MkDocs toolchain at `/opt/docs-venv`. It **runs as root** +(`/app/data` on this Cloudron is root-owned, as for the other custom apps here). + +## Building and pinning the agent image + +- **Source repo:** `/home/ubuntu/repos/woodpecker-agent` on the agent host. + ⚠️ **This repo has no git remote and has never been pushed to Gitea** — the CI + agent's build source exists only on the agent host. Treat that as a gap to close. +- **Tag convention:** `-` (e.g. `20261104-e070275`). Never a + floating tag. +- **Build:** `~/.hermes/scripts/cloudron_builder_build.sh registry.inference.coop/co-op/woodpecker-agent ` + (the script carries the registry auth triple, which is not optional). +- **Deploy:** write the new tag into `CloudronManifest.json`'s `dockerImage`, + commit, build `--no-cache`, then `cloudron update --app 5ac63873-…` **without** + `--image` (the manifest's pin wins). Verify by the running image tag **and** + something only the new build produces — never by exit code alone. +- `publishing-requirements.txt` mirrors the docs repo's + `publishing/requirements.txt`; **keep the two in step** when the docs toolchain + changes. + +## When a pipeline runs (and when it does not) + +- **Manual-trigger only.** Nothing here fires on push; a pipeline is started by + hand from the Woodpecker UI. +- **The pipeline is not yet green end to end.** +- **`git-lfs` is required by the clone step.** Woodpecker runs `git lfs fetch` + after cloning, so a clone step fails with + `git: 'lfs' is not a git command` on any image without it — this was added to + the agent image on 2026-11-04. If a rebuild ever drops it, clones break again. + +## Current state (2026-11-04): installed, agent connected, cannot run jobs + +- The server is up — `https://ci.inference.coop/` serves the Woodpecker UI. +- The agent app is `running`/`healthy` and its health endpoint answers — **but + that is liveness, not capability.** The log shows the agent's **gRPC job-collection + long-poll failing continuously**: the server answers the agent's RPC with + **HTTP 500** (an nginx `text/html` 500 body), so the agent retries with backoff + and **never receives work**. +- A healthy agent that cannot collect jobs is the exact trap to avoid reading as + progress. Score a CI worker by **the route that delivers it work**, not by its + health RPC. + +**Unresolved / needs an operator:** the server-side gRPC 500. Until the server +answers the agent's stream, no pipeline can execute regardless of repository +configuration. This is a server-side (or platform-proxy) problem, not an agent +image problem. + +## Enabling a repo (known gate) + +- Repos are connected through the **Gitea** source (`WOODPECKER_GITEA`). Enabling + a repo requires the repo owner to log in to Woodpecker with **Gitea OAuth** and + activate it. Standing up that OAuth login source is an **operator step** + (creating the Gitea OAuth application needs `write:user`, which the bot's Gitea + token does not have). +- Reading run/pipeline state is done from the server's **MySQL** database, not + from a public API. +- Note: the whole point of this CI is to take the docs build and custom-app builds + off the agent host — see `co-op/support#30` (moving `REGISTRY_BUILD_TOKEN` to CI). diff --git a/migration/cloudron-server-migration.md b/migration/cloudron-server-migration.md index 3cde571..519d397 100644 --- a/migration/cloudron-server-migration.md +++ b/migration/cloudron-server-migration.md @@ -508,3 +508,57 @@ access removed 2026-11-02 — retirement begun; app restored and safe in backups **Outstanding:** tonight 23:00 UTC first backup (verify `normal`, ~4.9 GiB); Cloudron subscription page check on the new dashboard; my primary model route re-heals on its own (fallback worked all night); old box keep powered-on/apps-stopped/backups-disabled for a few days, then decommission. + +--- + +# POST-MIGRATION ADDENDUM — 2026-11-04 + +Recorded by the infra-docs audit. This section is current state; everything above +is the as-it-happened record. + +## App count: 16 → 19 (three apps added since cutover) + +The migration restored **16 apps**. Since then the server carries **19** — the +three new ones are the docs site, the CI server and the CI agent. See +`app-inventory.md` for the full list. Anywhere above that says "16 apps" is the +state at cutover, not today. + +| New since 2026-11-02 | FQDN | Cloudron id | +|---|---|---| +| Docs site (Surfer) | `docs.inference.coop` | `85b7ce06-e2af-49a4-bd4f-a9c2e692e3ab` | +| Woodpecker CI server | `ci.inference.coop` | `4984f95e-0c0a-4af3-9ce1-4ca4f3e8483d` | +| Woodpecker CI agent (custom) | `ci-agent.inference.coop` | `5ac63873-cf2c-4ad9-acd6-81dc0e016c83` | + +## The DNS "split" is resolved — no hostname is served by the old box + +At one point `chat.inference.coop` and `portal.inference.coop` were observed still +resolving to the **old** box (`179.199.135.72`) while `gateway`/`my`/`docs` were on +the **new** box (`179.237.86.116`). **As of 2026-11-04 there is no such split:** +every hostname — `chat`, `portal`, `gateway`, `my`, `docs`, `ci`, `ci-agent`, +`matrix`, `git`, `www` and the apex — resolves to the **new** box. + +The zone is **wildcarded**, so a resolving name proves nothing by itself. The +decisive fact: a **made-up** hostname (`zzznope.inference.coop`) also answers the +**new** box `179.237.86.116` at the authoritative nameservers (`ns1.yoursrs.com`). +That means the **wildcard itself now points at the new box**, so no name is being +served by the old box via the wildcard either. + +## The old box still exists (rollback), and serves nothing + +- `179.199.135.72` is **still powered on** and answers on 22/80/443/587/993, **but + it hosts no running app.** A request with `Host: chat.inference.coop` returns + Cloudron's **"app down / Not Found" fallback page**, and its TLS certificate has + an **empty subject** — the signature of a Cloudron whose vhost's app is stopped, + exactly as documented ("powered on with apps stopped"). +- **No DNS reference points at it**, so it receives no real traffic. +- **Decommission is still pending** — this is the one open item from the cutover + that has not happened. Its backup schedule must remain **disabled** while it is + kept (two boxes on one S3 bucket interleave and prune each other's backups). + +## Still unresolved / for the operator + +- **Decommission the old box** (`179.199.135.72`) when rollback is no longer + wanted, after confirming the new box's backups are proven. +- **Member mailbox contents** — the long-standing unknown from §7 (whether + members' mail is inside the Cloudron backup) still needs a real IMAP login to + confirm. diff --git a/operator-token-runbook.md b/operator-token-runbook.md index fa438b7..6658d93 100644 --- a/operator-token-runbook.md +++ b/operator-token-runbook.md @@ -188,8 +188,9 @@ container). The `PYTHONPATH` env survives restores (it's in app config). The portal and gateway run their own housekeeping via the Cloudron scheduler addon, declared in each app's `CloudronManifest.json` under `addons.scheduler`: -- **portal**: `membership_sweep` (nightly 3am, `/app/code/sweep.sh` → - `POST /admin/sweep` with the **`ADMIN_TOKEN`** header — NOT `WEBHOOK_TOKEN`, +- **portal**: `membership_sweep` (three times daily — `0 3,11,19 * * *` UTC per + the live manifest; earlier notes said "nightly at 3am" and are stale. Runs + `/app/code/sweep.sh` → `POST /admin/sweep` with the **`ADMIN_TOKEN`** header — NOT `WEBHOOK_TOKEN`, which returns 403 on `/admin/*` since the 2026-10-28 cutover), `rag_refresh` (nightly 3:30am, `/app/code/rag_refresh.py` — pulls docs from the Gitea raw API and refreshes the chat knowledge base; needs `OWUI_API_KEY` + diff --git a/provider-pricing/README.md b/provider-pricing/README.md new file mode 100644 index 0000000..c711a0c --- /dev/null +++ b/provider-pricing/README.md @@ -0,0 +1,44 @@ +# Provider pricing + +What our model providers charge, and where the authoritative numbers live. + +## The authority + +**The gateway's own per-model costs are the authority** — they are what the +response headers (`x-litellm-response-cost`) and member billing both read. They +live in the LiteLLM config on the gateway's persistent volume +(`/app/data/config.yaml`), under each model's `model_info`, as **USD per token** +(the comments in that file quote per-1M rates). Read them live with +`GET https://gateway.inference.coop/model/info` (master key), or read the config +straight from the running app. + +**The member-facing mirror** is `co-op/docs/models.md`. When a rate changes, +update **both** the gateway config and `models.md` — the config is authoritative, +`models.md` is the mirror. + +A missing cost field makes a paid feature free; a stale one bills wrong; a missing +`cache_read_input_token_cost` bills cache hits at the full input rate. All three +have happened here. + +## The pages + +| File | Provider | Notes | +|---|---|---| +| `greenpt.md` | GreenPT | renewable energy; the default model lives here | +| `tinfoil.md` | Tinfoil | TEE; the local-proxy architecture lives here | +| `regolo.md` | Regolo | renewable + ZDR; includes our measured usage | +| `decision-model.md` | GreenPT "Metis" via `/v1/systemone` | a pass-through **route**, not a chat model | + +## The routing rule + +Members reach models through one gateway, which fronts all providers. The +per-request cost the gateway records is the number to trust; do not re-derive a +price from a provider's public page (they drift, and some are login-walled). + +## A note on currencies + +Regolo quotes in **EUR**; our gateway figures convert at **EUR × 1.16 → USD** +(the rate our numbers use — `regolo.md` documents the derivation). GreenPT's own +listed rates appear in **EUR with the same numerals the gateway carries** — +whether an EUR→USD conversion is intended there is **unverified**; confirm it +against the GreenPT invoice before treating the figures as final. diff --git a/provider-pricing/decision-model.md b/provider-pricing/decision-model.md new file mode 100644 index 0000000..5caf508 --- /dev/null +++ b/provider-pricing/decision-model.md @@ -0,0 +1,58 @@ +# The decision-model route — GreenPT "Metis" (`/v1/systemone`) + +The gateway exposes a **decision model** — GreenPT's **Metis** (System One API) — +as a **pass-through route**, not a chat model. + +## What it is, and why it is a route and not a model + +Metis is a **decision** endpoint, not a chat completion: it takes a `state` plus up +to 32 typed questions and returns **probabilities / choices / scores**. It +generates no text (`output_tokens` is always 0), so **it cannot be a `model_list` +entry** — LiteLLM's chat pipeline has nothing to parse. It is therefore exposed as +a pass-through: + +``` +path: /v1/systemone +target: https://api.greenpt.ai/v1/systemone +auth: true # members call it with their normal co-op API key +``` + +- **It is API-only and deliberately NOT a chat-picker model.** It never appears in + the chat model list. Do not add it there. +- **Members authenticate with their own co-op key**, never the provider key — the + route's `Authorization` header injects the provider key server-side. +- Hosted by GreenPT (policy-based privacy, EU, 100% renewable), same + `GREENPT_API_KEY` the green chat models use. + +## Pricing and the metering fix + +- **$0.12 per 1M input tokens; output is free.** + +Metering is **exact, from the token count GreenPT reports back**. Because Metis is +not a `model_list` entry, it is priced through the +`coop_passthrough_cost_per_token` map, keyed by the route path +(`/v1/systemone: 0.00000012`). `patch_passthrough_token_cost.py` (built into the +image) reads `usage.input_tokens` from the Metis response body at logging time and +writes the exact figure to member spend. + +**Why this exists — record it, it will recur:** **generic pass-through routes +originally logged nothing.** A route that is not a `model_list` entry gets no +automatic pricing, so its calls were metered as $0 — a paid feature silently free. +The pass-through accounting had to be fixed so spend lands against the **calling +member's key**. If another pass-through route is ever added, it needs the same +treatment or it bills nothing. + +`cost_per_request: 0.0001` on the route is now only a **fallback** — it applies if +a response arrives without a usable token count, so a call is never silently free. +Keep the two in sync. + +## Access control gotcha (community edition) + +The route is opened to any authenticated member key via +`coop_passthrough_open_to_keys` (consumed at build time by +`patch_passthrough_key_access.py`). **LiteLLM's own gate for `auth=true` +pass-through routes (`allowed_passthrough_routes`) is an Enterprise-only field**, +so on this community-edition proxy it can never be granted to member keys. The +only alternative, `auth: false`, would expose the route publicly and forward the +co-op's GreenPT key to anonymous callers. **Authentication is unchanged** — the +endpoint still requires a valid key. diff --git a/provider-pricing/greenpt.md b/provider-pricing/greenpt.md new file mode 100644 index 0000000..0e82092 --- /dev/null +++ b/provider-pricing/greenpt.md @@ -0,0 +1,44 @@ +# GreenPT pricing + +GreenPT is OpenAI-compatible, EU-hosted on **100% renewable energy**. LiteLLM +talks to it **directly** (no TEE proxy). Billing across the gateway: + +- `api_base`: `https://api.greenpt.ai/v1` +- key: `GREENPT_API_KEY` (gateway app env) + +## What we serve, and what it costs + +Rates as recorded in the gateway's authorititative config comment, **USD per 1M +tokens** (the gateway's `model_info` carries the per-token equivalents): + +| our model | input | output | cached input | +|---|---|---|---| +| `greenpt/deepseek-v4.1-flash` **(DEFAULT model)** | $0.22 | $1.10 | $0.011 | +| `greenpt/glm-5.3-flash` | $0.11 | $0.44 | $0.022 | +| `greenpt/glm-5.3` | $1.10 | $4.40 | $0.275 | + +- **`greenpt/deepseek-v4.1-flash` is the gateway's default model.** It is the same + model family as Tinfoil's `tinfoil/deepseek-v4-1-flash`, but on renewable energy + and **not** marked "Experimental" by the provider (Tinfoil's counterpart is), so + it is the default and Tinfoil's is its private counterpart. The policy trade is + **policy-based privacy (GreenPT) vs TEE (Tinfoil)** — say so plainly rather than + implying GreenPT is confidential computing. +- `greenpt/glm-5.3` is the strongest **agentic** model in the green tier (1M context). +- Note the **dotted slug** `glm-5.3-flash` on GreenPT, distinct from Tinfoil's + `glm-5-3-flash` (hyphenated). Getting a model slug wrong is a silent 404 at call + time. + +## Historical + +- **Removed 2026-10:** `green-l` / `green-r` — GreenPT now marks them "not + recommended". +- The config comment previously read "€0.22 in / €1.10 out" alongside the dollar + figures — see the currency note in `README.md`; the intended FX is unverified. + +## The fallback key + +`GREENPT_FALLBACK_API_KEY` (agent host) is the **same value** as the gateway's +`GREENPT_API_KEY` — verified by hash comparison. It powers the steward agent's +direct-to-provider fallback during a gateway outage. **Rotating one without the +other silently breaks the fallback** (a dedicated key is the cleaner end state). +See the token runbook. diff --git a/provider-pricing/regolo.md b/provider-pricing/regolo.md index 1838405..c791b8a 100644 --- a/provider-pricing/regolo.md +++ b/provider-pricing/regolo.md @@ -97,3 +97,21 @@ At these volumes the mispricing is small: apertus's input roughly 2.5x-ing on a **$0.82 lifetime** is about **$1/month** of exposure, and qwen's cut moves ~$0.6/month in members' favour. So this is **billing accuracy, not urgency** — but accuracy still matters, because the same numbers feed the cost headers members can now read. + +## UPDATE 2026-11-04 — the gateway was updated; both stale sections above are now resolved + +Checked against the live gateway config (`/app/data/config.yaml`, `model_info`): + +- `regolo/apertus-70b` = **1.16e-06 / 3.944e-06** per token + ($1.16 / $3.94 per 1M) — the new 2026-11-01 rate. The "~70 cent/M loss while the + gateway still bills $0.46" paragraph above no longer applies. +- `regolo/qwen3.5-122b` = **6.96e-07 / 3.944e-06** ($0.70 / $3.94 per 1M) — the new + rate. The "over-bills members by ~46 cents/M" paragraph no longer applies. +- `regolo/qwen-image` now carries **`input_cost_per_pixel: 1.74e-8`** — the EUR + figure converted at ×1.16, i.e. the "image model is priced in EUR-as-USD" open + inconsistency above is **fixed**. It is no longer priced at the raw EUR number. + +So the two "Until the gateway is updated" tables and the image-model section above +are **historical**. The usage measurement ($4.38/month, ~1.1% of inference spend) +and the subscription analysis stand. Keep this file dated-section-per-change; do +not delete the history, it shows the reasoning. diff --git a/provider-pricing/tinfoil.md b/provider-pricing/tinfoil.md new file mode 100644 index 0000000..a3c16be --- /dev/null +++ b/provider-pricing/tinfoil.md @@ -0,0 +1,84 @@ +# Tinfoil pricing and architecture + +Tinfoil serves models inside **TEE-protected enclaves**. The co-op does not +self-host it — Tinfoil is a hosted cloud service; models run in Tinfoil's TEEs. +Tinfoil is the co-op's **architecturally-private** tier (contrast: GreenPT and +Regolo are the renewable tier, with policy-based privacy, not TEE). + +## The architecture — read this before touching the Tinfoil route + +LiteLLM does **not** talk to Tinfoil's enclave directly. Every Tinfoil model in the +gateway config points at: + +``` +api_base: http://127.0.0.1:3301/v1 +``` + +That is a **local proxy** running **inside the gateway container** +(`/app/data/tinfoil-proxy -b 127.0.0.1 -p 3301`). LiteLLM speaks **plaintext +OpenAI** to the proxy; the proxy **verifies the enclave attestation and wraps the +request/response bodies in EHBP (HPKE)** before forwarding to the Tinfoil +enclave. The provider key is set via the environment variable `TINFOIL_API_KEY`. + +Consequences: + +- **The proxy is load-bearing.** If it is not running, every Tinfoil model fails; + LiteLLM's own config still looks correct. Check the process, not just the config. +- **`http://127.0.0.1:3301` is intentional, not a misconfiguration.** Do not + "fix" it to a public Tinfoil URL — the encryption happens in the proxy. +- The proxy binary lives on the persistent volume (`/app/data/tinfoil-proxy`), so + it survives restarts; it is started by the app's `start.sh`. + +## What we serve, and what it costs + +Rates as recorded in the gateway config, **USD per 1M tokens** unless noted: + +| our model | input | output | cached input | +|---|---|---|---| +| `tinfoil/deepseek-v4-1-flash` | $0.65 | $1.45 | $0.13 | +| `tinfoil/gpt-oss-120b` | $0.15 | $0.60 | — | +| `tinfoil/glm-5.3` | $1.80 | $5.75 | $0.45 | +| `tinfoil/whisper-large-v3-turbo` (STT) | $0 / 1M | — | **$0.01 per request** | +| `tinfoil/voxtral-tts` (TTS) | $0 / 1M | — | **$0.002 per request** | +| `tinfoil/nomic-embed-text` (embeddings) | $0.05 | $0 | — | + +- **`tinfoil/glm-5.3` replaced `tinfoil/glm-5-3-flash`** (Tinfoil deprecated the + flash variant on 2026-10-09). Text-only (no vision), reasoning + tool calling. +- **`tinfoil/deepseek-v4-1-flash` and the glm-flash variant carry an + "Experimental: may be deprecated with minimal notice" flag from Tinfoil** — + which is part of why the GreenPT counterpart is the default. + +### Audio is billed per REQUEST, and needs a build-time patch + +Tinfoil bills the audio models **per request** (its model list carries a +`requestPrice`), not per token. Reading only the two token prices made TTS look +free and left STT billed off an older catalog. **LiteLLM cannot price per request +on the audio paths**, so `patch_audio_request_cost.py` (built into the image via +the Dockerfile) makes the cost calculator honour `input_cost_per_request`. +**Re-check that build-log line after every LiteLLM upgrade.** + +One non-obvious load-bearing detail: for TTS, `input_cost_per_character: 0` is +**required**. Without it the TTS cost branch raises `ValueError("cost for tts call +is None")` and aborts the whole calculation, so the spend write is skipped and +speech bills nothing — whatever `input_cost_per_request` says. + +**Audio keep-alive** (the Cloudron scheduler task `audio_keepalive`, every 10 min) +pings the Tinfoil audio models so voice mode does not pay the ~24s TEE cold-start +after idle. It is **TTS-only by design**: the STT warm ping was disabled +2026-10-11 because it costs $0.01/request — roughly $52/month at 6 pings/hour. +`docs.inference.coop`-style "keep-alive economics" detail is in the script. + +### The embedding model is operator-only + +`tinfoil/nomic-embed-text` (768-dim) is **not** a member-facing model — it is +scoped to the co-op's own admin team (`model_info.team_id`) for the docs-RAG, so +member keys cannot call it. The portal also filters it out of the chat-facing +model list by its `mode: embedding`. + +## Fallbacks + +The router treats `tinfoil/deepseek-v4-1-flash` as the fallback for +`greenpt/deepseek-v4.1-flash`, and `tinfoil/gpt-oss-120b` as the fallback for +`tinfoil/deepseek-v4-1-flash`. The timeout is deliberately generous (180s) with +`TimeoutErrorRetries: 0` — DeepSeek generations routinely exceed 30s, and a +timed-out generation is **already billed** by Tinfoil, so re-hitting it re-bills.