diff --git a/README.md b/README.md index 0ba0ceb..47775d7 100644 --- a/README.md +++ b/README.md @@ -2,9 +2,12 @@ Operator- and maintainer-facing documentation for the Inference Cooperative. This is the material that describes how the co-op's infrastructure is run — -access tokens, scheduled tasks, maintenance procedures, and outage write-ups — -as opposed to the member-facing documentation published at -. +access tokens, scheduled tasks, maintenance procedures, outage write-ups, and +the server-migration record — as opposed to the member-facing documentation +published at . + +Anyone may read it; operators are the audience. Comment, or open an issue, +where something looks wrong — that review is the point. ## This repository is deliberately not a website @@ -24,16 +27,47 @@ Read it on the web here: | `maintenance/chat-task-models.md` | The small pinned model used for chat titles/tags/suggestions | | `maintenance/librechat-sandbox.md` | Operating the LibreChat evaluation instance (config, overlay image, key sync, pitfalls) | | `incidents/` | Post-incident write-ups, one file per incident | +| `migration/` | The Cloudron server-migration record: the plan, the DNS change sheet, and the captured pre-migration state | +| `provider-pricing/` | What our model providers charge, and when it changed | + +## The migration material + +`migration/` holds the record of moving Cloudron to its new server: the plan +([`cloudron-server-migration.md`](migration/cloudron-server-migration.md)), +the record-by-record cutover sheet +([`dns-change-sheet.md`](migration/dns-change-sheet.md)), and the captured +pre-migration state the restore checker compares against +([`migration-before-state.json`](migration/migration-before-state.json)). + +**The migration is complete (2026-11-02)** — this material is kept as the +record of how it was planned and done, not as pending work. Current +operational procedure lives in the co-op's `cloudron-ops` skill; the +member-facing description lives in `co-op/docs/infrastructure.md`. + +### Read the DNS sheet with this in mind + +Its record counts **cannot be trusted until reconciled against the registrar's +panel.** The zone is wildcarded, so *any* hostname resolves — a name that +answers proves nothing about whether a record exists. At least two listed +records turned out not to exist, and the real record inventory is only visible +in the registrar's own list. + +## No secrets here + +Audited 2026-11-01, including the full commit history: no credentials, tokens, +keys, member emails or account identifiers. Where a runbook needs a credential +to exist (a backup passphrase, an S3 key), it records **where the credential is +held**, never the credential. ## Where operators look - **Member-facing docs** live in [`co-op/docs`](https://git.inference.coop/co-op/docs) and are published at . -- **Migration runbooks** live in [`co-op/runbooks`](https://git.inference.coop/co-op/runbooks). - The member-facing docs link here from their "For operators" section. ## History -These files were moved here from `co-op/docs`, where they were previously -published on the member-facing site. The commit history for each file is -preserved in this repository. +The operator runbook, maintenance pages and incident write-ups were moved here +from `co-op/docs`, where they were previously published on the member-facing +site. The migration material came from `co-op/runbooks` (now archived), and was +brought across with its commit history intact via an unrelated-histories merge. diff --git a/migration/cloudron-server-migration.md b/migration/cloudron-server-migration.md new file mode 100644 index 0000000..3cde571 --- /dev/null +++ b/migration/cloudron-server-migration.md @@ -0,0 +1,510 @@ +# Cloudron server migration — Hostinger → Infomaniak + +> **COMPLETED 2026-11-02.** The migration is done: the co-op's services run on the new +> Infomaniak box, every hostname resolves there, 16 apps verified against the pre-migration +> baseline, and members were notified before and after the window. +> +> **This document is now HISTORY — retained deliberately, not pending work.** It is the record of +> how the move was planned and carried out, and it is worth keeping for the next one. Read it for +> what was learned, not for current state. Two things went stale the moment the cutover finished: +> anything marked "to do" below, and the mail architecture (see the note further down). +> +> **Where the live knowledge lives now:** the `cloudron-ops` skill for operational procedure, and + +> **The private twin `ops/runbooks` was ARCHIVED by the steward on 2026-11-02.** It held nothing this +> repository lacks (same five files, this one ahead) and is now read-only, so there is a single +> maintained copy. Archived rather than deleted: reversible, and the history stays inspectable. +> `co-op/docs/infrastructure.md` for the member-facing description. Those two are maintained; this +> file is not. + +--- + +Pre-flight and cutover runbook for moving the co-op's Cloudron server. Written +2026-10-29 from a live inventory of the server; revised 2026-10-30 after a read-only +review against live DNS, federation and backup state. **Owner:** Nathan. **Executor:** +whoever is at the console that weekend — assume it is not the person who wrote this. + +The server being migrated hosts `my.inference.coop` and **16 apps**. The agent host is a +separate machine (already on Infomaniak) and is NOT part of this migration. + +## 1. What Cloudron's backup covers — verified, not assumed + +- All 16 apps and their `/app/data` (localstorage), plus addons: postgres, mysql, + mongodb, redis. +- Cloudron platform config: domains, dashboard users, app env vars, TLS config, backup + target settings, scheduled tasks. +- **The container registry app, 3.8 GB of custom images** (`librechat`, `litellm`, + `admin-panel`, `member-portal`, `member-dashboard`). These exist nowhere else — if the + registry does not restore, the custom apps cannot start. +- Size check that confirms app data is included: ~5.8 GB of app data visible inside + containers vs ~4.6 GB per backup run. + +Apps: chat, chat-legacy (OpenWebUI), inference.coop (Surfer), forum (Loomio), matrix +(Synapse), gateway (LiteLLM), panel, registry, git (Gitea), vault (Vaultwarden), portal, +search (SearXNG), dashboard, newsletter (Listmonk), email (Cloudron Mail), cloud +(Nextcloud). + +## 2. What it does NOT cover + +- **The host OS layer.** Ubuntu itself, netplan, cloud-init, `sshd` config, + `authorized_keys`, hostname, PTR, provider firewall, hand-edited `/etc`. Build the new + box; do not expect to restore it. +- **The Cloudron licence/subscription** — must be moved to the new server. +- **DNS records.** Manual at yoursrs.com (domains.coop NS). Cloudron cannot manage them. +- **The backup target config and passphrase** — chicken-and-egg by design. +- **App data written to a host path instead of `/app/data`** — see the audit in §3. +- **Anything on the old box outside Cloudron** (e.g. `/etc` tweaks). Diff before + decommissioning. + +Not a gap, despite looking like one: our custom images (registry is an app, backed up), +and the agent's own state (different machine). + +## 3. Pre-flight + +Companion sheet: `dns-change-sheet.md` (every record, current value, target value). + +**Blocker — do this first.** The Vault app (`vault.inference.coop`) runs *on the server +being migrated*, and the backup passphrase is believed to live there. The credentials +needed to restore are inside the thing being restored. + +- [ ] Export the Vault contents and store them off-server (personal password manager or + offline media). Confirm the **Cloudron backup passphrase** is in your own hands. +- [ ] **Retrieve the S3 secretAccessKey — the off-box copy does NOT have it.** Checked + 2026-10-30: `~/.hermes/security/backup_sites_backup.json` carries `accessKeyId` and + nothing else. A restore begins by pointing the new Cloudron at the backup target, so + the *secret* must be in hand before the window — get it from the Infomaniak console + and store it with the passphrase, off-box. +- [x] The **restic** passphrase is safe: `~/.hermes/scripts/backup.sh` documents that + `RESTIC_PASSWORD` lives in the agent host's `.env` **and** the co-op Vault. The agent + host is not being migrated, so the agent-side backup stays recoverable. (The Vault, + however, is an app on the box being migrated — so the Vault copy alone is not + off-box until it is exported.) +- [x] New host capacity: **4 CPU, 12 GB RAM, 250 GB disk** (given 2026-10-30) — ample for + ~6 GB of app data plus images. Old box used ~17 GB of 58 GB. +- [x] OS: **Ubuntu 26.04 LTS is supported** — Cloudron's install docs specify a fresh + Ubuntu 26.04 x64 server. Minimums are 2 GB RAM / 20 GB disk; the new box has + 12 GB / 250 GB. Resolved 2026-10-30. +- [ ] **Version floor: install Cloudron ≥ 10.1.5**, the platform version this box runs + (`/api/v1/cloudron/status` → `10.1.5`; backup manifests read `box_v10.1.5`). The + earlier note of "9.0.4" was the *CLI* version, not the platform — a restore cannot go + backwards. +- [x] **Installed 2026-10-30 — banner shown, setup page live on the IPs.** Run `cloudron-setup` + with **no `--domain`**, and leave the browser wizard for cutover day. Verified from the agent + host: the box serves the setup page on **both** `https://179.237.86.116` and + `https://[2001:1600:18:206::89]` (HTTP 200), and **80/443 are open on v4** (the earlier + v4-rules gap is fixed). + **The certificate is NOT self-signed.** Cloudron obtained a **valid Let's Encrypt certificate + for the bare IP** — SAN `IP Address:179.237.86.116`, expires 2026-11-06, and `curl` verifies it + with no `-k`. That is LE's new short-lived IP-certificate feature. Consequence: **nothing is + attempted against `my.inference.coop` yet, so there is no ACME rate-limit exposure at all** — + the "watch Let's Encrypt limits" caveat in earlier drafts of this section is moot. Over IPv6 the + IP cert does not validate (the SAN covers the v4 address only), which is harmless: after + cutover the hostname certificates cover both families. + **There is no `cloudron` CLI on the server** — that is a separate npm tool for managing a remote + Cloudron, not part of the platform. For server-side diagnostics use the documented tool, + `sudo cloudron-support --troubleshoot`. The version needs no guessing: a fresh install pulls the + current release, which is 10.1.5 or newer, and anything ≥ 10.1.5 restores this backup. + **Ports on the new box: 25 inbound CONFIRMED OPEN** (operator, via the Infomaniak portal, 2026-11-01 — this is the receive path, and it is the half that cannot be replaced by a relay). + Opened by the operator on 2026-11-01 (4190 is not offered by the provider — ManageSieve, not needed; leave it). + **Do not read a timeout from the agent host as "blocked".** Checked 2026-11-01 on both address families: + 465/587/993/995 all time out while 80/443 answer in ~2 ms, and the same probe reaches the OLD box on those + ports instantly. But the new box has no mail app yet (the wizard is unsubmitted), so nothing is listening, + and Cloudron's own firewall drops rather than refuses by default — a permitted-but-empty port and a blocked + port are indistinguishable from outside. The 2026-10-30 IPv6 observation is the better evidence: those ports + were REFUSED there, which means the packet arrived, i.e. permitted. Verify for real once the mail app exists. Outbound 25 is **not needed**: RFC-wise, a smarthost relay carries our sending, and Scaleway Transactional Email is being set up for exactly that. Do not spend a support ticket on outbound 25. + (Previous note, kept for the date: "Ports still to open on the new box (checked 2026-10-30): 25, 465, 587, 993, 995, 4190.") + + **Provider tier — confirmed 2026-11-01: Infomaniak VPS Cloud** (not Lite; the two differ + in ways that matter here). What the tier provides, per Infomaniak's own product pages: + a **99.99% availability** commitment, **10 Gb/s** bandwidth, resizing **up and down**, and + **one snapshot included** (extensible) — so the snapshot question is answered, but **none + has been taken yet on the new box**: take one before submitting the Cloudron wizard, so + there is a known-good point to return to. The Lite tier, by contrast, has no snapshot, no + SLA, and closes outbound 25 outright; Cloud allows **port 25 "on justified request"**. + That last point is the one to cite if the Infomaniak support thread asks us to justify it. + 22, 80 and 443 answer today. Mail will not work on the new box until these are added, and + outbound 25 is the separate Infomaniak ticket. + **Leave for cutover day:** the domain, the plan/subscription (completing the wizard early can + move the subscription off the live server — the one step worth reading twice), and the + restore-from-backup itself. Looking at the setup page is harmless; submitting it is not. +- [ ] Take an Infomaniak **snapshot of the VPS right after the install** (snapshots: 0 of 1 used), so a + botched wizard on cutover day can roll back to a known-good installed platform. +- [ ] Confirm the Cloudron subscription can be moved to a new server (do it at cutover, + not early). +- [ ] **IPv6: DECIDED — repoint.** The new box has `2001:1600:18:206::89`, so all 18 + AAAA records move with the A records (§3 originally left this open). The zone carries + **AAAA records on 18 hostnames** — everything that has an A record — all pointing to + `2a02:4780:6e:83fd::1` (Hostinger's range). Counted 2026-10-30; the full list is in + `dns-change-sheet.md`. This is 18 deletions or repoints, not a footnote. If the new box has IPv6, repoint them; + if it does not, **delete the AAAA records** as part of the switch. Changing only + the A records leaves every IPv6-capable client — including other Matrix + homeservers and mail senders — still reaching the old box. +- [ ] Set **rDNS/PTR → `my.inference.coop` on BOTH `179.237.86.116` and + `2001:1600:18:206::89`**. Cloudron cannot do this — PTR belongs to the IP owner and + exists only in the Infomaniak console. The console lists reverse DNS as unconfigured for both. + Not time-sensitive — do it now. If IPv6 sending causes trouble later, disable v6 + egress for SMTP rather than removing the AAAA. +- [ ] **Open these in the INFOMANIAK console, not on the host** (measured 2026-10-30: 22 + answers, 80/443 are silently dropped — a provider firewall default). Cloudron manages + the host's own iptables but explicitly does not touch the provider firewall: its docs + say "ensure the firewall does not block port 80 (HTTP) and 443 (HTTPS)". Ports: + **22, 80, 443, 25, 465, 587, 993** (plus 995 and 4190 to match the old box; leave + 8448, 110 and 143 closed). Verified 2026-10-30 against what the old box serves: + 22/80/443/465/587/993/995/4190 answer; 8448, 110, 143 and 25 are dropped. The 25 + result is not trustworthy — it came from a server whose network may block outbound + 25 — and it must be open, because the platform has **inbound mail enabled**: + `GET /api/v1/mail` → `{"domain":"inference.coop","inbound":true, + "dkimSelector":"cloudron","relay":{"provider":"cloudron-smtp"}}`. The DKIM selector + there matches the `cloudron._domainkey` TXT record, which is a useful cross-check. + + **How to verify rules before Cloudron is installed** — a blocked port *times out*, + a permitted port with no listener yet is *refused*. Both are informative; a timeout + is an absent rule. + **State on 2026-10-30:** the rules are applied on **IPv6** (80/443/465/587/993/995 + refused = permitted; 8448/110/143/4190 dropped) but **NOT on IPv4** — everything + except 22 still times out. Infomaniak's console appears to keep separate IPv4 and + IPv6 rule sets. This must be fixed on both: IPv4-only clients, and Let's Encrypt's + HTTP-01 validation, are the ones that matter most, and 4190 (ManageSieve) stays shut + unless members need mail-filter clients. + **Port 25, settled from Cloudron's docs:** *inbound* 25 is **required to receive + email** and must be opened on both addresses. *Outbound* 25 is **not required if a + relay/smart host is configured** — Cloudron's own guidance is "request your provider + to unblock this port **or set up a smart host**", and this box already has + `relay: {"provider": "cloudron-smtp"}`. Confirm on the dashboard's Email page that + the relay is a real remote smarthost (host + port 465/587); if it is, Infomaniak's + outbound-25 restriction is irrelevant and worth no support ticket. (mail delivery + — test from outside, e.g. `telnet my.inference.coop 25` once DNS moves; note that + many networks block *outbound* 25, so a closed result from inside another server is + not conclusive). **8448 is intentionally closed** — federation is delegated to + 443 via the apex `.well-known` (verified live: + `inference.coop/.well-known/matrix/server` → `{"m.server":"matrix.inference.coop:443"}`). + Do not "fix" 8448. Do not add ufw rules that fight Docker; the old box runs with + ufw inactive and Cloudron-managed iptables. +- [ ] **DMARC is `p=reject; pct=100`.** Strict. After the restore, any DKIM/SPF + misalignment means outbound mail is *discarded by recipients*, not junked. The + DKIM TXT record (`cloudron._domainkey`) must match the key inside the restored + mail app's data — verify both rather than assuming. +- [ ] **Mount audit on the Cloudron host** — the one class of data the backup cannot see: + + sudo docker ps -q | xargs -I{} sudo docker inspect {} \ + --format '{{.Name}} {{range .Mounts}}{{.Source}}=>{{.Destination}} {{end}}' \ + | tr ' ' '\n' | grep -v /app/data + + **RESOLVED 2026-10-30:** the host mount is Nextcloud's `/var/lib/nextcloud`, **3 MB**, + holding only `preview/` (regenerable thumbnails) — not a data-loss risk. Re-run this + audit only if apps change before the cutover; it is now a confirm, not an unknown. +- [ ] Leave the **single snapshot slot free** for a post-restore, pre-DNS-switch snapshot — + a bad restore then becomes a revert instead of a rebuild. +- [ ] Record the highest backup run number before the freeze. (As of 2026-10-30 the + newest run is healthy at ~4.6 GB and the backup space was enlarged, so the quota + pressure that produced the `EntityTooLarge` failures is relieved — but run growth + is still the number to watch: 2.6 → 4.6 GB per run over two weeks.) +- [ ] Lower all A **and AAAA** record TTLs to 300 s **a day early**. The zone TTL is + **3600 s** — an earlier note read 760–1642 s, which was a decrementing *cache + countdown*, not the configured TTL. At 3600 s, propagation after the switch takes + up to an hour, which is exactly the window where the old box must not be accepting + writes (§4). +- [ ] List external allowlists that carry our IP: Regolo, GreenPT, Tinfoil, any mail relay, + and inbound webhooks. Update after cutover. +- [ ] SSH host keys change with the box. Accept the new fingerprint deliberately, in + advance, so a strict-host-key refusal doesn't stop you mid-migration. +- [ ] Announce the window in `#updates`. + +## 3a. The "before" snapshot + +`migration-before-state.json` in this repo is a machine-diffable record of the current +server: 16 apps with their manifest ids, image tags, state, health and **env var names** +(never values); the backup sites, schedule and retention; the newest run's id, state, +integrity signature and size; Cloudron's version; and data fingerprints (member count, +API-key count, ledger rows, `metrics_daily` rows, Gitea repo count). Capture nothing +further; after the restore, diff it. A silently-empty app is exactly what a diff catches +and a spot-check misses. + +## Credentials needed on the day + +Status as of 2026-10-30: + +- [x] **Cloudron backup passphrase** — off-box (confirmed 2026-10-30), in the operator's KeePass vault. + A restore asks for this first; without it the backup is unreadable. This was the real blocker. +- [x] **Vaultwarden master password + admin credentials** — off-box in the same KeePass vault + (2026-10-30). The Vault's own database is inside the Cloudron backup, so a restore returns it + whole; the master password is what makes the restored contents readable. Note the shape of this: + `vault.inference.coop` runs `com.github.bitwardenrs` — **Vaultwarden**, the open-source Bitwarden + server — so official Bitwarden clients already work against it. There is nothing to sync. +- [x] **S3 `secretAccessKey`** — **confirmed 2026-11-01: held externally by the operator along with the rest of the Infomaniak backup keys.** (A restore is unreadable without it, which is why it was flagged.) This is the one credential + absent from the co-op's own off-box JSON (that carries `accessKeyId` only), and a restore begins + by pointing the new Cloudron at the backup target — so it must be typeable before the window. +- [ ] **Vault export** (optional insurance) — the contents already ride in the Cloudron backup, so this + is not the source of truth. Cheap protection against the app restoring but its DB coming back + damaged. +- [x] **restic passphrase** (agent host, different backup) — in that host's `.env`, and that host is + not the one migrating, so it stays recoverable either way. + +**Continuity, not just safety.** All of the above live in one person's KeePass file. For a +co-operative that is a governance risk as much as an ops one: if the one custodian is unavailable, so +is the infrastructure. The cheap fixes are a second custodian (or a sealed shared copy), and a line in +this repo recording *where* the break-glass copies are held — never the secrets themselves. + +## The announcement, ready to post in #updates + +> **Scheduled maintenance: , , – UTC.** We are moving our servers to a new host +> in Switzerland. Chat, the portal, the API and the website will be unavailable during the window. No +> data is being changed or removed — this is a move, not a rebuild. If anything still looks wrong +> afterwards, reply here and we will sort it out. + +## The all-clear email — drafted 2026-11-02, ready to paste into Listmonk + +Sent from the usual identity (`Inference Cooperative `). Send after DNS +propagation completes (~08:00 UTC on switch day) so links in the email resolve for everyone. + +> Hi all, +> +> Last night we promised to write as soon as the move was complete and everything checked out. This +> is that email. +> +> The co-op's services now run entirely on our own infrastructure, hosted by Infomaniak in +> Switzerland — the renewable energy, Swiss privacy, worker-owned, foundation-governed setup we +> described when we warned you about downtime. Chat, the member portal, the dashboard, code hosting, +> the vault, cloud storage and this newsletter all came back whole from the final backup, and we +> verified them one by one before switching over the domain. +> +> We're sorry for the outage. A server move is a rare event for a small co-op — we don't expect +> another one anytime soon — and the payoff is infrastructure that belongs to us, on our terms. +> +> If anything looks off in the next few days, reply here and we'll sort it out. +> +> We hope you enjoyed the AI-free Sunday night. Everything is back where you left it. +> +> — Inference Cooperative + +## 4. Cutover, in order + +1. **Freeze writes.** Pause the portal's `membership_sweep` and `rag_refresh`, hold + Listmonk campaigns. +2. **Final backup on the old box.** Run "Backup now" and confirm state `normal`. + Everything after this timestamp is *lost* in a forward migration. +3. **Stop the apps on the old box** — all of them, including mail. Keep the *machine* + powered on for rollback, but nothing may accept writes from here on: during the + up-to-one-hour DNS propagation, a still-serving old box silently collects Matrix + messages, Gitea pushes, API spend and mail into a database that gets thrown away. + Clients and mail senders retry; that is the correct failure mode. +4. **Stop the old box's backup schedule** (Settings → Backups → disable). Two Cloudrons + writing one target interleave and produce `EntityTooLarge` quota failures. +5. **Build the new box and restore** from the Infomaniak S3 target, using the passphrase + from §3. Expect TLS to fail until DNS moves — reach the dashboard by IP or a temporary + override. +6. **Switch the A records — and the AAAA records** (repoint or delete, per the §3 + decision). +7. Wait out the TTL (up to an hour at 3600 s if not lowered), then confirm certificates + issued. +8. **Re-enable backups on the new box — once daily at 23:00, retention 7 days.** (This is + now applied on the old box — schedule `00 00 23 * * *`, verified — re-check it on the + new box after restore rather than assuming.) +9. Update external allowlists. The old box stays powered on with apps stopped, for + rollback only. + +## 5. Post-restore verification + +Run `python3 ~/.hermes/scripts/verify-restore.py` from the **agent** host — it stays up through the +migration, so it is the one machine that can judge the new box. It diffs the restored server against +`migration-before-state.json`: app inventory, **env var NAMES** (a lost var comes back as an app that +will not start), image tags, backup target and schedule, portal data fingerprints (row counts must +never drop), DNS, federation, mail ports, and whether each site answers. Exit 1 = at least one +regression. `--quiet` prints failures and warnings only; `--only ` runs one group. + +Run it **before** cutover too: DNS and port-25 failures are expected then, and they are exactly what +cutover is meant to clear. The 2026-10-30 dry run gave 50 pass / 1 warn / 13 fail, all 13 explained +by pre-cutover DNS plus port 25 — the point is that no *unexplained* failure appears. + +- [ ] Dashboard loads with a valid certificate; all 16 apps `running`. +- [ ] **One custom app loads** — e.g. `panel.inference.coop`. This is the proof the + registry restored; without it nothing custom starts. +- [ ] **The apex `.well-known` answers** — `curl https://inference.coop/.well-known/matrix/server` + must return `{"m.server":"matrix.inference.coop:443"}`. It is served by the *website* + app, so if that app's data is missing, federation breaks while + `matrix.inference.coop` itself still looks healthy. Then: + `curl -s https://matrix.inference.coop/_matrix/federation/v1/version`. +- [ ] Data spot-checks: Matrix login + a room loads; Gitea repo list; portal + `/admin/overview` returns the expected member count; gateway `/v1/models`. +- [ ] Mail: MX resolves to the new IP; SPF still `a:my.inference.coop`; the DKIM TXT + (`cloudron._domainkey`) matches the restored key; PTR correct; **send to an + external mailbox and confirm it actually arrived** (DMARC is `p=reject` — a silent + discard is the failure mode, not a junk folder); receive from outside; **test + port 25 inbound from a third network**. +- [ ] Agent API access: `cloudron list` from the agent host. The API token lives in + Cloudron's config so it should survive; if it does not, mint a new one and update the + agent's `.env`. +- [ ] Schedules fire **once**, not twice: portal sweeps, Listmonk, agent digests. +- [ ] First backup on the new box is `normal` and in the expected size range. + +## 6. Rollback + +- Keep the old box **powered on with apps stopped** for several days. Rollback = point DNS + back and start the apps. +- Do not delete old snapshots from S3 until the new box has at least one verified backup + run. +- Know the asymmetry: anything written between the final backup and the DNS switch exists + **only on the old box**. Rolling forward discards it; rolling back keeps it. §4 step 3 + (stop apps) is what keeps that window empty rather than merely short. + +## 7. Open unknowns — resolve before cutover + +- **Are MAILBOX CONTENTS inside the Cloudron backup?** Mail is ON for + `inference.coop` with inbound enabled, and the mail server is part of the +*platform*, not the Mail app's container (that app listens only on 3000 — it is a + UI, not an MTA). Platform config is backed up, so the DKIM key and mail settings + survive; whether members' mailboxes do is **unverified** and would be silent, + permanent data loss if they do not. Confirm against Cloudron's docs or the + dashboard's backup contents before the freeze. This is now the most consequential + open question in this runbook. +- ~~Whether the Vault export (§3 blocker) has actually been done.~~ **Closed 2026-11-01:** the operator confirms access to all relevant passwords outside the server. The full Vault export remains optional — the vault's own database is inside the Cloudron backup, so a restore returns it. +- The Cloudron host's OS version, disk and RAM (not visible over the API). +- Whether any provider API key is bound to our current IP. +- Whether the subscription permits a server move. + +## 8. Pitfalls this server has already produced + +- **Quota failures:** 4.82 GB per run twice daily hit the Infomaniak ceiling + (`Upload exceeds quota`). The space has since been enlarged and the schedule is now + once daily — both verified — but per-run size grows (2.6 → 4.6 GB in two weeks), mostly + registry image blobs. If the ceiling ever returns, the durable fix is getting the + rebuildable images out of every snapshot, not deleting backups. +- **The once-a-day change cannot be made over the API** — dashboard only. Historically it + was reported as done and was not. Verified applied 2026-10-30. +- **`cloudron update --image` reads the manifest from the current directory.** Running it + from the wrong repo fails with `manifest id does not match`, which looks like a server + problem and is not. +- **The agent host is not migrated** but will see errors while DNS moves. It needs no + config change: every reference is a hostname, including `--server my.inference.coop`. + Its 30-minute webhook watch will fire during the window and may raise false alarms; + the daily digest now compares against its previous run before flagging, so one-off + transients should not reach you as decisions. +- **A DNS cache countdown is not a zone TTL.** Reading 760–1642 s off a resolver answer + and writing it into this runbook as "the TTL" was wrong — the configured TTL was 3600 s + all along. Query the zone at the registrar (or trust a fresh authoritative answer) for + planning numbers. + +--- + +# STATE OF PLAY — 2026-11-02 (AS IT WAS BEFORE THE CUTOVER — superseded) + +Written before the cutover for whoever picked it up next, including a fresh agent session with none of +the history. It is kept verbatim because it shows what was believed at the time and what turned out to +matter. **One thing below is now wrong on purpose-left-alone grounds: mail.** Listmonk has since moved to +Scaleway TEM (`smtp.tem.scaleway.com`, username = the DKIM selector UUID) — see the `cloudron-ops` skill +for the current arrangement rather than trusting this section. + +## Mail: receiving on the box, sending through a relay + +- **Inbound port 25 is OPEN** (operator confirmed in the Infomaniak portal, 2026-11-01). 465/587/993/995 + were opened too; 4190 is not offered and is not needed. **Outbound 25 is not required** — do not spend + a support ticket on it. +- **Scaleway Transactional Email is proven working.** A real message relayed through it produced: + `spf=pass`, `dkim=pass … header.s=ede94f37-204c-41ee-b388-e940520316d1` (the UUID selector in the DNS + sheet — 2048-bit), `dmarc=pass policy.published-domain-policy=reject`, and + `iprev=pass (phan-van4.scw-tem.cloud)`. SPF carries `include:_spf.tem.scaleway.com`. +- **BUT the newsletter does NOT use it yet.** Listmonk is still configured `host=mail, port=2465` — the + Cloudron mail app. It is in "copy settings from Cloudron" mode, so its own SMTP settings are overwritten + and the host keeps reverting (its `msg_retry_delay` does persist, which is a useful tell). **Do not + assume a newsletter sent through Scaleway; check the host before relying on it.** +- The **portal** also still sends through the Cloudron mail addon (`CLOUDRON_MAIL_*`). Moving it to the + relay needs its own SMTP env vars plus a code change, deliberately not done yet. +- **Consequence for the window:** the newsletter and the portal can only send **before** the box goes down + or **after** it comes back. Not during. + +## DNS: 11 records, two of them wildcards + +See `dns-change-sheet.md`, rebuilt 2026-11-01 from the registrar's panel. The important shape: + +- Only **11 records exist**. `*.inference.coop` A and AAAA carry **every application hostname** — chat, + portal, panel, dashboard, newsletter, gateway, vault, git, registry, files, forum, email, docs, my, + status. They are NOT individual records. **Move the wildcards or every app stays behind.** +- `server.inference.coop` is an A record nothing documents. Identify before moving. +- TTL is 3600 everywhere; lower it before the window. +- **Never read a timeout from the agent host as "blocked"** (see the port-25 lesson in §7). The agent host + cannot observe outbound 25 at all. + +## Credentials: all off-box, all confirmed + +Cloudron backup passphrase, Infomaniak S3 `secretAccessKey`, and the Vault passwords are held externally +by the operator (confirmed 2026-11-01). A restore is unreadable without the first two. + +## Not done yet + +- Cloudron **setup wizard on the new box** — still unsubmitted. +- **No snapshot** on the new box (the tier includes one — free rollback point). +- **No restore rehearsal** has been attempted. +- `verify-restore.py` is fixed (both blind spots) and the baseline is fresh as of `267db1f`, committed to + this repo. Run it after any restore; its two remaining FAILs pre-cutover are the expected + "not migrated yet" signature. + +## CAUTION: two Matrix identities + +The backup contains Synapse **and its signing key**. Restore it and the new box becomes a second +homeserver claiming the same server name as the old one, signing with the same key. Federation peers see +one identity behaving as two. **Keep the restored apps stopped except while actively testing.** + +## The agent running this + +The steward bot runs on a **third machine** (`179.237.103.165`) — neither the old nor the new Cloudron +box — so it stays up throughout. Its **primary model route goes through the gateway on the old box** +(`gateway.inference.coop` → `179.199.135.72`) and will fail during the move; its **fallback is GreenPT +direct** (`api.greenpt.ai`, `glm-5.3-flash`, `GREENPT_FALLBACK_API_KEY`), verified working 2026-11-02. +Tooling that talks to `git.`, `gateway.` or `portal.inference.coop` will be unavailable mid-move. + +--- + +# RESTORE COMPLETE — 2026-11-02 ~06:30 UTC + +All 16 apps restored on the new box (179.237.86.116) via per-app restore from the final backup +`box_v10.1.6_b4fd6c91` (03:57 UTC). Data verified per app, not per state flag: + +- matrix 35 rooms / 28 users, original signing key `ed25519:a_jSIu` — identity preserved +- portal members 91 / api_keys 74 / ledger 24; newsletter 91 subscribers, 8 campaigns +- forum 89 users / 18 discussions / 348 comments; gitea `code` org 5/5 public repos +- dashboard carries the post-baseline deploy (image 20261102-b352b1f, CREDIT_PACK_URL) — the + baseline is stale here, not the restore + +What the restore survived, in order: root disk too small (data moved to /mnt/data), docker store +wiped to fix overlay2 (images re-pulled, `cloudron` network recreated, firewall refreshed by +reboot), `cloudron-syslog` restart after the platformdata move, and `127.0.0.1 +registry.inference.coop` in the new box's /etc/hosts so custom images don't reach the dead box. + +**Two lessons for next time, now in skill `cloudron-ops`:** (1) check which filesystem Cloudron +actually uses, not the advertised disk size — nothing in the installer flags an unused 245 GB +volume; (2) `repair` recreates containers but never restores data — the data path is per-app +restore from the box backup entry. Never batch-parallel restores. + +**Still open at time of writing:** DNS on the old box (apex, wildcard A/AAAA, `www` → 179.237.86.116 +/ 2001:1600:18:206::89; `server.inference.coop` untouched); TTLs still 3600; agent host has a +temporary `/etc/hosts` override `my.inference.coop → 179.237.86.116` that MUST be removed after the +DNS switch or verify-restore.py reports fiction; old box powered on, apps stopped, backup schedule +disabled; first new-box backup expected 23:00 UTC — verify it is `normal` and expected size. + +--- + +# CUTOVER COMPLETE — 2026-11-02 07:45 UTC + +DNS switched ~06:30 UTC (apex + wildcard A/AAAA + www → 179.237.86.116 / 2001:1600:18:206::89; +all TXT/MX untouched; no AAAA added for www — it has none). `server.inference.coop` (the one +unidentified record) was also repointed to the new box — answered authoritatively 07:53 UTC — so the +old box has **no remaining DNS references**. Agent-host override removed 06:44. Final verify-restore.py: **66 pass / 7 warn / 1 FAIL** — the single FAIL is +the dashboard's stale baseline (post-baseline deploy, expected). Key sync (chat↔portal) healed 07:12 +after the new box's resolver cache expired; git push done via +`git -c http.curloptResolve=git.inference.coop:443:179.237.86.116 push` (bypasses stale resolver — +keep this trick). Baseline recaptured against the new box at 07:53 UTC (original in git history); +verify-restore.py then reports **67 pass / 7 warn / 0 FAIL**. + +**Mail required one more step:** after the docker-store wipe, the platform mail container is NOT +recreated at boot, and every mail API route (and even the mail-domain list) dies on +"Unable to find container mail" — the container lookup happens before any reconfigure. Fix: +`cloudron-support --check-services` then `--recreate-containers` (sanctioned, no data loss); mail +ports 465/587/993/995 reopened immediately; port 25 is untestable from the agent host (outbound 25 +blocked — the runbook's known blind spot). Member mailbox CONTENTS still need a real IMAP login to +confirm (the runbook's long-standing unknown). All-clear email drafted as Listmonk campaign 18 +("All-clear after the move (draft)", Members list, status draft). chat-legacy (Open WebUI): member +access removed 2026-11-02 — retirement begun; app restored and safe in backups. + +**Outstanding:** tonight 23:00 UTC first backup (verify `normal`, ~4.9 GiB); Cloudron subscription +page check on the new dashboard; my primary model route re-heals on its own (fallback worked all +night); old box keep powered-on/apps-stopped/backups-disabled for a few days, then decommission. diff --git a/migration/dns-change-sheet.md b/migration/dns-change-sheet.md new file mode 100644 index 0000000..d2db105 --- /dev/null +++ b/migration/dns-change-sheet.md @@ -0,0 +1,86 @@ +# DNS change sheet — inference.coop + +**Rebuilt 2026-11-01 from the registrar's own panel** (domains.coop; nameservers `ns1/ns2.yoursrs.com`). +The previous version listed record counts that could not be reproduced and at least two records that did +not exist, so it is replaced rather than edited. + +## Why the panel was the only source + +- **AXFR is refused**, so the zone cannot be pulled. +- **The zone is wildcarded.** A made-up hostname (`zz-random-check-9f2.inference.coop`) returns the old + box's address. So a name that *resolves* proves nothing about whether a record exists, and any hostname + absent from the list below is served by the wildcard. +- Only 11 records exist. Everything else is `*`. + +## The complete inventory (11 records) + +| # | Type | Hostname | Value | TTL | Moves? | +|---|------|----------|-------|-----|--------| +| 1 | A | `inference.coop` | `179.199.135.72` | 3600 | **YES** → new box | +| 2 | AAAA | `inference.coop` | `2a02:4780:6e:83fd::1` | 3600 | **YES** → new box | +| 3 | A | `*` | `179.199.135.72` | 3600 | **YES** → new box | +| 4 | AAAA | `*` | `2a02:4780:6e:83fd::1` | 3600 | **YES** → new box | +| 5 | A | `server` | `179.199.135.72` | 3600 | **CHECK FIRST** — nothing else references it | +| 6 | A | `www` | `179.199.135.72` | 3600 | **YES** → new box | +| 7 | MX (pri 10) | `inference.coop` | `my.inference.coop` | 3600 | stays (follows record 1/2) | +| 8 | TXT | `inference.coop` | `v=spf1 a:my.inference.coop include:_spf.tem.scaleway.com ~all` | 3600 | stays | +| 9 | TXT | `_dmarc` | `v=DMARC1; p=reject; pct=100` | 3600 | stays | +| 10 | TXT | `cloudron._domainkey` | `v=DKIM1; t=s; p=MIGfMA0GCSqGSIb3…` (1024-bit; full value recovered 2026-11-01 and restored) | 3600 | stays | +| 11 | TXT | `ede94f37-204c-41ee-b3…` | `v=DKIM1; h=sha256; k=rsa; p=MIIBIjA…` (2048-bit) | 3600 | stays | +| 12 | TXT | `_atproto` | `did=did:plc:4leras5objxiqtnbjzoholds` | 3600 | stays | + +(The table shows 12 rows because `www` was added as a separate item; rows 1–11 are the panel's 11. +Reconcile the numbering against the panel when editing.) + +## The four records that matter at cutover + +**Records 1–4 and 6**: apex A, apex AAAA, wildcard A, wildcard AAAA, and `www`. Change these and the site +moves. Nothing else in the zone needs touching for the migration. + +**The wildcards are the item the old sheet omitted entirely.** Because `*` carries both A and AAAA records, +every hostname not listed above — `chat`, `portal`, `panel`, `dashboard`, `newsletter`, `gateway`, +`vault`, `git`, `registry`, `files`, `forum`, `email`, `docs`, `my`, `status` — resolves *only* via the +wildcard. They are not individual records and must not be created as such; they move when the wildcards +move. A partial cutover that changes the apex but not `*` leaves every application on the old box. + +## Two DKIM selectors, for different senders + +- **`cloudron._domainkey`** — the box's own mail signing (1024-bit). Used by the portal and by Listmonk, + both of which relay through the Cloudron mail app. +- **`ede94f37-…`** — a UUID-named selector with a 2048-bit key. UUID selectors are what a service + generates for itself, and this is almost certainly **Scaleway Transactional Email's**. Scaleway's own + domain verification passed, which is the authority on whether it is right. + +*Name and value for row 11 are truncated in the panel screenshot. Read the full name in the panel (or +from Scaleway's domain-verification page) before editing that row.* + +## What the zone does NOT contain + +No CNAME, no CAA, no SRV, no `_acme-challenge` entries. If any of those are ever needed, they are new +records, not moves. (`docs.inference.coop` has no record of its own — it is served by the wildcard, which +is why it answers with the Cloudron dashboard.) + +## Before the window + +1. **Lower the TTL on records 1–4 and 6 from 3600 to ~300.** At 3600 a mistake takes an hour to correct; + at 300 it takes five minutes. Do this a day ahead so the old TTL expires, then restore it afterwards. +2. ~~**Establish what `server.inference.coop` is used for.**~~ **Resolved 2026-11-02:** it was repointed + to the new box with the rest (confirmed authoritatively on both NS). Its purpose is still + unidentified, but it no longer references the old box — see the runbook's CUTOVER COMPLETE section. +3. **Confirm the full name of record 11** (above). + +## Verification after the move + +Query the **authoritative** nameservers, not a resolver, and query the specific name — a wildcard answer +cannot be distinguished from a real record otherwise: + +``` +dig +short A inference.coop @ns1.yoursrs.com +dig +short AAAA inference.coop @ns1.yoursrs.com +dig +short A zz-random-check-9f2.inference.coop @ns1.yoursrs.com # wildcard, must now be the NEW box +dig +short MX inference.coop @ns1.yoursrs.com +``` + +Then `verify-restore.py`. Its two known blind spots (the `data` group crashing on a missing PATH, and the +mail check asserting a port-25 state it cannot observe from the agent host) are still unfixed — fix them +before relying on a clean run. diff --git a/migration/migration-before-state.json b/migration/migration-before-state.json new file mode 100644 index 0000000..f6c5901 --- /dev/null +++ b/migration/migration-before-state.json @@ -0,0 +1,248 @@ +{ + "app_count": 16, + "apps": { + "4023ccbf-d57a-4500-97df-14de9cf3516e": { + "env_names": [ + "ALLOW_EMAIL_LOGIN", + "APP_TITLE", + "IMAGE_GEN_OAI_BASEURL", + "IMAGE_GEN_OAI_MODEL", + "SEARXNG_INSTANCE_URL", + "STT_API_KEY", + "SYSTEM_IMAGE_KEY", + "TTS_API_KEY" + ], + "health": "healthy", + "image": "20261031-2a98892", + "location": "chat.inference.coop", + "manifest": "ai.librechat.cloudronapp@1.4.5", + "state": null + }, + "50ae6c97-e851-4d17-b013-51ca7a4e5be0": { + "env_names": [ + "DEFAULT_PROMPT_SUGGESTIONS", + "ENABLE_ADMIN_CHAT_ACCESS", + "ENABLE_ADMIN_EXPORT", + "PYTHONPATH", + "RAG_EMBEDDING_ENGINE", + "RAG_EMBEDDING_MODEL", + "RAG_OPENAI_API_BASE_URL", + "RAG_OPENAI_API_KEY" + ], + "health": "healthy", + "image": "202610300419310000", + "location": "chat-legacy.inference.coop", + "manifest": "com.openwebui.cloudronapp@3.7.9", + "state": null + }, + "55f5f904-3233-44de-ad7b-5dc7c8533b75": { + "env_names": [], + "health": "healthy", + "image": "202610221509540000", + "location": "inference.coop", + "manifest": "io.cloudron.surfer@7.4.0", + "state": null + }, + "8b1a972c-08af-4e92-852c-d573c37b01ad": { + "env_names": [ + "B3_API_KEY", + "LOOMIO_DISABLE_DEMO_GROUPS" + ], + "health": "healthy", + "image": "202611010321240000", + "location": "forum.inference.coop", + "manifest": "org.loomio.cloudronapp@1.30.0", + "state": null + }, + "aa6f4597-6206-4509-b32e-c89750b2c1d8": { + "env_names": [], + "health": "healthy", + "image": "202610290719420000", + "location": "matrix.inference.coop", + "manifest": "org.matrix.synapse@1.147.1", + "state": null + }, + "adababd4-2427-4bdd-9bf7-6bdad0302610": { + "env_names": [ + "GREENPT_API_KEY", + "LITELLM_CORS_ORIGINS", + "LITELLM_MIGRATION_DIR", + "OLLAMA_API_KEY", + "TINFOIL_API_KEY", + "TINFOIL_USER_CACHE_SECRET" + ], + "health": "healthy", + "image": "1.0.0-1793493736807", + "location": "gateway.inference.coop", + "manifest": "ai.coop.litellm@1.0.0", + "state": null + }, + "ae187071-7a43-4a6e-8a02-986609a66a1a": { + "env_names": [ + "ADMIN_TOKEN", + "ADMIN_USERNAMES", + "PORTAL_BASE" + ], + "health": "healthy", + "image": "20261101-624c891", + "location": "panel.inference.coop", + "manifest": "coop.inference.admin-panel@0.1.0", + "state": null + }, + "c257ba3c-6737-4375-ad51-ea19586bf3f0": { + "env_names": [], + "health": "healthy", + "image": "202610221511140000", + "location": "registry.inference.coop", + "manifest": "io.cloudron.container-registry@1.4.0", + "state": null + }, + "c91c13aa-30ed-4af9-9599-7a87ab4f221d": { + "env_names": [], + "health": "healthy", + "image": "202611011719500000", + "location": "git.inference.coop", + "manifest": "io.gitea.cloudronapp@1.44.0", + "state": null + }, + "cb838bcf-f2e2-4670-b117-dca04b0ee957": { + "env_names": [], + "health": "healthy", + "image": "202610211943370000", + "location": "vault.inference.coop", + "manifest": "com.github.bitwardenrs@1.26.5", + "state": null + }, + "d4f05e4b-75c6-492e-9faf-7afb0712fc94": { + "env_names": [ + "ADMIN_TOKEN", + "ADMIN_TOKEN_LEGACY", + "BROKER_SECRET", + "BUDGET_NOTICES", + "CLOUDRON_TOKEN", + "DOCS_KNOWLEDGE_ID", + "INACTIVE_GROUP_ID", + "LIBRECHAT_SYNC_TOKEN", + "LISTMONK_API_TOKEN", + "LISTMONK_API_USER", + "LISTMONK_LIST_ID", + "LITELLM_BASE", + "LITELLM_DB_HOST", + "LITELLM_DB_NAME", + "LITELLM_DB_PASSWORD", + "LITELLM_DB_PORT", + "LITELLM_DB_USER", + "LITELLM_MASTER_KEY", + "LOOMIO_API_KEY", + "LOOMIO_B3_KEY", + "LOOMIO_GROUP_ID", + "MANUAL_MEMBERS", + "MATRIX_BOT_TOKEN", + "MATRIX_EXTRA_ROOMS", + "MATRIX_GENERAL_ROOM", + "MATRIX_HOMESERVER", + "MATRIX_SPACE_ID", + "MEMBERS_GROUP_ID", + "MEMBER_BUDGET", + "OC_OAUTH_CLIENT_ID", + "OC_OAUTH_CLIENT_SECRET", + "OC_PERSONAL_TOKEN", + "OWUI_API_KEY", + "OWUI_BASE", + "OWUI_JWT_SECRET", + "OWUI_MEMBERS_GROUP_ID", + "PORTAL_KEK", + "PORTAL_SECRET", + "WEBHOOK_TOKEN" + ], + "health": "healthy", + "image": "20261101-6a7eaf0", + "location": "portal.inference.coop", + "manifest": "coop.inference.member-portal@0.1.0", + "state": null + }, + "d7c73fff-ee42-4db0-98c0-4ce7a8ab35cc": { + "env_names": [], + "health": "healthy", + "image": "202610252319260000", + "location": "search.inference.coop", + "manifest": "io.github.ascimoo.searx@2.119.0", + "state": null + }, + "daf8750e-2683-4104-83ea-f77d1b4601e8": { + "env_names": [ + "BROKER_SECRET", + "CREDIT_PACK_URL", + "PORTAL_BASE" + ], + "health": "healthy", + "image": "20261102-b352b1f", + "location": "dashboard.inference.coop", + "manifest": "coop.inference.member-dashboard@0.1.0", + "state": null + }, + "eed2217c-b165-430e-bc29-7ccfef9b11e4": { + "env_names": [], + "health": "healthy", + "image": "202610220221130000", + "location": "newsletter.inference.coop", + "manifest": "app.listmonk.cloudronapp@1.17.1", + "state": null + }, + "fbb256b9-ead3-41d8-bee0-d5ea87149436": { + "env_names": [], + "health": "healthy", + "image": "202610261602550000", + "location": "email.inference.coop", + "manifest": "io.cloudron.mail@1.6.1", + "state": null + }, + "fd261d67-b790-424f-9e82-f74883ee8585": { + "env_names": [], + "health": "healthy", + "image": "202610211606270000", + "location": "cloud.inference.coop", + "manifest": "com.nextcloud.cloudronapp@5.9.0", + "state": null + } + }, + "backup_runs_retained": 10, + "backup_sites": [ + { + "bucket": "default", + "endpoint": "https://s3.swiss-backup02.infomaniak.com/", + "name": "Infomaniak Encrypted S3", + "provider": "s3-v4-compat", + "retention_days": 7, + "schedule": "00 00 23 * * *" + }, + { + "bucket": null, + "endpoint": "filesystem", + "name": "Default", + "provider": "filesystem", + "retention_days": 2, + "schedule": "never" + } + ], + "captured_at": "2026-11-02T07:53:09Z", + "cloudron_version": "10.1.6", + "gitea_code_repos": 5, + "newest_backup": { + "bytes": 5295323496, + "created": "2026-11-01T23:13:20.000Z", + "has_integrity_signature": true, + "id": "box_v10.1.6_3b21b999", + "state": "normal" + }, + "portal_tables": { + "budget_notices": 3, + "credit_ledger": 24, + "held_credits": 1, + "member_api_keys": 74, + "members": 91, + "metrics_daily": 60, + "pending_members": 3, + "sqlite_sequence": 2 + } +} diff --git a/provider-pricing/regolo.md b/provider-pricing/regolo.md new file mode 100644 index 0000000..1838405 --- /dev/null +++ b/provider-pricing/regolo.md @@ -0,0 +1,99 @@ +# Regolo pricing: what changed, and where our numbers live + +Source: Regolo's announcement email (link goes to their SSO portal, so the tables below are +entered by hand) plus their public pricing page. + +## Effective 2026-11-01 (from the notice email), EUR per 1M tokens + +| model | input | output | +|---|---|---| +| gpt-oss-20b | €0.10 | €0.20 | +| qwen3.5-9b | €0.10 | €0.20 | +| brick-complexity-pro (SEMANTIC SMART ROUTER) | €0.10 | €0.20 | +| gpt-oss-120b | €0.20 | €0.70 | +| mistral-small-4-119b | €0.20 | €0.70 | +| gemma4-31b | €0.40 | €0.60 | +| qwen3.8-27b | €0.50 | €3.40 | +| **qwen3.5-122b** | **€0.60** | **€3.40** | +| **apertus-70b** | **€1.00** | **€3.40** | +| glm5.2 | €2.30 | €6.90 | + +## Before (public pricing page, read 2026-10-31), EUR per 1M tokens + +apertus-70b €0.40 / €2.10 · qwen3.5-122b €1.00 / €4.20 · gpt-oss-120b €1.00 / €4.20 · +gpt-oss-20b €0.10 / €0.42 · gemma4-31b €0.40 / €2.10 · glm5.2 €2.00 / €5.20 · +mistral-small-4-119b €0.50 / €2.10 · qwen3.8-27b €0.50 / €2.10 · qwen3.5-9b €0.07 / €0.35 · +brick-complexity-pro €0.10 / €0.40 · brick-v1-beta €0.00 / €0.00 + +Image generation, unchanged by the notice: **€0.000000015 per pixel** (with the +subscription note: Qwen-Image is *not included* in any plan). + +## What we serve, and the change it needs + +Peg: EUR x 1.16 -> USD, the rate our numbers already use. + +| our model | old USD/M | new USD/M (from 2026-11-01) | direction | +|---|---|---|---| +| regolo/apertus-70b | $0.46 / $2.44 | **$1.16 / $3.94** | input +152%, output +62% | +| regolo/qwen3.5-122b | $1.16 / $4.87 | **$0.70 / $3.94** | input -40%, output -19% | + +**Until the gateway is updated**, apertus bills $0.46/M input while costing ~$1.16/M — a ~70 cent/M +loss on the co-op — and qwen over-bills members by ~46 cents/M input. Both directions argue for +updating promptly rather than waiting. + +## Where the numbers live + +The gateway's per-model costs are the authority, and they are what the response headers +(`x-litellm-response-cost`, shipped 2026-10-31) and member billing both read. Read them with +`GET https://gateway.inference.coop/model/info` (master key): as of 2026-10-31, +`regolo/apertus-70b` = 4.6e-07 / 2.44e-06 and `regolo/qwen3.5-122b` = 1.16e-06 / 4.87e-06 +per token. Member-facing copy is `docs/models.md`. + +## Open inconsistency: the image model is priced in EUR-as-USD + +`regolo/qwen-image` carries **no** input/output cost, and a live 512x512 call billed +**$0.00393216** — which is 262,144 px x EUR 0.000000015, i.e. the EUR figure applied as if it +were dollars. The two chat models convert at 1.16; the image model does not, so it is ~15.5% +under cost. Small per image (~$0.0007), but it is the same class of error as the chat prices and +should be fixed in the same pass (x 1.16 -> $0.0000000174 per pixel). + +## Process note + +Keep a dated section per notice. The next change should be a diff against this file rather than a +re-derivation — the email link is a login wall, so the numbers only reach us when the operator +forwards them. + + +## Our actual Regolo usage (measured 2026-10-31, from LiteLLM SpendLogs) + +Source: `"LiteLLM_SpendLogs"` in the gateway's Postgres, reached with python + `psycopg` inside the +gateway container (there is no `psql` there). Models were first used 2026-10-16. + +| model | requests | spend | window | +|---|---|---|---| +| regolo/qwen3.5-122b | 195 | $2.61 | 2026-10-16 → 10-31 | +| regolo/qwen-image | 94 | $0.96 | 2026-10-16 → 10-31 | +| regolo/apertus-70b | 157 | $0.82 | 2026-10-16 → 10-31 | +| **all Regolo** | **452** | **$4.38** | last 30 days; ~all of it | + +Context: total gateway spend is **$426.34** lifetime (104,947 requests, since 2026-09-04), and +**~$393 in the last 30 days** — greenpt/glm $231.42, deepseek $146.14, gpt-oss $10.32, +Regolo **$4.38**, other $0.79. So **Regolo is ~1.1% of inference spend, about $4.38/month.** + +## Should we take a subscription? + +**Almost certainly not at current usage.** A plan has to beat ~$4.38/month to be worth it, and +Regolo's plan prices are **not published** — the pricing page has a SUBSCRIPTIONS section with no +figures, which is why the announcement email links into their account portal. Worth one short +question to them via "Talk to an engineer", not a project. + +One thing that is public and useful: their **30-day free trial** grants **1M tokens/day** across +"All Core Models". Our entire Regolo usage is roughly **120k tokens/day** (≈3.6M/month), i.e. well +inside that, so the trial — and likely a low tier — would cover everything we currently do. + +## The 2026-11-01 change, in money terms + +At these volumes the mispricing is small: apertus's input roughly 2.5x-ing on a model that has cost +**$0.82 lifetime** is about **$1/month** of exposure, and qwen's cut moves ~$0.6/month in members' +favour. So this is **billing accuracy, not urgency** — but accuracy still matters, because the same +numbers feed the cost headers members can now read.