commit eec9572a3f09926b8608e904116494607562a18a Author: Inferencebot Date: Thu Oct 29 22:46:40 2026 +0000 Add Cloudron server migration runbook (Hostinger → Infomaniak) Pre-flight, ordered cutover, post-restore verification, rollback, and the list of what Cloudron's backup does not cover. Written from a live inventory of the server on 2026-10-29; open unknowns are marked rather than guessed. diff --git a/README.md b/README.md new file mode 100644 index 0000000..4305880 --- /dev/null +++ b/README.md @@ -0,0 +1,6 @@ +# Runbooks + +Operational runbooks for the co-op's infrastructure. Private: these contain +topology, addresses, and procedure detail that should not be public. + +- [Cloudron server migration — Hostinger → Infomaniak](cloudron-server-migration.md) diff --git a/cloudron-server-migration.md b/cloudron-server-migration.md new file mode 100644 index 0000000..67c31f7 --- /dev/null +++ b/cloudron-server-migration.md @@ -0,0 +1,138 @@ +# Cloudron server migration — Hostinger → Infomaniak + +Pre-flight and cutover runbook for moving the co-op's Cloudron server. Written +2026-10-29 from a live inventory of the server. **Owner:** Nathan. **Executor:** whoever +is at the console that weekend — assume it is not the person who wrote this. + +The server being migrated hosts `my.inference.coop` and **16 apps**. The agent host is a +separate machine (already on Infomaniak) and is NOT part of this migration. + +## 1. What Cloudron's backup covers — verified, not assumed + +- All 16 apps and their `/app/data` (localstorage), plus addons: postgres, mysql, + mongodb, redis. +- Cloudron platform config: domains, dashboard users, app env vars, TLS config, backup + target settings, scheduled tasks. +- **The container registry app, 3.8 GB of custom images** (`librechat`, `litellm`, + `admin-panel`, `member-portal`, `member-dashboard`). These exist nowhere else — if the + registry does not restore, the custom apps cannot start. +- Size check that confirms app data is included: ~5.8 GB of app data visible inside + containers vs ~4.8 GB per backup run. + +Apps: chat, chat-legacy (OpenWebUI), inference.coop (Surfer), forum (Loomio), matrix +(Synapse), gateway (LiteLLM), panel, registry, git (Gitea), vault (Vaultwarden), portal, +search (SearXNG), dashboard, newsletter (Listmonk), email (Cloudron Mail), cloud +(Nextcloud). + +## 2. What it does NOT cover + +- **The host OS layer.** Ubuntu itself, netplan, cloud-init, `sshd` config, + `authorized_keys`, hostname, PTR, provider firewall, hand-edited `/etc`. Build the new + box; do not expect to restore it. +- **The Cloudron licence/subscription** — must be moved to the new server. +- **DNS records.** Manual at yoursrs.com (domains.coop NS). Cloudron cannot manage them. +- **The backup target config and passphrase** — chicken-and-egg by design. +- **App data written to a host path instead of `/app/data`** — see the audit in §3. +- **Anything on the old box outside Cloudron** (e.g. `/etc` tweaks). Diff before + decommissioning. + +Not a gap, despite looking like one: our custom images (registry is an app, backed up), +and the agent's own state (different machine). + +## 3. Pre-flight + +**Blocker — do this first.** The Vault app (`vault.inference.coop`) runs *on the server +being migrated*, and the backup passphrase is believed to live there. The credentials +needed to restore are inside the thing being restored. + +- [ ] Export the Vault contents and store them off-server (personal password manager or + offline media). Confirm the **Cloudron backup passphrase** is in your own hands. +- [ ] Confirm the backup target's S3 access key is available off-box. The agent host has + `~/.hermes/security/backup_sites_backup.json` (0600) — check whether it carries the + key; if not, retrieve it from the Infomaniak console. +- [ ] New host capacity: `df -h`, `free -m`, `lsb_release -a`, `cloudron --version`. + Must hold ~6 GB of app data plus Docker images and growth; check swap exists. +- [ ] OS: same or newer Ubuntu than the old box. Cloudron install version must be ≥ the + old box's (9.0.4). +- [ ] Confirm the Cloudron subscription can be moved to a new server. +- [ ] Ask Infomaniak to set **rDNS/PTR for the new IP to `my.inference.coop`** — mail + deliverability depends on it. +- [ ] Open on the new box: 22, 80, 443, 465, 587, 993. Match the old box's exposure + (`ufw` is inactive there; Cloudron manages its own iptables — do not add ufw rules + that fight Docker). +- [ ] **Mount audit on the Cloudron host** — the one class of data the backup cannot see: + + sudo docker ps -q | xargs -I{} sudo docker inspect {} \ + --format '{{.Name}} {{range .Mounts}}{{.Source}}=>{{.Destination}} {{end}}' \ + | tr ' ' '\n' | grep -v /app/data + + A container showed `/var/lib/nextcloud` mounted from the host. Identify its owner and + size; if it holds real data, copy it separately. +- [ ] Record the highest backup run number before the freeze. +- [ ] Lower all A-record TTLs to 300 s **a day early** (currently 760–1642 s ≈ 13–27 min). +- [ ] List external allowlists that carry our IP: Regolo, GreenPT, Tinfoil, any mail relay, + and inbound webhooks. Update after cutover. +- [ ] Announce the window in `#updates`. + +## 4. Cutover, in order + +1. **Freeze writes.** Pause the portal's `membership_sweep` and `rag_refresh`, hold + Listmonk campaigns. Chat cannot be frozen — note the exact snapshot time. +2. **Final backup on the old box.** Run "Backup now" and confirm state `normal`. + Everything after this timestamp is *lost* in a forward migration. +3. **Stop the old box's backup schedule** (Settings → Backups → disable). Two Cloudrons + writing one target interleave and produce `EntityTooLarge` quota failures. +4. **Build the new box and restore** from the Infomaniak S3 target, using the passphrase + from §3. Expect TLS to fail until DNS moves — reach the dashboard by IP or a temporary + override. +5. **Switch the A records** to the new IP. +6. Wait out the TTL, then confirm certificates issued. +7. **Re-enable backups on the new box — once daily, retention 7 days.** (The approved + change was never applied; the API has no route for it, so do it in the dashboard.) +8. Update external allowlists. Leave the old box running read-only. + +## 5. Post-restore verification + +- [ ] Dashboard loads with a valid certificate; all 16 apps `running`. +- [ ] **One custom app loads** — e.g. `panel.inference.coop`. This is the proof the + registry restored; without it nothing custom starts. +- [ ] Data spot-checks: Matrix login + a room loads; Gitea repo list; portal + `/admin/overview` returns the expected member count; gateway `/v1/models`. +- [ ] Mail: send to and receive from an external address; MX resolves to the new IP; SPF + still `a:my.inference.coop`; DKIM keys survived (they are in app data); PTR correct. +- [ ] Federation: `curl -s https://matrix.inference.coop/_matrix/federation/v1/version`. + Other homeservers cache the old IP and will retry it for a while — that is expected. +- [ ] Agent API access: `cloudron list` from the agent host. The API token lives in + Cloudron's config so it should survive; if it does not, mint a new one and update the + agent's `.env`. +- [ ] Schedules fire **once**, not twice: portal sweeps, Listmonk, agent digests. +- [ ] First backup on the new box is `normal` and in the expected size range. + +## 6. Rollback + +- Keep the old box powered on, read-only, for several days. +- Rollback = point DNS back. Do not delete old snapshots from S3 until the new box has at + least one verified backup run. +- Know the asymmetry: anything written between the final backup and the DNS switch exists + **only on the old box**. Rolling forward discards it; rolling back keeps it. Keep that + window short and quiet. + +## 7. Open unknowns — resolve before cutover + +- Which app owns the `/var/lib/nextcloud` host mount, and how much data is in it. +- The Cloudron host's OS version, disk and RAM (not visible over the API). +- Whether the S3 access key is in the off-box JSON or only inside Cloudron. +- Whether any provider API key is bound to our current IP. +- Whether the subscription permits a server move. + +## 8. Pitfalls this server has already produced + +- **Quota failures:** 4.82 GB per run twice daily hit the Infomaniak ceiling + (`Upload exceeds quota`). Retention is 7 days; schedule must be once daily. +- **The once-a-day change cannot be made over the API** — dashboard only. Historically it + was reported as done and was not. +- **`cloudron update --image` reads the manifest from the current directory.** Running it + from the wrong repo fails with `manifest id does not match`, which looks like a server + problem and is not. +- **The agent host is not migrated** but will see errors while DNS moves. It needs no + config change: every reference is a hostname, including `--server my.inference.coop`.