runbooks/cloudron-server-migration.md

Cloudron server migration — Hostinger → Infomaniak

Pre-flight and cutover runbook for moving the co-op's Cloudron server. Written 2026-10-29 from a live inventory of the server. Owner: Nathan. Executor: whoever is at the console that weekend — assume it is not the person who wrote this.

The server being migrated hosts my.inference.coop and 16 apps. The agent host is a separate machine (already on Infomaniak) and is NOT part of this migration.

1. What Cloudron's backup covers — verified, not assumed

• All 16 apps and their /app/data (localstorage), plus addons: postgres, mysql, mongodb, redis.
• Cloudron platform config: domains, dashboard users, app env vars, TLS config, backup target settings, scheduled tasks.
• The container registry app, 3.8 GB of custom images (librechat, litellm, admin-panel, member-portal, member-dashboard). These exist nowhere else — if the registry does not restore, the custom apps cannot start.
• Size check that confirms app data is included: ~5.8 GB of app data visible inside containers vs ~4.8 GB per backup run.

Apps: chat, chat-legacy (OpenWebUI), inference.coop (Surfer), forum (Loomio), matrix (Synapse), gateway (LiteLLM), panel, registry, git (Gitea), vault (Vaultwarden), portal, search (SearXNG), dashboard, newsletter (Listmonk), email (Cloudron Mail), cloud (Nextcloud).

2. What it does NOT cover

• The host OS layer. Ubuntu itself, netplan, cloud-init, sshd config, authorized_keys, hostname, PTR, provider firewall, hand-edited /etc. Build the new box; do not expect to restore it.
• The Cloudron licence/subscription — must be moved to the new server.
• DNS records. Manual at yoursrs.com (domains.coop NS). Cloudron cannot manage them.
• The backup target config and passphrase — chicken-and-egg by design.
• App data written to a host path instead of /app/data — see the audit in §3.
• Anything on the old box outside Cloudron (e.g. /etc tweaks). Diff before decommissioning.

Not a gap, despite looking like one: our custom images (registry is an app, backed up), and the agent's own state (different machine).

3. Pre-flight

Blocker — do this first. The Vault app (vault.inference.coop) runs on the server being migrated, and the backup passphrase is believed to live there. The credentials needed to restore are inside the thing being restored.

• Export the Vault contents and store them off-server (personal password manager or offline media). Confirm the Cloudron backup passphrase is in your own hands.

• Confirm the backup target's S3 access key is available off-box. The agent host has ~/.hermes/security/backup_sites_backup.json (0600) — check whether it carries the key; if not, retrieve it from the Infomaniak console.

• New host capacity: df -h, free -m, lsb_release -a, cloudron --version. Must hold ~6 GB of app data plus Docker images and growth; check swap exists.

• OS: same or newer Ubuntu than the old box. Cloudron install version must be ≥ the old box's (9.0.4).

• Confirm the Cloudron subscription can be moved to a new server.

• Ask Infomaniak to set rDNS/PTR for the new IP to my.inference.coop — mail deliverability depends on it.

• Open on the new box: 22, 80, 443, 465, 587, 993. Match the old box's exposure (ufw is inactive there; Cloudron manages its own iptables — do not add ufw rules that fight Docker).

• Mount audit on the Cloudron host — the one class of data the backup cannot see:

  sudo docker ps -q | xargs -I{} sudo docker inspect {} \
    --format '{{.Name}} {{range .Mounts}}{{.Source}}=>{{.Destination}} {{end}}' \
    | tr ' ' '\n' | grep -v /app/data
  
  A container showed `/var/lib/nextcloud` mounted from the host. Identify its owner and
  size; if it holds real data, copy it separately.
  

• Record the highest backup run number before the freeze.

• Lower all A-record TTLs to 300 s a day early (currently 760–1642 s ≈ 13–27 min).

• List external allowlists that carry our IP: Regolo, GreenPT, Tinfoil, any mail relay, and inbound webhooks. Update after cutover.

• Announce the window in #updates.

4. Cutover, in order

1. Freeze writes. Pause the portal's membership_sweep and rag_refresh, hold Listmonk campaigns. Chat cannot be frozen — note the exact snapshot time.
2. Final backup on the old box. Run "Backup now" and confirm state normal. Everything after this timestamp is lost in a forward migration.
3. Stop the old box's backup schedule (Settings → Backups → disable). Two Cloudrons writing one target interleave and produce EntityTooLarge quota failures.
4. Build the new box and restore from the Infomaniak S3 target, using the passphrase from §3. Expect TLS to fail until DNS moves — reach the dashboard by IP or a temporary override.
5. Switch the A records to the new IP.
6. Wait out the TTL, then confirm certificates issued.
7. Re-enable backups on the new box — once daily, retention 7 days. (The approved change was never applied; the API has no route for it, so do it in the dashboard.)
8. Update external allowlists. Leave the old box running read-only.

5. Post-restore verification

• Dashboard loads with a valid certificate; all 16 apps running.
• One custom app loads — e.g. panel.inference.coop. This is the proof the registry restored; without it nothing custom starts.
• Data spot-checks: Matrix login + a room loads; Gitea repo list; portal /admin/overview returns the expected member count; gateway /v1/models.
• Mail: send to and receive from an external address; MX resolves to the new IP; SPF still a:my.inference.coop; DKIM keys survived (they are in app data); PTR correct.
• Federation: curl -s https://matrix.inference.coop/_matrix/federation/v1/version. Other homeservers cache the old IP and will retry it for a while — that is expected.
• Agent API access: cloudron list from the agent host. The API token lives in Cloudron's config so it should survive; if it does not, mint a new one and update the agent's .env.
• Schedules fire once, not twice: portal sweeps, Listmonk, agent digests.
• First backup on the new box is normal and in the expected size range.

6. Rollback

• Keep the old box powered on, read-only, for several days.
• Rollback = point DNS back. Do not delete old snapshots from S3 until the new box has at least one verified backup run.
• Know the asymmetry: anything written between the final backup and the DNS switch exists only on the old box. Rolling forward discards it; rolling back keeps it. Keep that window short and quiet.

7. Open unknowns — resolve before cutover

• Which app owns the /var/lib/nextcloud host mount, and how much data is in it.
• The Cloudron host's OS version, disk and RAM (not visible over the API).
• Whether the S3 access key is in the off-box JSON or only inside Cloudron.
• Whether any provider API key is bound to our current IP.
• Whether the subscription permits a server move.

8. Pitfalls this server has already produced

• Quota failures: 4.82 GB per run twice daily hit the Infomaniak ceiling (Upload exceeds quota). Retention is 7 days; schedule must be once daily.
• The once-a-day change cannot be made over the API — dashboard only. Historically it was reported as done and was not.
• cloudron update --image reads the manifest from the current directory. Running it from the wrong repo fails with manifest id does not match, which looks like a server problem and is not.
• The agent host is not migrated but will see errors while DNS moves. It needs no config change: every reference is a hostname, including --server my.inference.coop.