runbooks/cloudron-server-migration.md

Cloudron server migration — Hostinger → Infomaniak

Pre-flight and cutover runbook for moving the co-op's Cloudron server. Written 2026-10-29 from a live inventory of the server; revised 2026-10-30 after a read-only review against live DNS, federation and backup state. Owner: Nathan. Executor: whoever is at the console that weekend — assume it is not the person who wrote this.

The server being migrated hosts my.inference.coop and 16 apps. The agent host is a separate machine (already on Infomaniak) and is NOT part of this migration.

1. What Cloudron's backup covers — verified, not assumed

• All 16 apps and their /app/data (localstorage), plus addons: postgres, mysql, mongodb, redis.
• Cloudron platform config: domains, dashboard users, app env vars, TLS config, backup target settings, scheduled tasks.
• The container registry app, 3.8 GB of custom images (librechat, litellm, admin-panel, member-portal, member-dashboard). These exist nowhere else — if the registry does not restore, the custom apps cannot start.
• Size check that confirms app data is included: ~5.8 GB of app data visible inside containers vs ~4.6 GB per backup run.

Apps: chat, chat-legacy (OpenWebUI), inference.coop (Surfer), forum (Loomio), matrix (Synapse), gateway (LiteLLM), panel, registry, git (Gitea), vault (Vaultwarden), portal, search (SearXNG), dashboard, newsletter (Listmonk), email (Cloudron Mail), cloud (Nextcloud).

2. What it does NOT cover

• The host OS layer. Ubuntu itself, netplan, cloud-init, sshd config, authorized_keys, hostname, PTR, provider firewall, hand-edited /etc. Build the new box; do not expect to restore it.
• The Cloudron licence/subscription — must be moved to the new server.
• DNS records. Manual at yoursrs.com (domains.coop NS). Cloudron cannot manage them.
• The backup target config and passphrase — chicken-and-egg by design.
• App data written to a host path instead of /app/data — see the audit in §3.
• Anything on the old box outside Cloudron (e.g. /etc tweaks). Diff before decommissioning.

Not a gap, despite looking like one: our custom images (registry is an app, backed up), and the agent's own state (different machine).

3. Pre-flight

Companion sheet: dns-change-sheet.md (every record, current value, target value).

Blocker — do this first. The Vault app (vault.inference.coop) runs on the server being migrated, and the backup passphrase is believed to live there. The credentials needed to restore are inside the thing being restored.

• Export the Vault contents and store them off-server (personal password manager or offline media). Confirm the Cloudron backup passphrase is in your own hands.

• Confirm the backup target's S3 access key is available off-box. The agent host has ~/.hermes/security/backup_sites_backup.json (0600) — check whether it carries the key; if not, retrieve it from the Infomaniak console.

• New host capacity: 4 CPU, 12 GB RAM, 250 GB disk (given 2026-10-30) — ample for ~6 GB of app data plus images. Old box used ~17 GB of 58 GB.

• OS: Ubuntu 26.04 LTS is supported — Cloudron's install docs specify a fresh Ubuntu 26.04 x64 server. Minimums are 2 GB RAM / 20 GB disk; the new box has 12 GB / 250 GB. Resolved 2026-10-30.

• Version floor: install Cloudron ≥ 10.1.5, the platform version this box runs (/api/v1/cloudron/status → 10.1.5; backup manifests read box_v10.1.5). The earlier note of "9.0.4" was the CLI version, not the platform — a restore cannot go backwards.

• Install now, stop before the wizard. cloudron-setup --domain my.inference.coop installs Docker, the platform and the CLI, and needs no DNS of its own — certificate issuance will simply fail and retry until DNS moves (harmless, but it does mean ACME attempts against a hostname still pointing at Hostinger; watch Let's Encrypt rate limits). Leave the browser setup wizard for cutover day: that is where the restore-from-backup and the subscription/plan are chosen, and completing it early risks moving the subscription off the live server. Verify that risk before touching the wizard — it is the one step worth reading twice.

• Confirm the Cloudron subscription can be moved to a new server (do it at cutover, not early).

• IPv6: DECIDED — repoint. The new box has 2001:1600:18:206::89, so all 18 AAAA records move with the A records (§3 originally left this open). The zone carries AAAA records on 18 hostnames — everything that has an A record — all pointing to 2a02:4780:6e:83fd::1 (Hostinger's range). Counted 2026-10-30; the full list is in dns-change-sheet.md. This is 18 deletions or repoints, not a footnote. If the new box has IPv6, repoint them; if it does not, delete the AAAA records as part of the switch. Changing only the A records leaves every IPv6-capable client — including other Matrix homeservers and mail senders — still reaching the old box.

• Set rDNS/PTR → my.inference.coop on BOTH 179.237.86.116 and 2001:1600:18:206::89. Cloudron cannot do this — PTR belongs to the IP owner and exists only in the Infomaniak console. The console lists reverse DNS as unconfigured for both. Not time-sensitive — do it now. If IPv6 sending causes trouble later, disable v6 egress for SMTP rather than removing the AAAA.

• Open these in the INFOMANIAK console, not on the host (measured 2026-10-30: 22 answers, 80/443 are silently dropped — a provider firewall default). Cloudron manages the host's own iptables but explicitly does not touch the provider firewall: its docs say "ensure the firewall does not block port 80 (HTTP) and 443 (HTTPS)". Ports: 22, 80, 443, 25, 465, 587, 993 (plus 995 and 4190 to match the old box; leave 8448, 110 and 143 closed). Verified 2026-10-30 against what the old box serves: 22/80/443/465/587/993/995/4190 answer; 8448, 110, 143 and 25 are dropped. The 25 result is not trustworthy — it came from a server whose network may block outbound 25 — and it must be open, because the platform has inbound mail enabled: GET /api/v1/mail → {"domain":"inference.coop","inbound":true, "dkimSelector":"cloudron","relay":{"provider":"cloudron-smtp"}}. The DKIM selector there matches the cloudron._domainkey TXT record, which is a useful cross-check.

  **How to verify rules before Cloudron is installed** — a blocked port *times out*,
  a permitted port with no listener yet is *refused*. Both are informative; a timeout
  is an absent rule.
  **State on 2026-10-30:** the rules are applied on **IPv6** (80/443/465/587/993/995
  refused = permitted; 8448/110/143/4190 dropped) but **NOT on IPv4** — everything
  except 22 still times out. Infomaniak's console appears to keep separate IPv4 and
  IPv6 rule sets. This must be fixed on both: IPv4-only clients, and Let's Encrypt's
  HTTP-01 validation, are the ones that matter most, and 4190 (ManageSieve) stays shut
  unless members need mail-filter clients.
  **Port 25, settled from Cloudron's docs:** *inbound* 25 is **required to receive
  email** and must be opened on both addresses. *Outbound* 25 is **not required if a
  relay/smart host is configured** — Cloudron's own guidance is "request your provider
  to unblock this port **or set up a smart host**", and this box already has
  `relay: {"provider": "cloudron-smtp"}`. Confirm on the dashboard's Email page that
  the relay is a real remote smarthost (host + port 465/587); if it is, Infomaniak's
  outbound-25 restriction is irrelevant and worth no support ticket. (mail delivery
  — test from outside, e.g. `telnet my.inference.coop 25` once DNS moves; note that
  many networks block *outbound* 25, so a closed result from inside another server is
  not conclusive). **8448 is intentionally closed** — federation is delegated to
  443 via the apex `.well-known` (verified live:
  `inference.coop/.well-known/matrix/server` → `{"m.server":"matrix.inference.coop:443"}`).
  Do not "fix" 8448. Do not add ufw rules that fight Docker; the old box runs with
  ufw inactive and Cloudron-managed iptables.
  

• DMARC is p=reject; pct=100. Strict. After the restore, any DKIM/SPF misalignment means outbound mail is discarded by recipients, not junked. The DKIM TXT record (cloudron._domainkey) must match the key inside the restored mail app's data — verify both rather than assuming.

• Mount audit on the Cloudron host — the one class of data the backup cannot see:

  sudo docker ps -q | xargs -I{} sudo docker inspect {} \
    --format '{{.Name}} {{range .Mounts}}{{.Source}}=>{{.Destination}} {{end}}' \
    | tr ' ' '\n' | grep -v /app/data
  
  **RESOLVED 2026-10-30:** the host mount is Nextcloud's `/var/lib/nextcloud`, **3 MB**,
  holding only `preview/` (regenerable thumbnails) — not a data-loss risk. Re-run this
  audit only if apps change before the cutover; it is now a confirm, not an unknown.
  

• Leave the single snapshot slot free for a post-restore, pre-DNS-switch snapshot — a bad restore then becomes a revert instead of a rebuild.

• Record the highest backup run number before the freeze. (As of 2026-10-30 the newest run is healthy at ~4.6 GB and the backup space was enlarged, so the quota pressure that produced the EntityTooLarge failures is relieved — but run growth is still the number to watch: 2.6 → 4.6 GB per run over two weeks.)

• Lower all A and AAAA record TTLs to 300 s a day early. The zone TTL is 3600 s — an earlier note read 760–1642 s, which was a decrementing cache countdown, not the configured TTL. At 3600 s, propagation after the switch takes up to an hour, which is exactly the window where the old box must not be accepting writes (§4).

• List external allowlists that carry our IP: Regolo, GreenPT, Tinfoil, any mail relay, and inbound webhooks. Update after cutover.

• SSH host keys change with the box. Accept the new fingerprint deliberately, in advance, so a strict-host-key refusal doesn't stop you mid-migration.

• Announce the window in #updates.

4. Cutover, in order

1. Freeze writes. Pause the portal's membership_sweep and rag_refresh, hold Listmonk campaigns.
2. Final backup on the old box. Run "Backup now" and confirm state normal. Everything after this timestamp is lost in a forward migration.
3. Stop the apps on the old box — all of them, including mail. Keep the machine powered on for rollback, but nothing may accept writes from here on: during the up-to-one-hour DNS propagation, a still-serving old box silently collects Matrix messages, Gitea pushes, API spend and mail into a database that gets thrown away. Clients and mail senders retry; that is the correct failure mode.
4. Stop the old box's backup schedule (Settings → Backups → disable). Two Cloudrons writing one target interleave and produce EntityTooLarge quota failures.
5. Build the new box and restore from the Infomaniak S3 target, using the passphrase from §3. Expect TLS to fail until DNS moves — reach the dashboard by IP or a temporary override.
6. Switch the A records — and the AAAA records (repoint or delete, per the §3 decision).
7. Wait out the TTL (up to an hour at 3600 s if not lowered), then confirm certificates issued.
8. Re-enable backups on the new box — once daily at 23:00, retention 7 days. (This is now applied on the old box — schedule 00 00 23 * * *, verified — re-check it on the new box after restore rather than assuming.)
9. Update external allowlists. The old box stays powered on with apps stopped, for rollback only.

5. Post-restore verification

• Dashboard loads with a valid certificate; all 16 apps running.
• One custom app loads — e.g. panel.inference.coop. This is the proof the registry restored; without it nothing custom starts.
• The apex .well-known answers — curl https://inference.coop/.well-known/matrix/server must return {"m.server":"matrix.inference.coop:443"}. It is served by the website app, so if that app's data is missing, federation breaks while matrix.inference.coop itself still looks healthy. Then: curl -s https://matrix.inference.coop/_matrix/federation/v1/version.
• Data spot-checks: Matrix login + a room loads; Gitea repo list; portal /admin/overview returns the expected member count; gateway /v1/models.
• Mail: MX resolves to the new IP; SPF still a:my.inference.coop; the DKIM TXT (cloudron._domainkey) matches the restored key; PTR correct; send to an external mailbox and confirm it actually arrived (DMARC is p=reject — a silent discard is the failure mode, not a junk folder); receive from outside; test port 25 inbound from a third network.
• Agent API access: cloudron list from the agent host. The API token lives in Cloudron's config so it should survive; if it does not, mint a new one and update the agent's .env.
• Schedules fire once, not twice: portal sweeps, Listmonk, agent digests.
• First backup on the new box is normal and in the expected size range.

6. Rollback

• Keep the old box powered on with apps stopped for several days. Rollback = point DNS back and start the apps.
• Do not delete old snapshots from S3 until the new box has at least one verified backup run.
• Know the asymmetry: anything written between the final backup and the DNS switch exists only on the old box. Rolling forward discards it; rolling back keeps it. §4 step 3 (stop apps) is what keeps that window empty rather than merely short.

7. Open unknowns — resolve before cutover

• Are MAILBOX CONTENTS inside the Cloudron backup? Mail is ON for inference.coop with inbound enabled, and the mail server is part of the platform, not the Mail app's container (that app listens only on 3000 — it is a UI, not an MTA). Platform config is backed up, so the DKIM key and mail settings survive; whether members' mailboxes do is unverified and would be silent, permanent data loss if they do not. Confirm against Cloudron's docs or the dashboard's backup contents before the freeze. This is now the most consequential open question in this runbook.
• Whether the Vault export (§3 blocker) has actually been done.
• The Cloudron host's OS version, disk and RAM (not visible over the API).
• Whether the S3 access key is in the off-box JSON or only inside Cloudron.
• Whether any provider API key is bound to our current IP.
• Whether the subscription permits a server move.

8. Pitfalls this server has already produced

• Quota failures: 4.82 GB per run twice daily hit the Infomaniak ceiling (Upload exceeds quota). The space has since been enlarged and the schedule is now once daily — both verified — but per-run size grows (2.6 → 4.6 GB in two weeks), mostly registry image blobs. If the ceiling ever returns, the durable fix is getting the rebuildable images out of every snapshot, not deleting backups.
• The once-a-day change cannot be made over the API — dashboard only. Historically it was reported as done and was not. Verified applied 2026-10-30.
• cloudron update --image reads the manifest from the current directory. Running it from the wrong repo fails with manifest id does not match, which looks like a server problem and is not.
• The agent host is not migrated but will see errors while DNS moves. It needs no config change: every reference is a hostname, including --server my.inference.coop. Its 30-minute webhook watch will fire during the window and may raise false alarms; the daily digest now compares against its previous run before flagging, so one-off transients should not reach you as decisions.
• A DNS cache countdown is not a zone TTL. Reading 760–1642 s off a resolver answer and writing it into this runbook as "the TTL" was wrong — the configured TTL was 3600 s all along. Query the zone at the registrar (or trust a fresh authoritative answer) for planning numbers.