runbooks/cloudron-server-migration.md

# Cloudron server migration — Hostinger → Infomaniak

Pre-flight and cutover runbook for moving the co-op's Cloudron server. Written
2026-10-29 from a live inventory of the server; revised 2026-10-30 after a read-only
review against live DNS, federation and backup state. **Owner:** Nathan. **Executor:**
whoever is at the console that weekend — assume it is not the person who wrote this.

The server being migrated hosts `my.inference.coop` and **16 apps**. The agent host is a
separate machine (already on Infomaniak) and is NOT part of this migration.

## 1. What Cloudron's backup covers — verified, not assumed

- All 16 apps and their `/app/data` (localstorage), plus addons: postgres, mysql,
  mongodb, redis.
- Cloudron platform config: domains, dashboard users, app env vars, TLS config, backup
  target settings, scheduled tasks.
- **The container registry app, 3.8 GB of custom images** (`librechat`, `litellm`,
  `admin-panel`, `member-portal`, `member-dashboard`). These exist nowhere else — if the
  registry does not restore, the custom apps cannot start.
- Size check that confirms app data is included: ~5.8 GB of app data visible inside
  containers vs ~4.6 GB per backup run.

Apps: chat, chat-legacy (OpenWebUI), inference.coop (Surfer), forum (Loomio), matrix
(Synapse), gateway (LiteLLM), panel, registry, git (Gitea), vault (Vaultwarden), portal,
search (SearXNG), dashboard, newsletter (Listmonk), email (Cloudron Mail), cloud
(Nextcloud).

## 2. What it does NOT cover

- **The host OS layer.** Ubuntu itself, netplan, cloud-init, `sshd` config,
  `authorized_keys`, hostname, PTR, provider firewall, hand-edited `/etc`. Build the new
  box; do not expect to restore it.
- **The Cloudron licence/subscription** — must be moved to the new server.
- **DNS records.** Manual at yoursrs.com (domains.coop NS). Cloudron cannot manage them.
- **The backup target config and passphrase** — chicken-and-egg by design.
- **App data written to a host path instead of `/app/data`** — see the audit in §3.
- **Anything on the old box outside Cloudron** (e.g. `/etc` tweaks). Diff before
  decommissioning.

Not a gap, despite looking like one: our custom images (registry is an app, backed up),
and the agent's own state (different machine).

## 3. Pre-flight

**Blocker — do this first.** The Vault app (`vault.inference.coop`) runs *on the server
being migrated*, and the backup passphrase is believed to live there. The credentials
needed to restore are inside the thing being restored.

- [ ] Export the Vault contents and store them off-server (personal password manager or
      offline media). Confirm the **Cloudron backup passphrase** is in your own hands.
- [ ] Confirm the backup target's S3 access key is available off-box. The agent host has
      `~/.hermes/security/backup_sites_backup.json` (0600) — check whether it carries the
      key; if not, retrieve it from the Infomaniak console.
- [ ] New host capacity: `df -h`, `free -m`, `lsb_release -a`, `cloudron --version`.
      Must hold ~6 GB of app data plus Docker images and growth; check swap exists.
- [ ] OS: same or newer Ubuntu than the old box. Cloudron install version must be ≥ the
      old box's (9.0.4).
- [ ] Confirm the Cloudron subscription can be moved to a new server.
- [ ] **Decide the IPv6 question before anything else DNS-related.** The zone carries
      **AAAA records**: `inference.coop`, `my`, `matrix` all point to
      `2a02:4780:6e:83fd::1` (Hostinger's range). If the new box has IPv6, repoint them;
      if it does not, **delete the AAAA records** as part of the switch. Changing only
      the A records leaves every IPv6-capable client — including other Matrix
      homeservers and mail senders — still reaching the old box.
- [ ] Ask Infomaniak to set **rDNS/PTR for the new IP to `my.inference.coop`** — mail
      deliverability depends on it.
- [ ] Open on the new box: **22, 80, 443, 465, 587, 993, and 25 inbound** (mail delivery
      — test from outside, e.g. `telnet my.inference.coop 25` once DNS moves; note that
      many networks block *outbound* 25, so a closed result from inside another server is
      not conclusive). **8448 is intentionally closed** — federation is delegated to
      443 via the apex `.well-known` (verified live:
      `inference.coop/.well-known/matrix/server` → `{"m.server":"matrix.inference.coop:443"}`).
      Do not "fix" 8448. Do not add ufw rules that fight Docker; the old box runs with
      ufw inactive and Cloudron-managed iptables.
- [ ] **DMARC is `p=reject; pct=100`.** Strict. After the restore, any DKIM/SPF
      misalignment means outbound mail is *discarded by recipients*, not junked. The
      DKIM TXT record (`cloudron._domainkey`) must match the key inside the restored
      mail app's data — verify both rather than assuming.
- [ ] **Mount audit on the Cloudron host** — the one class of data the backup cannot see:

      sudo docker ps -q | xargs -I{} sudo docker inspect {} \
        --format '{{.Name}} {{range .Mounts}}{{.Source}}=>{{.Destination}} {{end}}' \
        | tr ' ' '\n' | grep -v /app/data

      A container showed `/var/lib/nextcloud` mounted from the host. Identify its owner and
      size; if it holds real data, copy it separately.
- [ ] Record the highest backup run number before the freeze. (As of 2026-10-30 the
      newest run is healthy at ~4.6 GB and the backup space was enlarged, so the quota
      pressure that produced the `EntityTooLarge` failures is relieved — but run growth
      is still the number to watch: 2.6 → 4.6 GB per run over two weeks.)
- [ ] Lower all A **and AAAA** record TTLs to 300 s **a day early**. The zone TTL is
      **3600 s** — an earlier note read 760–1642 s, which was a decrementing *cache
      countdown*, not the configured TTL. At 3600 s, propagation after the switch takes
      up to an hour, which is exactly the window where the old box must not be accepting
      writes (§4).
- [ ] List external allowlists that carry our IP: Regolo, GreenPT, Tinfoil, any mail relay,
      and inbound webhooks. Update after cutover.
- [ ] SSH host keys change with the box. Accept the new fingerprint deliberately, in
      advance, so a strict-host-key refusal doesn't stop you mid-migration.
- [ ] Announce the window in `#updates`.

## 4. Cutover, in order

1. **Freeze writes.** Pause the portal's `membership_sweep` and `rag_refresh`, hold
   Listmonk campaigns.
2. **Final backup on the old box.** Run "Backup now" and confirm state `normal`.
   Everything after this timestamp is *lost* in a forward migration.
3. **Stop the apps on the old box** — all of them, including mail. Keep the *machine*
   powered on for rollback, but nothing may accept writes from here on: during the
   up-to-one-hour DNS propagation, a still-serving old box silently collects Matrix
   messages, Gitea pushes, API spend and mail into a database that gets thrown away.
   Clients and mail senders retry; that is the correct failure mode.
4. **Stop the old box's backup schedule** (Settings → Backups → disable). Two Cloudrons
   writing one target interleave and produce `EntityTooLarge` quota failures.
5. **Build the new box and restore** from the Infomaniak S3 target, using the passphrase
   from §3. Expect TLS to fail until DNS moves — reach the dashboard by IP or a temporary
   override.
6. **Switch the A records — and the AAAA records** (repoint or delete, per the §3
   decision).
7. Wait out the TTL (up to an hour at 3600 s if not lowered), then confirm certificates
   issued.
8. **Re-enable backups on the new box — once daily at 23:00, retention 7 days.** (This is
   now applied on the old box — schedule `00 00 23 * * *`, verified — re-check it on the
   new box after restore rather than assuming.)
9. Update external allowlists. The old box stays powered on with apps stopped, for
   rollback only.

## 5. Post-restore verification

- [ ] Dashboard loads with a valid certificate; all 16 apps `running`.
- [ ] **One custom app loads** — e.g. `panel.inference.coop`. This is the proof the
      registry restored; without it nothing custom starts.
- [ ] **The apex `.well-known` answers** — `curl https://inference.coop/.well-known/matrix/server`
      must return `{"m.server":"matrix.inference.coop:443"}`. It is served by the *website*
      app, so if that app's data is missing, federation breaks while
      `matrix.inference.coop` itself still looks healthy. Then:
      `curl -s https://matrix.inference.coop/_matrix/federation/v1/version`.
- [ ] Data spot-checks: Matrix login + a room loads; Gitea repo list; portal
      `/admin/overview` returns the expected member count; gateway `/v1/models`.
- [ ] Mail: MX resolves to the new IP; SPF still `a:my.inference.coop`; the DKIM TXT
      (`cloudron._domainkey`) matches the restored key; PTR correct; **send to an
      external mailbox and confirm it actually arrived** (DMARC is `p=reject` — a silent
      discard is the failure mode, not a junk folder); receive from outside; **test
      port 25 inbound from a third network**.
- [ ] Agent API access: `cloudron list` from the agent host. The API token lives in
      Cloudron's config so it should survive; if it does not, mint a new one and update the
      agent's `.env`.
- [ ] Schedules fire **once**, not twice: portal sweeps, Listmonk, agent digests.
- [ ] First backup on the new box is `normal` and in the expected size range.

## 6. Rollback

- Keep the old box **powered on with apps stopped** for several days. Rollback = point DNS
  back and start the apps.
- Do not delete old snapshots from S3 until the new box has at least one verified backup
  run.
- Know the asymmetry: anything written between the final backup and the DNS switch exists
  **only on the old box**. Rolling forward discards it; rolling back keeps it. §4 step 3
  (stop apps) is what keeps that window empty rather than merely short.

## 7. Open unknowns — resolve before cutover

- Whether the Vault export (§3 blocker) has actually been done.
- Which app owns the `/var/lib/nextcloud` host mount, and how much data is in it.
- The Cloudron host's OS version, disk and RAM (not visible over the API).
- Whether the S3 access key is in the off-box JSON or only inside Cloudron.
- Whether any provider API key is bound to our current IP.
- Whether the subscription permits a server move.
- Whether the new Infomaniak box has IPv6 at all (decides the AAAA action).

## 8. Pitfalls this server has already produced

- **Quota failures:** 4.82 GB per run twice daily hit the Infomaniak ceiling
  (`Upload exceeds quota`). The space has since been enlarged and the schedule is now
  once daily — both verified — but per-run size grows (2.6 → 4.6 GB in two weeks), mostly
  registry image blobs. If the ceiling ever returns, the durable fix is getting the
  rebuildable images out of every snapshot, not deleting backups.
- **The once-a-day change cannot be made over the API** — dashboard only. Historically it
  was reported as done and was not. Verified applied 2026-10-30.
- **`cloudron update --image` reads the manifest from the current directory.** Running it
  from the wrong repo fails with `manifest id does not match`, which looks like a server
  problem and is not.
- **The agent host is not migrated** but will see errors while DNS moves. It needs no
  config change: every reference is a hostname, including `--server my.inference.coop`.
  Its 30-minute webhook watch will fire during the window and may raise false alarms;
  the daily digest now compares against its previous run before flagging, so one-off
  transients should not reach you as decisions.
- **A DNS cache countdown is not a zone TTL.** Reading 760–1642 s off a resolver answer
  and writing it into this runbook as "the TTL" was wrong — the configured TTL was 3600 s
  all along. Query the zone at the registrar (or trust a fresh authoritative answer) for
  planning numbers.