runbooks/cloudron-server-migration.md

# Cloudron server migration — Hostinger → Infomaniak

Pre-flight and cutover runbook for moving the co-op's Cloudron server. Written
2026-10-29 from a live inventory of the server. **Owner:** Nathan. **Executor:** whoever
is at the console that weekend — assume it is not the person who wrote this.

The server being migrated hosts `my.inference.coop` and **16 apps**. The agent host is a
separate machine (already on Infomaniak) and is NOT part of this migration.

## 1. What Cloudron's backup covers — verified, not assumed

- All 16 apps and their `/app/data` (localstorage), plus addons: postgres, mysql,
  mongodb, redis.
- Cloudron platform config: domains, dashboard users, app env vars, TLS config, backup
  target settings, scheduled tasks.
- **The container registry app, 3.8 GB of custom images** (`librechat`, `litellm`,
  `admin-panel`, `member-portal`, `member-dashboard`). These exist nowhere else — if the
  registry does not restore, the custom apps cannot start.
- Size check that confirms app data is included: ~5.8 GB of app data visible inside
  containers vs ~4.8 GB per backup run.

Apps: chat, chat-legacy (OpenWebUI), inference.coop (Surfer), forum (Loomio), matrix
(Synapse), gateway (LiteLLM), panel, registry, git (Gitea), vault (Vaultwarden), portal,
search (SearXNG), dashboard, newsletter (Listmonk), email (Cloudron Mail), cloud
(Nextcloud).

## 2. What it does NOT cover

- **The host OS layer.** Ubuntu itself, netplan, cloud-init, `sshd` config,
  `authorized_keys`, hostname, PTR, provider firewall, hand-edited `/etc`. Build the new
  box; do not expect to restore it.
- **The Cloudron licence/subscription** — must be moved to the new server.
- **DNS records.** Manual at yoursrs.com (domains.coop NS). Cloudron cannot manage them.
- **The backup target config and passphrase** — chicken-and-egg by design.
- **App data written to a host path instead of `/app/data`** — see the audit in §3.
- **Anything on the old box outside Cloudron** (e.g. `/etc` tweaks). Diff before
  decommissioning.

Not a gap, despite looking like one: our custom images (registry is an app, backed up),
and the agent's own state (different machine).

## 3. Pre-flight

**Blocker — do this first.** The Vault app (`vault.inference.coop`) runs *on the server
being migrated*, and the backup passphrase is believed to live there. The credentials
needed to restore are inside the thing being restored.

- [ ] Export the Vault contents and store them off-server (personal password manager or
      offline media). Confirm the **Cloudron backup passphrase** is in your own hands.
- [ ] Confirm the backup target's S3 access key is available off-box. The agent host has
      `~/.hermes/security/backup_sites_backup.json` (0600) — check whether it carries the
      key; if not, retrieve it from the Infomaniak console.
- [ ] New host capacity: `df -h`, `free -m`, `lsb_release -a`, `cloudron --version`.
      Must hold ~6 GB of app data plus Docker images and growth; check swap exists.
- [ ] OS: same or newer Ubuntu than the old box. Cloudron install version must be ≥ the
      old box's (9.0.4).
- [ ] Confirm the Cloudron subscription can be moved to a new server.
- [ ] Ask Infomaniak to set **rDNS/PTR for the new IP to `my.inference.coop`** — mail
      deliverability depends on it.
- [ ] Open on the new box: 22, 80, 443, 465, 587, 993. Match the old box's exposure
      (`ufw` is inactive there; Cloudron manages its own iptables — do not add ufw rules
      that fight Docker).
- [ ] **Mount audit on the Cloudron host** — the one class of data the backup cannot see:

      sudo docker ps -q | xargs -I{} sudo docker inspect {} \
        --format '{{.Name}} {{range .Mounts}}{{.Source}}=>{{.Destination}} {{end}}' \
        | tr ' ' '\n' | grep -v /app/data

      A container showed `/var/lib/nextcloud` mounted from the host. Identify its owner and
      size; if it holds real data, copy it separately.
- [ ] Record the highest backup run number before the freeze.
- [ ] Lower all A-record TTLs to 300 s **a day early** (currently 760–1642 s ≈ 13–27 min).
- [ ] List external allowlists that carry our IP: Regolo, GreenPT, Tinfoil, any mail relay,
      and inbound webhooks. Update after cutover.
- [ ] Announce the window in `#updates`.

## 4. Cutover, in order

1. **Freeze writes.** Pause the portal's `membership_sweep` and `rag_refresh`, hold
   Listmonk campaigns. Chat cannot be frozen — note the exact snapshot time.
2. **Final backup on the old box.** Run "Backup now" and confirm state `normal`.
   Everything after this timestamp is *lost* in a forward migration.
3. **Stop the old box's backup schedule** (Settings → Backups → disable). Two Cloudrons
   writing one target interleave and produce `EntityTooLarge` quota failures.
4. **Build the new box and restore** from the Infomaniak S3 target, using the passphrase
   from §3. Expect TLS to fail until DNS moves — reach the dashboard by IP or a temporary
   override.
5. **Switch the A records** to the new IP.
6. Wait out the TTL, then confirm certificates issued.
7. **Re-enable backups on the new box — once daily, retention 7 days.** (The approved
   change was never applied; the API has no route for it, so do it in the dashboard.)
8. Update external allowlists. Leave the old box running read-only.

## 5. Post-restore verification

- [ ] Dashboard loads with a valid certificate; all 16 apps `running`.
- [ ] **One custom app loads** — e.g. `panel.inference.coop`. This is the proof the
      registry restored; without it nothing custom starts.
- [ ] Data spot-checks: Matrix login + a room loads; Gitea repo list; portal
      `/admin/overview` returns the expected member count; gateway `/v1/models`.
- [ ] Mail: send to and receive from an external address; MX resolves to the new IP; SPF
      still `a:my.inference.coop`; DKIM keys survived (they are in app data); PTR correct.
- [ ] Federation: `curl -s https://matrix.inference.coop/_matrix/federation/v1/version`.
      Other homeservers cache the old IP and will retry it for a while — that is expected.
- [ ] Agent API access: `cloudron list` from the agent host. The API token lives in
      Cloudron's config so it should survive; if it does not, mint a new one and update the
      agent's `.env`.
- [ ] Schedules fire **once**, not twice: portal sweeps, Listmonk, agent digests.
- [ ] First backup on the new box is `normal` and in the expected size range.

## 6. Rollback

- Keep the old box powered on, read-only, for several days.
- Rollback = point DNS back. Do not delete old snapshots from S3 until the new box has at
  least one verified backup run.
- Know the asymmetry: anything written between the final backup and the DNS switch exists
  **only on the old box**. Rolling forward discards it; rolling back keeps it. Keep that
  window short and quiet.

## 7. Open unknowns — resolve before cutover

- Which app owns the `/var/lib/nextcloud` host mount, and how much data is in it.
- The Cloudron host's OS version, disk and RAM (not visible over the API).
- Whether the S3 access key is in the off-box JSON or only inside Cloudron.
- Whether any provider API key is bound to our current IP.
- Whether the subscription permits a server move.

## 8. Pitfalls this server has already produced

- **Quota failures:** 4.82 GB per run twice daily hit the Infomaniak ceiling
  (`Upload exceeds quota`). Retention is 7 days; schedule must be once daily.
- **The once-a-day change cannot be made over the API** — dashboard only. Historically it
  was reported as done and was not.
- **`cloudron update --image` reads the manifest from the current directory.** Running it
  from the wrong repo fails with `manifest id does not match`, which looks like a server
  problem and is not.
- **The agent host is not migrated** but will see errors while DNS moves. It needs no
  config change: every reference is a hostname, including `--server my.inference.coop`.