Add Cloudron server migration runbook (Hostinger → Infomaniak)
Pre-flight, ordered cutover, post-restore verification, rollback, and the list of what Cloudron's backup does not cover. Written from a live inventory of the server on 2026-10-29; open unknowns are marked rather than guessed.
This commit is contained in:
inference-bot committed
commit
eec9572a3f
2 files changed
+144
No files matched your search
@@ -0,0 +1,6 @@
|
||||
# Runbooks
|
||||
|
||||
Operational runbooks for the co-op's infrastructure. Private: these contain
|
||||
topology, addresses, and procedure detail that should not be public.
|
||||
|
||||
- [Cloudron server migration — Hostinger → Infomaniak](cloudron-server-migration.md)
|
||||
@@ -0,0 +1,138 @@
|
||||
# Cloudron server migration — Hostinger → Infomaniak
|
||||
|
||||
Pre-flight and cutover runbook for moving the co-op's Cloudron server. Written
|
||||
2026-10-29 from a live inventory of the server. **Owner:** Nathan. **Executor:** whoever
|
||||
is at the console that weekend — assume it is not the person who wrote this.
|
||||
|
||||
The server being migrated hosts `my.inference.coop` and **16 apps**. The agent host is a
|
||||
separate machine (already on Infomaniak) and is NOT part of this migration.
|
||||
|
||||
## 1. What Cloudron's backup covers — verified, not assumed
|
||||
|
||||
- All 16 apps and their `/app/data` (localstorage), plus addons: postgres, mysql,
|
||||
mongodb, redis.
|
||||
- Cloudron platform config: domains, dashboard users, app env vars, TLS config, backup
|
||||
target settings, scheduled tasks.
|
||||
- **The container registry app, 3.8 GB of custom images** (`librechat`, `litellm`,
|
||||
`admin-panel`, `member-portal`, `member-dashboard`). These exist nowhere else — if the
|
||||
registry does not restore, the custom apps cannot start.
|
||||
- Size check that confirms app data is included: ~5.8 GB of app data visible inside
|
||||
containers vs ~4.8 GB per backup run.
|
||||
|
||||
Apps: chat, chat-legacy (OpenWebUI), inference.coop (Surfer), forum (Loomio), matrix
|
||||
(Synapse), gateway (LiteLLM), panel, registry, git (Gitea), vault (Vaultwarden), portal,
|
||||
search (SearXNG), dashboard, newsletter (Listmonk), email (Cloudron Mail), cloud
|
||||
(Nextcloud).
|
||||
|
||||
## 2. What it does NOT cover
|
||||
|
||||
- **The host OS layer.** Ubuntu itself, netplan, cloud-init, `sshd` config,
|
||||
`authorized_keys`, hostname, PTR, provider firewall, hand-edited `/etc`. Build the new
|
||||
box; do not expect to restore it.
|
||||
- **The Cloudron licence/subscription** — must be moved to the new server.
|
||||
- **DNS records.** Manual at yoursrs.com (domains.coop NS). Cloudron cannot manage them.
|
||||
- **The backup target config and passphrase** — chicken-and-egg by design.
|
||||
- **App data written to a host path instead of `/app/data`** — see the audit in §3.
|
||||
- **Anything on the old box outside Cloudron** (e.g. `/etc` tweaks). Diff before
|
||||
decommissioning.
|
||||
|
||||
Not a gap, despite looking like one: our custom images (registry is an app, backed up),
|
||||
and the agent's own state (different machine).
|
||||
|
||||
## 3. Pre-flight
|
||||
|
||||
**Blocker — do this first.** The Vault app (`vault.inference.coop`) runs *on the server
|
||||
being migrated*, and the backup passphrase is believed to live there. The credentials
|
||||
needed to restore are inside the thing being restored.
|
||||
|
||||
- [ ] Export the Vault contents and store them off-server (personal password manager or
|
||||
offline media). Confirm the **Cloudron backup passphrase** is in your own hands.
|
||||
- [ ] Confirm the backup target's S3 access key is available off-box. The agent host has
|
||||
`~/.hermes/security/backup_sites_backup.json` (0600) — check whether it carries the
|
||||
key; if not, retrieve it from the Infomaniak console.
|
||||
- [ ] New host capacity: `df -h`, `free -m`, `lsb_release -a`, `cloudron --version`.
|
||||
Must hold ~6 GB of app data plus Docker images and growth; check swap exists.
|
||||
- [ ] OS: same or newer Ubuntu than the old box. Cloudron install version must be ≥ the
|
||||
old box's (9.0.4).
|
||||
- [ ] Confirm the Cloudron subscription can be moved to a new server.
|
||||
- [ ] Ask Infomaniak to set **rDNS/PTR for the new IP to `my.inference.coop`** — mail
|
||||
deliverability depends on it.
|
||||
- [ ] Open on the new box: 22, 80, 443, 465, 587, 993. Match the old box's exposure
|
||||
(`ufw` is inactive there; Cloudron manages its own iptables — do not add ufw rules
|
||||
that fight Docker).
|
||||
- [ ] **Mount audit on the Cloudron host** — the one class of data the backup cannot see:
|
||||
|
||||
sudo docker ps -q | xargs -I{} sudo docker inspect {} \
|
||||
--format '{{.Name}} {{range .Mounts}}{{.Source}}=>{{.Destination}} {{end}}' \
|
||||
| tr ' ' '\n' | grep -v /app/data
|
||||
|
||||
A container showed `/var/lib/nextcloud` mounted from the host. Identify its owner and
|
||||
size; if it holds real data, copy it separately.
|
||||
- [ ] Record the highest backup run number before the freeze.
|
||||
- [ ] Lower all A-record TTLs to 300 s **a day early** (currently 760–1642 s ≈ 13–27 min).
|
||||
- [ ] List external allowlists that carry our IP: Regolo, GreenPT, Tinfoil, any mail relay,
|
||||
and inbound webhooks. Update after cutover.
|
||||
- [ ] Announce the window in `#updates`.
|
||||
|
||||
## 4. Cutover, in order
|
||||
|
||||
1. **Freeze writes.** Pause the portal's `membership_sweep` and `rag_refresh`, hold
|
||||
Listmonk campaigns. Chat cannot be frozen — note the exact snapshot time.
|
||||
2. **Final backup on the old box.** Run "Backup now" and confirm state `normal`.
|
||||
Everything after this timestamp is *lost* in a forward migration.
|
||||
3. **Stop the old box's backup schedule** (Settings → Backups → disable). Two Cloudrons
|
||||
writing one target interleave and produce `EntityTooLarge` quota failures.
|
||||
4. **Build the new box and restore** from the Infomaniak S3 target, using the passphrase
|
||||
from §3. Expect TLS to fail until DNS moves — reach the dashboard by IP or a temporary
|
||||
override.
|
||||
5. **Switch the A records** to the new IP.
|
||||
6. Wait out the TTL, then confirm certificates issued.
|
||||
7. **Re-enable backups on the new box — once daily, retention 7 days.** (The approved
|
||||
change was never applied; the API has no route for it, so do it in the dashboard.)
|
||||
8. Update external allowlists. Leave the old box running read-only.
|
||||
|
||||
## 5. Post-restore verification
|
||||
|
||||
- [ ] Dashboard loads with a valid certificate; all 16 apps `running`.
|
||||
- [ ] **One custom app loads** — e.g. `panel.inference.coop`. This is the proof the
|
||||
registry restored; without it nothing custom starts.
|
||||
- [ ] Data spot-checks: Matrix login + a room loads; Gitea repo list; portal
|
||||
`/admin/overview` returns the expected member count; gateway `/v1/models`.
|
||||
- [ ] Mail: send to and receive from an external address; MX resolves to the new IP; SPF
|
||||
still `a:my.inference.coop`; DKIM keys survived (they are in app data); PTR correct.
|
||||
- [ ] Federation: `curl -s https://matrix.inference.coop/_matrix/federation/v1/version`.
|
||||
Other homeservers cache the old IP and will retry it for a while — that is expected.
|
||||
- [ ] Agent API access: `cloudron list` from the agent host. The API token lives in
|
||||
Cloudron's config so it should survive; if it does not, mint a new one and update the
|
||||
agent's `.env`.
|
||||
- [ ] Schedules fire **once**, not twice: portal sweeps, Listmonk, agent digests.
|
||||
- [ ] First backup on the new box is `normal` and in the expected size range.
|
||||
|
||||
## 6. Rollback
|
||||
|
||||
- Keep the old box powered on, read-only, for several days.
|
||||
- Rollback = point DNS back. Do not delete old snapshots from S3 until the new box has at
|
||||
least one verified backup run.
|
||||
- Know the asymmetry: anything written between the final backup and the DNS switch exists
|
||||
**only on the old box**. Rolling forward discards it; rolling back keeps it. Keep that
|
||||
window short and quiet.
|
||||
|
||||
## 7. Open unknowns — resolve before cutover
|
||||
|
||||
- Which app owns the `/var/lib/nextcloud` host mount, and how much data is in it.
|
||||
- The Cloudron host's OS version, disk and RAM (not visible over the API).
|
||||
- Whether the S3 access key is in the off-box JSON or only inside Cloudron.
|
||||
- Whether any provider API key is bound to our current IP.
|
||||
- Whether the subscription permits a server move.
|
||||
|
||||
## 8. Pitfalls this server has already produced
|
||||
|
||||
- **Quota failures:** 4.82 GB per run twice daily hit the Infomaniak ceiling
|
||||
(`Upload exceeds quota`). Retention is 7 days; schedule must be once daily.
|
||||
- **The once-a-day change cannot be made over the API** — dashboard only. Historically it
|
||||
was reported as done and was not.
|
||||
- **`cloudron update --image` reads the manifest from the current directory.** Running it
|
||||
from the wrong repo fails with `manifest id does not match`, which looks like a server
|
||||
problem and is not.
|
||||
- **The agent host is not migrated** but will see errors while DNS moves. It needs no
|
||||
config change: every reference is a hostname, including `--server my.inference.coop`.
|
||||
Reference in new issue
Block a user