From a2088e39763ef959cde0004b198222d5882805bc Mon Sep 17 00:00:00 2001 From: Inferencebot Date: Fri, 30 Oct 2026 15:32:14 +0000 Subject: [PATCH] migration runbook: fix two errors and five gaps found in a read-only review Verified against live DNS, federation and backup state (no privileged commands): - IPv6: the zone carries AAAA records (2a02:4780:6e:83fd::1, Hostinger) that the plan never mentioned. Switching A records only leaves IPv6 clients - including other homeservers and mail senders - on the old box. Repoint or delete; decide pre-flight. - TTL: the runbook said 760-1642s, which was a decrementing cache countdown, not the zone TTL (3600s). Propagation is up to an hour, which is why the next item matters. - "Leave the old box running read-only" was undefined and dangerous: during the TTL window a still-serving old box silently collects Matrix messages, Gitea pushes, API spend and mail into a database that gets thrown away. Now step 3: stop the apps after the final backup; keep the machine powered for rollback only. - Port 25 inbound was missing from the port list (mail delivery); 8448 noted as intentionally closed - federation is delegated to 443 via the apex .well-known, verified live. - DMARC is p=reject pct=100: post-restore mail misalignment means discarded mail, not junk. DKIM TXT must be verified against the restored key; test delivery to an external mailbox, not just that the app accepts the send. - Post-restore federation check now includes the apex .well-known (served by the website app), without which federation breaks while matrix.inference.coop looks fine. - Stale text removed: the once-daily backup schedule IS applied (00 00 23 * * *, verified), and the backup space was enlarged - newest run healthy at 4.6 GB. --- cloudron-server-migration.md | 107 ++++++++++++++++++++++++++--------- 1 file changed, 80 insertions(+), 27 deletions(-) diff --git a/cloudron-server-migration.md b/cloudron-server-migration.md index 67c31f7..078e0a1 100644 --- a/cloudron-server-migration.md +++ b/cloudron-server-migration.md @@ -1,8 +1,9 @@ # Cloudron server migration — Hostinger → Infomaniak Pre-flight and cutover runbook for moving the co-op's Cloudron server. Written -2026-10-29 from a live inventory of the server. **Owner:** Nathan. **Executor:** whoever -is at the console that weekend — assume it is not the person who wrote this. +2026-10-29 from a live inventory of the server; revised 2026-10-30 after a read-only +review against live DNS, federation and backup state. **Owner:** Nathan. **Executor:** +whoever is at the console that weekend — assume it is not the person who wrote this. The server being migrated hosts `my.inference.coop` and **16 apps**. The agent host is a separate machine (already on Infomaniak) and is NOT part of this migration. @@ -17,7 +18,7 @@ separate machine (already on Infomaniak) and is NOT part of this migration. `admin-panel`, `member-portal`, `member-dashboard`). These exist nowhere else — if the registry does not restore, the custom apps cannot start. - Size check that confirms app data is included: ~5.8 GB of app data visible inside - containers vs ~4.8 GB per backup run. + containers vs ~4.6 GB per backup run. Apps: chat, chat-legacy (OpenWebUI), inference.coop (Surfer), forum (Loomio), matrix (Synapse), gateway (LiteLLM), panel, registry, git (Gitea), vault (Vaultwarden), portal, @@ -55,11 +56,26 @@ needed to restore are inside the thing being restored. - [ ] OS: same or newer Ubuntu than the old box. Cloudron install version must be ≥ the old box's (9.0.4). - [ ] Confirm the Cloudron subscription can be moved to a new server. +- [ ] **Decide the IPv6 question before anything else DNS-related.** The zone carries + **AAAA records**: `inference.coop`, `my`, `matrix` all point to + `2a02:4780:6e:83fd::1` (Hostinger's range). If the new box has IPv6, repoint them; + if it does not, **delete the AAAA records** as part of the switch. Changing only + the A records leaves every IPv6-capable client — including other Matrix + homeservers and mail senders — still reaching the old box. - [ ] Ask Infomaniak to set **rDNS/PTR for the new IP to `my.inference.coop`** — mail deliverability depends on it. -- [ ] Open on the new box: 22, 80, 443, 465, 587, 993. Match the old box's exposure - (`ufw` is inactive there; Cloudron manages its own iptables — do not add ufw rules - that fight Docker). +- [ ] Open on the new box: **22, 80, 443, 465, 587, 993, and 25 inbound** (mail delivery + — test from outside, e.g. `telnet my.inference.coop 25` once DNS moves; note that + many networks block *outbound* 25, so a closed result from inside another server is + not conclusive). **8448 is intentionally closed** — federation is delegated to + 443 via the apex `.well-known` (verified live: + `inference.coop/.well-known/matrix/server` → `{"m.server":"matrix.inference.coop:443"}`). + Do not "fix" 8448. Do not add ufw rules that fight Docker; the old box runs with + ufw inactive and Cloudron-managed iptables. +- [ ] **DMARC is `p=reject; pct=100`.** Strict. After the restore, any DKIM/SPF + misalignment means outbound mail is *discarded by recipients*, not junked. The + DKIM TXT record (`cloudron._domainkey`) must match the key inside the restored + mail app's data — verify both rather than assuming. - [ ] **Mount audit on the Cloudron host** — the one class of data the backup cannot see: sudo docker ps -q | xargs -I{} sudo docker inspect {} \ @@ -68,40 +84,64 @@ needed to restore are inside the thing being restored. A container showed `/var/lib/nextcloud` mounted from the host. Identify its owner and size; if it holds real data, copy it separately. -- [ ] Record the highest backup run number before the freeze. -- [ ] Lower all A-record TTLs to 300 s **a day early** (currently 760–1642 s ≈ 13–27 min). +- [ ] Record the highest backup run number before the freeze. (As of 2026-10-30 the + newest run is healthy at ~4.6 GB and the backup space was enlarged, so the quota + pressure that produced the `EntityTooLarge` failures is relieved — but run growth + is still the number to watch: 2.6 → 4.6 GB per run over two weeks.) +- [ ] Lower all A **and AAAA** record TTLs to 300 s **a day early**. The zone TTL is + **3600 s** — an earlier note read 760–1642 s, which was a decrementing *cache + countdown*, not the configured TTL. At 3600 s, propagation after the switch takes + up to an hour, which is exactly the window where the old box must not be accepting + writes (§4). - [ ] List external allowlists that carry our IP: Regolo, GreenPT, Tinfoil, any mail relay, and inbound webhooks. Update after cutover. +- [ ] SSH host keys change with the box. Accept the new fingerprint deliberately, in + advance, so a strict-host-key refusal doesn't stop you mid-migration. - [ ] Announce the window in `#updates`. ## 4. Cutover, in order 1. **Freeze writes.** Pause the portal's `membership_sweep` and `rag_refresh`, hold - Listmonk campaigns. Chat cannot be frozen — note the exact snapshot time. + Listmonk campaigns. 2. **Final backup on the old box.** Run "Backup now" and confirm state `normal`. Everything after this timestamp is *lost* in a forward migration. -3. **Stop the old box's backup schedule** (Settings → Backups → disable). Two Cloudrons +3. **Stop the apps on the old box** — all of them, including mail. Keep the *machine* + powered on for rollback, but nothing may accept writes from here on: during the + up-to-one-hour DNS propagation, a still-serving old box silently collects Matrix + messages, Gitea pushes, API spend and mail into a database that gets thrown away. + Clients and mail senders retry; that is the correct failure mode. +4. **Stop the old box's backup schedule** (Settings → Backups → disable). Two Cloudrons writing one target interleave and produce `EntityTooLarge` quota failures. -4. **Build the new box and restore** from the Infomaniak S3 target, using the passphrase +5. **Build the new box and restore** from the Infomaniak S3 target, using the passphrase from §3. Expect TLS to fail until DNS moves — reach the dashboard by IP or a temporary override. -5. **Switch the A records** to the new IP. -6. Wait out the TTL, then confirm certificates issued. -7. **Re-enable backups on the new box — once daily, retention 7 days.** (The approved - change was never applied; the API has no route for it, so do it in the dashboard.) -8. Update external allowlists. Leave the old box running read-only. +6. **Switch the A records — and the AAAA records** (repoint or delete, per the §3 + decision). +7. Wait out the TTL (up to an hour at 3600 s if not lowered), then confirm certificates + issued. +8. **Re-enable backups on the new box — once daily at 23:00, retention 7 days.** (This is + now applied on the old box — schedule `00 00 23 * * *`, verified — re-check it on the + new box after restore rather than assuming.) +9. Update external allowlists. The old box stays powered on with apps stopped, for + rollback only. ## 5. Post-restore verification - [ ] Dashboard loads with a valid certificate; all 16 apps `running`. - [ ] **One custom app loads** — e.g. `panel.inference.coop`. This is the proof the registry restored; without it nothing custom starts. +- [ ] **The apex `.well-known` answers** — `curl https://inference.coop/.well-known/matrix/server` + must return `{"m.server":"matrix.inference.coop:443"}`. It is served by the *website* + app, so if that app's data is missing, federation breaks while + `matrix.inference.coop` itself still looks healthy. Then: + `curl -s https://matrix.inference.coop/_matrix/federation/v1/version`. - [ ] Data spot-checks: Matrix login + a room loads; Gitea repo list; portal `/admin/overview` returns the expected member count; gateway `/v1/models`. -- [ ] Mail: send to and receive from an external address; MX resolves to the new IP; SPF - still `a:my.inference.coop`; DKIM keys survived (they are in app data); PTR correct. -- [ ] Federation: `curl -s https://matrix.inference.coop/_matrix/federation/v1/version`. - Other homeservers cache the old IP and will retry it for a while — that is expected. +- [ ] Mail: MX resolves to the new IP; SPF still `a:my.inference.coop`; the DKIM TXT + (`cloudron._domainkey`) matches the restored key; PTR correct; **send to an + external mailbox and confirm it actually arrived** (DMARC is `p=reject` — a silent + discard is the failure mode, not a junk folder); receive from outside; **test + port 25 inbound from a third network**. - [ ] Agent API access: `cloudron list` from the agent host. The API token lives in Cloudron's config so it should survive; if it does not, mint a new one and update the agent's `.env`. @@ -110,29 +150,42 @@ needed to restore are inside the thing being restored. ## 6. Rollback -- Keep the old box powered on, read-only, for several days. -- Rollback = point DNS back. Do not delete old snapshots from S3 until the new box has at - least one verified backup run. +- Keep the old box **powered on with apps stopped** for several days. Rollback = point DNS + back and start the apps. +- Do not delete old snapshots from S3 until the new box has at least one verified backup + run. - Know the asymmetry: anything written between the final backup and the DNS switch exists - **only on the old box**. Rolling forward discards it; rolling back keeps it. Keep that - window short and quiet. + **only on the old box**. Rolling forward discards it; rolling back keeps it. §4 step 3 + (stop apps) is what keeps that window empty rather than merely short. ## 7. Open unknowns — resolve before cutover +- Whether the Vault export (§3 blocker) has actually been done. - Which app owns the `/var/lib/nextcloud` host mount, and how much data is in it. - The Cloudron host's OS version, disk and RAM (not visible over the API). - Whether the S3 access key is in the off-box JSON or only inside Cloudron. - Whether any provider API key is bound to our current IP. - Whether the subscription permits a server move. +- Whether the new Infomaniak box has IPv6 at all (decides the AAAA action). ## 8. Pitfalls this server has already produced - **Quota failures:** 4.82 GB per run twice daily hit the Infomaniak ceiling - (`Upload exceeds quota`). Retention is 7 days; schedule must be once daily. + (`Upload exceeds quota`). The space has since been enlarged and the schedule is now + once daily — both verified — but per-run size grows (2.6 → 4.6 GB in two weeks), mostly + registry image blobs. If the ceiling ever returns, the durable fix is getting the + rebuildable images out of every snapshot, not deleting backups. - **The once-a-day change cannot be made over the API** — dashboard only. Historically it - was reported as done and was not. + was reported as done and was not. Verified applied 2026-10-30. - **`cloudron update --image` reads the manifest from the current directory.** Running it from the wrong repo fails with `manifest id does not match`, which looks like a server problem and is not. - **The agent host is not migrated** but will see errors while DNS moves. It needs no config change: every reference is a hostname, including `--server my.inference.coop`. + Its 30-minute webhook watch will fire during the window and may raise false alarms; + the daily digest now compares against its previous run before flagging, so one-off + transients should not reach you as decisions. +- **A DNS cache countdown is not a zone TTL.** Reading 760–1642 s off a resolver answer + and writing it into this runbook as "the TTL" was wrong — the configured TTL was 3600 s + all along. Query the zone at the registrar (or trust a fresh authoritative answer) for + planning numbers.