migration runbook: fix two errors and five gaps found in a read-only review

Verified against live DNS, federation and backup state (no privileged commands):

- IPv6: the zone carries AAAA records (2a02:4780:6e:83fd::1, Hostinger) that the plan
  never mentioned. Switching A records only leaves IPv6 clients - including other
  homeservers and mail senders - on the old box. Repoint or delete; decide pre-flight.
- TTL: the runbook said 760-1642s, which was a decrementing cache countdown, not the
  zone TTL (3600s). Propagation is up to an hour, which is why the next item matters.
- "Leave the old box running read-only" was undefined and dangerous: during the TTL
  window a still-serving old box silently collects Matrix messages, Gitea pushes, API
  spend and mail into a database that gets thrown away. Now step 3: stop the apps after
  the final backup; keep the machine powered for rollback only.
- Port 25 inbound was missing from the port list (mail delivery); 8448 noted as
  intentionally closed - federation is delegated to 443 via the apex .well-known,
  verified live.
- DMARC is p=reject pct=100: post-restore mail misalignment means discarded mail, not
  junk. DKIM TXT must be verified against the restored key; test delivery to an external
  mailbox, not just that the app accepts the send.
- Post-restore federation check now includes the apex .well-known (served by the
  website app), without which federation breaks while matrix.inference.coop looks fine.
- Stale text removed: the once-daily backup schedule IS applied (00 00 23 * * *,
  verified), and the backup space was enlarged - newest run healthy at 4.6 GB.

cloudron-server-migration.md

@@ -1,8 +1,9 @@
 # Cloudron server migration — Hostinger → Infomaniak
 
 Pre-flight and cutover runbook for moving the co-op's Cloudron server. Written
-2026-10-29 from a live inventory of the server. **Owner:** Nathan. **Executor:** whoever
-is at the console that weekend — assume it is not the person who wrote this.
+2026-10-29 from a live inventory of the server; revised 2026-10-30 after a read-only
+review against live DNS, federation and backup state. **Owner:** Nathan. **Executor:**
+whoever is at the console that weekend — assume it is not the person who wrote this.
 
 The server being migrated hosts `my.inference.coop` and **16 apps**. The agent host is a
 separate machine (already on Infomaniak) and is NOT part of this migration.
@@ -17,7 +18,7 @@ separate machine (already on Infomaniak) and is NOT part of this migration.
   `admin-panel`, `member-portal`, `member-dashboard`). These exist nowhere else — if the
   registry does not restore, the custom apps cannot start.
 - Size check that confirms app data is included: ~5.8 GB of app data visible inside
-  containers vs ~4.8 GB per backup run.
+  containers vs ~4.6 GB per backup run.
 
 Apps: chat, chat-legacy (OpenWebUI), inference.coop (Surfer), forum (Loomio), matrix
 (Synapse), gateway (LiteLLM), panel, registry, git (Gitea), vault (Vaultwarden), portal,
@@ -55,11 +56,26 @@ needed to restore are inside the thing being restored.
 - [ ] OS: same or newer Ubuntu than the old box. Cloudron install version must be ≥ the
       old box's (9.0.4).
 - [ ] Confirm the Cloudron subscription can be moved to a new server.
+- [ ] **Decide the IPv6 question before anything else DNS-related.** The zone carries
+      **AAAA records**: `inference.coop`, `my`, `matrix` all point to
+      `2a02:4780:6e:83fd::1` (Hostinger's range). If the new box has IPv6, repoint them;
+      if it does not, **delete the AAAA records** as part of the switch. Changing only
+      the A records leaves every IPv6-capable client — including other Matrix
+      homeservers and mail senders — still reaching the old box.
 - [ ] Ask Infomaniak to set **rDNS/PTR for the new IP to `my.inference.coop`** — mail
       deliverability depends on it.
-- [ ] Open on the new box: 22, 80, 443, 465, 587, 993. Match the old box's exposure
-      (`ufw` is inactive there; Cloudron manages its own iptables — do not add ufw rules
-      that fight Docker).
+- [ ] Open on the new box: **22, 80, 443, 465, 587, 993, and 25 inbound** (mail delivery
+      — test from outside, e.g. `telnet my.inference.coop 25` once DNS moves; note that
+      many networks block *outbound* 25, so a closed result from inside another server is
+      not conclusive). **8448 is intentionally closed** — federation is delegated to
+      443 via the apex `.well-known` (verified live:
+      `inference.coop/.well-known/matrix/server` → `{"m.server":"matrix.inference.coop:443"}`).
+      Do not "fix" 8448. Do not add ufw rules that fight Docker; the old box runs with
+      ufw inactive and Cloudron-managed iptables.
+- [ ] **DMARC is `p=reject; pct=100`.** Strict. After the restore, any DKIM/SPF
+      misalignment means outbound mail is *discarded by recipients*, not junked. The
+      DKIM TXT record (`cloudron._domainkey`) must match the key inside the restored
+      mail app's data — verify both rather than assuming.
 - [ ] **Mount audit on the Cloudron host** — the one class of data the backup cannot see:
 
       sudo docker ps -q | xargs -I{} sudo docker inspect {} \
@@ -68,40 +84,64 @@ needed to restore are inside the thing being restored.
 
       A container showed `/var/lib/nextcloud` mounted from the host. Identify its owner and
       size; if it holds real data, copy it separately.
-- [ ] Record the highest backup run number before the freeze.
-- [ ] Lower all A-record TTLs to 300 s **a day early** (currently 760–1642 s ≈ 13–27 min).
+- [ ] Record the highest backup run number before the freeze. (As of 2026-10-30 the
+      newest run is healthy at ~4.6 GB and the backup space was enlarged, so the quota
+      pressure that produced the `EntityTooLarge` failures is relieved — but run growth
+      is still the number to watch: 2.6 → 4.6 GB per run over two weeks.)
+- [ ] Lower all A **and AAAA** record TTLs to 300 s **a day early**. The zone TTL is
+      **3600 s** — an earlier note read 760–1642 s, which was a decrementing *cache
+      countdown*, not the configured TTL. At 3600 s, propagation after the switch takes
+      up to an hour, which is exactly the window where the old box must not be accepting
+      writes (§4).
 - [ ] List external allowlists that carry our IP: Regolo, GreenPT, Tinfoil, any mail relay,
       and inbound webhooks. Update after cutover.
+- [ ] SSH host keys change with the box. Accept the new fingerprint deliberately, in
+      advance, so a strict-host-key refusal doesn't stop you mid-migration.
 - [ ] Announce the window in `#updates`.
 
 ## 4. Cutover, in order
 
 1. **Freeze writes.** Pause the portal's `membership_sweep` and `rag_refresh`, hold
-   Listmonk campaigns. Chat cannot be frozen — note the exact snapshot time.
+   Listmonk campaigns.
 2. **Final backup on the old box.** Run "Backup now" and confirm state `normal`.
    Everything after this timestamp is *lost* in a forward migration.
-3. **Stop the old box's backup schedule** (Settings → Backups → disable). Two Cloudrons
+3. **Stop the apps on the old box** — all of them, including mail. Keep the *machine*
+   powered on for rollback, but nothing may accept writes from here on: during the
+   up-to-one-hour DNS propagation, a still-serving old box silently collects Matrix
+   messages, Gitea pushes, API spend and mail into a database that gets thrown away.
+   Clients and mail senders retry; that is the correct failure mode.
+4. **Stop the old box's backup schedule** (Settings → Backups → disable). Two Cloudrons
    writing one target interleave and produce `EntityTooLarge` quota failures.
-4. **Build the new box and restore** from the Infomaniak S3 target, using the passphrase
+5. **Build the new box and restore** from the Infomaniak S3 target, using the passphrase
    from §3. Expect TLS to fail until DNS moves — reach the dashboard by IP or a temporary
    override.
-5. **Switch the A records** to the new IP.
-6. Wait out the TTL, then confirm certificates issued.
-7. **Re-enable backups on the new box — once daily, retention 7 days.** (The approved
-   change was never applied; the API has no route for it, so do it in the dashboard.)
-8. Update external allowlists. Leave the old box running read-only.
+6. **Switch the A records — and the AAAA records** (repoint or delete, per the §3
+   decision).
+7. Wait out the TTL (up to an hour at 3600 s if not lowered), then confirm certificates
+   issued.
+8. **Re-enable backups on the new box — once daily at 23:00, retention 7 days.** (This is
+   now applied on the old box — schedule `00 00 23 * * *`, verified — re-check it on the
+   new box after restore rather than assuming.)
+9. Update external allowlists. The old box stays powered on with apps stopped, for
+   rollback only.
 
 ## 5. Post-restore verification
 
 - [ ] Dashboard loads with a valid certificate; all 16 apps `running`.
 - [ ] **One custom app loads** — e.g. `panel.inference.coop`. This is the proof the
       registry restored; without it nothing custom starts.
+- [ ] **The apex `.well-known` answers** — `curl https://inference.coop/.well-known/matrix/server`
+      must return `{"m.server":"matrix.inference.coop:443"}`. It is served by the *website*
+      app, so if that app's data is missing, federation breaks while
+      `matrix.inference.coop` itself still looks healthy. Then:
+      `curl -s https://matrix.inference.coop/_matrix/federation/v1/version`.
 - [ ] Data spot-checks: Matrix login + a room loads; Gitea repo list; portal
       `/admin/overview` returns the expected member count; gateway `/v1/models`.
-- [ ] Mail: send to and receive from an external address; MX resolves to the new IP; SPF
-      still `a:my.inference.coop`; DKIM keys survived (they are in app data); PTR correct.
-- [ ] Federation: `curl -s https://matrix.inference.coop/_matrix/federation/v1/version`.
-      Other homeservers cache the old IP and will retry it for a while — that is expected.
+- [ ] Mail: MX resolves to the new IP; SPF still `a:my.inference.coop`; the DKIM TXT
+      (`cloudron._domainkey`) matches the restored key; PTR correct; **send to an
+      external mailbox and confirm it actually arrived** (DMARC is `p=reject` — a silent
+      discard is the failure mode, not a junk folder); receive from outside; **test
+      port 25 inbound from a third network**.
 - [ ] Agent API access: `cloudron list` from the agent host. The API token lives in
       Cloudron's config so it should survive; if it does not, mint a new one and update the
       agent's `.env`.
@@ -110,29 +150,42 @@ needed to restore are inside the thing being restored.
 
 ## 6. Rollback
 
-- Keep the old box powered on, read-only, for several days.
-- Rollback = point DNS back. Do not delete old snapshots from S3 until the new box has at
-  least one verified backup run.
+- Keep the old box **powered on with apps stopped** for several days. Rollback = point DNS
+  back and start the apps.
+- Do not delete old snapshots from S3 until the new box has at least one verified backup
+  run.
 - Know the asymmetry: anything written between the final backup and the DNS switch exists
-  **only on the old box**. Rolling forward discards it; rolling back keeps it. Keep that
-  window short and quiet.
+  **only on the old box**. Rolling forward discards it; rolling back keeps it. §4 step 3
+  (stop apps) is what keeps that window empty rather than merely short.
 
 ## 7. Open unknowns — resolve before cutover
 
+- Whether the Vault export (§3 blocker) has actually been done.
 - Which app owns the `/var/lib/nextcloud` host mount, and how much data is in it.
 - The Cloudron host's OS version, disk and RAM (not visible over the API).
 - Whether the S3 access key is in the off-box JSON or only inside Cloudron.
 - Whether any provider API key is bound to our current IP.
 - Whether the subscription permits a server move.
+- Whether the new Infomaniak box has IPv6 at all (decides the AAAA action).
 
 ## 8. Pitfalls this server has already produced
 
 - **Quota failures:** 4.82 GB per run twice daily hit the Infomaniak ceiling
-  (`Upload exceeds quota`). Retention is 7 days; schedule must be once daily.
+  (`Upload exceeds quota`). The space has since been enlarged and the schedule is now
+  once daily — both verified — but per-run size grows (2.6 → 4.6 GB in two weeks), mostly
+  registry image blobs. If the ceiling ever returns, the durable fix is getting the
+  rebuildable images out of every snapshot, not deleting backups.
 - **The once-a-day change cannot be made over the API** — dashboard only. Historically it
-  was reported as done and was not.
+  was reported as done and was not. Verified applied 2026-10-30.
 - **`cloudron update --image` reads the manifest from the current directory.** Running it
   from the wrong repo fails with `manifest id does not match`, which looks like a server
   problem and is not.
 - **The agent host is not migrated** but will see errors while DNS moves. It needs no
   config change: every reference is a hostname, including `--server my.inference.coop`.
+  Its 30-minute webhook watch will fire during the window and may raise false alarms;
+  the daily digest now compares against its previous run before flagging, so one-off
+  transients should not reach you as decisions.
+- **A DNS cache countdown is not a zone TTL.** Reading 760–1642 s off a resolver answer
+  and writing it into this runbook as "the TTL" was wrong — the configured TTL was 3600 s
+  all along. Query the zone at the registrar (or trust a fresh authoritative answer) for
+  planning numbers.