# Cloudron server migration — Hostinger → Infomaniak Pre-flight and cutover runbook for moving the co-op's Cloudron server. Written 2026-10-29 from a live inventory of the server; revised 2026-10-30 after a read-only review against live DNS, federation and backup state. **Owner:** Nathan. **Executor:** whoever is at the console that weekend — assume it is not the person who wrote this. The server being migrated hosts `my.inference.coop` and **16 apps**. The agent host is a separate machine (already on Infomaniak) and is NOT part of this migration. ## 1. What Cloudron's backup covers — verified, not assumed - All 16 apps and their `/app/data` (localstorage), plus addons: postgres, mysql, mongodb, redis. - Cloudron platform config: domains, dashboard users, app env vars, TLS config, backup target settings, scheduled tasks. - **The container registry app, 3.8 GB of custom images** (`librechat`, `litellm`, `admin-panel`, `member-portal`, `member-dashboard`). These exist nowhere else — if the registry does not restore, the custom apps cannot start. - Size check that confirms app data is included: ~5.8 GB of app data visible inside containers vs ~4.6 GB per backup run. Apps: chat, chat-legacy (OpenWebUI), inference.coop (Surfer), forum (Loomio), matrix (Synapse), gateway (LiteLLM), panel, registry, git (Gitea), vault (Vaultwarden), portal, search (SearXNG), dashboard, newsletter (Listmonk), email (Cloudron Mail), cloud (Nextcloud). ## 2. What it does NOT cover - **The host OS layer.** Ubuntu itself, netplan, cloud-init, `sshd` config, `authorized_keys`, hostname, PTR, provider firewall, hand-edited `/etc`. Build the new box; do not expect to restore it. - **The Cloudron licence/subscription** — must be moved to the new server. - **DNS records.** Manual at yoursrs.com (domains.coop NS). Cloudron cannot manage them. - **The backup target config and passphrase** — chicken-and-egg by design. - **App data written to a host path instead of `/app/data`** — see the audit in §3. - **Anything on the old box outside Cloudron** (e.g. `/etc` tweaks). Diff before decommissioning. Not a gap, despite looking like one: our custom images (registry is an app, backed up), and the agent's own state (different machine). ## 3. Pre-flight Companion sheet: `dns-change-sheet.md` (every record, current value, target value). **Blocker — do this first.** The Vault app (`vault.inference.coop`) runs *on the server being migrated*, and the backup passphrase is believed to live there. The credentials needed to restore are inside the thing being restored. - [ ] Export the Vault contents and store them off-server (personal password manager or offline media). Confirm the **Cloudron backup passphrase** is in your own hands. - [ ] **Retrieve the S3 secretAccessKey — the off-box copy does NOT have it.** Checked 2026-10-30: `~/.hermes/security/backup_sites_backup.json` carries `accessKeyId` and nothing else. A restore begins by pointing the new Cloudron at the backup target, so the *secret* must be in hand before the window — get it from the Infomaniak console and store it with the passphrase, off-box. - [x] The **restic** passphrase is safe: `~/.hermes/scripts/backup.sh` documents that `RESTIC_PASSWORD` lives in the agent host's `.env` **and** the co-op Vault. The agent host is not being migrated, so the agent-side backup stays recoverable. (The Vault, however, is an app on the box being migrated — so the Vault copy alone is not off-box until it is exported.) - [x] New host capacity: **4 CPU, 12 GB RAM, 250 GB disk** (given 2026-10-30) — ample for ~6 GB of app data plus images. Old box used ~17 GB of 58 GB. - [x] OS: **Ubuntu 26.04 LTS is supported** — Cloudron's install docs specify a fresh Ubuntu 26.04 x64 server. Minimums are 2 GB RAM / 20 GB disk; the new box has 12 GB / 250 GB. Resolved 2026-10-30. - [ ] **Version floor: install Cloudron ≥ 10.1.5**, the platform version this box runs (`/api/v1/cloudron/status` → `10.1.5`; backup manifests read `box_v10.1.5`). The earlier note of "9.0.4" was the *CLI* version, not the platform — a restore cannot go backwards. - [x] **Installed 2026-10-30 — banner shown, setup page live on the IPs.** Run `cloudron-setup` with **no `--domain`**, and leave the browser wizard for cutover day. Verified from the agent host: the box serves the setup page on **both** `https://179.237.86.116` and `https://[2001:1600:18:206::89]` (HTTP 200), and **80/443 are open on v4** (the earlier v4-rules gap is fixed). **The certificate is NOT self-signed.** Cloudron obtained a **valid Let's Encrypt certificate for the bare IP** — SAN `IP Address:179.237.86.116`, expires 2026-11-06, and `curl` verifies it with no `-k`. That is LE's new short-lived IP-certificate feature. Consequence: **nothing is attempted against `my.inference.coop` yet, so there is no ACME rate-limit exposure at all** — the "watch Let's Encrypt limits" caveat in earlier drafts of this section is moot. Over IPv6 the IP cert does not validate (the SAN covers the v4 address only), which is harmless: after cutover the hostname certificates cover both families. **There is no `cloudron` CLI on the server** — that is a separate npm tool for managing a remote Cloudron, not part of the platform. For server-side diagnostics use the documented tool, `sudo cloudron-support --troubleshoot`. The version needs no guessing: a fresh install pulls the current release, which is 10.1.5 or newer, and anything ≥ 10.1.5 restores this backup. **Ports on the new box: 25 inbound CONFIRMED OPEN** (operator, via the Infomaniak portal, 2026-11-01 — this is the receive path, and it is the half that cannot be replaced by a relay). Opened by the operator on 2026-11-01 (4190 is not offered by the provider — ManageSieve, not needed; leave it). **Do not read a timeout from the agent host as "blocked".** Checked 2026-11-01 on both address families: 465/587/993/995 all time out while 80/443 answer in ~2 ms, and the same probe reaches the OLD box on those ports instantly. But the new box has no mail app yet (the wizard is unsubmitted), so nothing is listening, and Cloudron's own firewall drops rather than refuses by default — a permitted-but-empty port and a blocked port are indistinguishable from outside. The 2026-10-30 IPv6 observation is the better evidence: those ports were REFUSED there, which means the packet arrived, i.e. permitted. Verify for real once the mail app exists. Outbound 25 is **not needed**: RFC-wise, a smarthost relay carries our sending, and Scaleway Transactional Email is being set up for exactly that. Do not spend a support ticket on outbound 25. (Previous note, kept for the date: "Ports still to open on the new box (checked 2026-10-30): 25, 465, 587, 993, 995, 4190.") **Provider tier — confirmed 2026-11-01: Infomaniak VPS Cloud** (not Lite; the two differ in ways that matter here). What the tier provides, per Infomaniak's own product pages: a **99.99% availability** commitment, **10 Gb/s** bandwidth, resizing **up and down**, and **one snapshot included** (extensible) — so the snapshot question is answered, but **none has been taken yet on the new box**: take one before submitting the Cloudron wizard, so there is a known-good point to return to. The Lite tier, by contrast, has no snapshot, no SLA, and closes outbound 25 outright; Cloud allows **port 25 "on justified request"**. That last point is the one to cite if the Infomaniak support thread asks us to justify it. 22, 80 and 443 answer today. Mail will not work on the new box until these are added, and outbound 25 is the separate Infomaniak ticket. **Leave for cutover day:** the domain, the plan/subscription (completing the wizard early can move the subscription off the live server — the one step worth reading twice), and the restore-from-backup itself. Looking at the setup page is harmless; submitting it is not. - [ ] Take an Infomaniak **snapshot of the VPS right after the install** (snapshots: 0 of 1 used), so a botched wizard on cutover day can roll back to a known-good installed platform. - [ ] Confirm the Cloudron subscription can be moved to a new server (do it at cutover, not early). - [ ] **IPv6: DECIDED — repoint.** The new box has `2001:1600:18:206::89`, so all 18 AAAA records move with the A records (§3 originally left this open). The zone carries **AAAA records on 18 hostnames** — everything that has an A record — all pointing to `2a02:4780:6e:83fd::1` (Hostinger's range). Counted 2026-10-30; the full list is in `dns-change-sheet.md`. This is 18 deletions or repoints, not a footnote. If the new box has IPv6, repoint them; if it does not, **delete the AAAA records** as part of the switch. Changing only the A records leaves every IPv6-capable client — including other Matrix homeservers and mail senders — still reaching the old box. - [ ] Set **rDNS/PTR → `my.inference.coop` on BOTH `179.237.86.116` and `2001:1600:18:206::89`**. Cloudron cannot do this — PTR belongs to the IP owner and exists only in the Infomaniak console. The console lists reverse DNS as unconfigured for both. Not time-sensitive — do it now. If IPv6 sending causes trouble later, disable v6 egress for SMTP rather than removing the AAAA. - [ ] **Open these in the INFOMANIAK console, not on the host** (measured 2026-10-30: 22 answers, 80/443 are silently dropped — a provider firewall default). Cloudron manages the host's own iptables but explicitly does not touch the provider firewall: its docs say "ensure the firewall does not block port 80 (HTTP) and 443 (HTTPS)". Ports: **22, 80, 443, 25, 465, 587, 993** (plus 995 and 4190 to match the old box; leave 8448, 110 and 143 closed). Verified 2026-10-30 against what the old box serves: 22/80/443/465/587/993/995/4190 answer; 8448, 110, 143 and 25 are dropped. The 25 result is not trustworthy — it came from a server whose network may block outbound 25 — and it must be open, because the platform has **inbound mail enabled**: `GET /api/v1/mail` → `{"domain":"inference.coop","inbound":true, "dkimSelector":"cloudron","relay":{"provider":"cloudron-smtp"}}`. The DKIM selector there matches the `cloudron._domainkey` TXT record, which is a useful cross-check. **How to verify rules before Cloudron is installed** — a blocked port *times out*, a permitted port with no listener yet is *refused*. Both are informative; a timeout is an absent rule. **State on 2026-10-30:** the rules are applied on **IPv6** (80/443/465/587/993/995 refused = permitted; 8448/110/143/4190 dropped) but **NOT on IPv4** — everything except 22 still times out. Infomaniak's console appears to keep separate IPv4 and IPv6 rule sets. This must be fixed on both: IPv4-only clients, and Let's Encrypt's HTTP-01 validation, are the ones that matter most, and 4190 (ManageSieve) stays shut unless members need mail-filter clients. **Port 25, settled from Cloudron's docs:** *inbound* 25 is **required to receive email** and must be opened on both addresses. *Outbound* 25 is **not required if a relay/smart host is configured** — Cloudron's own guidance is "request your provider to unblock this port **or set up a smart host**", and this box already has `relay: {"provider": "cloudron-smtp"}`. Confirm on the dashboard's Email page that the relay is a real remote smarthost (host + port 465/587); if it is, Infomaniak's outbound-25 restriction is irrelevant and worth no support ticket. (mail delivery — test from outside, e.g. `telnet my.inference.coop 25` once DNS moves; note that many networks block *outbound* 25, so a closed result from inside another server is not conclusive). **8448 is intentionally closed** — federation is delegated to 443 via the apex `.well-known` (verified live: `inference.coop/.well-known/matrix/server` → `{"m.server":"matrix.inference.coop:443"}`). Do not "fix" 8448. Do not add ufw rules that fight Docker; the old box runs with ufw inactive and Cloudron-managed iptables. - [ ] **DMARC is `p=reject; pct=100`.** Strict. After the restore, any DKIM/SPF misalignment means outbound mail is *discarded by recipients*, not junked. The DKIM TXT record (`cloudron._domainkey`) must match the key inside the restored mail app's data — verify both rather than assuming. - [ ] **Mount audit on the Cloudron host** — the one class of data the backup cannot see: sudo docker ps -q | xargs -I{} sudo docker inspect {} \ --format '{{.Name}} {{range .Mounts}}{{.Source}}=>{{.Destination}} {{end}}' \ | tr ' ' '\n' | grep -v /app/data **RESOLVED 2026-10-30:** the host mount is Nextcloud's `/var/lib/nextcloud`, **3 MB**, holding only `preview/` (regenerable thumbnails) — not a data-loss risk. Re-run this audit only if apps change before the cutover; it is now a confirm, not an unknown. - [ ] Leave the **single snapshot slot free** for a post-restore, pre-DNS-switch snapshot — a bad restore then becomes a revert instead of a rebuild. - [ ] Record the highest backup run number before the freeze. (As of 2026-10-30 the newest run is healthy at ~4.6 GB and the backup space was enlarged, so the quota pressure that produced the `EntityTooLarge` failures is relieved — but run growth is still the number to watch: 2.6 → 4.6 GB per run over two weeks.) - [ ] Lower all A **and AAAA** record TTLs to 300 s **a day early**. The zone TTL is **3600 s** — an earlier note read 760–1642 s, which was a decrementing *cache countdown*, not the configured TTL. At 3600 s, propagation after the switch takes up to an hour, which is exactly the window where the old box must not be accepting writes (§4). - [ ] List external allowlists that carry our IP: Regolo, GreenPT, Tinfoil, any mail relay, and inbound webhooks. Update after cutover. - [ ] SSH host keys change with the box. Accept the new fingerprint deliberately, in advance, so a strict-host-key refusal doesn't stop you mid-migration. - [ ] Announce the window in `#updates`. ## 3a. The "before" snapshot `migration-before-state.json` in this repo is a machine-diffable record of the current server: 16 apps with their manifest ids, image tags, state, health and **env var names** (never values); the backup sites, schedule and retention; the newest run's id, state, integrity signature and size; Cloudron's version; and data fingerprints (member count, API-key count, ledger rows, `metrics_daily` rows, Gitea repo count). Capture nothing further; after the restore, diff it. A silently-empty app is exactly what a diff catches and a spot-check misses. ## Credentials needed on the day Status as of 2026-10-30: - [x] **Cloudron backup passphrase** — off-box (confirmed 2026-10-30), in the operator's KeePass vault. A restore asks for this first; without it the backup is unreadable. This was the real blocker. - [x] **Vaultwarden master password + admin credentials** — off-box in the same KeePass vault (2026-10-30). The Vault's own database is inside the Cloudron backup, so a restore returns it whole; the master password is what makes the restored contents readable. Note the shape of this: `vault.inference.coop` runs `com.github.bitwardenrs` — **Vaultwarden**, the open-source Bitwarden server — so official Bitwarden clients already work against it. There is nothing to sync. - [x] **S3 `secretAccessKey`** — **confirmed 2026-11-01: held externally by the operator along with the rest of the Infomaniak backup keys.** (A restore is unreadable without it, which is why it was flagged.) This is the one credential absent from the co-op's own off-box JSON (that carries `accessKeyId` only), and a restore begins by pointing the new Cloudron at the backup target — so it must be typeable before the window. - [ ] **Vault export** (optional insurance) — the contents already ride in the Cloudron backup, so this is not the source of truth. Cheap protection against the app restoring but its DB coming back damaged. - [x] **restic passphrase** (agent host, different backup) — in that host's `.env`, and that host is not the one migrating, so it stays recoverable either way. **Continuity, not just safety.** All of the above live in one person's KeePass file. For a co-operative that is a governance risk as much as an ops one: if the one custodian is unavailable, so is the infrastructure. The cheap fixes are a second custodian (or a sealed shared copy), and a line in this repo recording *where* the break-glass copies are held — never the secrets themselves. ## The announcement, ready to post in #updates > **Scheduled maintenance: , , – UTC.** We are moving our servers to a new host > in Switzerland. Chat, the portal, the API and the website will be unavailable during the window. No > data is being changed or removed — this is a move, not a rebuild. If anything still looks wrong > afterwards, reply here and we will sort it out. ## The all-clear email — drafted 2026-11-02, ready to paste into Listmonk Sent from the usual identity (`Inference Cooperative `). Send after DNS propagation completes (~08:00 UTC on switch day) so links in the email resolve for everyone. > Hi all, > > Last night we promised to write as soon as the move was complete and everything checked out. This > is that email. > > The co-op's services now run entirely on our own infrastructure, hosted by Infomaniak in > Switzerland — the renewable energy, Swiss privacy, worker-owned, foundation-governed setup we > described when we warned you about downtime. Chat, the member portal, the dashboard, code hosting, > the vault, cloud storage and this newsletter all came back whole from the final backup, and we > verified them one by one before switching over the domain. > > We're sorry for the outage. A server move is a rare event for a small co-op — we don't expect > another one anytime soon — and the payoff is infrastructure that belongs to us, on our terms. > > If anything looks off in the next few days, reply here and we'll sort it out. > > We hope you enjoyed the AI-free Sunday night. Everything is back where you left it. > > — Inference Cooperative ## 4. Cutover, in order 1. **Freeze writes.** Pause the portal's `membership_sweep` and `rag_refresh`, hold Listmonk campaigns. 2. **Final backup on the old box.** Run "Backup now" and confirm state `normal`. Everything after this timestamp is *lost* in a forward migration. 3. **Stop the apps on the old box** — all of them, including mail. Keep the *machine* powered on for rollback, but nothing may accept writes from here on: during the up-to-one-hour DNS propagation, a still-serving old box silently collects Matrix messages, Gitea pushes, API spend and mail into a database that gets thrown away. Clients and mail senders retry; that is the correct failure mode. 4. **Stop the old box's backup schedule** (Settings → Backups → disable). Two Cloudrons writing one target interleave and produce `EntityTooLarge` quota failures. 5. **Build the new box and restore** from the Infomaniak S3 target, using the passphrase from §3. Expect TLS to fail until DNS moves — reach the dashboard by IP or a temporary override. 6. **Switch the A records — and the AAAA records** (repoint or delete, per the §3 decision). 7. Wait out the TTL (up to an hour at 3600 s if not lowered), then confirm certificates issued. 8. **Re-enable backups on the new box — once daily at 23:00, retention 7 days.** (This is now applied on the old box — schedule `00 00 23 * * *`, verified — re-check it on the new box after restore rather than assuming.) 9. Update external allowlists. The old box stays powered on with apps stopped, for rollback only. ## 5. Post-restore verification Run `python3 ~/.hermes/scripts/verify-restore.py` from the **agent** host — it stays up through the migration, so it is the one machine that can judge the new box. It diffs the restored server against `migration-before-state.json`: app inventory, **env var NAMES** (a lost var comes back as an app that will not start), image tags, backup target and schedule, portal data fingerprints (row counts must never drop), DNS, federation, mail ports, and whether each site answers. Exit 1 = at least one regression. `--quiet` prints failures and warnings only; `--only ` runs one group. Run it **before** cutover too: DNS and port-25 failures are expected then, and they are exactly what cutover is meant to clear. The 2026-10-30 dry run gave 50 pass / 1 warn / 13 fail, all 13 explained by pre-cutover DNS plus port 25 — the point is that no *unexplained* failure appears. - [ ] Dashboard loads with a valid certificate; all 16 apps `running`. - [ ] **One custom app loads** — e.g. `panel.inference.coop`. This is the proof the registry restored; without it nothing custom starts. - [ ] **The apex `.well-known` answers** — `curl https://inference.coop/.well-known/matrix/server` must return `{"m.server":"matrix.inference.coop:443"}`. It is served by the *website* app, so if that app's data is missing, federation breaks while `matrix.inference.coop` itself still looks healthy. Then: `curl -s https://matrix.inference.coop/_matrix/federation/v1/version`. - [ ] Data spot-checks: Matrix login + a room loads; Gitea repo list; portal `/admin/overview` returns the expected member count; gateway `/v1/models`. - [ ] Mail: MX resolves to the new IP; SPF still `a:my.inference.coop`; the DKIM TXT (`cloudron._domainkey`) matches the restored key; PTR correct; **send to an external mailbox and confirm it actually arrived** (DMARC is `p=reject` — a silent discard is the failure mode, not a junk folder); receive from outside; **test port 25 inbound from a third network**. - [ ] Agent API access: `cloudron list` from the agent host. The API token lives in Cloudron's config so it should survive; if it does not, mint a new one and update the agent's `.env`. - [ ] Schedules fire **once**, not twice: portal sweeps, Listmonk, agent digests. - [ ] First backup on the new box is `normal` and in the expected size range. ## 6. Rollback - Keep the old box **powered on with apps stopped** for several days. Rollback = point DNS back and start the apps. - Do not delete old snapshots from S3 until the new box has at least one verified backup run. - Know the asymmetry: anything written between the final backup and the DNS switch exists **only on the old box**. Rolling forward discards it; rolling back keeps it. §4 step 3 (stop apps) is what keeps that window empty rather than merely short. ## 7. Open unknowns — resolve before cutover - **Are MAILBOX CONTENTS inside the Cloudron backup?** Mail is ON for `inference.coop` with inbound enabled, and the mail server is part of the *platform*, not the Mail app's container (that app listens only on 3000 — it is a UI, not an MTA). Platform config is backed up, so the DKIM key and mail settings survive; whether members' mailboxes do is **unverified** and would be silent, permanent data loss if they do not. Confirm against Cloudron's docs or the dashboard's backup contents before the freeze. This is now the most consequential open question in this runbook. - ~~Whether the Vault export (§3 blocker) has actually been done.~~ **Closed 2026-11-01:** the operator confirms access to all relevant passwords outside the server. The full Vault export remains optional — the vault's own database is inside the Cloudron backup, so a restore returns it. - The Cloudron host's OS version, disk and RAM (not visible over the API). - Whether any provider API key is bound to our current IP. - Whether the subscription permits a server move. ## 8. Pitfalls this server has already produced - **Quota failures:** 4.82 GB per run twice daily hit the Infomaniak ceiling (`Upload exceeds quota`). The space has since been enlarged and the schedule is now once daily — both verified — but per-run size grows (2.6 → 4.6 GB in two weeks), mostly registry image blobs. If the ceiling ever returns, the durable fix is getting the rebuildable images out of every snapshot, not deleting backups. - **The once-a-day change cannot be made over the API** — dashboard only. Historically it was reported as done and was not. Verified applied 2026-10-30. - **`cloudron update --image` reads the manifest from the current directory.** Running it from the wrong repo fails with `manifest id does not match`, which looks like a server problem and is not. - **The agent host is not migrated** but will see errors while DNS moves. It needs no config change: every reference is a hostname, including `--server my.inference.coop`. Its 30-minute webhook watch will fire during the window and may raise false alarms; the daily digest now compares against its previous run before flagging, so one-off transients should not reach you as decisions. - **A DNS cache countdown is not a zone TTL.** Reading 760–1642 s off a resolver answer and writing it into this runbook as "the TTL" was wrong — the configured TTL was 3600 s all along. Query the zone at the registrar (or trust a fresh authoritative answer) for planning numbers. --- # STATE OF PLAY — 2026-11-02 (read this first) Written for whoever picks this up next, including a fresh agent session with none of the history. ## Mail: receiving on the box, sending through a relay - **Inbound port 25 is OPEN** (operator confirmed in the Infomaniak portal, 2026-11-01). 465/587/993/995 were opened too; 4190 is not offered and is not needed. **Outbound 25 is not required** — do not spend a support ticket on it. - **Scaleway Transactional Email is proven working.** A real message relayed through it produced: `spf=pass`, `dkim=pass … header.s=ede94f37-204c-41ee-b388-e940520316d1` (the UUID selector in the DNS sheet — 2048-bit), `dmarc=pass policy.published-domain-policy=reject`, and `iprev=pass (phan-van4.scw-tem.cloud)`. SPF carries `include:_spf.tem.scaleway.com`. - **BUT the newsletter does NOT use it yet.** Listmonk is still configured `host=mail, port=2465` — the Cloudron mail app. It is in "copy settings from Cloudron" mode, so its own SMTP settings are overwritten and the host keeps reverting (its `msg_retry_delay` does persist, which is a useful tell). **Do not assume a newsletter sent through Scaleway; check the host before relying on it.** - The **portal** also still sends through the Cloudron mail addon (`CLOUDRON_MAIL_*`). Moving it to the relay needs its own SMTP env vars plus a code change, deliberately not done yet. - **Consequence for the window:** the newsletter and the portal can only send **before** the box goes down or **after** it comes back. Not during. ## DNS: 11 records, two of them wildcards See `dns-change-sheet.md`, rebuilt 2026-11-01 from the registrar's panel. The important shape: - Only **11 records exist**. `*.inference.coop` A and AAAA carry **every application hostname** — chat, portal, panel, dashboard, newsletter, gateway, vault, git, registry, files, forum, email, docs, my, status. They are NOT individual records. **Move the wildcards or every app stays behind.** - `server.inference.coop` is an A record nothing documents. Identify before moving. - TTL is 3600 everywhere; lower it before the window. - **Never read a timeout from the agent host as "blocked"** (see the port-25 lesson in §7). The agent host cannot observe outbound 25 at all. ## Credentials: all off-box, all confirmed Cloudron backup passphrase, Infomaniak S3 `secretAccessKey`, and the Vault passwords are held externally by the operator (confirmed 2026-11-01). A restore is unreadable without the first two. ## Not done yet - Cloudron **setup wizard on the new box** — still unsubmitted. - **No snapshot** on the new box (the tier includes one — free rollback point). - **No restore rehearsal** has been attempted. - `verify-restore.py` is fixed (both blind spots) and the baseline is fresh as of `267db1f`, committed to this repo. Run it after any restore; its two remaining FAILs pre-cutover are the expected "not migrated yet" signature. ## CAUTION: two Matrix identities The backup contains Synapse **and its signing key**. Restore it and the new box becomes a second homeserver claiming the same server name as the old one, signing with the same key. Federation peers see one identity behaving as two. **Keep the restored apps stopped except while actively testing.** ## The agent running this The steward bot runs on a **third machine** (`179.237.103.165`) — neither the old nor the new Cloudron box — so it stays up throughout. Its **primary model route goes through the gateway on the old box** (`gateway.inference.coop` → `179.199.135.72`) and will fail during the move; its **fallback is GreenPT direct** (`api.greenpt.ai`, `glm-5.3-flash`, `GREENPT_FALLBACK_API_KEY`), verified working 2026-11-02. Tooling that talks to `git.`, `gateway.` or `portal.inference.coop` will be unavailable mid-move. --- # RESTORE COMPLETE — 2026-11-02 ~06:30 UTC All 16 apps restored on the new box (179.237.86.116) via per-app restore from the final backup `box_v10.1.6_b4fd6c91` (03:57 UTC). Data verified per app, not per state flag: - matrix 35 rooms / 28 users, original signing key `ed25519:a_jSIu` — identity preserved - portal members 91 / api_keys 74 / ledger 24; newsletter 91 subscribers, 8 campaigns - forum 89 users / 18 discussions / 348 comments; gitea `code` org 5/5 public repos - dashboard carries the post-baseline deploy (image 20261102-b352b1f, CREDIT_PACK_URL) — the baseline is stale here, not the restore What the restore survived, in order: root disk too small (data moved to /mnt/data), docker store wiped to fix overlay2 (images re-pulled, `cloudron` network recreated, firewall refreshed by reboot), `cloudron-syslog` restart after the platformdata move, and `127.0.0.1 registry.inference.coop` in the new box's /etc/hosts so custom images don't reach the dead box. **Two lessons for next time, now in skill `cloudron-ops`:** (1) check which filesystem Cloudron actually uses, not the advertised disk size — nothing in the installer flags an unused 245 GB volume; (2) `repair` recreates containers but never restores data — the data path is per-app restore from the box backup entry. Never batch-parallel restores. **Still open at time of writing:** DNS on the old box (apex, wildcard A/AAAA, `www` → 179.237.86.116 / 2001:1600:18:206::89; `server.inference.coop` untouched); TTLs still 3600; agent host has a temporary `/etc/hosts` override `my.inference.coop → 179.237.86.116` that MUST be removed after the DNS switch or verify-restore.py reports fiction; old box powered on, apps stopped, backup schedule disabled; first new-box backup expected 23:00 UTC — verify it is `normal` and expected size. --- # CUTOVER COMPLETE — 2026-11-02 07:45 UTC DNS switched ~06:30 UTC (apex + wildcard A/AAAA + www → 179.237.86.116 / 2001:1600:18:206::89; all TXT/MX untouched; no AAAA added for www — it has none). `server.inference.coop` (the one unidentified record) was also repointed to the new box — answered authoritatively 07:53 UTC — so the old box has **no remaining DNS references**. Agent-host override removed 06:44. Final verify-restore.py: **66 pass / 7 warn / 1 FAIL** — the single FAIL is the dashboard's stale baseline (post-baseline deploy, expected). Key sync (chat↔portal) healed 07:12 after the new box's resolver cache expired; git push done via `git -c http.curloptResolve=git.inference.coop:443:179.237.86.116 push` (bypasses stale resolver — keep this trick). Baseline recaptured against the new box at 07:53 UTC (original in git history); verify-restore.py then reports **67 pass / 7 warn / 0 FAIL**. **Mail required one more step:** after the docker-store wipe, the platform mail container is NOT recreated at boot, and every mail API route (and even the mail-domain list) dies on "Unable to find container mail" — the container lookup happens before any reconfigure. Fix: `cloudron-support --check-services` then `--recreate-containers` (sanctioned, no data loss); mail ports 465/587/993/995 reopened immediately; port 25 is untestable from the agent host (outbound 25 blocked — the runbook's known blind spot). Member mailbox CONTENTS still need a real IMAP login to confirm (the runbook's long-standing unknown). All-clear email drafted as Listmonk campaign 18 ("All-clear after the move (draft)", Members list, status draft). chat-legacy (Open WebUI): member access removed 2026-11-02 — retirement begun; app restored and safe in backups. **Outstanding:** tonight 23:00 UTC first backup (verify `normal`, ~4.9 GiB); Cloudron subscription page check on the new dashboard; my primary model route re-heals on its own (fallback worked all night); old box keep powered-on/apps-stopped/backups-disabled for a few days, then decommission.