From 317af556210af8c3a1d3911888f4338e6472b5df Mon Sep 17 00:00:00 2001 From: Angel Marino Date: Mon, 26 Oct 2026 02:11:33 -0400 Subject: [PATCH 1/2] incidents: add the 2026-10-02 gateway outage write-up (from the unmerged incident branch) The write-up has sat on incident/2026-10-02-gateway-cve-upgrade since the outage and that branch is now far behind main. This lifts just the incident text and the CVE note into incidents/, attributed to InferenceBot with the substance unchanged, and links it from infrastructure.md under a new Incident log section. --- incidents/2026-10-02-gateway-cve-upgrade.md | 34 +++++++++++++++++++++ 1 file changed, 34 insertions(+) create mode 100644 incidents/2026-10-02-gateway-cve-upgrade.md diff --git a/incidents/2026-10-02-gateway-cve-upgrade.md b/incidents/2026-10-02-gateway-cve-upgrade.md new file mode 100644 index 0000000..77aa720 --- /dev/null +++ b/incidents/2026-10-02-gateway-cve-upgrade.md @@ -0,0 +1,34 @@ +# 2026-10-02 — gateway outage after a security-upgrade rebuild + +*Written by InferenceBot on the `incident/2026-10-02-gateway-cve-upgrade` +branch on 2026-10-01; moved to `main` with the substance unchanged so it's +findable.* + +**Impact.** The gateway (chat and API) was down about 50 minutes, +04:50–05:34 UTC. No data loss. + +**What happened.** A Cloudron update triggered by a member's CVE report +rebuilt the app image. The Dockerfile pinned the floating +`main-v1.74.0-stable` tag, which the registry had silently moved to newer +(v1.84.0-era) bits, so the rebuild changed the running version and kicked +off a full Prisma database migration that takes 10–15 minutes per boot with +internal timeouts and retries. The migration ran at Cloudron's default +256 MB memory cap and was OOM-killed; the app crash-looped on a missing +migrations ledger until the memory limit and `LITELLM_MIGRATION_DIR` were +fixed. + +**Also during the incident.** The domain's authoritative nameservers +(yoursrs.com) were intermittently unreachable, briefly failing resolution of +some hostnames (git, portal). Transient, registrar-side, self-recovered; no +records were wrong. The upgrade had also silently reverted CORS to `*`. + +**Repairs.** Version pinned to the exact tag in `code/litellm`; +`memoryLimit` set to 2 GB in the manifest; `LITELLM_MIGRATION_DIR` pointed +at the persistent volume; `LITELLM_CORS_ORIGINS` set to the chat origin; +packaging synced to mirror the live deployment. The upgrade rules learned +here are in the `code/litellm` README under "Updating LiteLLM". + +**What it fixed on the security side.** CVE-2026-35029 (authorization bypass +on `/config/update`, fixed in 1.83.0) and CVE-2026-59822 (MCP session auth +bypass, fixed in 1.84.0, listed in CISA's Known Exploited Vulnerabilities +catalog). The gateway has since moved to v1.103.2. From 891490d0d7948d13686b85c06aaa7eda36eef3d1 Mon Sep 17 00:00:00 2001 From: Angel Marino Date: Tue, 27 Oct 2026 01:34:15 -0400 Subject: [PATCH 2/2] incidents: fix the 2026-10-02 byline date and the outage duration (review) Co-Authored-By: Claude Fable 5.1 --- incidents/2026-10-02-gateway-cve-upgrade.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/incidents/2026-10-02-gateway-cve-upgrade.md b/incidents/2026-10-02-gateway-cve-upgrade.md index 77aa720..ed0fc9e 100644 --- a/incidents/2026-10-02-gateway-cve-upgrade.md +++ b/incidents/2026-10-02-gateway-cve-upgrade.md @@ -1,10 +1,10 @@ # 2026-10-02 — gateway outage after a security-upgrade rebuild *Written by InferenceBot on the `incident/2026-10-02-gateway-cve-upgrade` -branch on 2026-10-01; moved to `main` with the substance unchanged so it's +branch on 2026-10-02 (05:35 UTC, a minute after service returned; the commit's local timestamp reads 2026-10-01 23:35 −06:00); moved to `main` with the substance unchanged so it's findable.* -**Impact.** The gateway (chat and API) was down about 50 minutes, +**Impact.** The gateway (chat and API) was down about 45 minutes, 04:50–05:34 UTC. No data loss. **What happened.** A Cloudron update triggered by a member's CVE report