incidents: add the 2026-10-02 gateway outage write-up (from the unmerged incident branch)
The write-up has sat on incident/2026-10-02-gateway-cve-upgrade since the outage and that branch is now far behind main. This lifts just the incident text and the CVE note into incidents/, attributed to InferenceBot with the substance unchanged, and links it from infrastructure.md under a new Incident log section.
This commit is contained in:
1 parent
87e0a085f4
commit
cdb705d89e
2 files changed
+39
No files matched your search
@@ -0,0 +1,34 @@
|
|||||||
|
# 2026-10-02 — gateway outage after a security-upgrade rebuild
|
||||||
|
|
||||||
|
*Written by InferenceBot on the `incident/2026-10-02-gateway-cve-upgrade`
|
||||||
|
branch on 2026-10-01; moved to `main` with the substance unchanged so it's
|
||||||
|
findable.*
|
||||||
|
|
||||||
|
**Impact.** The gateway (chat and API) was down about 50 minutes,
|
||||||
|
04:50–05:34 UTC. No data loss.
|
||||||
|
|
||||||
|
**What happened.** A Cloudron update triggered by a member's CVE report
|
||||||
|
rebuilt the app image. The Dockerfile pinned the floating
|
||||||
|
`main-v1.74.0-stable` tag, which the registry had silently moved to newer
|
||||||
|
(v1.84.0-era) bits, so the rebuild changed the running version and kicked
|
||||||
|
off a full Prisma database migration that takes 10–15 minutes per boot with
|
||||||
|
internal timeouts and retries. The migration ran at Cloudron's default
|
||||||
|
256 MB memory cap and was OOM-killed; the app crash-looped on a missing
|
||||||
|
migrations ledger until the memory limit and `LITELLM_MIGRATION_DIR` were
|
||||||
|
fixed.
|
||||||
|
|
||||||
|
**Also during the incident.** The domain's authoritative nameservers
|
||||||
|
(yoursrs.com) were intermittently unreachable, briefly failing resolution of
|
||||||
|
some hostnames (git, portal). Transient, registrar-side, self-recovered; no
|
||||||
|
records were wrong. The upgrade had also silently reverted CORS to `*`.
|
||||||
|
|
||||||
|
**Repairs.** Version pinned to the exact tag in `code/litellm`;
|
||||||
|
`memoryLimit` set to 2 GB in the manifest; `LITELLM_MIGRATION_DIR` pointed
|
||||||
|
at the persistent volume; `LITELLM_CORS_ORIGINS` set to the chat origin;
|
||||||
|
packaging synced to mirror the live deployment. The upgrade rules learned
|
||||||
|
here are in the `code/litellm` README under "Updating LiteLLM".
|
||||||
|
|
||||||
|
**What it fixed on the security side.** CVE-2026-35029 (authorization bypass
|
||||||
|
on `/config/update`, fixed in 1.83.0) and CVE-2026-59822 (MCP session auth
|
||||||
|
bypass, fixed in 1.84.0, listed in CISA's Known Exploited Vulnerabilities
|
||||||
|
catalog). The gateway has since moved to v1.103.2.
|
||||||
@@ -204,3 +204,8 @@ through it; the other providers are called directly.
|
|||||||
Maintaining the co-op's infrastructure — rotating Cloudron tokens, editing the
|
Maintaining the co-op's infrastructure — rotating Cloudron tokens, editing the
|
||||||
portal's environment, and troubleshooting the provisioning pipeline — is
|
portal's environment, and troubleshooting the provisioning pipeline — is
|
||||||
documented in the [operator runbook](operator-token-runbook.md).
|
documented in the [operator runbook](operator-token-runbook.md).
|
||||||
|
|
||||||
|
## Incident log
|
||||||
|
|
||||||
|
Write-ups for outages live in [`incidents/`](incidents/). One so far:
|
||||||
|
[2026-10-02 — gateway outage after a security-upgrade rebuild](incidents/2026-10-02-gateway-cve-upgrade.md).
|
||||||
Reference in new issue
Block a user