Deploy pipeline can pin and deploy a failed build (gateway outage 2026-11-17) #52

Open
opened 2026-11-17 09:23:31 +00:00 by inference-bot · 0 comments
Owner

What happened. An update attempt pinned a LiteLLM image whose build had failed, Cloudron could not pull it (404 manifest unknown), and the gateway app was left stopped after the failed switch. It was restored by pinning the last good image (20261116-7a167a8) and updating again.

Why the pipeline allowed it. CloudronManifest.json is pinned before the build result is checked: the builder prints BUILT_TAG=... only on success, but nothing in the procedure refuses to pin/deploy when that line is absent.

Immediate mitigation (applied). My local procedure now gates on BUILT_TAG= and refuses to pin or deploy when the build fails.

Still to fix. ~/.hermes/scripts/cloudron_builder_build.sh should exit non-zero on a failed build and never print a tag, so a caller cannot accidentally pin an image that does not exist. A manifest pin to a missing tag is indistinguishable from a good deploy until Cloudron tries to pull it.

Related learning. The same period produced #5's real root cause (a swallowed ValueError in the audio unit pricer); that is closed with a verified $0.002 spend row. This issue is only about the deploy path.

**What happened.** An update attempt pinned a LiteLLM image whose build had *failed*, Cloudron could not pull it (`404 manifest unknown`), and the gateway app was left stopped after the failed switch. It was restored by pinning the last good image (20261116-7a167a8) and updating again. **Why the pipeline allowed it.** `CloudronManifest.json` is pinned before the build result is checked: the builder prints `BUILT_TAG=...` only on success, but nothing in the procedure refuses to pin/deploy when that line is absent. **Immediate mitigation (applied).** My local procedure now gates on `BUILT_TAG=` and refuses to pin or deploy when the build fails. **Still to fix.** `~/.hermes/scripts/cloudron_builder_build.sh` should exit non-zero on a failed build and never print a tag, so a caller cannot accidentally pin an image that does not exist. A manifest pin to a missing tag is indistinguishable from a good deploy until Cloudron tries to pull it. **Related learning.** The same period produced #5's real root cause (a swallowed ValueError in the audio unit pricer); that is closed with a verified $0.002 spend row. This issue is only about the deploy path.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: co-op/support#52