What happened. An update attempt pinned a LiteLLM image whose build had failed, Cloudron could not pull it (404 manifest unknown), and the gateway app was left stopped after the failed switch. It was restored by pinning the last good image (20261116-7a167a8) and updating again.
Why the pipeline allowed it.CloudronManifest.json is pinned before the build result is checked: the builder prints BUILT_TAG=... only on success, but nothing in the procedure refuses to pin/deploy when that line is absent.
Immediate mitigation (applied). My local procedure now gates on BUILT_TAG= and refuses to pin or deploy when the build fails.
Still to fix.~/.hermes/scripts/cloudron_builder_build.sh should exit non-zero on a failed build and never print a tag, so a caller cannot accidentally pin an image that does not exist. A manifest pin to a missing tag is indistinguishable from a good deploy until Cloudron tries to pull it.
Related learning. The same period produced #5's real root cause (a swallowed ValueError in the audio unit pricer); that is closed with a verified $0.002 spend row. This issue is only about the deploy path.
**What happened.** An update attempt pinned a LiteLLM image whose build had *failed*, Cloudron could not pull it (`404 manifest unknown`), and the gateway app was left stopped after the failed switch. It was restored by pinning the last good image (20261116-7a167a8) and updating again.
**Why the pipeline allowed it.** `CloudronManifest.json` is pinned before the build result is checked: the builder prints `BUILT_TAG=...` only on success, but nothing in the procedure refuses to pin/deploy when that line is absent.
**Immediate mitigation (applied).** My local procedure now gates on `BUILT_TAG=` and refuses to pin or deploy when the build fails.
**Still to fix.** `~/.hermes/scripts/cloudron_builder_build.sh` should exit non-zero on a failed build and never print a tag, so a caller cannot accidentally pin an image that does not exist. A manifest pin to a missing tag is indistinguishable from a good deploy until Cloudron tries to pull it.
**Related learning.** The same period produced #5's real root cause (a swallowed ValueError in the audio unit pricer); that is closed with a verified $0.002 spend row. This issue is only about the deploy path.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
What happened. An update attempt pinned a LiteLLM image whose build had failed, Cloudron could not pull it (
404 manifest unknown), and the gateway app was left stopped after the failed switch. It was restored by pinning the last good image (20261116-7a167a8) and updating again.Why the pipeline allowed it.
CloudronManifest.jsonis pinned before the build result is checked: the builder printsBUILT_TAG=...only on success, but nothing in the procedure refuses to pin/deploy when that line is absent.Immediate mitigation (applied). My local procedure now gates on
BUILT_TAG=and refuses to pin or deploy when the build fails.Still to fix.
~/.hermes/scripts/cloudron_builder_build.shshould exit non-zero on a failed build and never print a tag, so a caller cannot accidentally pin an image that does not exist. A manifest pin to a missing tag is indistinguishable from a good deploy until Cloudron tries to pull it.Related learning. The same period produced #5's real root cause (a swallowed ValueError in the audio unit pricer); that is closed with a verified $0.002 spend row. This issue is only about the deploy path.