publishing: one canonical build venv; drift check; deploy-failure alerting
The poller's private venv (.docs-publisher/.venv-docs) diverged from the canonical one twice, each time silently delaying a deploy. Consolidate: - poll-watcher.sh builds with the ONE canonical venv (~/repos/docs/.venv-docs) while keeping its dedicated checkout (uncommitted edits in a working clone must never block or leak into a deploy). - New drift check: publishing/ + mkdocs.yml are compared with the canonical checkout before every build; a mismatch fails the deploy loudly instead of shipping logic the canonical source cannot reproduce. - Pre-deploy gate: build.sh --check-only verifies the pinned toolchain and every configured plugin (self-heals missing pins, fails loudly naming an unpinned plugin). - Alerting: 3 consecutive deploy failures -> marker file + ONE Matrix message to the InferenceBot HQ room (hourly cap); counter resets on next success. - The watcher re-execs itself when a deploy changed poll-watcher.sh, so the running loop always matches the deployed one. - webhook receiver (dormant) repointed at the canonical venv too; unit files and README document the layout.
This commit is contained in:
1 parent
1691f79621
commit
20d384be87
5 files changed
+222
-21
No files matched your search
@@ -105,3 +105,45 @@ app; the agent-host mechanisms were chosen because they add no new Cloudron app.
|
||||
Each deploy writes `meta.json` into the site with the commit it was built
|
||||
from, so a deploy can be verified from the served site itself:
|
||||
https://docs.inference.coop/meta.json
|
||||
|
||||
## One build environment (2026-11-12)
|
||||
|
||||
The deploy used to have TWO build environments — this repo's `.venv-docs` and
|
||||
a second private venv in the poller's checkout at
|
||||
`/home/ubuntu/.docs-publisher/.venv-docs`. They diverged twice, each time
|
||||
silently delaying a deploy (the brand.css cache-busting confusion, then a
|
||||
missing `mkdocs-redirects` that failed every build for ~15 minutes while the
|
||||
poller retried without telling anyone). Consolidated:
|
||||
|
||||
- **ONE venv.** Every trigger — the poller, the (dormant) webhook receiver, a
|
||||
manual `publishing/build.sh`, CI — builds with the canonical checkout's
|
||||
venv: `~/repos/docs/.venv-docs`. The poller's private venv was deleted.
|
||||
- **The dedicated checkout stays, by design.** The poller still builds from
|
||||
`/home/ubuntu/.docs-publisher`, its own clean clone: uncommitted edits in a
|
||||
working clone must never block a deploy or leak into the published site.
|
||||
- **Drift check.** Before every build the poller compares `publishing/` and
|
||||
`mkdocs.yml` between its checkout and the canonical one; any mismatch fails
|
||||
the deploy loudly (with a diff and the fix — `git -C ~/repos/docs pull
|
||||
--ff-only`), and the failure counts toward the alert below. A canonical
|
||||
checkout that lags behind main trips this too, on purpose: a deploy of
|
||||
logic the canonical source cannot reproduce is the failure mode this exists
|
||||
to prevent. A missing canonical checkout is fatal as well.
|
||||
- **Plugin gate + self-heal.** `publishing/requirements.txt` pins every
|
||||
package the build needs (`mkdocs`, `mkdocs-material`, `mkdocs-redirects`).
|
||||
`build.sh` verifies the venv against it before every build and installs
|
||||
anything missing or off-version (fast local no-op when current); it also
|
||||
fails loudly if `mkdocs.yml` configures a plugin that `requirements.txt`
|
||||
does not pin. The poller runs this as its pre-deploy check via
|
||||
`build.sh --check-only`.
|
||||
- **Failure alerting.** After 3 consecutive deploy failures the poller writes
|
||||
a marker (`~/.docs-deploy.alert`) and sends ONE message to the InferenceBot
|
||||
HQ Matrix room naming the failing commit — never more than one message per
|
||||
hour (`~/.docs-deploy.last-alert`). The counter (`~/.docs-deploy.failcount`)
|
||||
resets on the next successful deploy. A deploy that succeeds silently
|
||||
resets everything; nothing announces routine successes.
|
||||
|
||||
Files: `~/.docs-deploy.state` (last deployed commit), `~/.docs-deploy.lock`
|
||||
(deploy lock), `~/.docs-deploy.failcount`, `~/.docs-deploy.alert`,
|
||||
`~/.docs-deploy.last-alert`. Manual deploys should respect the lock:
|
||||
`flock -n ~/.docs-deploy.lock bash publishing/build.sh && flock -n
|
||||
~/.docs-deploy.lock bash publishing/deploy.sh`.
|
||||
Reference in new issue
Block a user