publishing: one canonical build venv; drift check; deploy-failure alerting

The poller's private venv (.docs-publisher/.venv-docs) diverged from the
canonical one twice, each time silently delaying a deploy. Consolidate:

- poll-watcher.sh builds with the ONE canonical venv
  (~/repos/docs/.venv-docs) while keeping its dedicated checkout (uncommitted
  edits in a working clone must never block or leak into a deploy).
- New drift check: publishing/ + mkdocs.yml are compared with the canonical
  checkout before every build; a mismatch fails the deploy loudly instead of
  shipping logic the canonical source cannot reproduce.
- Pre-deploy gate: build.sh --check-only verifies the pinned toolchain and
  every configured plugin (self-heals missing pins, fails loudly naming an
  unpinned plugin).
- Alerting: 3 consecutive deploy failures -> marker file + ONE Matrix message
  to the InferenceBot HQ room (hourly cap); counter resets on next success.
- The watcher re-execs itself when a deploy changed poll-watcher.sh, so the
  running loop always matches the deployed one.
- webhook receiver (dormant) repointed at the canonical venv too; unit files
  and README document the layout.
This commit is contained in:
inference-bot committed 2026-11-12 03:23:33 +00:00
1 parent 1691f79621
commit 20d384be87
5 files changed
+222 -21

No files matched your search

+42
View File
@@ -105,3 +105,45 @@ app; the agent-host mechanisms were chosen because they add no new Cloudron app.
Each deploy writes `meta.json` into the site with the commit it was built
from, so a deploy can be verified from the served site itself:
https://docs.inference.coop/meta.json
## One build environment (2026-11-12)
The deploy used to have TWO build environments — this repo's `.venv-docs` and
a second private venv in the poller's checkout at
`/home/ubuntu/.docs-publisher/.venv-docs`. They diverged twice, each time
silently delaying a deploy (the brand.css cache-busting confusion, then a
missing `mkdocs-redirects` that failed every build for ~15 minutes while the
poller retried without telling anyone). Consolidated:
- **ONE venv.** Every trigger — the poller, the (dormant) webhook receiver, a
manual `publishing/build.sh`, CI — builds with the canonical checkout's
venv: `~/repos/docs/.venv-docs`. The poller's private venv was deleted.
- **The dedicated checkout stays, by design.** The poller still builds from
`/home/ubuntu/.docs-publisher`, its own clean clone: uncommitted edits in a
working clone must never block a deploy or leak into the published site.
- **Drift check.** Before every build the poller compares `publishing/` and
`mkdocs.yml` between its checkout and the canonical one; any mismatch fails
the deploy loudly (with a diff and the fix — `git -C ~/repos/docs pull
--ff-only`), and the failure counts toward the alert below. A canonical
checkout that lags behind main trips this too, on purpose: a deploy of
logic the canonical source cannot reproduce is the failure mode this exists
to prevent. A missing canonical checkout is fatal as well.
- **Plugin gate + self-heal.** `publishing/requirements.txt` pins every
package the build needs (`mkdocs`, `mkdocs-material`, `mkdocs-redirects`).
`build.sh` verifies the venv against it before every build and installs
anything missing or off-version (fast local no-op when current); it also
fails loudly if `mkdocs.yml` configures a plugin that `requirements.txt`
does not pin. The poller runs this as its pre-deploy check via
`build.sh --check-only`.
- **Failure alerting.** After 3 consecutive deploy failures the poller writes
a marker (`~/.docs-deploy.alert`) and sends ONE message to the InferenceBot
HQ Matrix room naming the failing commit — never more than one message per
hour (`~/.docs-deploy.last-alert`). The counter (`~/.docs-deploy.failcount`)
resets on the next successful deploy. A deploy that succeeds silently
resets everything; nothing announces routine successes.
Files: `~/.docs-deploy.state` (last deployed commit), `~/.docs-deploy.lock`
(deploy lock), `~/.docs-deploy.failcount`, `~/.docs-deploy.alert`,
`~/.docs-deploy.last-alert`. Manual deploys should respect the lock:
`flock -n ~/.docs-deploy.lock bash publishing/build.sh && flock -n
~/.docs-deploy.lock bash publishing/deploy.sh`.