Files
docs/publishing/README.md
T
inference-bot 20d384be87 publishing: one canonical build venv; drift check; deploy-failure alerting
The poller's private venv (.docs-publisher/.venv-docs) diverged from the
canonical one twice, each time silently delaying a deploy. Consolidate:

- poll-watcher.sh builds with the ONE canonical venv
  (~/repos/docs/.venv-docs) while keeping its dedicated checkout (uncommitted
  edits in a working clone must never block or leak into a deploy).
- New drift check: publishing/ + mkdocs.yml are compared with the canonical
  checkout before every build; a mismatch fails the deploy loudly instead of
  shipping logic the canonical source cannot reproduce.
- Pre-deploy gate: build.sh --check-only verifies the pinned toolchain and
  every configured plugin (self-heals missing pins, fails loudly naming an
  unpinned plugin).
- Alerting: 3 consecutive deploy failures -> marker file + ONE Matrix message
  to the InferenceBot HQ room (hourly cap); counter resets on next success.
- The watcher re-execs itself when a deploy changed poll-watcher.sh, so the
  running loop always matches the deployed one.
- webhook receiver (dormant) repointed at the canonical venv too; unit files
  and README document the layout.
2026-11-12 03:23:33 +00:00

7.5 KiB

Publishing the documentation site

The Markdown in this repository is the single source of truth. This directory is the publishing layer that turns it into a member-readable static site and puts it on the co-op's docs Surfer app at https://docs.inference.coop/. Documents are never edited here — a change to a document is a normal commit or pull request against the Markdown at the repository root.

What is built

File Purpose
../mkdocs.yml Site configuration (MkDocs + Material for MkDocs)
brand.css Co-op palette + self-hosted Figtree webfont (SIL OFL 1.1)
fonts/ Figtree woff2 files and the OFL licence text
logo.svg Co-op logo (shared with the website repo)
build.sh Stage the Markdown into .site-src/, render to .site-build/
hooks.py MkDocs hooks: directory-link rewriting + content-hashed stylesheet URLs
deploy.sh Push the build into the Surfer app container
poll-watcher.sh Polls Gitea main every 60s; deploys on change (agent host)
docs-deploy-poll.service systemd unit for the poller
webhook-receiver.py Gitea webhook → pull → build → deploy (agent host)
webhook-receiver.service systemd unit for the receiver
requirements.txt Pinned Python build toolchain

Nothing external is loaded by the site: no CDN, no third-party font service. Fonts are served from this repository's own output.

Cache-proof stylesheets

brand.css and typography.css are listed under extra_css, so at a fixed URL they would be vulnerable to a stale browser cache after a deploy — a warm cache keeps serving the old file because the URL never changes. hooks.py's on_post_build fixes this at build time: it copies each stylesheet to <name>.<sha256-8>.css in the output and rewrites the <link> in every built page to the hashed name. A change to a stylesheet's content therefore always produces a new URL. The authored sources keep their names and locations; the hashed copies exist only in the build output. The hashed copy sits in the same directory as the original, so the @font-face url("fonts/…") references keep resolving. The deploy is unchanged.

Build

publishing/build.sh          # creates .venv-docs on first run

Output lands in .site-build/ (gitignored). mkdocs build --strict is what CI-style runs should use, so broken links fail the build.

Deploy

CLOUDRON_TOKEN=... publishing/deploy.sh

Writes the build into the docs Surfer app (Cloudron app docs.inference.coop, io.cloudron.surfer) at:

That is now the one canonical location. The docs used to be a pair of directories inside the marketing Surfer app (inference.coop) as well, at /app/data/public-docs/ and /app/data/public/docs/; both were deleted when this app took over, so https://inference.coop/docs/ no longer serves the docs (nothing on the marketing site linked to it).

Override the destination with SURFER_APP (the app's Cloudron location) and DOCS_DEST (path inside the container) if you ever need to.

Automatic rebuilds (push to main)

Two mechanisms keep the site in step with main; both run on the agent host and deploy into the docs Cloudron app above.

Polling (running now). docs-deploy-poll.service polls Gitea every 60 seconds and deploys whenever main moves. A push appears on the site within about a minute, with no human action. The Gitea webhook alone was not enough because the agent host's cloud firewall only admits inbound SSH — a webhook POST is silently dropped — while outbound access to Gitea works.

The service builds from a dedicated clean clone at /home/ubuntu/.docs-publisher, not from anyone's working copy, so uncommitted edits in a working clone can never block a deploy. Edit, commit, push — that is the whole deployment story.

Webhook (wired, waiting on one port). A Gitea webhook is registered on co-op/docs (push events, branch main) pointing at the receiver below. The moment the Infomaniak panel allows inbound TCP to the agent host — port 8787 from the Cloudron box 179.237.86.116 is enough — start the receiver and pushes deploy instantly instead of within a minute:

sudo systemctl enable --now docs-deploy-webhook
tail -f /home/ubuntu/docs-deploy.log        # what both mechanisms are doing

The receiver authenticates webhooks with an HMAC-SHA256 shared secret (DEPLOY_WEBHOOK_SECRET in the agent host's env file) and only accepts pushes to main of co-op/docs from the Cloudron box. Woodpecker CI (Cloudron catalog) could do the same job inside Cloudron, at the cost of running another app; the agent-host mechanisms were chosen because they add no new Cloudron app.

Each deploy writes meta.json into the site with the commit it was built from, so a deploy can be verified from the served site itself: https://docs.inference.coop/meta.json

One build environment (2026-11-12)

The deploy used to have TWO build environments — this repo's .venv-docs and a second private venv in the poller's checkout at /home/ubuntu/.docs-publisher/.venv-docs. They diverged twice, each time silently delaying a deploy (the brand.css cache-busting confusion, then a missing mkdocs-redirects that failed every build for ~15 minutes while the poller retried without telling anyone). Consolidated:

  • ONE venv. Every trigger — the poller, the (dormant) webhook receiver, a manual publishing/build.sh, CI — builds with the canonical checkout's venv: ~/repos/docs/.venv-docs. The poller's private venv was deleted.
  • The dedicated checkout stays, by design. The poller still builds from /home/ubuntu/.docs-publisher, its own clean clone: uncommitted edits in a working clone must never block a deploy or leak into the published site.
  • Drift check. Before every build the poller compares publishing/ and mkdocs.yml between its checkout and the canonical one; any mismatch fails the deploy loudly (with a diff and the fix — git -C ~/repos/docs pull --ff-only), and the failure counts toward the alert below. A canonical checkout that lags behind main trips this too, on purpose: a deploy of logic the canonical source cannot reproduce is the failure mode this exists to prevent. A missing canonical checkout is fatal as well.
  • Plugin gate + self-heal. publishing/requirements.txt pins every package the build needs (mkdocs, mkdocs-material, mkdocs-redirects). build.sh verifies the venv against it before every build and installs anything missing or off-version (fast local no-op when current); it also fails loudly if mkdocs.yml configures a plugin that requirements.txt does not pin. The poller runs this as its pre-deploy check via build.sh --check-only.
  • Failure alerting. After 3 consecutive deploy failures the poller writes a marker (~/.docs-deploy.alert) and sends ONE message to the InferenceBot HQ Matrix room naming the failing commit — never more than one message per hour (~/.docs-deploy.last-alert). The counter (~/.docs-deploy.failcount) resets on the next successful deploy. A deploy that succeeds silently resets everything; nothing announces routine successes.

Files: ~/.docs-deploy.state (last deployed commit), ~/.docs-deploy.lock (deploy lock), ~/.docs-deploy.failcount, ~/.docs-deploy.alert, ~/.docs-deploy.last-alert. Manual deploys should respect the lock: flock -n ~/.docs-deploy.lock bash publishing/build.sh && flock -n ~/.docs-deploy.lock bash publishing/deploy.sh.