0 Fork 0

Files

T

inference-bot 2eb88f55b4 docs: infrastructure audit — app inventory, backups, CI, provider pricing, agent fleet

Audit of the operator docs against live reality (2026-11-04):
- app-inventory.md: all 19 Cloudron apps with id, fqdn and registry image pin.
  Corrects the portal path (code/member-portal, not co-op/member-portal) and the
  panel path (local/coop.inference.admin-panel, not code/admin-panel).
- maintenance/backups.md: the two backup systems; the deliberate per-user restic
  cache split and the --group-by "" retention fix, with why.
- maintenance/woodpecker-ci.md: CI server + agent, image build/pin, manual-only
  trigger, git-lfs requirement, and its current not-yet-green state (agent's
  job-collection long-poll gets HTTP 500 from the server).
- provider-pricing/: index + GreenPT, Tinfoil (local EHBP proxy architecture) and
  the Metis /v1/systemone decision-model route; regolo.md updated to record the
  gateway prices were applied.
- agent-fleet.md: steward, infrastructure-bot and socialbot (Unix users, gateway
  units, teams); member-bot tier 2 planned but not yet provisioned.
- README: file list; operator-token-runbook: correct the membership_sweep cadence
  (3x daily, not nightly); migration doc: post-migration addendum (16->19 apps,
  DNS split resolved, old box still on with apps stopped).

2026-11-04 14:46:12 +00:00

5.3 KiB

Raw Permalink Blame History

Woodpecker CI (server + agent)

Self-hosted CI for the co-op's repos. As of 2026-11-04 the CI is installed but NOT working end to end — read the Current state section before trusting it for anything.

Two Cloudron apps, added 2026-11-03:

App	FQDN	Cloudron id	Image
Woodpecker server	`ci.inference.coop`	`4984f95e-0c0a-4af3-9ce1-4ca4f3e8483d`	`cloudron/org.woodpecker_ci.cloudronapp:202610300219490000` (store, v2.22.1 / upstream 3.20.1)
Woodpecker agent	`ci-agent.inference.coop`	`5ac63873-cf2c-4ad9-acd6-81dc0e016c83`	`registry.inference.coop/co-op/woodpecker-agent:20261104-e070275` (custom)

The server is the stock Cloudron app. It uses the mysql addon for storage and exposes GRPC_PORT 9000 to the agent. Its config lives in /app/data/env.sh (WOODPECKER_GITEA, WOODPECKER_ADMIN, WOODPECKER_AGENT_SECRET, WOODPECKER_OPEN, …). No env vars are set through the app's Cloudron env (env-keys is empty) — everything is in that file.
The agent is a custom app. Its config is set as Cloudron env (names only): WOODPECKER_SERVER = ci.inference.coop:9000, WOODPECKER_GRPC_SECURE=true, WOODPECKER_BACKEND=local, WOODPECKER_MAX_WORKFLOWS=2, WOODPECKER_AGENT_CONFIG_FILE=/app/data/agent.conf, plus the shared WOODPECKER_AGENT_SECRET.

Why the agent image is custom (and why the backend is `local`)

Cloudron apps have no Docker socket, so the default docker backend cannot run and the agent uses the local backend: each pipeline step executes as a shell command inside the agent container, and a step's image: field selects the shell, not a container image. Consequence: every tool a pipeline step needs must be baked into the agent image, and the Cloudron rootfs is read-only at run time, so nothing can install at run time.

The image is FROM woodpeckerci/woodpecker-agent:v3.20.1-alpine (the alpine variant — the default scratch image has no shell). It adds bash, git, git-lfs, tar, ca-certificates, python3, node/npm, the cloudron CLI, and a pinned MkDocs toolchain at /opt/docs-venv. It runs as root (/app/data on this Cloudron is root-owned, as for the other custom apps here).

Building and pinning the agent image

Source repo: /home/ubuntu/repos/woodpecker-agent on the agent host. ⚠️ This repo has no git remote and has never been pushed to Gitea — the CI agent's build source exists only on the agent host. Treat that as a gap to close.
Tag convention: <yyyymmdd>-<gitsha> (e.g. 20261104-e070275). Never a floating tag.
Build: ~/.hermes/scripts/cloudron_builder_build.sh registry.inference.coop/co-op/woodpecker-agent <tag> <dir> (the script carries the registry auth triple, which is not optional).
Deploy: write the new tag into CloudronManifest.json's dockerImage, commit, build --no-cache, then cloudron update --app 5ac63873-… without --image (the manifest's pin wins). Verify by the running image tag and something only the new build produces — never by exit code alone.
publishing-requirements.txt mirrors the docs repo's publishing/requirements.txt; keep the two in step when the docs toolchain changes.

When a pipeline runs (and when it does not)

Manual-trigger only. Nothing here fires on push; a pipeline is started by hand from the Woodpecker UI.
The pipeline is not yet green end to end.
git-lfs is required by the clone step. Woodpecker runs git lfs fetch after cloning, so a clone step fails with git: 'lfs' is not a git command on any image without it — this was added to the agent image on 2026-11-04. If a rebuild ever drops it, clones break again.

Current state (2026-11-04): installed, agent connected, cannot run jobs

The server is up — https://ci.inference.coop/ serves the Woodpecker UI.
The agent app is running/healthy and its health endpoint answers — but that is liveness, not capability. The log shows the agent's gRPC job-collection long-poll failing continuously: the server answers the agent's RPC with HTTP 500 (an nginx text/html 500 body), so the agent retries with backoff and never receives work.
A healthy agent that cannot collect jobs is the exact trap to avoid reading as progress. Score a CI worker by the route that delivers it work, not by its health RPC.

Unresolved / needs an operator: the server-side gRPC 500. Until the server answers the agent's stream, no pipeline can execute regardless of repository configuration. This is a server-side (or platform-proxy) problem, not an agent image problem.

Enabling a repo (known gate)

Repos are connected through the Gitea source (WOODPECKER_GITEA). Enabling a repo requires the repo owner to log in to Woodpecker with Gitea OAuth and activate it. Standing up that OAuth login source is an operator step (creating the Gitea OAuth application needs write:user, which the bot's Gitea token does not have).
Reading run/pipeline state is done from the server's MySQL database, not from a public API.
Note: the whole point of this CI is to take the docs build and custom-app builds off the agent host — see co-op/support#30 (moving REGISTRY_BUILD_TOKEN to CI).

5.3 KiB Raw Permalink Blame History