inference-bot
2eb88f55b4
Audit of the operator docs against live reality (2026-11-04): - app-inventory.md: all 19 Cloudron apps with id, fqdn and registry image pin. Corrects the portal path (code/member-portal, not co-op/member-portal) and the panel path (local/coop.inference.admin-panel, not code/admin-panel). - maintenance/backups.md: the two backup systems; the deliberate per-user restic cache split and the --group-by "" retention fix, with why. - maintenance/woodpecker-ci.md: CI server + agent, image build/pin, manual-only trigger, git-lfs requirement, and its current not-yet-green state (agent's job-collection long-poll gets HTTP 500 from the server). - provider-pricing/: index + GreenPT, Tinfoil (local EHBP proxy architecture) and the Metis /v1/systemone decision-model route; regolo.md updated to record the gateway prices were applied. - agent-fleet.md: steward, infrastructure-bot and socialbot (Unix users, gateway units, teams); member-bot tier 2 planned but not yet provisioned. - README: file list; operator-token-runbook: correct the membership_sweep cadence (3x daily, not nightly); migration doc: post-migration addendum (16->19 apps, DNS split resolved, old box still on with apps stopped).
95 lines
5.3 KiB
Markdown
95 lines
5.3 KiB
Markdown
# Woodpecker CI (server + agent)
|
|
|
|
Self-hosted CI for the co-op's repos. **As of 2026-11-04 the CI is installed but
|
|
NOT working end to end** — read the *Current state* section before trusting it for
|
|
anything.
|
|
|
|
Two Cloudron apps, added 2026-11-03:
|
|
|
|
| App | FQDN | Cloudron id | Image |
|
|
|---|---|---|---|
|
|
| Woodpecker **server** | `ci.inference.coop` | `4984f95e-0c0a-4af3-9ce1-4ca4f3e8483d` | `cloudron/org.woodpecker_ci.cloudronapp:202610300219490000` (store, v2.22.1 / upstream 3.20.1) |
|
|
| Woodpecker **agent** | `ci-agent.inference.coop` | `5ac63873-cf2c-4ad9-acd6-81dc0e016c83` | `registry.inference.coop/co-op/woodpecker-agent:20261104-e070275` (**custom**) |
|
|
|
|
- The **server** is the stock Cloudron app. It uses the `mysql` addon for storage
|
|
and exposes `GRPC_PORT` 9000 to the agent. Its config lives in
|
|
`/app/data/env.sh` (`WOODPECKER_GITEA`, `WOODPECKER_ADMIN`,
|
|
`WOODPECKER_AGENT_SECRET`, `WOODPECKER_OPEN`, …). **No env vars are set through
|
|
the app's Cloudron env** (`env-keys` is empty) — everything is in that file.
|
|
- The **agent** is a custom app. Its config is set as Cloudron env (names only):
|
|
`WOODPECKER_SERVER` = `ci.inference.coop:9000`, `WOODPECKER_GRPC_SECURE=true`,
|
|
`WOODPECKER_BACKEND=local`, `WOODPECKER_MAX_WORKFLOWS=2`,
|
|
`WOODPECKER_AGENT_CONFIG_FILE=/app/data/agent.conf`, plus the shared
|
|
`WOODPECKER_AGENT_SECRET`.
|
|
|
|
## Why the agent image is custom (and why the backend is `local`)
|
|
|
|
Cloudron apps have **no Docker socket**, so the default `docker` backend cannot
|
|
run and the agent uses the **`local` backend**: each pipeline step executes as a
|
|
shell command *inside the agent container*, and a step's `image:` field selects
|
|
the **shell**, not a container image. Consequence: **every tool a pipeline step
|
|
needs must be baked into the agent image**, and the Cloudron rootfs is read-only
|
|
at run time, so nothing can install at run time.
|
|
|
|
The image is `FROM woodpeckerci/woodpecker-agent:v3.20.1-alpine` (the **alpine**
|
|
variant — the default `scratch` image has no shell). It adds `bash`, `git`,
|
|
**`git-lfs`**, `tar`, `ca-certificates`, `python3`, `node`/`npm`, the `cloudron`
|
|
CLI, and a pinned MkDocs toolchain at `/opt/docs-venv`. It **runs as root**
|
|
(`/app/data` on this Cloudron is root-owned, as for the other custom apps here).
|
|
|
|
## Building and pinning the agent image
|
|
|
|
- **Source repo:** `/home/ubuntu/repos/woodpecker-agent` on the agent host.
|
|
⚠️ **This repo has no git remote and has never been pushed to Gitea** — the CI
|
|
agent's build source exists only on the agent host. Treat that as a gap to close.
|
|
- **Tag convention:** `<yyyymmdd>-<gitsha>` (e.g. `20261104-e070275`). Never a
|
|
floating tag.
|
|
- **Build:** `~/.hermes/scripts/cloudron_builder_build.sh registry.inference.coop/co-op/woodpecker-agent <tag> <dir>`
|
|
(the script carries the registry auth triple, which is not optional).
|
|
- **Deploy:** write the new tag into `CloudronManifest.json`'s `dockerImage`,
|
|
commit, build `--no-cache`, then `cloudron update --app 5ac63873-…` **without**
|
|
`--image` (the manifest's pin wins). Verify by the running image tag **and**
|
|
something only the new build produces — never by exit code alone.
|
|
- `publishing-requirements.txt` mirrors the docs repo's
|
|
`publishing/requirements.txt`; **keep the two in step** when the docs toolchain
|
|
changes.
|
|
|
|
## When a pipeline runs (and when it does not)
|
|
|
|
- **Manual-trigger only.** Nothing here fires on push; a pipeline is started by
|
|
hand from the Woodpecker UI.
|
|
- **The pipeline is not yet green end to end.**
|
|
- **`git-lfs` is required by the clone step.** Woodpecker runs `git lfs fetch`
|
|
after cloning, so a clone step fails with
|
|
`git: 'lfs' is not a git command` on any image without it — this was added to
|
|
the agent image on 2026-11-04. If a rebuild ever drops it, clones break again.
|
|
|
|
## Current state (2026-11-04): installed, agent connected, cannot run jobs
|
|
|
|
- The server is up — `https://ci.inference.coop/` serves the Woodpecker UI.
|
|
- The agent app is `running`/`healthy` and its health endpoint answers — **but
|
|
that is liveness, not capability.** The log shows the agent's **gRPC job-collection
|
|
long-poll failing continuously**: the server answers the agent's RPC with
|
|
**HTTP 500** (an nginx `text/html` 500 body), so the agent retries with backoff
|
|
and **never receives work**.
|
|
- A healthy agent that cannot collect jobs is the exact trap to avoid reading as
|
|
progress. Score a CI worker by **the route that delivers it work**, not by its
|
|
health RPC.
|
|
|
|
**Unresolved / needs an operator:** the server-side gRPC 500. Until the server
|
|
answers the agent's stream, no pipeline can execute regardless of repository
|
|
configuration. This is a server-side (or platform-proxy) problem, not an agent
|
|
image problem.
|
|
|
|
## Enabling a repo (known gate)
|
|
|
|
- Repos are connected through the **Gitea** source (`WOODPECKER_GITEA`). Enabling
|
|
a repo requires the repo owner to log in to Woodpecker with **Gitea OAuth** and
|
|
activate it. Standing up that OAuth login source is an **operator step**
|
|
(creating the Gitea OAuth application needs `write:user`, which the bot's Gitea
|
|
token does not have).
|
|
- Reading run/pipeline state is done from the server's **MySQL** database, not
|
|
from a public API.
|
|
- Note: the whole point of this CI is to take the docs build and custom-app builds
|
|
off the agent host — see `co-op/support#30` (moving `REGISTRY_BUILD_TOKEN` to CI).
|