# Woodpecker CI (server + agent)

Self-hosted CI for the co-op's repos. **As of 2026-11-04 the CI is installed but
NOT working end to end** — read the *Current state* section before trusting it for
anything.

Two Cloudron apps, added 2026-11-03:

| App | FQDN | Cloudron id | Image |
|---|---|---|---|
| Woodpecker **server** | `ci.inference.coop` | `4984f95e-0c0a-4af3-9ce1-4ca4f3e8483d` | `cloudron/org.woodpecker_ci.cloudronapp:202610300219490000` (store, v2.22.1 / upstream 3.20.1) |
| Woodpecker **agent** | `ci-agent.inference.coop` | `5ac63873-cf2c-4ad9-acd6-81dc0e016c83` | `registry.inference.coop/co-op/woodpecker-agent:20261104-e070275` (**custom**) |

- The **server** is the stock Cloudron app. It uses the `mysql` addon for storage
  and exposes `GRPC_PORT` 9000 to the agent. Its config lives in
  `/app/data/env.sh` (`WOODPECKER_GITEA`, `WOODPECKER_ADMIN`,
  `WOODPECKER_AGENT_SECRET`, `WOODPECKER_OPEN`, …). **No env vars are set through
  the app's Cloudron env** (`env-keys` is empty) — everything is in that file.
- The **agent** is a custom app. Its config is set as Cloudron env (names only):
  `WOODPECKER_SERVER` = `ci.inference.coop:9000`, `WOODPECKER_GRPC_SECURE=true`,
  `WOODPECKER_BACKEND=local`, `WOODPECKER_MAX_WORKFLOWS=2`,
  `WOODPECKER_AGENT_CONFIG_FILE=/app/data/agent.conf`, plus the shared
  `WOODPECKER_AGENT_SECRET`.

## Why the agent image is custom (and why the backend is `local`)

Cloudron apps have **no Docker socket**, so the default `docker` backend cannot
run and the agent uses the **`local` backend**: each pipeline step executes as a
shell command *inside the agent container*, and a step's `image:` field selects
the **shell**, not a container image. Consequence: **every tool a pipeline step
needs must be baked into the agent image**, and the Cloudron rootfs is read-only
at run time, so nothing can install at run time.

The image is `FROM woodpeckerci/woodpecker-agent:v3.20.1-alpine` (the **alpine**
variant — the default `scratch` image has no shell). It adds `bash`, `git`,
**`git-lfs`**, `tar`, `ca-certificates`, `python3`, `node`/`npm`, the `cloudron`
CLI, and a pinned MkDocs toolchain at `/opt/docs-venv`. It **runs as root**
(`/app/data` on this Cloudron is root-owned, as for the other custom apps here).

## Building and pinning the agent image

- **Source repo:** `/home/ubuntu/repos/woodpecker-agent` on the agent host.
  ⚠️ **This repo has no git remote and has never been pushed to Gitea** — the CI
  agent's build source exists only on the agent host. Treat that as a gap to close.
- **Tag convention:** `<yyyymmdd>-<gitsha>` (e.g. `20261104-e070275`). Never a
  floating tag.
- **Build:** `~/.hermes/scripts/cloudron_builder_build.sh registry.inference.coop/co-op/woodpecker-agent <tag> <dir>`
  (the script carries the registry auth triple, which is not optional).
- **Deploy:** write the new tag into `CloudronManifest.json`'s `dockerImage`,
  commit, build `--no-cache`, then `cloudron update --app 5ac63873-…` **without**
  `--image` (the manifest's pin wins). Verify by the running image tag **and**
  something only the new build produces — never by exit code alone.
- `publishing-requirements.txt` mirrors the docs repo's
  `publishing/requirements.txt`; **keep the two in step** when the docs toolchain
  changes.

## When a pipeline runs (and when it does not)

- **Manual-trigger only.** Nothing here fires on push; a pipeline is started by
  hand from the Woodpecker UI.
- **The pipeline is not yet green end to end.**
- **`git-lfs` is required by the clone step.** Woodpecker runs `git lfs fetch`
  after cloning, so a clone step fails with
  `git: 'lfs' is not a git command` on any image without it — this was added to
  the agent image on 2026-11-04. If a rebuild ever drops it, clones break again.

## Current state (2026-11-04): installed, agent connected, cannot run jobs

- The server is up — `https://ci.inference.coop/` serves the Woodpecker UI.
- The agent app is `running`/`healthy` and its health endpoint answers — **but
  that is liveness, not capability.** The log shows the agent's **gRPC job-collection
  long-poll failing continuously**: the server answers the agent's RPC with
  **HTTP 500** (an nginx `text/html` 500 body), so the agent retries with backoff
  and **never receives work**.
- A healthy agent that cannot collect jobs is the exact trap to avoid reading as
  progress. Score a CI worker by **the route that delivers it work**, not by its
  health RPC.

**Unresolved / needs an operator:** the server-side gRPC 500. Until the server
answers the agent's stream, no pipeline can execute regardless of repository
configuration. This is a server-side (or platform-proxy) problem, not an agent
image problem.

## Enabling a repo (known gate)

- Repos are connected through the **Gitea** source (`WOODPECKER_GITEA`). Enabling
  a repo requires the repo owner to log in to Woodpecker with **Gitea OAuth** and
  activate it. Standing up that OAuth login source is an **operator step**
  (creating the Gitea OAuth application needs `write:user`, which the bot's Gitea
  token does not have).
- Reading run/pipeline state is done from the server's **MySQL** database, not
  from a public API.
- Note: the whole point of this CI is to take the docs build and custom-app builds
  off the agent host — see `co-op/support#30` (moving `REGISTRY_BUILD_TOKEN` to CI).