docs: infrastructure audit — app inventory, backups, CI, provider pricing, agent fleet
Audit of the operator docs against live reality (2026-11-04): - app-inventory.md: all 19 Cloudron apps with id, fqdn and registry image pin. Corrects the portal path (code/member-portal, not co-op/member-portal) and the panel path (local/coop.inference.admin-panel, not code/admin-panel). - maintenance/backups.md: the two backup systems; the deliberate per-user restic cache split and the --group-by "" retention fix, with why. - maintenance/woodpecker-ci.md: CI server + agent, image build/pin, manual-only trigger, git-lfs requirement, and its current not-yet-green state (agent's job-collection long-poll gets HTTP 500 from the server). - provider-pricing/: index + GreenPT, Tinfoil (local EHBP proxy architecture) and the Metis /v1/systemone decision-model route; regolo.md updated to record the gateway prices were applied. - agent-fleet.md: steward, infrastructure-bot and socialbot (Unix users, gateway units, teams); member-bot tier 2 planned but not yet provisioned. - README: file list; operator-token-runbook: correct the membership_sweep cadence (3x daily, not nightly); migration doc: post-migration addendum (16->19 apps, DNS split resolved, old box still on with apps stopped).
This commit is contained in:
1 parent
8b33d823c1
commit
2eb88f55b4
12 files changed
+688
-3
No files matched your search
@@ -0,0 +1,94 @@
|
||||
# Woodpecker CI (server + agent)
|
||||
|
||||
Self-hosted CI for the co-op's repos. **As of 2026-11-04 the CI is installed but
|
||||
NOT working end to end** — read the *Current state* section before trusting it for
|
||||
anything.
|
||||
|
||||
Two Cloudron apps, added 2026-11-03:
|
||||
|
||||
| App | FQDN | Cloudron id | Image |
|
||||
|---|---|---|---|
|
||||
| Woodpecker **server** | `ci.inference.coop` | `4984f95e-0c0a-4af3-9ce1-4ca4f3e8483d` | `cloudron/org.woodpecker_ci.cloudronapp:202610300219490000` (store, v2.22.1 / upstream 3.20.1) |
|
||||
| Woodpecker **agent** | `ci-agent.inference.coop` | `5ac63873-cf2c-4ad9-acd6-81dc0e016c83` | `registry.inference.coop/co-op/woodpecker-agent:20261104-e070275` (**custom**) |
|
||||
|
||||
- The **server** is the stock Cloudron app. It uses the `mysql` addon for storage
|
||||
and exposes `GRPC_PORT` 9000 to the agent. Its config lives in
|
||||
`/app/data/env.sh` (`WOODPECKER_GITEA`, `WOODPECKER_ADMIN`,
|
||||
`WOODPECKER_AGENT_SECRET`, `WOODPECKER_OPEN`, …). **No env vars are set through
|
||||
the app's Cloudron env** (`env-keys` is empty) — everything is in that file.
|
||||
- The **agent** is a custom app. Its config is set as Cloudron env (names only):
|
||||
`WOODPECKER_SERVER` = `ci.inference.coop:9000`, `WOODPECKER_GRPC_SECURE=true`,
|
||||
`WOODPECKER_BACKEND=local`, `WOODPECKER_MAX_WORKFLOWS=2`,
|
||||
`WOODPECKER_AGENT_CONFIG_FILE=/app/data/agent.conf`, plus the shared
|
||||
`WOODPECKER_AGENT_SECRET`.
|
||||
|
||||
## Why the agent image is custom (and why the backend is `local`)
|
||||
|
||||
Cloudron apps have **no Docker socket**, so the default `docker` backend cannot
|
||||
run and the agent uses the **`local` backend**: each pipeline step executes as a
|
||||
shell command *inside the agent container*, and a step's `image:` field selects
|
||||
the **shell**, not a container image. Consequence: **every tool a pipeline step
|
||||
needs must be baked into the agent image**, and the Cloudron rootfs is read-only
|
||||
at run time, so nothing can install at run time.
|
||||
|
||||
The image is `FROM woodpeckerci/woodpecker-agent:v3.20.1-alpine` (the **alpine**
|
||||
variant — the default `scratch` image has no shell). It adds `bash`, `git`,
|
||||
**`git-lfs`**, `tar`, `ca-certificates`, `python3`, `node`/`npm`, the `cloudron`
|
||||
CLI, and a pinned MkDocs toolchain at `/opt/docs-venv`. It **runs as root**
|
||||
(`/app/data` on this Cloudron is root-owned, as for the other custom apps here).
|
||||
|
||||
## Building and pinning the agent image
|
||||
|
||||
- **Source repo:** `/home/ubuntu/repos/woodpecker-agent` on the agent host.
|
||||
⚠️ **This repo has no git remote and has never been pushed to Gitea** — the CI
|
||||
agent's build source exists only on the agent host. Treat that as a gap to close.
|
||||
- **Tag convention:** `<yyyymmdd>-<gitsha>` (e.g. `20261104-e070275`). Never a
|
||||
floating tag.
|
||||
- **Build:** `~/.hermes/scripts/cloudron_builder_build.sh registry.inference.coop/co-op/woodpecker-agent <tag> <dir>`
|
||||
(the script carries the registry auth triple, which is not optional).
|
||||
- **Deploy:** write the new tag into `CloudronManifest.json`'s `dockerImage`,
|
||||
commit, build `--no-cache`, then `cloudron update --app 5ac63873-…` **without**
|
||||
`--image` (the manifest's pin wins). Verify by the running image tag **and**
|
||||
something only the new build produces — never by exit code alone.
|
||||
- `publishing-requirements.txt` mirrors the docs repo's
|
||||
`publishing/requirements.txt`; **keep the two in step** when the docs toolchain
|
||||
changes.
|
||||
|
||||
## When a pipeline runs (and when it does not)
|
||||
|
||||
- **Manual-trigger only.** Nothing here fires on push; a pipeline is started by
|
||||
hand from the Woodpecker UI.
|
||||
- **The pipeline is not yet green end to end.**
|
||||
- **`git-lfs` is required by the clone step.** Woodpecker runs `git lfs fetch`
|
||||
after cloning, so a clone step fails with
|
||||
`git: 'lfs' is not a git command` on any image without it — this was added to
|
||||
the agent image on 2026-11-04. If a rebuild ever drops it, clones break again.
|
||||
|
||||
## Current state (2026-11-04): installed, agent connected, cannot run jobs
|
||||
|
||||
- The server is up — `https://ci.inference.coop/` serves the Woodpecker UI.
|
||||
- The agent app is `running`/`healthy` and its health endpoint answers — **but
|
||||
that is liveness, not capability.** The log shows the agent's **gRPC job-collection
|
||||
long-poll failing continuously**: the server answers the agent's RPC with
|
||||
**HTTP 500** (an nginx `text/html` 500 body), so the agent retries with backoff
|
||||
and **never receives work**.
|
||||
- A healthy agent that cannot collect jobs is the exact trap to avoid reading as
|
||||
progress. Score a CI worker by **the route that delivers it work**, not by its
|
||||
health RPC.
|
||||
|
||||
**Unresolved / needs an operator:** the server-side gRPC 500. Until the server
|
||||
answers the agent's stream, no pipeline can execute regardless of repository
|
||||
configuration. This is a server-side (or platform-proxy) problem, not an agent
|
||||
image problem.
|
||||
|
||||
## Enabling a repo (known gate)
|
||||
|
||||
- Repos are connected through the **Gitea** source (`WOODPECKER_GITEA`). Enabling
|
||||
a repo requires the repo owner to log in to Woodpecker with **Gitea OAuth** and
|
||||
activate it. Standing up that OAuth login source is an **operator step**
|
||||
(creating the Gitea OAuth application needs `write:user`, which the bot's Gitea
|
||||
token does not have).
|
||||
- Reading run/pipeline state is done from the server's **MySQL** database, not
|
||||
from a public API.
|
||||
- Note: the whole point of this CI is to take the docs build and custom-app builds
|
||||
off the agent host — see `co-op/support#30` (moving `REGISTRY_BUILD_TOKEN` to CI).
|
||||
Reference in new issue
Block a user