librechat-sandbox: manifest-pinned deploy flow, builder-cache + ECONNRESET pitfalls, branding overlay, keyless image path; current deployed tag
This commit is contained in:
Nathan Schneider committed
1 parent
bac352f50e
commit
6a0d092e53
1 file changed
+65
-34
@@ -7,14 +7,16 @@ The LibreChat evaluation instance at **librechat-test.inference.coop**
|
||||
The single source of truth for the deployment is the repo
|
||||
**[co-op/librechat-custom](https://git.inference.coop/co-op/librechat-custom)**
|
||||
— its README is the full runbook (config deploy, image build/deploy, rollback,
|
||||
env vars, known state). This doc is the operator-facing summary.
|
||||
env vars, branding overlay, known state). This doc is the operator-facing
|
||||
summary.
|
||||
|
||||
## Architecture: what lives where
|
||||
|
||||
- **App config** — `/app/data/librechat.yaml` (persistent volume, mirrored
|
||||
byte-identically in the repo). Holds the "Inference Co-op" custom endpoint
|
||||
(gateway + per-member keys), the 8-model chat picker, speech, web search,
|
||||
the docs MCP server, and a forward-compatible theme block (see below).
|
||||
(gateway), the 8-model chat picker, speech, web search, and the two MCP
|
||||
servers. Volume edits deploy via `cloudron exec` (base64), never via the
|
||||
image — the image never touches `/app/data`.
|
||||
- **Env file** — `/app/data/env` (persistent volume). **`start.sh` sources it
|
||||
AFTER the process env, so it wins over `cloudron env`** — that is where
|
||||
`APP_TITLE="Inference Co-op"`, `ALLOW_EMAIL_LOGIN=false` (password login
|
||||
@@ -26,59 +28,83 @@ env vars, known state). This doc is the operator-facing summary.
|
||||
`IMAGE_GEN_OAI_MODEL` (agent image tool → gateway), `SEARXNG_INSTANCE_URL`.
|
||||
- **MCP servers** — `/app/data/mcp/server.py` (docs search, `search_docs`,
|
||||
starts at boot) and `/app/data/mcp/image_server.py` (image generation,
|
||||
`generate_image`, per-user startup — see below); both mirrored in the repo
|
||||
under `mcp/`, stdlib-only Python.
|
||||
- **Brand assets** (login logo, favicons, PWA icons, page title) — baked into
|
||||
a pinned overlay image `FROM` the official appstore image, built from the
|
||||
repo via `cloudron builder build` and deployed with
|
||||
`cloudron update --image ... --force`. Deployed tag:
|
||||
`registry.inference.coop/co-op/librechat-custom:20261019-5e214e4`.
|
||||
`generate_image`, `startup: false`); both mirrored in the repo under
|
||||
`mcp/`, stdlib-only Python.
|
||||
- **Branding overlay** (logo, favicons, PWA icons, `<title>`, `custom.css`,
|
||||
`custom2.js`) — baked into a pinned overlay image `FROM` the official
|
||||
appstore image. Deployed tag (last verified 2026-10-19):
|
||||
`registry.inference.coop/co-op/librechat-custom:20261019-fc9b11e`.
|
||||
- **Branding CSS/JS** — `custom.css` overrides LibreChat's Tailwind theme
|
||||
variables on `html.light`/`html.dark` (cream surfaces, co-op green accents)
|
||||
and hides the register/sign-up prompt via
|
||||
`p:has(> a[href*="register"])`; `custom2.js` is a MutationObserver that
|
||||
hides the prompt when React renders it (the auth form renders after
|
||||
`load`). The `interface.theme` block in the yaml remains
|
||||
forward-compatible only (inert on v0.8.8).
|
||||
|
||||
## Image generation (two member-facing paths)
|
||||
|
||||
1. **MCP tool (easiest)**: the `inference-image` MCP server exposes
|
||||
`generate_image` in the composer's **+** palette — no agent setup. Each
|
||||
member's own co-op API key is injected per-user (LibreChat
|
||||
`customUserVars` → `env`, `startup: false` because the key only exists
|
||||
after the member saves it in the MCP panel), so images bill to the
|
||||
member's own balance (~1–2¢ per 1024² image via `regolo/qwen-image`).
|
||||
1. **MCP tool (default, no key ever)**: the `inference-image` MCP server
|
||||
exposes `generate_image` in the composer's **+** palette — no agent setup,
|
||||
no key prompt. The server uses the member's `IC_API_KEY` if LibreChat
|
||||
injected one, otherwise sends a **keyless request** (the gateway bills the
|
||||
request; per-member billing is NOT enforced on this path).
|
||||
`customUserVars` were removed from the live yaml on 2026-10-19 — members
|
||||
are never asked for a key.
|
||||
2. **Agent tool**: LibreChat's built-in "OpenAI Image Tools"
|
||||
(`IMAGE_GEN_OAI_BASEURL`/`IMAGE_GEN_OAI_MODEL` → gateway; per-user keys).
|
||||
(`IMAGE_GEN_OAI_BASEURL`/`IMAGE_GEN_OAI_MODEL` → gateway; members paste
|
||||
their own key — v0.8.8 limitation).
|
||||
|
||||
## Update policy
|
||||
|
||||
- The app is now **image-sourced** (the `--force` update moved it off the
|
||||
- The app is **image-sourced** (the `--force` update moved it off the
|
||||
appstore source), so Cloudron will not auto-update it. This is deliberate —
|
||||
same lockdown model as LiteLLM.
|
||||
- **Image deploys**: build `--no-cache`, write the new tag into
|
||||
`CloudronManifest.json.dockerImage` (the only editable field), commit, then
|
||||
`cloudron update --app <id> --force`. **`--image` alone is IGNORED** when
|
||||
the manifest sits in the deploy directory. **The remote builder's cache can
|
||||
serve stale COPY layers even when file content changed** (six consecutive
|
||||
"Build successful" runs did on 2026-10-19) — so after every image deploy,
|
||||
verify: the Cloudron API's `manifest.dockerImage`, container-file md5 vs
|
||||
repo md5, the served HTML, and the rendered DOM. Full checklist in the
|
||||
repo README.
|
||||
- **When a new appstore package ships**: pull its manifest copy into the repo
|
||||
(`GET /api/v1/apps/<id> → .manifest`), bump the Dockerfile `FROM` to the new
|
||||
`manifest.dockerImage`, rebuild, verify in the sandbox, then update. A
|
||||
newer LibreChat also activates the already-present `interface.theme` block
|
||||
(co-op colors) automatically — the theme engine does not exist in v0.8.8
|
||||
and the block is silently ignored today.
|
||||
- **Rollback**: `cloudron update --app <id> --image <previous-tag> --force`
|
||||
(or the original appstore image). Cloudron also snapshots the app before
|
||||
every image update.
|
||||
(`GET /api/v1/apps/<id>` → `.manifest`), repoint `dockerImage` to the
|
||||
rebuilt overlay, bump the Dockerfile `FROM` to the new `manifest.dockerImage`
|
||||
(pinned), rebuild, verify in the sandbox, then update.
|
||||
- **Rollback**: repoint `CloudronManifest.json.dockerImage` to the previous
|
||||
tag and rerun `cloudron update --force` (or the original appstore image).
|
||||
Cloudron also snapshots the app before every image update.
|
||||
|
||||
## Hard-won pitfalls (do not re-learn these)
|
||||
|
||||
- `cloudron update` deploys **`manifest.dockerImage`** from the manifest in
|
||||
the deploy directory — the `--image` flag alone is ignored. The repo
|
||||
the deploy directory — the `--image` flag alone is ignored (and the update
|
||||
output still prints the `--image` value, so it *looks* right). The repo
|
||||
manifest is a byte-identical copy of the OFFICIAL appstore manifest except
|
||||
`dockerImage`; editing anything else in it (or hand-writing one that omits
|
||||
the addons) strips `postgresql`/`mongodb`/`oidc`/`sendmail`/`localstorage`
|
||||
from the app and crash-loops it (`mkdir: Read-only file system`). That is
|
||||
what broke the sandbox on 2026-10-18.
|
||||
- The remote builder's layer cache can pin COPY steps to old file content
|
||||
even across "successful" rebuilds — use `--no-cache` after asset edits and
|
||||
confirm by comparing container-file md5 against the repo file.
|
||||
- Deploy `librechat.yaml` only via base64 round-trip. A `sed` insert once
|
||||
broke the YAML indentation and LibreChat **silently fell back to defaults**
|
||||
(all custom branding/models/MCP gone) with a single log line.
|
||||
- Reading large files from the container over `cloudron exec` with a plain
|
||||
`cat` resets the connection (`ECONNRESET`) — pipe through
|
||||
`gzip | base64 | fold` and decode locally.
|
||||
- `cloudron env set` alone does NOT disable the password form —
|
||||
`ALLOW_EMAIL_LOGIN` must be flipped in `/app/data/env` (precedence above).
|
||||
- Image generation in LibreChat is an **Agent tool ("OpenAI Image Tools")**,
|
||||
not a chat-picker button; image models in the chat picker were the source
|
||||
of the old 500 errors and are deliberately excluded.
|
||||
- No `branding:` block exists in LibreChat's config schema; the login logo
|
||||
and favicons come from the image overlay.
|
||||
- Image generation in LibreChat is the **MCP tool** (composer **+** palette);
|
||||
the agent "OpenAI Image Tools" path also exists. Image models in the chat
|
||||
picker were the source of the old 500 errors and are deliberately excluded.
|
||||
- No `branding:` block exists in LibreChat's config schema, and the login
|
||||
page has no `.login-page`/`.primary` classes to target — theming happens by
|
||||
overriding the Tailwind CSS variables on `html.light`/`html.dark`
|
||||
(`--surface-primary`, `--accent-primary`, …) from the overlay `custom.css`.
|
||||
|
||||
## Verification (after any change)
|
||||
|
||||
@@ -92,5 +118,10 @@ Expected: `Custom config file loaded`, `[MCP][inference-docs]` initialized
|
||||
with tool `search_docs`, `inference-image` registered (its `Tools: undefined`
|
||||
boot line is expected — per-user startup), `[schedules] engine started`,
|
||||
`appTitle: Inference Co-op`, `emailLoginEnabled: false`. The login page shows
|
||||
the co-op logo, the co-op favicon, and a single "Login with Inference
|
||||
Cooperative" (Cloudron SSO) path with no email/password form.
|
||||
the co-op logo, the co-op favicon, a single "Login with Inference
|
||||
Cooperative" (Cloudron SSO) path with no email/password form, **no
|
||||
"Don't have an account?" line**, and the cream/green co-op color scheme.
|
||||
|
||||
After an IMAGE deploy, additionally run the four-step verification in the
|
||||
repo README (API `manifest.dockerImage`, container md5 vs repo md5, served
|
||||
HTML tags, rendered DOM colors).
|
||||
Reference in new issue
Block a user