Nathan Schneider
34a23a34f3
Documents the verified setup: config/env/MCP on the persistent volume, brand assets via the pinned overlay image, /app/data/env precedence over cloudron env, the manifest.dockerImage deploy mechanism, image-gen as an agent tool, and the 2026-10-18 crash-loop root cause (hand-edited manifest).
83 lines
4.5 KiB
Markdown
83 lines
4.5 KiB
Markdown
# LibreChat sandbox maintenance
|
|
|
|
The LibreChat evaluation instance at **librechat-test.inference.coop**
|
|
(Cloudron app `4023ccbf-d57a-4500-97df-14de9cf3516e`, appstore package
|
|
`ai.librechat.cloudronapp@1.4.1`, which bundles **LibreChat v0.8.8**).
|
|
|
|
The single source of truth for the deployment is the repo
|
|
**[co-op/librechat-custom](https://git.inference.coop/co-op/librechat-custom)**
|
|
— its README is the full runbook (config deploy, image build/deploy, rollback,
|
|
env vars, known state). This doc is the operator-facing summary.
|
|
|
|
## Architecture: what lives where
|
|
|
|
- **App config** — `/app/data/librechat.yaml` (persistent volume, mirrored
|
|
byte-identically in the repo). Holds the "Inference Co-op" custom endpoint
|
|
(gateway + per-member keys), the 8-model chat picker, speech, web search,
|
|
the docs MCP server, and a forward-compatible theme block (see below).
|
|
- **Env file** — `/app/data/env` (persistent volume). **`start.sh` sources it
|
|
AFTER the process env, so it wins over `cloudron env`** — that is where
|
|
`APP_TITLE="Inference Co-op"`, `ALLOW_EMAIL_LOGIN=false` (password login
|
|
removed; Cloudron SSO is the only login path), and
|
|
`SCHEDULES_SINGLE_PROCESS=true` must live.
|
|
- **Other env vars** — via `cloudron env`: `TTS_API_KEY`/`STT_API_KEY`
|
|
(dedicated LiteLLM virtual key, alias `librechat-speech`, $5/30d, audio
|
|
models only — never the gateway master key), `IMAGE_GEN_OAI_BASEURL` +
|
|
`IMAGE_GEN_OAI_MODEL` (agent image tool → gateway), `SEARXNG_INSTANCE_URL`.
|
|
- **Docs MCP server** — `/app/data/mcp/server.py` (volume, mirrored in the
|
|
repo at `mcp/server.py`; stdlib-only Python; tool `search_docs`).
|
|
- **Brand assets** (login logo, favicons, PWA icons, page title) — baked into
|
|
a pinned overlay image `FROM` the official appstore image, built from the
|
|
repo via `cloudron builder build` and deployed with
|
|
`cloudron update --image ... --force`. Deployed tag:
|
|
`registry.inference.coop/co-op/librechat-custom:20261019-070dec3`.
|
|
|
|
## Update policy
|
|
|
|
- The app is now **image-sourced** (the `--force` update moved it off the
|
|
appstore source), so Cloudron will not auto-update it. This is deliberate —
|
|
same lockdown model as LiteLLM.
|
|
- **When a new appstore package ships**: pull its manifest copy into the repo
|
|
(`GET /api/v1/apps/<id> → .manifest`), bump the Dockerfile `FROM` to the new
|
|
`manifest.dockerImage`, rebuild, verify in the sandbox, then update. A
|
|
newer LibreChat also activates the already-present `interface.theme` block
|
|
(co-op colors) automatically — the theme engine does not exist in v0.8.8
|
|
and the block is silently ignored today.
|
|
- **Rollback**: `cloudron update --app <id> --image <previous-tag> --force`
|
|
(or the original appstore image). Cloudron also snapshots the app before
|
|
every image update.
|
|
|
|
## Hard-won pitfalls (do not re-learn these)
|
|
|
|
- `cloudron update` deploys **`manifest.dockerImage`** from the manifest in
|
|
the deploy directory — the `--image` flag alone is ignored. The repo
|
|
manifest is a byte-identical copy of the OFFICIAL appstore manifest except
|
|
`dockerImage`; editing anything else in it (or hand-writing one that omits
|
|
the addons) strips `postgresql`/`mongodb`/`oidc`/`sendmail`/`localstorage`
|
|
from the app and crash-loops it (`mkdir: Read-only file system`). That is
|
|
what broke the sandbox on 2026-10-18.
|
|
- Deploy `librechat.yaml` only via base64 round-trip. A `sed` insert once
|
|
broke the YAML indentation and LibreChat **silently fell back to defaults**
|
|
(all custom branding/models/MCP gone) with a single log line.
|
|
- `cloudron env set` alone does NOT disable the password form —
|
|
`ALLOW_EMAIL_LOGIN` must be flipped in `/app/data/env` (precedence above).
|
|
- Image generation in LibreChat is an **Agent tool ("OpenAI Image Tools")**,
|
|
not a chat-picker button; image models in the chat picker were the source
|
|
of the old 500 errors and are deliberately excluded.
|
|
- No `branding:` block exists in LibreChat's config schema; the login logo
|
|
and favicons come from the image overlay.
|
|
|
|
## Verification (after any change)
|
|
|
|
```bash
|
|
cloudron logs --app 4023ccbf-d57a-4500-97df-14de9cf3516e --lines 60 | \
|
|
grep -iE 'custom config|error|invalid|MCP|schedules'
|
|
curl -s https://librechat-test.inference.coop/api/config # appTitle, emailLoginEnabled
|
|
```
|
|
|
|
Expected: `Custom config file loaded`, `[MCP] Initialized with 1 configured
|
|
server and 1 tool`, `[schedules] engine started`, `appTitle: Inference
|
|
Co-op`, `emailLoginEnabled: false`. The login page shows the co-op logo, the
|
|
co-op favicon, and a single "Login with Inference Cooperative" (Cloudron SSO)
|
|
path with no email/password form.
|