LibreChat sandbox: rewrite maintenance doc for the volume-config + overlay-image architecture
Documents the verified setup: config/env/MCP on the persistent volume, brand assets via the pinned overlay image, /app/data/env precedence over cloudron env, the manifest.dockerImage deploy mechanism, image-gen as an agent tool, and the 2026-10-18 crash-loop root cause (hand-edited manifest).
This commit is contained in:
Nathan Schneider committed
1 parent
fa979a5ed9
commit
34a23a34f3
1 file changed
+76
-26
@@ -1,32 +1,82 @@
|
||||
# LibreChat sandbox maintenance
|
||||
|
||||
**Current image**
|
||||
- Docker image: `cloudron/ai.librechat.cloudronapp:202610150224320000`
|
||||
- Deployed as app **0752e430‑aa36‑436e‑8668‑ef18da474f47** (LibreChat sandbox at `https://librechat-test.inference.coop`).
|
||||
The LibreChat evaluation instance at **librechat-test.inference.coop**
|
||||
(Cloudron app `4023ccbf-d57a-4500-97df-14de9cf3516e`, appstore package
|
||||
`ai.librechat.cloudronapp@1.4.1`, which bundles **LibreChat v0.8.8**).
|
||||
|
||||
**Automatic updates**
|
||||
- The "Automatic updates" toggle in the Cloudron UI has been turned **off** (as of 2026‑10‑18). This prevents the sandbox from pulling newer Docker images automatically.
|
||||
The single source of truth for the deployment is the repo
|
||||
**[co-op/librechat-custom](https://git.inference.coop/co-op/librechat-custom)**
|
||||
— its README is the full runbook (config deploy, image build/deploy, rollback,
|
||||
env vars, known state). This doc is the operator-facing summary.
|
||||
|
||||
**Version‑pinning workflow**
|
||||
1. **Create a version tag** when a new LibreChat release is approved (e.g.
|
||||
`cloudron/ai.librechat.cloudronapp:202610250100000000`).
|
||||
2. **Push the tagged image** to the co‑op registry (e.g. `registry.inference.coop/librechat:sandbox‑2026‑10‑25`).
|
||||
```bash
|
||||
docker pull cloudron/ai.librechat.cloudronapp:<new‑tag>
|
||||
docker tag cloudron/ai.librechat.cloudronapp:<new‑tag> registry.inference.coop/librechat:sandbox‑<date>
|
||||
docker login registry.inference.coop # use registry‑bot credentials
|
||||
docker push registry.inference.coop/librechat:sandbox‑<date>
|
||||
```
|
||||
3. **Redeploy** the sandbox with the pinned tag:
|
||||
```bash
|
||||
cloudron update --app 0752e430-aa36-436e-8668-ef18da474f47 \
|
||||
--image registry.inference.coop/librechat:sandbox‑<date>
|
||||
```
|
||||
4. **Verify** the app is running (`cloudron status --app <id>`) and that the image field reflects the new tag.
|
||||
5. **Record** the change in this document (date, tag, reason) and optionally open a GitHub/Gitea issue linking to the upstream LibreChat release.
|
||||
## Architecture: what lives where
|
||||
|
||||
**Rollback**
|
||||
- To roll back, repeat step 3 with the previous image tag.
|
||||
- **App config** — `/app/data/librechat.yaml` (persistent volume, mirrored
|
||||
byte-identically in the repo). Holds the "Inference Co-op" custom endpoint
|
||||
(gateway + per-member keys), the 8-model chat picker, speech, web search,
|
||||
the docs MCP server, and a forward-compatible theme block (see below).
|
||||
- **Env file** — `/app/data/env` (persistent volume). **`start.sh` sources it
|
||||
AFTER the process env, so it wins over `cloudron env`** — that is where
|
||||
`APP_TITLE="Inference Co-op"`, `ALLOW_EMAIL_LOGIN=false` (password login
|
||||
removed; Cloudron SSO is the only login path), and
|
||||
`SCHEDULES_SINGLE_PROCESS=true` must live.
|
||||
- **Other env vars** — via `cloudron env`: `TTS_API_KEY`/`STT_API_KEY`
|
||||
(dedicated LiteLLM virtual key, alias `librechat-speech`, $5/30d, audio
|
||||
models only — never the gateway master key), `IMAGE_GEN_OAI_BASEURL` +
|
||||
`IMAGE_GEN_OAI_MODEL` (agent image tool → gateway), `SEARXNG_INSTANCE_URL`.
|
||||
- **Docs MCP server** — `/app/data/mcp/server.py` (volume, mirrored in the
|
||||
repo at `mcp/server.py`; stdlib-only Python; tool `search_docs`).
|
||||
- **Brand assets** (login logo, favicons, PWA icons, page title) — baked into
|
||||
a pinned overlay image `FROM` the official appstore image, built from the
|
||||
repo via `cloudron builder build` and deployed with
|
||||
`cloudron update --image ... --force`. Deployed tag:
|
||||
`registry.inference.coop/co-op/librechat-custom:20261019-070dec3`.
|
||||
|
||||
**Future maintainers**
|
||||
- All steps are captured in this doc and in the corresponding issue in `co‑op/support` (Issue #24). The `co‑op/docs` repository should contain this file; keep it up‑to‑date whenever the sandbox image changes or the automatic‑update flag is toggled.
|
||||
## Update policy
|
||||
|
||||
- The app is now **image-sourced** (the `--force` update moved it off the
|
||||
appstore source), so Cloudron will not auto-update it. This is deliberate —
|
||||
same lockdown model as LiteLLM.
|
||||
- **When a new appstore package ships**: pull its manifest copy into the repo
|
||||
(`GET /api/v1/apps/<id> → .manifest`), bump the Dockerfile `FROM` to the new
|
||||
`manifest.dockerImage`, rebuild, verify in the sandbox, then update. A
|
||||
newer LibreChat also activates the already-present `interface.theme` block
|
||||
(co-op colors) automatically — the theme engine does not exist in v0.8.8
|
||||
and the block is silently ignored today.
|
||||
- **Rollback**: `cloudron update --app <id> --image <previous-tag> --force`
|
||||
(or the original appstore image). Cloudron also snapshots the app before
|
||||
every image update.
|
||||
|
||||
## Hard-won pitfalls (do not re-learn these)
|
||||
|
||||
- `cloudron update` deploys **`manifest.dockerImage`** from the manifest in
|
||||
the deploy directory — the `--image` flag alone is ignored. The repo
|
||||
manifest is a byte-identical copy of the OFFICIAL appstore manifest except
|
||||
`dockerImage`; editing anything else in it (or hand-writing one that omits
|
||||
the addons) strips `postgresql`/`mongodb`/`oidc`/`sendmail`/`localstorage`
|
||||
from the app and crash-loops it (`mkdir: Read-only file system`). That is
|
||||
what broke the sandbox on 2026-10-18.
|
||||
- Deploy `librechat.yaml` only via base64 round-trip. A `sed` insert once
|
||||
broke the YAML indentation and LibreChat **silently fell back to defaults**
|
||||
(all custom branding/models/MCP gone) with a single log line.
|
||||
- `cloudron env set` alone does NOT disable the password form —
|
||||
`ALLOW_EMAIL_LOGIN` must be flipped in `/app/data/env` (precedence above).
|
||||
- Image generation in LibreChat is an **Agent tool ("OpenAI Image Tools")**,
|
||||
not a chat-picker button; image models in the chat picker were the source
|
||||
of the old 500 errors and are deliberately excluded.
|
||||
- No `branding:` block exists in LibreChat's config schema; the login logo
|
||||
and favicons come from the image overlay.
|
||||
|
||||
## Verification (after any change)
|
||||
|
||||
```bash
|
||||
cloudron logs --app 4023ccbf-d57a-4500-97df-14de9cf3516e --lines 60 | \
|
||||
grep -iE 'custom config|error|invalid|MCP|schedules'
|
||||
curl -s https://librechat-test.inference.coop/api/config # appTitle, emailLoginEnabled
|
||||
```
|
||||
|
||||
Expected: `Custom config file loaded`, `[MCP] Initialized with 1 configured
|
||||
server and 1 tool`, `[schedules] engine started`, `appTitle: Inference
|
||||
Co-op`, `emailLoginEnabled: false`. The login page shows the co-op logo, the
|
||||
co-op favicon, and a single "Login with Inference Cooperative" (Cloudron SSO)
|
||||
path with no email/password form.
|
||||
Reference in new issue
Block a user