From 34a23a34f3db635e14062037541ff4f0dd105c9e Mon Sep 17 00:00:00 2001 From: Nathan Schneider Date: Sun, 18 Oct 2026 20:31:03 -0600 Subject: [PATCH] LibreChat sandbox: rewrite maintenance doc for the volume-config + overlay-image architecture Documents the verified setup: config/env/MCP on the persistent volume, brand assets via the pinned overlay image, /app/data/env precedence over cloudron env, the manifest.dockerImage deploy mechanism, image-gen as an agent tool, and the 2026-10-18 crash-loop root cause (hand-edited manifest). --- maintenance/librechat-sandbox.md | 102 +++++++++++++++++++++++-------- 1 file changed, 76 insertions(+), 26 deletions(-) diff --git a/maintenance/librechat-sandbox.md b/maintenance/librechat-sandbox.md index c6a5127..c0ef630 100644 --- a/maintenance/librechat-sandbox.md +++ b/maintenance/librechat-sandbox.md @@ -1,32 +1,82 @@ # LibreChat sandbox maintenance -**Current image** -- Docker image: `cloudron/ai.librechat.cloudronapp:202610150224320000` -- Deployed as app **0752e430‑aa36‑436e‑8668‑ef18da474f47** (LibreChat sandbox at `https://librechat-test.inference.coop`). +The LibreChat evaluation instance at **librechat-test.inference.coop** +(Cloudron app `4023ccbf-d57a-4500-97df-14de9cf3516e`, appstore package +`ai.librechat.cloudronapp@1.4.1`, which bundles **LibreChat v0.8.8**). -**Automatic updates** -- The "Automatic updates" toggle in the Cloudron UI has been turned **off** (as of 2026‑10‑18). This prevents the sandbox from pulling newer Docker images automatically. +The single source of truth for the deployment is the repo +**[co-op/librechat-custom](https://git.inference.coop/co-op/librechat-custom)** +— its README is the full runbook (config deploy, image build/deploy, rollback, +env vars, known state). This doc is the operator-facing summary. -**Version‑pinning workflow** -1. **Create a version tag** when a new LibreChat release is approved (e.g. - `cloudron/ai.librechat.cloudronapp:202610250100000000`). -2. **Push the tagged image** to the co‑op registry (e.g. `registry.inference.coop/librechat:sandbox‑2026‑10‑25`). - ```bash - docker pull cloudron/ai.librechat.cloudronapp: - docker tag cloudron/ai.librechat.cloudronapp: registry.inference.coop/librechat:sandbox‑ - docker login registry.inference.coop # use registry‑bot credentials - docker push registry.inference.coop/librechat:sandbox‑ - ``` -3. **Redeploy** the sandbox with the pinned tag: - ```bash - cloudron update --app 0752e430-aa36-436e-8668-ef18da474f47 \ - --image registry.inference.coop/librechat:sandbox‑ - ``` -4. **Verify** the app is running (`cloudron status --app `) and that the image field reflects the new tag. -5. **Record** the change in this document (date, tag, reason) and optionally open a GitHub/Gitea issue linking to the upstream LibreChat release. +## Architecture: what lives where -**Rollback** -- To roll back, repeat step 3 with the previous image tag. +- **App config** — `/app/data/librechat.yaml` (persistent volume, mirrored + byte-identically in the repo). Holds the "Inference Co-op" custom endpoint + (gateway + per-member keys), the 8-model chat picker, speech, web search, + the docs MCP server, and a forward-compatible theme block (see below). +- **Env file** — `/app/data/env` (persistent volume). **`start.sh` sources it + AFTER the process env, so it wins over `cloudron env`** — that is where + `APP_TITLE="Inference Co-op"`, `ALLOW_EMAIL_LOGIN=false` (password login + removed; Cloudron SSO is the only login path), and + `SCHEDULES_SINGLE_PROCESS=true` must live. +- **Other env vars** — via `cloudron env`: `TTS_API_KEY`/`STT_API_KEY` + (dedicated LiteLLM virtual key, alias `librechat-speech`, $5/30d, audio + models only — never the gateway master key), `IMAGE_GEN_OAI_BASEURL` + + `IMAGE_GEN_OAI_MODEL` (agent image tool → gateway), `SEARXNG_INSTANCE_URL`. +- **Docs MCP server** — `/app/data/mcp/server.py` (volume, mirrored in the + repo at `mcp/server.py`; stdlib-only Python; tool `search_docs`). +- **Brand assets** (login logo, favicons, PWA icons, page title) — baked into + a pinned overlay image `FROM` the official appstore image, built from the + repo via `cloudron builder build` and deployed with + `cloudron update --image ... --force`. Deployed tag: + `registry.inference.coop/co-op/librechat-custom:20261019-070dec3`. -**Future maintainers** -- All steps are captured in this doc and in the corresponding issue in `co‑op/support` (Issue #24). The `co‑op/docs` repository should contain this file; keep it up‑to‑date whenever the sandbox image changes or the automatic‑update flag is toggled. +## Update policy + +- The app is now **image-sourced** (the `--force` update moved it off the + appstore source), so Cloudron will not auto-update it. This is deliberate — + same lockdown model as LiteLLM. +- **When a new appstore package ships**: pull its manifest copy into the repo + (`GET /api/v1/apps/ → .manifest`), bump the Dockerfile `FROM` to the new + `manifest.dockerImage`, rebuild, verify in the sandbox, then update. A + newer LibreChat also activates the already-present `interface.theme` block + (co-op colors) automatically — the theme engine does not exist in v0.8.8 + and the block is silently ignored today. +- **Rollback**: `cloudron update --app --image --force` + (or the original appstore image). Cloudron also snapshots the app before + every image update. + +## Hard-won pitfalls (do not re-learn these) + +- `cloudron update` deploys **`manifest.dockerImage`** from the manifest in + the deploy directory — the `--image` flag alone is ignored. The repo + manifest is a byte-identical copy of the OFFICIAL appstore manifest except + `dockerImage`; editing anything else in it (or hand-writing one that omits + the addons) strips `postgresql`/`mongodb`/`oidc`/`sendmail`/`localstorage` + from the app and crash-loops it (`mkdir: Read-only file system`). That is + what broke the sandbox on 2026-10-18. +- Deploy `librechat.yaml` only via base64 round-trip. A `sed` insert once + broke the YAML indentation and LibreChat **silently fell back to defaults** + (all custom branding/models/MCP gone) with a single log line. +- `cloudron env set` alone does NOT disable the password form — + `ALLOW_EMAIL_LOGIN` must be flipped in `/app/data/env` (precedence above). +- Image generation in LibreChat is an **Agent tool ("OpenAI Image Tools")**, + not a chat-picker button; image models in the chat picker were the source + of the old 500 errors and are deliberately excluded. +- No `branding:` block exists in LibreChat's config schema; the login logo + and favicons come from the image overlay. + +## Verification (after any change) + +```bash +cloudron logs --app 4023ccbf-d57a-4500-97df-14de9cf3516e --lines 60 | \ + grep -iE 'custom config|error|invalid|MCP|schedules' +curl -s https://librechat-test.inference.coop/api/config # appTitle, emailLoginEnabled +``` + +Expected: `Custom config file loaded`, `[MCP] Initialized with 1 configured +server and 1 tool`, `[schedules] engine started`, `appTitle: Inference +Co-op`, `emailLoginEnabled: false`. The login page shows the co-op logo, the +co-op favicon, and a single "Login with Inference Cooperative" (Cloudron SSO) +path with no email/password form.