157 lines
8.9 KiB
Markdown
157 lines
8.9 KiB
Markdown
# LibreChat sandbox maintenance
|
|
|
|
The LibreChat evaluation instance at **librechat-test.inference.coop**
|
|
(Cloudron app `4023ccbf-d57a-4500-97df-14de9cf3516e`, appstore package
|
|
`ai.librechat.cloudronapp@1.4.1`, which bundles **LibreChat v0.8.8**).
|
|
|
|
The single source of truth for the deployment is the repo
|
|
**[co-op/librechat-custom](https://git.inference.coop/co-op/librechat-custom)**
|
|
— its README is the full runbook (config deploy, image build/deploy, rollback,
|
|
env vars, branding overlay, known state). This doc is the operator-facing
|
|
summary.
|
|
|
|
## Architecture: what lives where
|
|
|
|
- **App config** — `/app/data/librechat.yaml` (persistent volume, mirrored
|
|
byte-identically in the repo). Holds the "Inference Co-op" custom endpoint
|
|
(gateway), the 8-model chat picker, speech, web search, and the two MCP
|
|
servers. Volume edits deploy via `cloudron exec` (base64), never via the
|
|
image — the image never touches `/app/data`.
|
|
- **Env file** — `/app/data/env` (persistent volume). **`start.sh` sources it
|
|
AFTER the process env, so it wins over `cloudron env`** — that is where
|
|
`APP_TITLE="Inference Co-op"`, `ALLOW_EMAIL_LOGIN=false` (password login
|
|
removed; Cloudron SSO is the only login path), and
|
|
`SCHEDULES_SINGLE_PROCESS=true` must live.
|
|
- **Other env vars** — via `cloudron env`: `TTS_API_KEY`/`STT_API_KEY`
|
|
(dedicated LiteLLM virtual key, alias `librechat-speech`, $5/30d, audio
|
|
models only — never the gateway master key), `IMAGE_GEN_OAI_BASEURL` +
|
|
`IMAGE_GEN_OAI_MODEL` (agent image tool → gateway), `SEARXNG_INSTANCE_URL`.
|
|
- **MCP servers** — `/app/data/mcp/server.py` (docs search, `search_docs`,
|
|
starts at boot) and `/app/data/mcp/image_server.py` (image generation,
|
|
`generate_image`, `startup: false`); both mirrored in the repo under
|
|
`mcp/`, stdlib-only Python.
|
|
- **Branding overlay** (logo, favicons, PWA icons, `<title>`, `custom.css`,
|
|
`custom2.js`) — baked into a pinned overlay image `FROM` the official
|
|
appstore image. Deployed tag (last verified 2026-10-19):
|
|
`registry.inference.coop/co-op/librechat-custom:20261019-fc9b11e`.
|
|
- **Branding CSS/JS** — `custom.css` overrides LibreChat's Tailwind theme
|
|
variables on `html.light`/`html.dark` using the HOMEPAGE's canonical
|
|
palette (`--cream #faf8f5`, `--wash #eef4ea`, `--green-deep #5b8c5a`,
|
|
`--green-mid #6ba86b`, `--ink #2d3327`, `--muted #6b7a62`) and brands
|
|
the loading flash (`#loading-container`); the login logo is the green
|
|
pine-dots vector (white-plate variant was invisible on cream — the
|
|
'flash on load' glitch), cache-busted via `?v=3` in the client bundle.
|
|
and hides the register/sign-up prompt via
|
|
`p:has(> a[href*="register"])`; `custom2.js` is a MutationObserver that
|
|
hides the prompt when React renders it (the auth form renders after
|
|
`load`). The `interface.theme` block in the yaml remains
|
|
forward-compatible only (inert on v0.8.8).
|
|
|
|
## Image generation (two member-facing paths)
|
|
|
|
1. **MCP tool (default, no key ever, no gate)**: the `inference-image` MCP
|
|
server exposes `generate_image` in the composer's **+** palette — no agent
|
|
setup, no key prompt. Images bill to the co-op SYSTEM key
|
|
(`SYSTEM_IMAGE_KEY`, LiteLLM alias `system-image-fallback`, $5/30d,
|
|
`regolo/qwen-image` only). No `customUserVars` gate — it made the credential
|
|
REQUIRED and blocked the whole server for any not-yet-seeded member
|
|
(first-login gap) before the fallback could run, and v0.8.8 has no optional
|
|
flag. Per-member image billing is therefore deferred.
|
|
2. **Agent tool**: LibreChat's built-in "OpenAI Image Tools"
|
|
(`IMAGE_GEN_OAI_BASEURL`/`IMAGE_GEN_OAI_MODEL` → gateway; members paste
|
|
their own key — v0.8.8 limitation).
|
|
|
|
## Update policy
|
|
|
|
- The app is **image-sourced** (the `--force` update moved it off the
|
|
appstore source), so Cloudron will not auto-update it. This is deliberate —
|
|
same lockdown model as LiteLLM.
|
|
- **Image deploys**: build `--no-cache`, write the new tag into
|
|
`CloudronManifest.json.dockerImage` (the only editable field), commit, then
|
|
`cloudron update --app <id> --force`. **`--image` alone is IGNORED** when
|
|
the manifest sits in the deploy directory. **The remote builder's cache can
|
|
serve stale COPY layers even when file content changed** (six consecutive
|
|
"Build successful" runs did on 2026-10-19) — so after every image deploy,
|
|
verify: the Cloudron API's `manifest.dockerImage`, container-file md5 vs
|
|
repo md5, the served HTML, and the rendered DOM. Full checklist in the
|
|
repo README.
|
|
- **When a new appstore package ships**: pull its manifest copy into the repo
|
|
(`GET /api/v1/apps/<id>` → `.manifest`), repoint `dockerImage` to the
|
|
rebuilt overlay, bump the Dockerfile `FROM` to the new `manifest.dockerImage`
|
|
(pinned), rebuild, verify in the sandbox, then update.
|
|
- **Rollback**: repoint `CloudronManifest.json.dockerImage` to the previous
|
|
tag and rerun `cloudron update --force` (or the original appstore image).
|
|
Cloudron also snapshots the app before every image update.
|
|
|
|
## Hard-won pitfalls (do not re-learn these)
|
|
|
|
- `cloudron update` deploys **`manifest.dockerImage`** from the manifest in
|
|
the deploy directory — the `--image` flag alone is ignored (and the update
|
|
output still prints the `--image` value, so it *looks* right). The repo
|
|
manifest is a byte-identical copy of the OFFICIAL appstore manifest except
|
|
`dockerImage`; editing anything else in it (or hand-writing one that omits
|
|
the addons) strips `postgresql`/`mongodb`/`oidc`/`sendmail`/`localstorage`
|
|
from the app and crash-loops it (`mkdir: Read-only file system`). That is
|
|
what broke the sandbox on 2026-10-18.
|
|
- The remote builder's layer cache can pin COPY steps to old file content
|
|
even across "successful" rebuilds — use `--no-cache` after asset edits and
|
|
confirm by comparing container-file md5 against the repo file.
|
|
- Deploy `librechat.yaml` only via base64 round-trip. A `sed` insert once
|
|
broke the YAML indentation and LibreChat **silently fell back to defaults**
|
|
(all custom branding/models/MCP gone) with a single log line.
|
|
- Reading large files from the container over `cloudron exec` with a plain
|
|
`cat` resets the connection (`ECONNRESET`) — pipe through
|
|
`gzip | base64 | fold` and decode locally.
|
|
- `cloudron env set` alone does NOT disable the password form —
|
|
`ALLOW_EMAIL_LOGIN` must be flipped in `/app/data/env` (precedence above).
|
|
- Image generation in LibreChat is the **MCP tool** (composer **+** palette);
|
|
the agent "OpenAI Image Tools" path also exists. Image models in the chat
|
|
picker were the source of the old 500 errors and are deliberately excluded.
|
|
- No `branding:` block exists in LibreChat's config schema, and the login
|
|
page has no `.login-page`/`.primary` classes to target — theming happens by
|
|
overriding the Tailwind CSS variables on `html.light`/`html.dark`
|
|
(`--surface-primary`, `--accent-primary`, …) from the overlay `custom.css`.
|
|
|
|
|
|
## Per-member key auto-load (no API-key prompts)
|
|
|
|
Members never enter an API key. The portal provisions each member's LiteLLM
|
|
key at onboarding; the sandbox seeds it into LibreChat's user-key store:
|
|
|
|
- `GET /admin/librechat/keys` on the portal returns active members' chat keys
|
|
(dedicated `X-Sync-Token` auth — `LIBRECHAT_SYNC_TOKEN` on both apps).
|
|
- `/app/data/sync-keys.js` (in the LibreChat container, mirrored in the repo
|
|
as `sync-keys.js`) fetches that feed and seeds the `keys` collection
|
|
(chat endpoint, `apiKey: "user_provided"`) and `pluginauths`
|
|
(`mcp:inference-image` → `IC_API_KEY`, injected per-user via
|
|
`customUserVars` so image generation bills to the member's own team).
|
|
- Members without a sandbox account are skipped until they first log in; run
|
|
the sync after onboarding waves and on a schedule (the command is in the
|
|
repo README). Idempotent.
|
|
- Encryption matches LibreChat's own (`CREDS_KEY`/`CREDS_IV`, read from
|
|
`/proc/1/environ` — exec sessions don't inherit the app env).
|
|
|
|
> Terminal output redacts values after `apiKey:` as `***` even for harmless
|
|
> strings (`user_provided`, `${TTS_API_KEY}`). Probe lengths/counts instead
|
|
> of trusting redacted output.
|
|
|
|
## Verification (after any change)
|
|
|
|
```bash
|
|
cloudron logs --app 4023ccbf-d57a-4500-97df-14de9cf3516e --lines 60 | \
|
|
grep -iE 'custom config|error|invalid|MCP|schedules'
|
|
curl -s https://librechat-test.inference.coop/api/config # appTitle, emailLoginEnabled
|
|
```
|
|
|
|
Expected: `Custom config file loaded`, `[MCP][inference-docs]` initialized
|
|
with tool `search_docs`, `inference-image` registered (its `Tools: undefined`
|
|
boot line is expected — per-user startup), `[schedules] engine started`,
|
|
`appTitle: Inference Co-op`, `emailLoginEnabled: false`. The login page shows
|
|
the co-op logo, the co-op favicon, a single "Login with Inference
|
|
Cooperative" (Cloudron SSO) path with no email/password form, **no
|
|
"Don't have an account?" line**, and the cream/green co-op color scheme.
|
|
|
|
After an IMAGE deploy, additionally run the four-step verification in the
|
|
repo README (API `manifest.dockerImage`, container md5 vs repo md5, served
|
|
HTML tags, rendered DOM colors).
|