LibreChat sandbox maintenance

The LibreChat evaluation instance at librechat-test.inference.coop (Cloudron app 4023ccbf-d57a-4500-97df-14de9cf3516e, appstore package ai.librechat.cloudronapp@1.4.1, which bundles LibreChat v0.8.8).

The single source of truth for the deployment is the repo co-op/librechat-custom — its README is the full runbook (config deploy, image build/deploy, rollback, env vars, branding overlay, known state). This doc is the operator-facing summary.

Architecture: what lives where

App config — /app/data/librechat.yaml (persistent volume, mirrored byte-identically in the repo). Holds the "Inference Co-op" custom endpoint (gateway), the 8-model chat picker, speech, web search, and the two MCP servers. Volume edits deploy via cloudron exec (base64), never via the image — the image never touches /app/data.
Env file — /app/data/env (persistent volume). start.sh sources it AFTER the process env, so it wins over cloudron env — that is where APP_TITLE="Inference Co-op", ALLOW_EMAIL_LOGIN=false (password login removed; Cloudron SSO is the only login path), and SCHEDULES_SINGLE_PROCESS=true must live.
Other env vars — via cloudron env: TTS_API_KEY/STT_API_KEY (dedicated LiteLLM virtual key, alias librechat-speech, $5/30d, audio models only — never the gateway master key), IMAGE_GEN_OAI_BASEURL + IMAGE_GEN_OAI_MODEL (agent image tool → gateway), SEARXNG_INSTANCE_URL.
MCP servers — /app/data/mcp/server.py (docs search, search_docs, starts at boot) and /app/data/mcp/image_server.py (image generation, generate_image, startup: false); both mirrored in the repo under mcp/, stdlib-only Python.
Branding overlay (logo, favicons, PWA icons, <title>, custom.css, custom2.js) — baked into a pinned overlay image FROM the official appstore image. Deployed tag (last verified 2026-10-19): registry.inference.coop/co-op/librechat-custom:20261019-fc9b11e.
Branding CSS/JS — custom.css overrides LibreChat's Tailwind theme variables on html.light/html.dark using the HOMEPAGE's canonical palette (--cream #faf8f5, --wash #eef4ea, --green-deep #5b8c5a, --green-mid #6ba86b, --ink #2d3327, --muted #6b7a62) and brands the loading flash (#loading-container); the login logo is the green pine-dots vector (white-plate variant was invisible on cream — the 'flash on load' glitch), cache-busted via ?v=3 in the client bundle. and hides the register/sign-up prompt via p:has(> a[href*="register"]); custom2.js is a MutationObserver that hides the prompt when React renders it (the auth form renders after load). The interface.theme block in the yaml remains forward-compatible only (inert on v0.8.8).

Image generation (two member-facing paths)

MCP tool (default, no key ever, no gate): the inference-image MCP server exposes generate_image in the composer's + palette — no agent setup, no key prompt. Images bill to the co-op SYSTEM key (SYSTEM_IMAGE_KEY, LiteLLM alias system-image-fallback, $5/30d, regolo/qwen-image only). No customUserVars gate — it made the credential REQUIRED and blocked the whole server for any not-yet-seeded member (first-login gap) before the fallback could run, and v0.8.8 has no optional flag. Per-member image billing is therefore deferred.
Agent tool: LibreChat's built-in "OpenAI Image Tools" (IMAGE_GEN_OAI_BASEURL/IMAGE_GEN_OAI_MODEL → gateway; members paste their own key — v0.8.8 limitation).

Update policy

The app is image-sourced (the --force update moved it off the appstore source), so Cloudron will not auto-update it. This is deliberate — same lockdown model as LiteLLM.
Image deploys: build --no-cache, write the new tag into CloudronManifest.json.dockerImage (the only editable field), commit, then cloudron update --app <id> --force. --image alone is IGNORED when the manifest sits in the deploy directory. The remote builder's cache can serve stale COPY layers even when file content changed (six consecutive "Build successful" runs did on 2026-10-19) — so after every image deploy, verify: the Cloudron API's manifest.dockerImage, container-file md5 vs repo md5, the served HTML, and the rendered DOM. Full checklist in the repo README.
When a new appstore package ships: pull its manifest copy into the repo (GET /api/v1/apps/<id> → .manifest), repoint dockerImage to the rebuilt overlay, bump the Dockerfile FROM to the new manifest.dockerImage (pinned), rebuild, verify in the sandbox, then update.
Rollback: repoint CloudronManifest.json.dockerImage to the previous tag and rerun cloudron update --force (or the original appstore image). Cloudron also snapshots the app before every image update.

Hard-won pitfalls (do not re-learn these)

cloudron update deploys manifest.dockerImage from the manifest in the deploy directory — the --image flag alone is ignored (and the update output still prints the --image value, so it looks right). The repo manifest is a byte-identical copy of the OFFICIAL appstore manifest except dockerImage; editing anything else in it (or hand-writing one that omits the addons) strips postgresql/mongodb/oidc/sendmail/localstorage from the app and crash-loops it (mkdir: Read-only file system). That is what broke the sandbox on 2026-10-18.
The remote builder's layer cache can pin COPY steps to old file content even across "successful" rebuilds — use --no-cache after asset edits and confirm by comparing container-file md5 against the repo file.
Deploy librechat.yaml only via base64 round-trip. A sed insert once broke the YAML indentation and LibreChat silently fell back to defaults (all custom branding/models/MCP gone) with a single log line.
Reading large files from the container over cloudron exec with a plain cat resets the connection (ECONNRESET) — pipe through gzip | base64 | fold and decode locally.
cloudron env set alone does NOT disable the password form — ALLOW_EMAIL_LOGIN must be flipped in /app/data/env (precedence above).
Image generation in LibreChat is the MCP tool (composer + palette); the agent "OpenAI Image Tools" path also exists. Image models in the chat picker were the source of the old 500 errors and are deliberately excluded.
No branding: block exists in LibreChat's config schema, and the login page has no .login-page/.primary classes to target — theming happens by overriding the Tailwind CSS variables on html.light/html.dark (--surface-primary, --accent-primary, …) from the overlay custom.css.

Per-member key auto-load (no API-key prompts)

Members never enter an API key. The portal provisions each member's LiteLLM key at onboarding; the sandbox seeds it into LibreChat's user-key store:

GET /admin/librechat/keys on the portal returns active members' chat keys (dedicated X-Sync-Token auth — LIBRECHAT_SYNC_TOKEN on both apps).
/app/data/sync-keys.js (in the LibreChat container, mirrored in the repo as sync-keys.js) fetches that feed and seeds the keys collection (chat endpoint, apiKey: "user_provided") and pluginauths (mcp:inference-image → IC_API_KEY, injected per-user via customUserVars so image generation bills to the member's own team).
Members without a sandbox account are skipped until they first log in; run the sync after onboarding waves and on a schedule (the command is in the repo README). Idempotent.
Encryption matches LibreChat's own (CREDS_KEY/CREDS_IV, read from /proc/1/environ — exec sessions don't inherit the app env).

Terminal output redacts values after apiKey: as *** even for harmless strings (user_provided, ${TTS_API_KEY}). Probe lengths/counts instead of trusting redacted output.

Verification (after any change)

cloudron logs --app 4023ccbf-d57a-4500-97df-14de9cf3516e --lines 60 | \
  grep -iE 'custom config|error|invalid|MCP|schedules'
curl -s https://librechat-test.inference.coop/api/config   # appTitle, emailLoginEnabled

Expected: Custom config file loaded, [MCP][inference-docs] initialized with tool search_docs, inference-image registered (its Tools: undefined boot line is expected — per-user startup), [schedules] engine started, appTitle: Inference Co-op, emailLoginEnabled: false. The login page shows the co-op logo, the co-op favicon, a single "Login with Inference Cooperative" (Cloudron SSO) path with no email/password form, no "Don't have an account?" line, and the cream/green co-op color scheme.

After an IMAGE deploy, additionally run the four-step verification in the repo README (API manifest.dockerImage, container md5 vs repo md5, served HTML tags, rendered DOM colors).

8.9 KiB Raw Blame History