LibreChat sandbox — Inference Co-op customization

Source of truth for the LibreChat evaluation instance at librechat-test.inference.coop (Cloudron app 4023ccbf-d57a-4500-97df-14de9cf3516e, appstore package ai.librechat.cloudronapp@1.4.1, which bundles LibreChat v0.8.8 on image cloudron/ai.librechat.cloudronapp:202610150224320000).

The deployment has two independent layers — know which one you are changing:

1. Persistent volume (/app/data/…) — behavior: librechat.yaml, env, mcp/*.py. Survives image updates. Edited via cloudron exec, then mirrored into this repo (see "Volume changes" below).
2. Overlay image (this repo's Dockerfile) — appearance only: logo, favicons, custom.css, custom2.js, <title>. Rebuilt and redeployed to change branding.

The md5s of librechat.yaml, mcp/server.py, and mcp/image_server.py in this repo are kept byte-identical to the live volume files. Verify with:

APP=4023ccbf-d57a-4500-97df-14de9cf3516e
cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec --app $APP \
  -- md5sum /app/data/librechat.yaml /app/data/mcp/server.py /app/data/mcp/image_server.py
md5sum librechat.yaml mcp/server.py mcp/image_server.py

Architecture: what lives where

Layer	Location	Survives image update?	Repo mirror
App config	/app/data/librechat.yaml (volume)	yes	librechat.yaml
Env vars (env-file wins!)	/app/data/env (volume, sourced LAST by start.sh)	yes	documented below
Env vars (Cloudron-managed)	cloudron env (process env)	yes	documented below
MCP servers	/app/data/mcp/server.py + mcp/image_server.py (volume)	yes	mcp/
Brand assets + branding CSS/JS	baked into image overlay	no — rebuilt per image	assets/
CloudronManifest.json	repo only (controls what cloudron update deploys)	n/a	official appstore manifest except dockerImage

Precedence gotcha: start.sh sources /run/librechat/env AFTER the process env, and /app/data/env is appended into it — so a value defined in /app/data/env overrides the same cloudron env variable. That is why ALLOW_EMAIL_LOGIN=false must be set in /app/data/env, not (only) via cloudron env.

Volume changes (librechat.yaml, mcp/*.py, env)

Volume warning: edits you make to the files in this repo do not deploy with the image. The image never touches /app/data. Apply changes to the live files first, then mirror them here.

Deploy a config change (no image rebuild needed):

B64=$(base64 -w0 librechat.yaml)
cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec \
  --app 4023ccbf-d57a-4500-97df-14de9cf3516e \
  -- bash -c "echo '$B64' | base64 -d > /app/data/librechat.yaml"
cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" restart \
  --app 4023ccbf-d57a-4500-97df-14de9cf3516e --timeout 900

• Always deploy via base64 — heredocs and sed inserts have corrupted this file twice (literal $(...) / bad indentation → LibreChat silently falls back to defaults with only one log line).

• Always diff the repo file against the live one after deploying.

• Reading large live files back over cloudron exec cannot be done with a plain cat (a single large stdout stream resets the connection, ECONNRESET). Pipe it small instead:

  cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec \
    --app 4023ccbf-d57a-4500-97df-14de9cf3516e \
    -- bash -c "gzip -c /app/data/librechat.yaml | base64 -w0 | fold -w 100" > /tmp/live.b64
  tr -d '\n' < /tmp/live.b64 | base64 -d | gunzip > /tmp/live-librechat.yaml
  

Verify after every config change:

cloudron logs --app 4023ccbf-d57a-4500-97df-14de9cf3516e --lines 60 | grep -iE 'custom config|error|invalid|MCP'
curl -s https://librechat-test.inference.coop/api/config   # appTitle, emailLoginEnabled

Build + deploy the image (branding only)

Two traps have each silently eaten deploys of this overlay. Both look like success. Run the verification block below after every deploy.

1. cloudron update --image <tag> is IGNORED when this directory contains CloudronManifest.json. The CLI deploys manifest.dockerImage instead — while still printing your --image value in its output, so it looks right. The tag you build MUST be written into CloudronManifest.json.dockerImage before updating.
2. The remote builder's cache can serve stale COPY layers even when file content changed (observed 2026-10-19: six consecutive "Build successful" runs produced images with old assets). Build with --no-cache whenever anything under assets/ changed.

Full deploy sequence:

cd /path/to/librechat-custom      # the dir containing CloudronManifest.json
# 1. edit assets/ … then:
git add -A && git commit -m "…" && git push
SHA=$(git rev-parse --short HEAD)
# 2. build fresh layers (no-cache after asset edits)
cloudron builder build --no-cache \
  --repository registry.inference.coop/co-op/librechat-custom \
  --tag 20261019-$SHA
# 3. pin the manifest to the new tag — the ONLY field you may change
#    (edit "dockerImage" in CloudronManifest.json to :20261019-$SHA)
git add CloudronManifest.json && git commit -m "Pin manifest to $SHA" && git push
# 4. deploy — the manifest drives it
cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" update \
  --app 4023ccbf-d57a-4500-97df-14de9cf3516e --force

Rollback: repoint CloudronManifest.json.dockerImage to the previous tag (every build is tagged 20261019-<git-sha>; the base image cloudron/ai.librechat.cloudronapp:<tag> is the pre-overlay fallback) and rerun step 4.

Verify after every image deploy (all four, or it didn't happen)

APP=4023ccbf-d57a-4500-97df-14de9cf3516e
# 1. What is ACTUALLY running? (CLI update output is not evidence)
curl -s -H "Authorization: Bearer $CLOUDRON_TOKEN" \
  https://my.inference.coop/api/v1/apps/$APP \
  | python3 -c "import json,sys; print(json.load(sys.stdin)['manifest']['dockerImage'])"
# 2. Container files match the repo (catches stale builder cache)
cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec --app $APP \
  -- md5sum /app/code/librechat/client/dist/assets/custom.css
md5sum assets/custom.css
# 3. Served HTML carries the injection
curl -s https://librechat-test.inference.coop/login | grep -Eo 'assets/custom2?\.js|assets/custom\.css'
# 4. Rendered DOM shows it (browser devtools or a browser tool):
#    computed background of div.bg-surface-primary == rgb(250, 248, 245);
#    the "Don't have an account?" <p> computes display: none.

CloudronManifest.json rules

• The only field you may change is dockerImage — it must point at the overlay tag you want deployed.
• Everything else stays a byte-identical copy of the OFFICIAL appstore manifest. A hand-edited manifest that omits the addons (postgresql, mongodb, oidc, sendmail, localstorage) strips them from the running app and crash-loops it (mkdir: Read-only file system). That broke the sandbox on 2026-10-18 — the manifest, not the overlay, was the bug.
• When the appstore package updates, refresh the file with a new copy from the Cloudron API (GET /api/v1/apps/<id> → .manifest), repoint dockerImage to your rebuilt overlay, and bump the FROM tag in the Dockerfile to the new appstore dockerImage (pinned — never a floating tag).

Env vars (live state)

Set in /app/data/env (volume — these WIN over cloudron env):

• APP_TITLE="Inference Co-op" (was "LibreChat")
• ALLOW_EMAIL_LOGIN=false (was true) — password login removed; Cloudron SSO (OIDC) is the only login path
• SCHEDULES_SINGLE_PROCESS=true — sandbox runs a single replica; silences the scheduler's Redis guard error

Set via cloudron env set (process env):

• TTS_API_KEY / STT_API_KEY — dedicated LiteLLM virtual key (alias librechat-speech, $5 / 30d budget, restricted to tinfoil/voxtral-tts + tinfoil/whisper-large-v3-turbo). Regenerate with the gateway master key: POST /key/generate {"key_alias":"librechat-speech","models":[...],"max_budget":5,"budget_duration":"30d"} — never put the master key here.
• IMAGE_GEN_OAI_BASEURL=https://gateway.inference.coop/v1 and IMAGE_GEN_OAI_MODEL=regolo/qwen-image — the built-in agent "OpenAI Image Tools" points at the co-op gateway. No admin key is configured; members using the AGENT path paste their own key in the tool UI (v0.8.8 limitation). The MCP path below needs no key at all.
• SEARXNG_INSTANCE_URL=https://search.inference.coop — referenced from the YAML as ${SEARXNG_INSTANCE_URL} (self-hosted search; keyless JSON API).

Branding overlay (custom.css / custom2.js)

Injected into the built SPA by the Dockerfile (<link> + <script> after the <title> sed). What it does:

Palette (updated 2026-10-19): light --surface-primary is 234 243 234 (#eaf3ea) — the dashboard app's green wash background (Nathan's request to match the dashboard); secondary/hover 223 237 222, tertiary 213 228 211, border 205 224 203; loading flash matches. Dark mode: deep pine + green-light accents. Homepage variables remain the reference for other surfaces.

Fonts: Figtree (the homepage's OFL font) is self-hosted from the design kit (assets/fonts/figtree-*.woff2 + OFL.txt, served at /assets/fonts/) via @font-face in custom.css with a font-family override — LibreChat's built-in default is Inter. Never load fonts from a CDN.

Footer: build-time text (no-JS fallback) + runtime link upgrade — custom2.js rewrites the footer to "Inference Cooperative · Our AI, governed together" with a dataset guard so the MutationObserver doesn't loop.

UI strings: patch_ui_strings.py (build-time, idempotent, non-fatal) renames the "MCP Servers" button to "Tools" and replaces LibreChat's footer tagline ("Every AI for Everyone.") with the co-op footer ("Inference Cooperative · Our AI, governed together · inference.coop"), deleting stale .gz/.br siblings of edited files. Add new string replacements to the REPLACEMENTS list.

Palette source of truth: the co-op homepage's own CSS variables — --cream #faf8f5 (surfaces), --wash #eef4ea (panels/hover), --green-deep #5b8c5a (primary/submit), --green-mid #6ba86b (hover), --green-light #7ab87a (dark-mode accent), --ink #2d3327 (text), --muted #6b7a62 (secondary text), --border #e8e2d8. The html.light / html.dark blocks in custom.css map these onto LibreChat's Tailwind variables; keep them in sync when the homepage palette changes.

Logo & icons: assets/logo.svg is the GREEN pine-dots vector (extracted from the homepage's inline SVG — keep the xmlns attribute, a standalone .svg is invalid without it). The earlier white-plate PNG variant was invisible on the cream background and only "flashed" against the dark loading screen. Favicons/PWA icons are the cream-plate + green-dots design rasterized from design-assets/favicon.svg. The Dockerfile also cache-busts the logo URL in the client bundle (assets/logo.svg?v=3) and deletes the stale precompressed .gz/.br siblings — bump v= whenever the logo changes, or browsers/service workers may keep serving an old cached copy.

• Colors — LibreChat's Tailwind theme resolves every color through CSS variables set on html.light / html.dark (space-separated RGB): --surface-primary, --surface-secondary/-tertiary/-hover/-submit/-submit-hover, --accent-primary/-hover, --ring-primary, --link, --text-primary/-secondary, --border-light. custom.css overrides them (light: cream 250 248 245 surfaces + co-op green 91 140 90 accents; dark: matching co-op dark palette). To rebrand, edit the html.light / html.dark blocks — do not invent class names; .login-page, .primary etc. do not exist in the DOM. Read current values in devtools via getComputedStyle(document.documentElement).getPropertyValue('--surface-primary').
• No sign-up prompt — the "Don't have an account? Sign up" line is <p><a href="/register">Sign up</a></p>; hidden with p:has(> a[href*="register"]) { display: none !important; } (hiding only the <a> leaves the text). Any residual register/signup links are hidden too.
• custom2.js — the auth form is client-rendered AFTER load, so a plain load-time query matches nothing; a MutationObserver re-checks on every DOM change and hides the sign-up paragraph whenever React renders it.

There is also an interface.theme block in librechat.yaml — it is forward-compatible only: the theme engine ships in LibreChat releases AFTER 0.8.8 and the block is silently ignored today. When the appstore package bundles a newer LibreChat, revisit whether the CSS overlay is still needed.

Feature notes (LibreChat v0.8.8)

• Chat models: custom endpoint "Inference Co-op" → gateway, 8 chat modelSpecs (emoji convention shared with models.md). regolo/qwen-image is deliberately NOT in the chat picker (image models are not chat models; chat-calling it was the source of the old 500s).
• Image generation — two paths:
  1. MCP tool (default, no key ever): inference-image server (tool generate_image) — members select it in the composer's + palette and just ask. Per-member billing via customUserVars.IC_API_KEY (seeded from the portal) + env: IC_API_KEY: "{{IC_API_KEY}}"; a system-key fallback (system-image-fallback, $5/30d) covers the rare empty-value case. The customUserVars gate blocks a not-yet-seeded member until the next sync (≤10 min) — accepted as a short delay over billing everything to the system account.
  2. Agent tool: LibreChat's built-in "OpenAI Image Tools" (env IMAGE_GEN_OAI_BASEURL/IMAGE_GEN_OAI_MODEL → gateway; no admin key, members paste their own). Note LibreChat v0.8.8 renders the generated image only inside the agent conversation.
• MCP: inference-docs (tool search_docs, starts at boot) and inference-image (tool generate_image, startup: false). Members may add their own via the MCP Settings panel ("bring your own tools"). Permissions seeded via interface.mcpServers (use/create true, share false).
• Speech: TTS tinfoil/voxtral-tts (voice casual_female), STT tinfoil/whisper-large-v3-turbo, both via the gateway with the capped key.
• Web search: SearXNG (self-hosted) + keyless Keenable scraper, no reranker.

Per-member key auto-load (no API-key prompts)

Members never enter an API key. Chat and image generation resolve each member's own portal-provisioned key automatically:

1. The portal provisions a per-member LiteLLM key at onboarding and stores it (members.key_token).
2. GET /admin/librechat/keys on the portal returns those keys for ACTIVE members — authenticated by the dedicated X-Sync-Token header (LIBRECHAT_SYNC_TOKEN, set on BOTH apps; not the admin token).
3. sync-keys.js (this repo, deployed to /app/data/sync-keys.js) fetches that feed and upserts, per member with a sandbox account:
   • keys collection: {userId, name: "Inference Co-op", value: encrypt(JSON.stringify({apiKey}))} — consumed by the chat endpoint (apiKey: "user_provided" in the yaml).
   • pluginauths collection: {userId, pluginKey: "mcp_inference-image", authField: "IC_API_KEY", value: encrypt(key)} — consumed by the MCP customUserVars config so image generation bills to the member's own team. Encryption uses LibreChat's own scheme (CREDS_KEY/CREDS_IV, AES-256-CBC via @librechat/data-schemas encrypt()), read from the app process env (/proc/1/environ) because cloudron exec sessions don't inherit it.
4. Members who haven't logged in to the sandbox yet are skipped; the next run seeds them. Run hourly or nightly:

cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec \
  --app 4023ccbf-d57a-4500-97df-14de9cf3516e \
  -- bash -c "cd /app/code/librechat/api && node /app/data/sync-keys.js"

Idempotent. Verify with the DB probe: keys and pluginauths counts should equal the number of members who have logged in to the sandbox.

Output-redaction gotcha: terminal output renders values after apiKey: as *** even when the file contains user_provided or ${ENV_REF}. Never conclude a file contains a literal placeholder from redacted output — probe lengths/counts instead (tr -cd '*' < file | wc -c, awk -F'apiKey: ' '{print length($2)}').

Known state / limitations

• The [schedules] Redis guard error is resolved by SCHEDULES_SINGLE_PROCESS=true (above).
• The RAG API not reachable warning is expected (no rag_api addon); it only affects LibreChat's own file-RAG, which we do not use.
• The inference-image MCP server logs Tools: undefined at boot — expected: with startup: false it registers at boot and spawns per-user on first use in a conversation.
• Members' chat usage bills per-member via their endpoint key; the MCP image path is currently keyless and bills to the gateway (not enforced per-member). Wiring per-member key injection into LibreChat chat is an open task.
• Repo ↔ live parity last verified 2026-10-19: librechat.yaml, mcp/server.py, mcp/image_server.py md5s match the live volume; overlay image registry.inference.coop/co-op/librechat-custom:20261019-fc9b11e is deployed (container asset md5s match this repo).