LibreChat sandbox — Inference Co-op customization

Source of truth for the co-op's chat instance at chat.inference.coop (Cloudron app 4023ccbf-d57a-4500-97df-14de9cf3516e, appstore package ai.librechat.cloudronapp@1.4.1, which bundles LibreChat v0.8.8 on image cloudron/ai.librechat.cloudronapp:202610150224320000). librechat-test.inference.coop was the pre-swap hostname (the swap to chat. happened 2026-10-21; the test alias is still not provisioned).

The deployment has two independent layers — know which one you are changing:

1. Persistent volume (/app/data/…) — behavior: librechat.yaml, env, mcp/*.py. Survives image updates. Edited via cloudron exec, then mirrored into this repo (see "Volume changes" below).
2. Overlay image (this repo's Dockerfile) — appearance only: logo, favicons, custom.css, custom2.js, <title>. Rebuilt and redeployed to change branding.

The md5s of librechat.yaml, mcp/server.py, and mcp/image_server.py in this repo are kept byte-identical to the live volume files. Verify with:

APP=4023ccbf-d57a-4500-97df-14de9cf3516e
cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec --app $APP \
  -- md5sum /app/data/librechat.yaml /app/data/mcp/server.py /app/data/mcp/image_server.py
md5sum librechat.yaml mcp/server.py mcp/image_server.py

Architecture: what lives where

Layer	Location	Survives image update?	Repo mirror
App config	/app/data/librechat.yaml (volume)	yes	librechat.yaml
Env vars (env-file wins!)	/app/data/env (volume, sourced LAST by start.sh)	yes	documented below
Env vars (Cloudron-managed)	cloudron env (process env)	yes	documented below
MCP servers	/app/data/mcp/server.py + mcp/image_server.py (volume)	yes	mcp/
Per-member key seed	/app/data/sync-keys.js (volume; run by the manifest's scheduler task)	yes	sync-keys.js
Brand assets + branding CSS/JS	baked into image overlay	no — rebuilt per image	assets/
CloudronManifest.json	repo only (controls what cloudron update deploys)	n/a	official appstore manifest except dockerImage

Precedence gotcha: start.sh sources /run/librechat/env AFTER the process env, and /app/data/env is appended into it — so a value defined in /app/data/env overrides the same cloudron env variable. That is why ALLOW_EMAIL_LOGIN=false must be set in /app/data/env, not (only) via cloudron env.

Volume changes (librechat.yaml, mcp/*.py, env)

Volume warning: edits you make to the files in this repo do not deploy with the image. The image never touches /app/data. Apply changes to the live files first, then mirror them here.

Deploy a config change (no image rebuild needed):

B64=$(base64 -w0 librechat.yaml)
cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec \
  --app 4023ccbf-d57a-4500-97df-14de9cf3516e \
  -- bash -c "echo '$B64' | base64 -d > /app/data/librechat.yaml"
cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" restart \
  --app 4023ccbf-d57a-4500-97df-14de9cf3516e --timeout 900

• Always deploy via base64 — heredocs and sed inserts have corrupted this file twice (literal $(...) / bad indentation → LibreChat silently falls back to defaults with only one log line).

• Always diff the repo file against the live one after deploying.

• Reading large live files back over cloudron exec cannot be done with a plain cat (a single large stdout stream resets the connection, ECONNRESET). Pipe it small instead:

  cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec \
    --app 4023ccbf-d57a-4500-97df-14de9cf3516e \
    -- bash -c "gzip -c /app/data/librechat.yaml | base64 -w0 | fold -w 100" > /tmp/live.b64
  tr -d '\n' < /tmp/live.b64 | base64 -d | gunzip > /tmp/live-librechat.yaml
  

Verify after every config change:

cloudron logs --app 4023ccbf-d57a-4500-97df-14de9cf3516e --lines 60 | grep -iE 'custom config|error|invalid|MCP'
curl -s https://librechat-test.inference.coop/api/config   # appTitle, emailLoginEnabled

Build + deploy the image (branding only)

Two traps have each silently eaten deploys of this overlay. Both look like success. Run the verification block below after every deploy.

1. cloudron update --image <tag> is IGNORED when this directory contains CloudronManifest.json. The CLI deploys manifest.dockerImage instead — while still printing your --image value in its output, so it looks right. The tag you build MUST be written into CloudronManifest.json.dockerImage before updating.
2. The remote builder's cache can serve stale COPY layers even when file content changed (observed 2026-10-19: six consecutive "Build successful" runs produced images with old assets). Build with --no-cache whenever anything under assets/ changed.

Full deploy sequence:

cd /path/to/librechat-custom      # the dir containing CloudronManifest.json
# 1. edit assets/ … then:
git add -A && git commit -m "…" && git push
SHA=$(git rev-parse --short HEAD)
# 2. build fresh layers (no-cache after asset edits)
cloudron builder build --no-cache \
  --repository registry.inference.coop/co-op/librechat-custom \
  --tag 20261019-$SHA
# 3. pin the manifest to the new tag — the ONLY field you may change
#    (edit "dockerImage" in CloudronManifest.json to :20261019-$SHA)
git add CloudronManifest.json && git commit -m "Pin manifest to $SHA" && git push
# 4. deploy — the manifest drives it
cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" update \
  --app 4023ccbf-d57a-4500-97df-14de9cf3516e --force

./release.sh release runs this same sequence with both traps handled — it refuses to build from a dirty tree, always passes --no-cache, writes the tag into the manifest before deploying, then runs the verification block. build needs docker, so it runs on whoever's machine has it (deliberately not the agent VM); verify, volume and pin/deploy need only CLOUDRON_TOKEN. ./release.sh verify is safe to run any time.

Rollback: repoint CloudronManifest.json.dockerImage to the previous tag (every build is tagged 20261019-<git-sha>; the base image cloudron/ai.librechat.cloudronapp:<tag> is the pre-overlay fallback) and rerun step 4.

Verify after every image deploy (all four, or it didn't happen)

APP=4023ccbf-d57a-4500-97df-14de9cf3516e
# 1. What is ACTUALLY running? (CLI update output is not evidence)
curl -s -H "Authorization: Bearer $CLOUDRON_TOKEN" \
  https://my.inference.coop/api/v1/apps/$APP \
  | python3 -c "import json,sys; print(json.load(sys.stdin)['manifest']['dockerImage'])"
# 2. Container files match the repo (catches stale builder cache)
cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec --app $APP \
  -- md5sum /app/code/librechat/client/dist/assets/custom.css
md5sum assets/custom.css
# 3. Served HTML carries the injection
curl -s https://chat.inference.coop/login | grep -Eo 'assets/custom2?\.js|assets/custom\.css'
# 4. Rendered DOM shows it (browser devtools or a browser tool). The DEPLOYED
#    palette is the GREEN WASH with white text areas — an earlier version of this
#    block asserted cream (rgb(250,248,245)), which would have led you to "fix"
#    a correct deploy:
#      computed background of div.bg-surface-primary == rgb(234, 243, 234);
#      chat text areas / composer == rgb(255, 255, 255);
#      dark scheme == rgb(34, 39, 31);
#      the "Don't have an account?" <p> computes display: none.
# 5. The scheduler task's script exists in THIS image — a manifest repoint
#    deployed before its image lands leaves the key seed broken on schedule:
cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec --app $APP \
  -- test -f /app/data/sync-keys.js && echo "seed path present"

./release.sh verify runs checks 1-5 plus the seed task's own log lines and the running script's revision hash, and exits non-zero on any failure.

CloudronManifest.json rules

• The only field you may change is dockerImage — it must point at the overlay tag you want deployed.
• Everything else stays a byte-identical copy of the OFFICIAL appstore manifest. A hand-edited manifest that omits the addons (postgresql, mongodb, oidc, sendmail, localstorage) strips them from the running app and crash-loops it (mkdir: Read-only file system). That broke the sandbox on 2026-10-18 — the manifest, not the overlay, was the bug.
• When the appstore package updates, refresh the file with a new copy from the Cloudron API (GET /api/v1/apps/<id> → .manifest), repoint dockerImage to your rebuilt overlay, and bump the FROM tag in the Dockerfile to the new appstore dockerImage (pinned — never a floating tag).

Env vars (live state)

Set in /app/data/env (volume — these WIN over cloudron env):

• APP_TITLE="Inference Co-op" (was "LibreChat")
• ALLOW_EMAIL_LOGIN=false (was true) — password login removed; Cloudron SSO (OIDC) is the only login path
• SCHEDULES_SINGLE_PROCESS=true — sandbox runs a single replica; silences the scheduler's Redis guard error

Set via cloudron env set (process env):

• TTS_API_KEY / STT_API_KEY — dedicated LiteLLM virtual key (alias librechat-speech, $5 / 30d budget, restricted to tinfoil/voxtral-tts + tinfoil/whisper-large-v3-turbo). Regenerate with the gateway master key: POST /key/generate {"key_alias":"librechat-speech","models":[...],"max_budget":5,"budget_duration":"30d"} — never put the master key here.
• IMAGE_GEN_OAI_BASEURL=https://gateway.inference.coop/v1 and IMAGE_GEN_OAI_MODEL=regolo/qwen-image — the built-in agent "OpenAI Image Tools" points at the co-op gateway. No admin key is configured; members using the AGENT path paste their own key in the tool UI (v0.8.8 limitation). The MCP path below needs no key at all.
• SEARXNG_INSTANCE_URL=https://search.inference.coop — referenced from the YAML as ${SEARXNG_INSTANCE_URL} (self-hosted search; keyless JSON API).

Branding overlay (custom.css / custom2.js)

Injected into the built SPA by the Dockerfile (<link> + <script> after the <title> sed). What it does:

Palette (final): light --surface-primary is 250 248 245 (#faf8f5 cream) — the dashboard's true page background (the green wash #eaf3ea was tried as the full background and rejected as too dark; on the dashboard the green is accent-only). Sidebar/composer box = white cards, tertiary = #f4f2ec (dashboard chips), hover = #eaf3ea green wash, border = #e4e0d8, chat text areas (--surface-chat, --presentation) = white. Green (#5b8c5a family) stays on accents: buttons, links, focus rings. Loading flash = cream. Dark mode: deep pine + green-light accents. Homepage variables remain the reference for other surfaces.

Fonts: Figtree (the homepage's OFL font) is self-hosted from the design kit (assets/fonts/figtree-*.woff2 + OFL.txt, served at /assets/fonts/) via @font-face in custom.css with a font-family override — LibreChat's built-in default is Inter. Never load fonts from a CDN.

Footer: build-time text (no-JS fallback) + runtime link upgrade — custom2.js rewrites the footer to "Inference Cooperative · Our AI, governed together" with a dataset guard so the MutationObserver doesn't loop.

UI strings: patch_ui_strings.py (build-time, idempotent, non-fatal) renames the "MCP Servers" button to "Tools" and replaces LibreChat's footer tagline ("Every AI for Everyone.") with the co-op footer ("Inference Cooperative · Our AI, governed together · inference.coop"), deleting stale .gz/.br siblings of edited files. Add new string replacements to the REPLACEMENTS list.

Palette source of truth: the co-op homepage's own CSS variables — --cream #faf8f5 (surfaces), --wash #eef4ea (panels/hover), --green-deep #5b8c5a (primary/submit), --green-mid #6ba86b (hover), --green-light #7ab87a (dark-mode accent), --ink #2d3327 (text), --muted #6b7a62 (secondary text), --border #e8e2d8. The html.light / html.dark blocks in custom.css map these onto LibreChat's Tailwind variables; keep them in sync when the homepage palette changes.

Logo & icons: assets/logo.svg is the GREEN pine-dots vector (extracted from the homepage's inline SVG — keep the xmlns attribute, a standalone .svg is invalid without it). The earlier white-plate PNG variant was invisible on the cream background and only "flashed" against the dark loading screen. Favicons/PWA icons are the cream-plate + green-dots design rasterized from design-assets/favicon.svg. The Dockerfile also cache-busts the logo URL in the client bundle (assets/logo.svg?v=3) and deletes the stale precompressed .gz/.br siblings — bump v= whenever the logo changes, or browsers/service workers may keep serving an old cached copy.

• Colors — LibreChat's Tailwind theme resolves every color through CSS variables set on html.light / html.dark (space-separated RGB): --surface-primary, --surface-secondary/-tertiary/-hover/-submit/-submit-hover, --accent-primary/-hover, --ring-primary, --link, --text-primary/-secondary, --border-light. custom.css overrides them (light: cream 250 248 245 surfaces + co-op green 91 140 90 accents; dark: matching co-op dark palette). To rebrand, edit the html.light / html.dark blocks — do not invent class names; .login-page, .primary etc. do not exist in the DOM. Read current values in devtools via getComputedStyle(document.documentElement).getPropertyValue('--surface-primary').
• No sign-up prompt — the "Don't have an account? Sign up" line is <p><a href="/register">Sign up</a></p>; hidden with p:has(> a[href*="register"]) { display: none !important; } (hiding only the <a> leaves the text). Any residual register/signup links are hidden too.
• custom2.js — the auth form is client-rendered AFTER load, so a plain load-time query matches nothing; a MutationObserver re-checks on every DOM change and hides the sign-up paragraph whenever React renders it.

There is also an interface.theme block in librechat.yaml — it is forward-compatible only: the theme engine ships in LibreChat releases AFTER 0.8.8 and the block is silently ignored today. When the appstore package bundles a newer LibreChat, revisit whether the CSS overlay is still needed.

Feature notes (LibreChat v0.8.8)

• Chat models: custom endpoint "Inference Co-op" → gateway, 8 chat modelSpecs (emoji convention shared with models.md). regolo/qwen-image is deliberately NOT in the chat picker (image models are not chat models; chat-calling it was the source of the old 500s).
• Image generation — two paths:
  1. MCP tool (default, no key ever): inference-image server (tool generate_image) — members select it in the composer's + palette and just ask. Per-member billing via customUserVars.IC_API_KEY (seeded from the portal) + env: IC_API_KEY: "{{IC_API_KEY}}"; a system-key fallback (system-image-fallback, $5/30d) covers the rare empty-value case. The customUserVars gate blocks a not-yet-seeded member until the next sync (≤10 min) — accepted as a short delay over billing everything to the system account.
  2. Agent tool: LibreChat's built-in "OpenAI Image Tools" (env IMAGE_GEN_OAI_BASEURL/IMAGE_GEN_OAI_MODEL → gateway; no admin key, members paste their own). Note LibreChat v0.8.8 renders the generated image only inside the agent conversation.
• MCP: inference-docs (tool search_docs, starts at boot) and inference-image (tool generate_image, startup: false). Members may add their own via the MCP Settings panel ("bring your own tools"). Permissions seeded via interface.mcpServers (use/create true, share false).
• Speech: TTS tinfoil/voxtral-tts (voice casual_female), STT tinfoil/whisper-large-v3-turbo, both via the gateway with the capped key.
• Web search: SearXNG (self-hosted) + keyless Keenable scraper, no reranker.

Per-member key auto-load (no API-key prompts)

Members never enter an API key. Chat and image generation resolve each member's own portal-provisioned key automatically:

1. The portal provisions a per-member LiteLLM key at onboarding and stores it (members.key_token).
2. GET /admin/librechat/keys on the portal returns those keys for ACTIVE members — authenticated by the dedicated X-Sync-Token header (LIBRECHAT_SYNC_TOKEN, set on BOTH apps; not the admin token).
3. sync-keys.js (this repo, deployed to /app/data/sync-keys.js) fetches that feed and upserts, per member with a sandbox account:
   • keys collection: {userId, name: "Inference Co-op", value: encrypt(JSON.stringify({apiKey}))} — consumed by the chat endpoint (apiKey: "user_provided" in the yaml).
   • pluginauths collection: {userId, pluginKey: "mcp_inference-image", authField: "IC_API_KEY", value: encrypt(key)} — consumed by the MCP customUserVars config so image generation bills to the member's own team. Encryption uses LibreChat's own scheme (CREDS_KEY/CREDS_IV, AES-256-CBC via @librechat/data-schemas encrypt()).
4. Members who haven't logged in to the sandbox yet are skipped; the next run seeds them.

Scheduling (in this app, since 2026-10-25)

CloudronManifest.json declares the task in addons.scheduler:

"scheduler": {
  "librechat_key_sync": {
    "schedule": "*/10 * * * *",
    "command": "cd /app/code/librechat/api && node /app/data/sync-keys.js"
  }
}

Seeding therefore depends on nothing outside this app — it previously ran from a steward cron on another machine, which made a member-facing function depend on that machine's uptime and paths.

Two consequences of how the scheduler addon works, both learned the hard way:

• Each task runs in its own container. The app's process env is absent there, and /proc/1/environ belongs to the task launcher, not to LibreChat — so CREDS_KEY/CREDS_IV come from /app/data/.librechat-creds (root:cloudron 0640), written from the live app env:

  cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec \
    --app 4023ccbf-d57a-4500-97df-14de9cf3516e \
    -- sh -c 'umask 077; tr "\0" "\n" < /proc/1/environ | grep -E "^CREDS_(KEY|IV)=" > /app/data/.librechat-creds; chown root:cloudron /app/data/.librechat-creds; chmod 640 /app/data/.librechat-creds'
  

  If the app's key is ever rotated, re-run that. A stale key does not corrupt anything — the guard below refuses to write.

• It fails closed. sync-keys.js writes nothing unless CREDS_KEY first decrypts an existing key row (proof it is the app's real key) and unless each fresh ciphertext round-trips back to the plaintext. A credential problem exits non-zero, logs FATAL: ..., and writes /app/data/key-sync-diag.txt (uid, cwd, which env sources were readable) instead of overwriting working member keys.

Manual run (unchanged, useful after an onboarding wave):

cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec \
  --app 4023ccbf-d57a-4500-97df-14de9cf3516e \
  -- bash -c "cd /app/code/librechat/api && node /app/data/sync-keys.js"

Idempotent. Verify: the app log shows the task's output (creds check: decrypt-ok … / portal feed: N active member key(s) / done: N seeded, M skipped), and keys / pluginauths counts should equal the number of members who have logged in to the sandbox.

Output-redaction gotcha: terminal output renders values after apiKey: as *** even when the file contains user_provided or ${ENV_REF}. Never conclude a file contains a literal placeholder from redacted output — probe lengths/counts instead (tr -cd '*' < file | wc -c, awk -F'apiKey: ' '{print length($2)}').

Known state / limitations

• The [schedules] Redis guard error is resolved by SCHEDULES_SINGLE_PROCESS=true (above).
• The RAG API not reachable warning is expected (no rag_api addon); it only affects LibreChat's own file-RAG, which we do not use.
• The inference-image MCP server logs Tools: undefined at boot — expected: with startup: false it registers at boot and spawns per-user on first use in a conversation.
• Members' chat usage bills per-member via their endpoint key; the MCP image path is currently keyless and bills to the gateway (not enforced per-member). Wiring per-member key injection into LibreChat chat is an open task.
• Repo ↔ live parity last verified 2026-10-19: librechat.yaml, mcp/server.py, mcp/image_server.py md5s match the live volume; overlay image registry.inference.coop/co-op/librechat-custom:20261019-fc9b11e is deployed (container asset md5s match this repo).

PWA lessons (2026-10-22 — icon/band debugging)

• Builder COPY layers can be stale: after editing any asset, build with --no-cache and VERIFY the served file's md5 against the repo before reporting success. (The 20261022-df55d7f build shipped a stale maskable icon this way — the repo was right, the image wasn't.)
• PWA icon URLs must carry a per-build ?v=: they are served with max-age=172800, and remove/re-add of the PWA does not clear the phone's HTTP cache — launchers keep rendering the previous build's icon bytes. patch_index.py injects the query into every manifest icon src per build.
• Maskable icons: logo must sit inside the safe zone — ~66% of the canvas on a solid brand background. Full-bleed logos get cropped ("zoomed in").
• Firefox Android: crossorigin="use-credentials" on the manifest link breaks icon resolution (blank launcher icon) — patch_index.py strips it.
• Theme bars: the manifest theme_color is one static value; Chromium's nav bar and all Firefox bars use it even when the meta says otherwise. The durable fix is viewport-fit=cover + standalone-mode safe-area padding (custom.css) so the bands show the theme-correct page background; custom2.js additionally syncs one meta theme-color to the app's resolved theme.