LibreChat sandbox — Inference Co-op customization

Source of truth for the co-op's chat instance at chat.inference.coop (Cloudron app 4023ccbf-d57a-4500-97df-14de9cf3516e, appstore package ai.librechat.cloudronapp@1.4.5 — upstream LibreChat is unchanged at v0.8.8; the bump is uv 0.13.1 → 0.13.5 — on base image cloudron/ai.librechat.cloudronapp:202610230221180000). librechat-test.inference.coop was the pre-swap hostname (the swap to chat. happened 2026-10-21; the test alias is still not provisioned).

The deployment has two independent layers — know which one you are changing:

1. Persistent volume (/app/data/…) — behavior: librechat.yaml, env, mcp/*.py. Survives image updates. Edited via cloudron exec, then mirrored into this repo (see "Volume changes" below).
2. Overlay image (this repo's Dockerfile) — appearance only: logo, favicons, custom.css, custom2.js, <title>. Rebuilt and redeployed to change branding.

The md5s of librechat.yaml, mcp/server.py, and mcp/image_server.py in this repo are kept byte-identical to the live volume files. Verify with:

APP=4023ccbf-d57a-4500-97df-14de9cf3516e
cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec --app $APP \
  -- md5sum /app/data/librechat.yaml /app/data/mcp/server.py /app/data/mcp/image_server.py
md5sum librechat.yaml mcp/server.py mcp/image_server.py

Architecture: what lives where

Layer	Location	Survives image update?	Repo mirror
App config	/app/data/librechat.yaml (volume)	yes	librechat.yaml
Env vars (env-file wins!)	/app/data/env (volume, sourced LAST by start.sh)	yes	documented below
Env vars (Cloudron-managed)	cloudron env (process env)	yes	documented below
MCP servers	/app/data/mcp/server.py + mcp/image_server.py (volume)	yes	mcp/
Per-member key seed	sync-keys.js baked into the image at /app/code/sync-keys.js (run by the manifest's scheduler task)	no — rebuilt per image	sync-keys.js
Brand assets + branding CSS/JS	baked into image overlay	no — rebuilt per image	assets/
CloudronManifest.json	repo only (controls what cloudron update deploys)	n/a	official appstore manifest except dockerImage

Precedence gotcha: start.sh sources /run/librechat/env AFTER the process env, and /app/data/env is appended into it — so a value defined in /app/data/env overrides the same cloudron env variable. That is why ALLOW_EMAIL_LOGIN=false must be set in /app/data/env, not (only) via cloudron env.

Volume changes (librechat.yaml, mcp/*.py, env)

Volume warning: edits you make to the files in this repo do not deploy with the image. The image never touches /app/data. Apply changes to the live files first, then mirror them here.

Deploy a config change (no image rebuild needed):

B64=$(base64 -w0 librechat.yaml)
cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec \
  --app 4023ccbf-d57a-4500-97df-14de9cf3516e \
  -- bash -c "echo '$B64' | base64 -d > /app/data/librechat.yaml"
cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" restart \
  --app 4023ccbf-d57a-4500-97df-14de9cf3516e --timeout 900

• Always deploy via base64 — heredocs and sed inserts have corrupted this file twice (literal $(...) / bad indentation → LibreChat silently falls back to defaults with only one log line).

• Always diff the repo file against the live one after deploying.

• Reading large live files back over cloudron exec cannot be done with a plain cat (a single large stdout stream resets the connection, ECONNRESET). Pipe it small instead:

  cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec \
    --app 4023ccbf-d57a-4500-97df-14de9cf3516e \
    -- bash -c "gzip -c /app/data/librechat.yaml | base64 -w0 | fold -w 100" > /tmp/live.b64
  tr -d '\n' < /tmp/live.b64 | base64 -d | gunzip > /tmp/live-librechat.yaml
  

Verify after every config change:

cloudron logs --app 4023ccbf-d57a-4500-97df-14de9cf3516e --lines 60 | grep -iE 'custom config|error|invalid|MCP'
curl -s https://librechat-test.inference.coop/api/config   # appTitle, emailLoginEnabled

Build + deploy the image (branding only)

Two traps have each silently eaten deploys of this overlay. Both look like success. Run the verification block below after every deploy.

1. cloudron update --image <tag> is IGNORED when this directory contains CloudronManifest.json. The CLI deploys manifest.dockerImage instead — while still printing your --image value in its output, so it looks right. The tag you build MUST be written into CloudronManifest.json.dockerImage before updating.
2. The remote builder's cache can serve stale COPY layers even when file content changed (observed 2026-10-19: six consecutive "Build successful" runs produced images with old assets). Build with --no-cache whenever anything under assets/ changed.

Full deploy sequence:

cd /path/to/librechat-custom      # the dir containing CloudronManifest.json
# 1. edit assets/ … then:
git add -A && git commit -m "…" && git push
SHA=$(git rev-parse --short HEAD)
# 2. build fresh layers (no-cache after asset edits)
cloudron builder build --no-cache \
  --repository registry.inference.coop/co-op/librechat-custom \
  --tag 20261019-$SHA
# 3. pin the manifest to the new tag — the ONLY field you may change
#    (edit "dockerImage" in CloudronManifest.json to :20261019-$SHA)
git add CloudronManifest.json && git commit -m "Pin manifest to $SHA" && git push
# 4. deploy — the manifest drives it
cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" update \
  --app 4023ccbf-d57a-4500-97df-14de9cf3516e --force

./release.sh release runs this same sequence with both traps handled — it refuses to build from a dirty tree, always passes --no-cache, writes the tag into the manifest before deploying, then runs the verification block. Builds are remote (the registry's BuildKit does the work), so the only requirement is a registry login on the host running the command; verify, volume and pin/deploy need only CLOUDRON_TOKEN. ./release.sh verify is safe to run any time. If build reports no registry login, it prints both ways to bootstrap one (a narrow app password scoped to the registry app, or the CLI device flow).

Rollback: repoint CloudronManifest.json.dockerImage to the previous tag (every build is tagged 20261019-<git-sha>; the base image cloudron/ai.librechat.cloudronapp:<tag> is the pre-overlay fallback) and rerun step 4.

Verify after every image deploy (all four, or it didn't happen)

APP=4023ccbf-d57a-4500-97df-14de9cf3516e
# 1. What is ACTUALLY running? (CLI update output is not evidence)
curl -s -H "Authorization: Bearer $CLOUDRON_TOKEN" \
  https://my.inference.coop/api/v1/apps/$APP \
  | python3 -c "import json,sys; print(json.load(sys.stdin)['manifest']['dockerImage'])"
# 2. Container files match the repo (catches stale builder cache)
cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec --app $APP \
  -- md5sum /app/code/librechat/client/dist/assets/custom.css
md5sum assets/custom.css
# 3. Served HTML carries the injection
curl -s https://chat.inference.coop/login | grep -Eo 'assets/custom2?\.js|assets/custom\.css'
# 4. Rendered DOM shows it (browser devtools or a browser tool). The DEPLOYED
#    palette is CREAM with white text areas — the dashboard's own look. (An
#    intermediate version shipped the full green wash #eaf3ea as the page
#    background and read as "too green" — cream is the intended value, so this
#    block asserts cream; do not "fix" a cream deploy back to the wash):
#      computed background of div.bg-surface-primary == rgb(250, 248, 245);
#      chat text areas == rgb(255, 255, 255); composer band == cream;
#      dark scheme == rgb(34, 39, 31);
#      the "Don't have an account?" <p> computes display: none.
# 5. INSTALL-TIME colors, which no DOM check covers — the web manifest and the
#    index.html light meta must be #faf8f5, not the wash:
curl -s https://chat.inference.coop/manifest.webmanifest | grep -o '"theme_color":"[^"]*"'
curl -s https://chat.inference.coop/ | grep -o 'media="(prefers-color-scheme: light)" content="[^"]*"'
# 6. The scheduler task's script exists in THIS image — a manifest repoint
#    deployed before its image lands leaves the key seed broken on schedule:
cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec --app $APP \
  -- test -f /app/data/sync-keys.js && echo "seed path present"

./release.sh verify runs checks 1-5 plus the seed task's own log lines and the running script's revision hash, and exits non-zero on any failure.

Appstore pin bumps (base image)

The overlay's FROM is pinned to the exact appstore image the app runs. Bumping it is a two-line change in this repo (Dockerfile FROM + the manifest's version/changelog copied from the official manifest) plus a rebuild, but it re-runs every build-time patch against a new base layer, so treat the overlay as the risk, not the bump.

# what the appstore now offers (official manifest, authoritative)
curl -s https://api.cloudron.io/api/v1/apps/ai.librechat.cloudronapp \
  | python3 -c "import json,sys; m=json.load(sys.stdin)['data']['manifest']; print(m['version'], m.get('upstreamVersion'), m['changelog'].strip())"
# diff it against our copy — every field must match except dockerImage (+ scheduler)
# then: rebuild, pin, deploy, ./release.sh verify, and re-check the branding block above

Rollback is a one-line dockerImage repoint to the previous tag (built images stay in the registry), then ./release.sh deploy.

CloudronManifest.json rules

• The only field you may change is dockerImage — it must point at the overlay tag you want deployed.
• Everything else stays a byte-identical copy of the OFFICIAL appstore manifest. A hand-edited manifest that omits the addons (postgresql, mongodb, oidc, sendmail, localstorage) strips them from the running app and crash-loops it (mkdir: Read-only file system). That broke the sandbox on 2026-10-18 — the manifest, not the overlay, was the bug.
• When the appstore package updates, refresh the file with a new copy from the Cloudron API (GET /api/v1/apps/<id> → .manifest), repoint dockerImage to your rebuilt overlay, and bump the FROM tag in the Dockerfile to the new appstore dockerImage (pinned — never a floating tag).

Env vars (live state)

Set in /app/data/env (volume — these WIN over cloudron env):

• APP_TITLE="Inference Co-op" (was "LibreChat")
• ALLOW_EMAIL_LOGIN=false (was true) — password login removed; Cloudron SSO (OIDC) is the only login path
• SCHEDULES_SINGLE_PROCESS=true — sandbox runs a single replica; silences the scheduler's Redis guard error

Set via cloudron env set (process env):

• TTS_API_KEY / STT_API_KEY — dedicated LiteLLM virtual key (alias librechat-speech, $5 / 30d budget, restricted to tinfoil/voxtral-tts + tinfoil/whisper-large-v3-turbo). Regenerate with the gateway master key: POST /key/generate {"key_alias":"librechat-speech","models":[...],"max_budget":5,"budget_duration":"30d"} — never put the master key here.
• IMAGE_GEN_OAI_BASEURL=https://gateway.inference.coop/v1 and IMAGE_GEN_OAI_MODEL=regolo/qwen-image — the built-in agent "OpenAI Image Tools" points at the co-op gateway. No admin key is configured; members using the AGENT path paste their own key in the tool UI (v0.8.8 limitation). The MCP path below needs no key at all.
• SEARXNG_INSTANCE_URL=https://search.inference.coop — referenced from the YAML as ${SEARXNG_INSTANCE_URL} (self-hosted search; keyless JSON API).

Branding overlay (custom.css / custom2.js)

Injected into the built SPA by the Dockerfile (<link> + <script> after the <title> sed). What it does:

Palette (final, restored 2026-10-27): light --surface-primary is 250 248 245 (#faf8f5 cream) — the dashboard's true page background. Sibling surfaces: sidebar/composer box = white cards, tertiary = #f4f2ec (dashboard chips), hover = #f0ede6 (warm), border = #e4e0d8, --surface-chat = white, --presentation = cream (the composer band matches the page). Green (#5b8c5a family) stays on ACCENTS only: buttons, links (#4e7a4d), focus rings. Loading flash = cream. Dark mode: deep pine + green-light accents, unchanged. The green wash #eaf3ea is NOT a surface value anywhere — on the dashboard it appears only as badges/table headers. Applying it to --surface-primary is the regression that reads as "too green"; it was tried, rejected, restored by mistake in the Oct 21 rebuild, and removed again on Oct 27.

Fonts: Figtree (the homepage's OFL font) is self-hosted from the design kit (assets/fonts/figtree-*.woff2 + OFL.txt, served at /assets/fonts/) via @font-face in custom.css with a font-family override — LibreChat's built-in default is Inter. Never load fonts from a CDN.

Footer: build-time text (no-JS fallback) + runtime link upgrade — custom2.js rewrites the footer to "Inference Cooperative · Our AI, governed together" with a dataset guard so the MutationObserver doesn't loop.

UI strings: patch_ui_strings.py (build-time, idempotent, non-fatal) renames the "MCP Servers" button to "Tools" and replaces LibreChat's footer tagline ("Every AI for Everyone.") with the co-op footer ("Inference Cooperative · Our AI, governed together · inference.coop"), deleting stale .gz/.br siblings of edited files. Add new string replacements to the REPLACEMENTS list.

Palette source of truth: the DEPLOYED CSS of the co-op's own interfaces — dashboard, portal, panel — read from the running apps. All three are identical because they share one design system: --cream #faf8f5 (page background), #ffffff (cards), #f4f2ec/#f0ede6 (chips/hover), --green-deep #5b8c5a (primary/submit), --green-mid #6ba86b (hover), --green-light #7ab87a (dark-mode accent), --ink #2d3327 (text), --muted #6b7a62, --border #e4e0d8, --danger #b3543a. Read them from the apps rather than trusting either this file or the homepage (which carries an older --wash #eef4ea):

cloudron exec --app <dashboard-id> -- sh -c \
  "grep -rhoE -- '--[a-z0-9-]+: *[^;]+' /app/code | sort -u"

The html.light / html.dark blocks in custom.css map these onto LibreChat's Tailwind variables; keep them in sync when the design system changes. cloudron exec swallows a multi-line sh -c silently — base64 the script or keep it one line. Shared accessibility caveat, all four interfaces (not chat-specific): --muted on cream is 4.32:1, just under AA for body text, and white-on-green-deep buttons are 3.93:1 — fix everywhere or nowhere, so the interfaces stay consistent.

Logo & icons: assets/logo.svg is the GREEN pine-dots vector (extracted from the homepage's inline SVG — keep the xmlns attribute, a standalone .svg is invalid without it). The earlier white-plate PNG variant was invisible on the cream background and only "flashed" against the dark loading screen. Favicons/PWA icons are the cream-plate + green-dots design rasterized from design-assets/favicon.svg. The Dockerfile also cache-busts the logo URL in the client bundle (assets/logo.svg?v=3) and deletes the stale precompressed .gz/.br siblings — bump v= whenever the logo changes, or browsers/service workers may keep serving an old cached copy.

• Colors — LibreChat's Tailwind theme resolves every color through CSS variables set on html.light / html.dark (space-separated RGB): --surface-primary, --surface-secondary/-tertiary/-hover/-submit/-submit-hover, --accent-primary/-hover, --ring-primary, --link, --text-primary/-secondary, --border-light. custom.css overrides them (light: cream 250 248 245 surfaces + co-op green 91 140 90 accents; dark: matching co-op dark palette). To rebrand, edit the html.light / html.dark blocks — do not invent class names; .login-page, .primary etc. do not exist in the DOM. Read current values in devtools via getComputedStyle(document.documentElement).getPropertyValue('--surface-primary').
• No sign-up prompt — the "Don't have an account? Sign up" line is <p><a href="/register">Sign up</a></p>; hidden with p:has(> a[href*="register"]) { display: none !important; } (hiding only the <a> leaves the text). Any residual register/signup links are hidden too.
• custom2.js — the auth form is client-rendered AFTER load, so a plain load-time query matches nothing; a MutationObserver re-checks on every DOM change and hides the sign-up paragraph whenever React renders it.

There is also an interface.theme block in librechat.yaml — it is forward-compatible only: the theme engine ships in LibreChat releases AFTER 0.8.8 and the block is silently ignored today. When the appstore package bundles a newer LibreChat, revisit whether the CSS overlay is still needed.

Feature notes (LibreChat v0.8.8)

• Chat models: custom endpoint "Inference Co-op" → gateway, 8 chat modelSpecs (emoji convention shared with models.md). regolo/qwen-image is deliberately NOT in the chat picker (image models are not chat models; chat-calling it was the source of the old 500s).
• Image generation — two paths:
  1. MCP tool (default, no key ever): inference-image server (tool generate_image) — members select it in the composer's + palette and just ask. Per-member billing via customUserVars.IC_API_KEY (seeded from the portal) + env: IC_API_KEY: "{{IC_API_KEY}}"; a system-key fallback (system-image-fallback, $5/30d) covers the rare empty-value case. The customUserVars gate blocks a not-yet-seeded member until the next sync (≤10 min) — accepted as a short delay over billing everything to the system account.
  2. Agent tool: LibreChat's built-in "OpenAI Image Tools" (env IMAGE_GEN_OAI_BASEURL/IMAGE_GEN_OAI_MODEL → gateway; no admin key, members paste their own). Note LibreChat v0.8.8 renders the generated image only inside the agent conversation.
• MCP: inference-docs (tool search_docs, starts at boot) and inference-image (tool generate_image, startup: false). Members may add their own via the MCP Settings panel ("bring your own tools"). Permissions seeded via interface.mcpServers (use/create true, share false).
• Speech: TTS tinfoil/voxtral-tts (voice casual_female), STT tinfoil/whisper-large-v3-turbo, both via the gateway with the capped key.
• Web search: SearXNG (self-hosted) + keyless Keenable scraper, no reranker.

Per-member key auto-load (no API-key prompts)

Members never enter an API key. Chat and image generation resolve each member's own portal-provisioned key automatically:

1. The portal provisions a per-member LiteLLM key at onboarding and stores it (members.key_token).
2. GET /admin/librechat/keys on the portal returns those keys for ACTIVE members — authenticated by the dedicated X-Sync-Token header (LIBRECHAT_SYNC_TOKEN, set on BOTH apps; not the admin token).
3. sync-keys.js (this repo, deployed to /app/data/sync-keys.js) fetches that feed and upserts, per member with a sandbox account:
   • keys collection: {userId, name: "Inference Co-op", value: encrypt(JSON.stringify({apiKey}))} — consumed by the chat endpoint (apiKey: "user_provided" in the yaml).
   • pluginauths collection: {userId, pluginKey: "mcp_inference-image", authField: "IC_API_KEY", value: encrypt(key)} — consumed by the MCP customUserVars config so image generation bills to the member's own team. Encryption uses LibreChat's own scheme (CREDS_KEY/CREDS_IV, AES-256-CBC via @librechat/data-schemas encrypt()).
4. Members who haven't logged in to the sandbox yet are skipped; the next run seeds them.

Scheduling (in this app, since 2026-10-25)

CloudronManifest.json declares the task in addons.scheduler:

"scheduler": {
  "librechat_key_sync": {
    "schedule": "*/10 * * * *",
    "command": "cd /app/code/librechat/api && node /app/code/sync-keys.js"
  }
}

The script is baked into the image (COPY sync-keys.js /app/code/…), so it cannot be lost with a volume restore and cannot drift from this repo.

Ordering matters on the first deploy of this change: build the image → deploy (pin the new tag in the manifest first — the manifest's repointed command ships in the same cloudron update) → verify (./release.sh verify checks that the command path exists in the running image) → only then delete the volume copy /app/data/sync-keys.js, because rolling back to the previous image tag needs it back. After that, a change to this script requires a rebuild, not a volume push.

Seeding therefore depends on nothing outside this app — it previously ran from a steward cron on another machine, which made a member-facing function depend on that machine's uptime and paths.

Two consequences of how the scheduler addon works, both learned the hard way:

• Each task runs in its own container. The app's process env is absent there, and /proc/1/environ belongs to the task launcher, not to LibreChat — so CREDS_KEY/CREDS_IV come from /app/data/.librechat-creds (root:cloudron 0640), written from the live app env:

  cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec \
    --app 4023ccbf-d57a-4500-97df-14de9cf3516e \
    -- sh -c 'umask 077; tr "\0" "\n" < /proc/1/environ | grep -E "^CREDS_(KEY|IV)=" > /app/data/.librechat-creds; chown root:cloudron /app/data/.librechat-creds; chmod 640 /app/data/.librechat-creds'
  

  If the app's key is ever rotated, re-run that. A stale key does not corrupt anything — the guard below refuses to write.

• It fails closed. sync-keys.js writes nothing unless CREDS_KEY first decrypts an existing key row (proof it is the app's real key) and unless each fresh ciphertext round-trips back to the plaintext. A credential problem exits non-zero, logs FATAL: ..., and writes /app/data/key-sync-diag.txt (uid, cwd, which env sources were readable) instead of overwriting working member keys.

Manual run (unchanged, useful after an onboarding wave):

cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec \
  --app 4023ccbf-d57a-4500-97df-14de9cf3516e \
  -- bash -c "cd /app/code/librechat/api && node /app/data/sync-keys.js"

Idempotent. Verify: the app log shows the task's output (creds check: decrypt-ok … / portal feed: N active member key(s) / done: N seeded, M skipped), and keys / pluginauths counts should equal the number of members who have logged in to the sandbox.

Output-redaction gotcha: terminal output renders values after apiKey: as *** even when the file contains user_provided or ${ENV_REF}. Never conclude a file contains a literal placeholder from redacted output — probe lengths/counts instead (tr -cd '*' < file | wc -c, awk -F'apiKey: ' '{print length($2)}').

Known state / limitations

• The [schedules] Redis guard error is resolved by SCHEDULES_SINGLE_PROCESS=true (above).
• The RAG API not reachable warning is expected (no rag_api addon); it only affects LibreChat's own file-RAG, which we do not use.
• The inference-image MCP server logs Tools: undefined at boot — expected: with startup: false it registers at boot and spawns per-user on first use in a conversation.
• Members' chat usage bills per-member via their endpoint key; the MCP image path is currently keyless and bills to the gateway (not enforced per-member). Wiring per-member key injection into LibreChat chat is an open task.
• Repo ↔ live parity last verified 2026-10-19: librechat.yaml, mcp/server.py, mcp/image_server.py md5s match the live volume; overlay image registry.inference.coop/co-op/librechat-custom:20261019-fc9b11e is deployed (container asset md5s match this repo).

PWA lessons (2026-10-22 — icon/band debugging)

• Builder COPY layers can be stale: after editing any asset, build with --no-cache and VERIFY the served file's md5 against the repo before reporting success. (The 20261022-df55d7f build shipped a stale maskable icon this way — the repo was right, the image wasn't.)
• PWA icon URLs must carry a per-build ?v=: they are served with max-age=172800, and remove/re-add of the PWA does not clear the phone's HTTP cache — launchers keep rendering the previous build's icon bytes. patch_index.py injects the query into every manifest icon src per build.
• Maskable icons: logo must sit inside the safe zone — ~66% of the canvas on a solid brand background. Full-bleed logos get cropped ("zoomed in").
• Firefox Android: crossorigin="use-credentials" on the manifest link breaks icon resolution (blank launcher icon) — patch_index.py strips it.
• Theme bars: the manifest theme_color is one static value; Chromium's nav bar and all Firefox bars use it even when the meta says otherwise. The durable fix is viewport-fit=cover + standalone-mode safe-area padding (custom.css) so the bands show the theme-correct page background; custom2.js additionally syncs one meta theme-color to the app's resolved theme.