# LibreChat sandbox — Inference Co-op customization

Source of truth for the LibreChat evaluation instance at
**librechat-test.inference.coop** (Cloudron app `4023ccbf-d57a-4500-97df-14de9cf3516e`,
appstore package `ai.librechat.cloudronapp@1.4.1`, which bundles **LibreChat
v0.8.8** on image `cloudron/ai.librechat.cloudronapp:202610150224320000`).

The deployment has **two independent layers** — know which one you are changing:

1. **Persistent volume** (`/app/data/…`) — behavior: `librechat.yaml`, `env`,
   `mcp/*.py`. Survives image updates. Edited via `cloudron exec`, then
   mirrored into this repo (see "Volume changes" below).
2. **Overlay image** (this repo's Dockerfile) — appearance only: logo,
   favicons, `custom.css`, `custom2.js`, `<title>`. Rebuilt and redeployed to
   change branding.

The md5s of `librechat.yaml`, `mcp/server.py`, and `mcp/image_server.py` in
this repo are kept byte-identical to the live volume files. Verify with:

```bash
APP=4023ccbf-d57a-4500-97df-14de9cf3516e
cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec --app $APP \
  -- md5sum /app/data/librechat.yaml /app/data/mcp/server.py /app/data/mcp/image_server.py
md5sum librechat.yaml mcp/server.py mcp/image_server.py
```

## Architecture: what lives where

| Layer | Location | Survives image update? | Repo mirror |
|---|---|---|---|
| App config | `/app/data/librechat.yaml` (volume) | yes | `librechat.yaml` |
| Env vars (env-file wins!) | `/app/data/env` (volume, sourced LAST by start.sh) | yes | documented below |
| Env vars (Cloudron-managed) | `cloudron env` (process env) | yes | documented below |
| MCP servers | `/app/data/mcp/server.py` + `mcp/image_server.py` (volume) | yes | `mcp/` |
| Brand assets + branding CSS/JS | baked into image overlay | no — rebuilt per image | `assets/` |
| `CloudronManifest.json` | repo only (controls what `cloudron update` deploys) | n/a | official appstore manifest **except `dockerImage`** |

**Precedence gotcha:** `start.sh` sources `/run/librechat/env` AFTER the
process env, and `/app/data/env` is appended into it — so a value defined in
`/app/data/env` **overrides** the same `cloudron env` variable. That is why
`ALLOW_EMAIL_LOGIN=false` must be set in `/app/data/env`, not (only) via
`cloudron env`.

## Volume changes (librechat.yaml, mcp/*.py, env)

> **Volume warning:** edits you make to the files in this repo do **not**
> deploy with the image. The image never touches `/app/data`. Apply changes
> to the live files first, then mirror them here.

Deploy a config change (no image rebuild needed):

```bash
B64=$(base64 -w0 librechat.yaml)
cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec \
  --app 4023ccbf-d57a-4500-97df-14de9cf3516e \
  -- bash -c "echo '$B64' | base64 -d > /app/data/librechat.yaml"
cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" restart \
  --app 4023ccbf-d57a-4500-97df-14de9cf3516e --timeout 900
```

- Always deploy via base64 — heredocs and `sed` inserts have corrupted this
  file twice (literal `$(...)` / bad indentation → LibreChat silently falls
  back to defaults with only one log line).
- Always `diff` the repo file against the live one after deploying.
- **Reading large live files back over `cloudron exec`** cannot be done with
  a plain `cat` (a single large stdout stream resets the connection,
  `ECONNRESET`). Pipe it small instead:

  ```bash
  cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec \
    --app 4023ccbf-d57a-4500-97df-14de9cf3516e \
    -- bash -c "gzip -c /app/data/librechat.yaml | base64 -w0 | fold -w 100" > /tmp/live.b64
  tr -d '\n' < /tmp/live.b64 | base64 -d | gunzip > /tmp/live-librechat.yaml
  ```

Verify after every config change:

```bash
cloudron logs --app 4023ccbf-d57a-4500-97df-14de9cf3516e --lines 60 | grep -iE 'custom config|error|invalid|MCP'
curl -s https://librechat-test.inference.coop/api/config   # appTitle, emailLoginEnabled
```

## Build + deploy the image (branding only)

> **Two traps have each silently eaten deploys of this overlay.** Both look
> like success. Run the verification block below after every deploy.

1. **`cloudron update --image <tag>` is IGNORED when this directory contains
   `CloudronManifest.json`.** The CLI deploys `manifest.dockerImage` instead —
   while still printing your `--image` value in its output, so it *looks*
   right. The tag you build MUST be written into
   `CloudronManifest.json.dockerImage` before updating.
2. **The remote builder's cache can serve stale COPY layers even when file
   content changed** (observed 2026-10-19: six consecutive "Build successful"
   runs produced images with old assets). Build with `--no-cache` whenever
   anything under `assets/` changed.

Full deploy sequence:

```bash
cd /path/to/librechat-custom      # the dir containing CloudronManifest.json
# 1. edit assets/ … then:
git add -A && git commit -m "…" && git push
SHA=$(git rev-parse --short HEAD)
# 2. build fresh layers (no-cache after asset edits)
cloudron builder build --no-cache \
  --repository registry.inference.coop/co-op/librechat-custom \
  --tag 20261019-$SHA
# 3. pin the manifest to the new tag — the ONLY field you may change
#    (edit "dockerImage" in CloudronManifest.json to :20261019-$SHA)
git add CloudronManifest.json && git commit -m "Pin manifest to $SHA" && git push
# 4. deploy — the manifest drives it
cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" update \
  --app 4023ccbf-d57a-4500-97df-14de9cf3516e --force
```

Rollback: repoint `CloudronManifest.json.dockerImage` to the previous tag
(every build is tagged `20261019-<git-sha>`; the base image
`cloudron/ai.librechat.cloudronapp:<tag>` is the pre-overlay fallback) and
rerun step 4.

### Verify after every image deploy (all four, or it didn't happen)

```bash
APP=4023ccbf-d57a-4500-97df-14de9cf3516e
# 1. What is ACTUALLY running? (CLI update output is not evidence)
curl -s -H "Authorization: Bearer $CLOUDRON_TOKEN" \
  https://my.inference.coop/api/v1/apps/$APP \
  | python3 -c "import json,sys; print(json.load(sys.stdin)['app']['manifest']['dockerImage'])"
# 2. Container files match the repo (catches stale builder cache)
cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec --app $APP \
  -- md5sum /app/code/librechat/client/dist/assets/custom.css
md5sum assets/custom.css
# 3. Served HTML carries the injection
curl -s https://librechat-test.inference.coop/login | grep -Eo 'assets/custom2?\.js|assets/custom\.css'
# 4. Rendered DOM shows it (browser devtools or a browser tool):
#    computed background of div.bg-surface-primary == rgb(250, 248, 245);
#    the "Don't have an account?" <p> computes display: none.
```

### `CloudronManifest.json` rules

- The **only** field you may change is `dockerImage` — it must point at the
  overlay tag you want deployed.
- Everything else stays a byte-identical copy of the OFFICIAL appstore
  manifest. A hand-edited manifest that omits the addons (`postgresql`,
  `mongodb`, `oidc`, `sendmail`, `localstorage`) strips them from the running
  app and crash-loops it (`mkdir: Read-only file system`). That broke the
  sandbox on 2026-10-18 — the manifest, not the overlay, was the bug.
- When the appstore package updates, refresh the file with a new copy from
  the Cloudron API (`GET /api/v1/apps/<id>` → `.manifest`), repoint
  `dockerImage` to your rebuilt overlay, and bump the `FROM` tag in the
  Dockerfile to the new appstore `dockerImage` (pinned — never a floating tag).

## Env vars (live state)

Set in `/app/data/env` (volume — these WIN over cloudron env):

- `APP_TITLE="Inference Co-op"` (was `"LibreChat"`)
- `ALLOW_EMAIL_LOGIN=false` (was `true`) — password login removed; Cloudron
  SSO (OIDC) is the only login path
- `SCHEDULES_SINGLE_PROCESS=true` — sandbox runs a single replica; silences
  the scheduler's Redis guard error

Set via `cloudron env set` (process env):

- `TTS_API_KEY` / `STT_API_KEY` — **dedicated LiteLLM virtual key** (alias
  `librechat-speech`, $5 / 30d budget, restricted to `tinfoil/voxtral-tts` +
  `tinfoil/whisper-large-v3-turbo`). Regenerate with the gateway master key:
  `POST /key/generate {"key_alias":"librechat-speech","models":[...],"max_budget":5,"budget_duration":"30d"}`
  — never put the master key here.
- `IMAGE_GEN_OAI_BASEURL=https://gateway.inference.coop/v1` and
  `IMAGE_GEN_OAI_MODEL=regolo/qwen-image` — the built-in agent "OpenAI Image
  Tools" points at the co-op gateway. No admin key is configured; members
  using the AGENT path paste their own key in the tool UI (v0.8.8
  limitation). The MCP path below needs no key at all.
- `SEARXNG_INSTANCE_URL=https://search.inference.coop` — referenced from the
  YAML as `${SEARXNG_INSTANCE_URL}` (self-hosted search; keyless JSON API).

## Branding overlay (custom.css / custom2.js)

Injected into the built SPA by the Dockerfile (`<link>` + `<script>` after
the `<title>` sed). What it does:

- **Colors** — LibreChat's Tailwind theme resolves every color through CSS
  variables set on `html.light` / `html.dark` (space-separated RGB):
  `--surface-primary`, `--surface-secondary/-tertiary/-hover/-submit/-submit-hover`,
  `--accent-primary/-hover`, `--ring-primary`, `--link`,
  `--text-primary/-secondary`, `--border-light`. `custom.css` overrides them
  (light: cream `250 248 245` surfaces + co-op green `91 140 90` accents;
  dark: matching co-op dark palette). **To rebrand, edit the `html.light` /
  `html.dark` blocks** — do not invent class names; `.login-page`,
  `.primary` etc. do not exist in the DOM. Read current values in devtools
  via `getComputedStyle(document.documentElement).getPropertyValue('--surface-primary')`.
- **No sign-up prompt** — the "Don't have an account? Sign up" line is
  `<p><a href="/register">Sign up</a></p>`; hidden with
  `p:has(> a[href*="register"]) { display: none !important; }` (hiding only
  the `<a>` leaves the text). Any residual register/signup links are hidden
  too.
- **`custom2.js`** — the auth form is client-rendered AFTER `load`, so a
  plain load-time query matches nothing; a **MutationObserver** re-checks on
  every DOM change and hides the sign-up paragraph whenever React renders it.

There is also an `interface.theme` block in `librechat.yaml` — it is
**forward-compatible only**: the theme engine ships in LibreChat releases
AFTER 0.8.8 and the block is silently ignored today. When the appstore
package bundles a newer LibreChat, revisit whether the CSS overlay is still
needed.

## Feature notes (LibreChat v0.8.8)

- **Chat models**: custom endpoint "Inference Co-op" → gateway, 8 chat
  modelSpecs (emoji convention shared with models.md). `regolo/qwen-image`
  is deliberately NOT in the chat picker (image models are not chat models;
  chat-calling it was the source of the old 500s).
- **Image generation — two paths**:
  1. **MCP tool (default, no key ever)**: `inference-image` server (tool
     `generate_image`) — members select it in the composer's **+** palette
     and just ask. The server uses the member's `IC_API_KEY` if LibreChat
     injected one, otherwise sends a **keyless request** (the gateway bills
     the request; per-member billing is NOT enforced on this path).
     `customUserVars` were removed from the live yaml on 2026-10-19 —
     members are never asked for a key.
  2. **Agent tool**: LibreChat's built-in "OpenAI Image Tools" (env
     `IMAGE_GEN_OAI_BASEURL`/`IMAGE_GEN_OAI_MODEL` → gateway; no admin key,
     members paste their own). Note LibreChat v0.8.8 renders the generated
     image only inside the agent conversation.
- **MCP**: `inference-docs` (tool `search_docs`, starts at boot) and
  `inference-image` (tool `generate_image`, `startup: false`). Members may
  add their own via the MCP Settings panel ("bring your own tools").
  Permissions seeded via `interface.mcpServers` (use/create true, share false).
- **Speech**: TTS `tinfoil/voxtral-tts` (voice `casual_female`), STT
  `tinfoil/whisper-large-v3-turbo`, both via the gateway with the capped key.
- **Web search**: SearXNG (self-hosted) + keyless Keenable scraper, no reranker.

## Known state / limitations

- The `[schedules]` Redis guard error is resolved by
  `SCHEDULES_SINGLE_PROCESS=true` (above).
- The `RAG API not reachable` warning is expected (no rag_api addon); it only
  affects LibreChat's own file-RAG, which we do not use.
- The `inference-image` MCP server logs `Tools: undefined` at boot —
  expected: with `startup: false` it registers at boot and spawns per-user
  on first use in a conversation.
- Members' chat usage bills per-member via their endpoint key; the MCP image
  path is currently keyless and bills to the gateway (not enforced
  per-member). Wiring per-member key injection into LibreChat chat is an
  open task.
- Repo ↔ live parity last verified 2026-10-19: `librechat.yaml`,
  `mcp/server.py`, `mcp/image_server.py` md5s match the live volume; overlay
  image `registry.inference.coop/co-op/librechat-custom:20261019-fc9b11e` is
  deployed (container asset md5s match this repo).