librechat-custom/README.md

# LibreChat sandbox — Inference Co-op customization

Source of truth for the co-op's chat instance at
**chat.inference.coop** (Cloudron app `4023ccbf-d57a-4500-97df-14de9cf3516e`,
appstore package `ai.librechat.cloudronapp@1.4.5` — upstream LibreChat is
unchanged at **v0.8.8**; the bump is uv 0.13.1 → 0.13.5 — on base image
`cloudron/ai.librechat.cloudronapp:202610230221180000`).
`librechat-test.inference.coop` was the pre-swap hostname (the swap to
`chat.` happened 2026-10-21; the test alias is still not provisioned).

The deployment has **two independent layers** — know which one you are changing:

1. **Persistent volume** (`/app/data/…`) — behavior: `librechat.yaml`, `env`,
   `mcp/*.py`. Survives image updates. Edited via `cloudron exec`, then
   mirrored into this repo (see "Volume changes" below).
2. **Overlay image** (this repo's Dockerfile) — appearance only: logo,
   favicons, `custom.css`, `custom2.js`, `<title>`. Rebuilt and redeployed to
   change branding.

The md5s of `librechat.yaml`, `mcp/server.py`, and `mcp/image_server.py` in
this repo are kept byte-identical to the live volume files. Verify with:

```bash
APP=4023ccbf-d57a-4500-97df-14de9cf3516e
cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec --app $APP \
  -- md5sum /app/data/librechat.yaml /app/data/mcp/server.py /app/data/mcp/image_server.py
md5sum librechat.yaml mcp/server.py mcp/image_server.py
```

## Architecture: what lives where

| Layer | Location | Survives image update? | Repo mirror |
|---|---|---|---|
| App config | `/app/data/librechat.yaml` (volume) | yes | `librechat.yaml` |
| Env vars (env-file wins!) | `/app/data/env` (volume, sourced LAST by start.sh) | yes | documented below |
| Env vars (Cloudron-managed) | `cloudron env` (process env) | yes | documented below |
| MCP servers | `/app/data/mcp/server.py` + `mcp/image_server.py` (volume) | yes | `mcp/` |
| Per-member key seed | `sync-keys.js` baked into the image at `/app/code/sync-keys.js` (run by the manifest's `scheduler` task) | no — rebuilt per image | `sync-keys.js` |
| Brand assets + branding CSS/JS | baked into image overlay | no — rebuilt per image | `assets/` |
| `CloudronManifest.json` | repo only (controls what `cloudron update` deploys) | n/a | official appstore manifest **except `dockerImage`** |

**Precedence gotcha:** `start.sh` sources `/run/librechat/env` AFTER the
process env, and `/app/data/env` is appended into it — so a value defined in
`/app/data/env` **overrides** the same `cloudron env` variable. That is why
`ALLOW_EMAIL_LOGIN=false` must be set in `/app/data/env`, not (only) via
`cloudron env`.

## Volume changes (librechat.yaml, mcp/*.py, env)

> **Volume warning:** edits you make to the files in this repo do **not**
> deploy with the image. The image never touches `/app/data`. Apply changes
> to the live files first, then mirror them here.

Deploy a config change (no image rebuild needed):

```bash
B64=$(base64 -w0 librechat.yaml)
cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec \
  --app 4023ccbf-d57a-4500-97df-14de9cf3516e \
  -- bash -c "echo '$B64' | base64 -d > /app/data/librechat.yaml"
cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" restart \
  --app 4023ccbf-d57a-4500-97df-14de9cf3516e --timeout 900
```

- Always deploy via base64 — heredocs and `sed` inserts have corrupted this
  file twice (literal `$(...)` / bad indentation → LibreChat silently falls
  back to defaults with only one log line).
- Always `diff` the repo file against the live one after deploying.
- **Reading large live files back over `cloudron exec`** cannot be done with
  a plain `cat` (a single large stdout stream resets the connection,
  `ECONNRESET`). Pipe it small instead:

  ```bash
  cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec \
    --app 4023ccbf-d57a-4500-97df-14de9cf3516e \
    -- bash -c "gzip -c /app/data/librechat.yaml | base64 -w0 | fold -w 100" > /tmp/live.b64
  tr -d '\n' < /tmp/live.b64 | base64 -d | gunzip > /tmp/live-librechat.yaml
  ```

Verify after every config change:

```bash
cloudron logs --app 4023ccbf-d57a-4500-97df-14de9cf3516e --lines 60 | grep -iE 'custom config|error|invalid|MCP'
curl -s https://librechat-test.inference.coop/api/config   # appTitle, emailLoginEnabled
```

## Build + deploy the image (branding only)

> **Two traps have each silently eaten deploys of this overlay.** Both look
> like success. Run the verification block below after every deploy.

1. **`cloudron update --image <tag>` is IGNORED when this directory contains
   `CloudronManifest.json`.** The CLI deploys `manifest.dockerImage` instead —
   while still printing your `--image` value in its output, so it *looks*
   right. The tag you build MUST be written into
   `CloudronManifest.json.dockerImage` before updating.
2. **The remote builder's cache can serve stale COPY layers even when file
   content changed** (observed 2026-10-19: six consecutive "Build successful"
   runs produced images with old assets). Build with `--no-cache` whenever
   anything under `assets/` changed.

Full deploy sequence:

```bash
cd /path/to/librechat-custom      # the dir containing CloudronManifest.json
# 1. edit assets/ … then:
git add -A && git commit -m "…" && git push
SHA=$(git rev-parse --short HEAD)
# 2. build fresh layers (no-cache after asset edits)
cloudron builder build --no-cache \
  --repository registry.inference.coop/co-op/librechat-custom \
  --tag 20261019-$SHA
# 3. pin the manifest to the new tag — the ONLY field you may change
#    (edit "dockerImage" in CloudronManifest.json to :20261019-$SHA)
git add CloudronManifest.json && git commit -m "Pin manifest to $SHA" && git push
# 4. deploy — the manifest drives it
cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" update \
  --app 4023ccbf-d57a-4500-97df-14de9cf3516e --force
```

`./release.sh release` runs this same sequence with both traps handled — it
refuses to build from a dirty tree, always passes `--no-cache`, writes the tag
into the manifest before deploying, then runs the verification block. Builds are
remote (the registry's BuildKit does the work), so the only requirement is a
registry login on the host running the command; `verify`, `volume` and
`pin`/`deploy` need only `CLOUDRON_TOKEN`. `./release.sh verify` is safe to run
any time. If `build` reports no registry login, it prints both ways to bootstrap
one (a narrow app password scoped to the registry app, or the CLI device flow).

Rollback: repoint `CloudronManifest.json.dockerImage` to the previous tag
(every build is tagged `20261019-<git-sha>`; the base image
`cloudron/ai.librechat.cloudronapp:<tag>` is the pre-overlay fallback) and
rerun step 4.

### Verify after every image deploy (all four, or it didn't happen)

```bash
APP=4023ccbf-d57a-4500-97df-14de9cf3516e
# 1. What is ACTUALLY running? (CLI update output is not evidence)
curl -s -H "Authorization: Bearer $CLOUDRON_TOKEN" \
  https://my.inference.coop/api/v1/apps/$APP \
  | python3 -c "import json,sys; print(json.load(sys.stdin)['manifest']['dockerImage'])"
# 2. Container files match the repo (catches stale builder cache)
cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec --app $APP \
  -- md5sum /app/code/librechat/client/dist/assets/custom.css
md5sum assets/custom.css
# 3. Served HTML carries the injection
curl -s https://chat.inference.coop/login | grep -Eo 'assets/custom2?\.js|assets/custom\.css'
# 4. Rendered DOM shows it (browser devtools or a browser tool). The DEPLOYED
#    palette is the GREEN WASH with white text areas — an earlier version of this
#    block asserted cream (rgb(250,248,245)), which would have led you to "fix"
#    a correct deploy:
#      computed background of div.bg-surface-primary == rgb(234, 243, 234);
#      chat text areas / composer == rgb(255, 255, 255);
#      dark scheme == rgb(34, 39, 31);
#      the "Don't have an account?" <p> computes display: none.
# 5. The scheduler task's script exists in THIS image — a manifest repoint
#    deployed before its image lands leaves the key seed broken on schedule:
cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec --app $APP \
  -- test -f /app/data/sync-keys.js && echo "seed path present"
```

`./release.sh verify` runs checks 1-5 plus the seed task's own log lines and the
running script's revision hash, and exits non-zero on any failure.

### Appstore pin bumps (base image)

The overlay's `FROM` is pinned to the exact appstore image the app runs. Bumping
it is a two-line change in this repo (`Dockerfile` `FROM` + the manifest's
`version`/`changelog` copied from the official manifest) plus a rebuild, but it
re-runs every build-time patch against a new base layer, so treat the overlay as
the risk, not the bump.

```bash
# what the appstore now offers (official manifest, authoritative)
curl -s https://api.cloudron.io/api/v1/apps/ai.librechat.cloudronapp \
  | python3 -c "import json,sys; m=json.load(sys.stdin)['data']['manifest']; print(m['version'], m.get('upstreamVersion'), m['changelog'].strip())"
# diff it against our copy — every field must match except dockerImage (+ scheduler)
# then: rebuild, pin, deploy, ./release.sh verify, and re-check the branding block above
```

Rollback is a one-line `dockerImage` repoint to the previous tag (built images
stay in the registry), then `./release.sh deploy`.

### `CloudronManifest.json` rules

- The **only** field you may change is `dockerImage` — it must point at the
  overlay tag you want deployed.
- Everything else stays a byte-identical copy of the OFFICIAL appstore
  manifest. A hand-edited manifest that omits the addons (`postgresql`,
  `mongodb`, `oidc`, `sendmail`, `localstorage`) strips them from the running
  app and crash-loops it (`mkdir: Read-only file system`). That broke the
  sandbox on 2026-10-18 — the manifest, not the overlay, was the bug.
- When the appstore package updates, refresh the file with a new copy from
  the Cloudron API (`GET /api/v1/apps/<id>` → `.manifest`), repoint
  `dockerImage` to your rebuilt overlay, and bump the `FROM` tag in the
  Dockerfile to the new appstore `dockerImage` (pinned — never a floating tag).

## Env vars (live state)

Set in `/app/data/env` (volume — these WIN over cloudron env):

- `APP_TITLE="Inference Co-op"` (was `"LibreChat"`)
- `ALLOW_EMAIL_LOGIN=false` (was `true`) — password login removed; Cloudron
  SSO (OIDC) is the only login path
- `SCHEDULES_SINGLE_PROCESS=true` — sandbox runs a single replica; silences
  the scheduler's Redis guard error

Set via `cloudron env set` (process env):

- `TTS_API_KEY` / `STT_API_KEY` — **dedicated LiteLLM virtual key** (alias
  `librechat-speech`, $5 / 30d budget, restricted to `tinfoil/voxtral-tts` +
  `tinfoil/whisper-large-v3-turbo`). Regenerate with the gateway master key:
  `POST /key/generate {"key_alias":"librechat-speech","models":[...],"max_budget":5,"budget_duration":"30d"}`
  — never put the master key here.
- `IMAGE_GEN_OAI_BASEURL=https://gateway.inference.coop/v1` and
  `IMAGE_GEN_OAI_MODEL=regolo/qwen-image` — the built-in agent "OpenAI Image
  Tools" points at the co-op gateway. No admin key is configured; members
  using the AGENT path paste their own key in the tool UI (v0.8.8
  limitation). The MCP path below needs no key at all.
- `SEARXNG_INSTANCE_URL=https://search.inference.coop` — referenced from the
  YAML as `${SEARXNG_INSTANCE_URL}` (self-hosted search; keyless JSON API).

## Branding overlay (custom.css / custom2.js)

Injected into the built SPA by the Dockerfile (`<link>` + `<script>` after
the `<title>` sed). What it does:

**Palette (final)**: light `--surface-primary` is `250 248 245` (#faf8f5
cream) — the dashboard's true page background (the green wash #eaf3ea was
tried as the full background and rejected as too dark; on the dashboard the
green is accent-only). Sidebar/composer box = white cards, tertiary = #f4f2ec
(dashboard chips), hover = #eaf3ea green wash, border = #e4e0d8, chat text
areas (`--surface-chat`, `--presentation`) = white. Green (#5b8c5a family)
stays on accents: buttons, links, focus rings. Loading flash = cream.
Dark mode: deep pine + green-light accents. Homepage variables remain the
reference for other surfaces.

**Fonts**: Figtree (the homepage's OFL font) is self-hosted from the design
kit (`assets/fonts/figtree-*.woff2` + OFL.txt, served at `/assets/fonts/`)
via `@font-face` in `custom.css` with a `font-family` override — LibreChat's
built-in default is Inter. Never load fonts from a CDN.

**Footer**: build-time text (no-JS fallback) + runtime link upgrade —
`custom2.js` rewrites the footer to
"[Inference Cooperative](https://inference.coop) · Our AI, governed together"
with a `dataset` guard so the MutationObserver doesn't loop.

**UI strings**: `patch_ui_strings.py` (build-time, idempotent, non-fatal)
renames the "MCP Servers" button to **"Tools"** and replaces LibreChat's
footer tagline ("Every AI for Everyone.") with the co-op footer
("Inference Cooperative · Our AI, governed together · inference.coop"),
deleting stale `.gz`/`.br` siblings of edited files. Add new string
replacements to the `REPLACEMENTS` list.

**Palette source of truth**: the co-op homepage's own CSS variables —
`--cream #faf8f5` (surfaces), `--wash #eef4ea` (panels/hover),
`--green-deep #5b8c5a` (primary/submit), `--green-mid #6ba86b` (hover),
`--green-light #7ab87a` (dark-mode accent), `--ink #2d3327` (text),
`--muted #6b7a62` (secondary text), `--border #e8e2d8`. The `html.light` /
`html.dark` blocks in `custom.css` map these onto LibreChat's Tailwind
variables; keep them in sync when the homepage palette changes.

**Logo & icons**: `assets/logo.svg` is the GREEN pine-dots vector (extracted
from the homepage's inline SVG — keep the `xmlns` attribute, a standalone
.svg is invalid without it). The earlier white-plate PNG variant was
invisible on the cream background and only "flashed" against the dark
loading screen. Favicons/PWA icons are the cream-plate + green-dots design
rasterized from `design-assets/favicon.svg`. The Dockerfile also cache-busts
the logo URL in the client bundle (`assets/logo.svg?v=3`) and deletes the
stale precompressed `.gz`/`.br` siblings — bump `v=` whenever the logo
changes, or browsers/service workers may keep serving an old cached copy.

- **Colors** — LibreChat's Tailwind theme resolves every color through CSS
  variables set on `html.light` / `html.dark` (space-separated RGB):
  `--surface-primary`, `--surface-secondary/-tertiary/-hover/-submit/-submit-hover`,
  `--accent-primary/-hover`, `--ring-primary`, `--link`,
  `--text-primary/-secondary`, `--border-light`. `custom.css` overrides them
  (light: cream `250 248 245` surfaces + co-op green `91 140 90` accents;
  dark: matching co-op dark palette). **To rebrand, edit the `html.light` /
  `html.dark` blocks** — do not invent class names; `.login-page`,
  `.primary` etc. do not exist in the DOM. Read current values in devtools
  via `getComputedStyle(document.documentElement).getPropertyValue('--surface-primary')`.
- **No sign-up prompt** — the "Don't have an account? Sign up" line is
  `<p><a href="/register">Sign up</a></p>`; hidden with
  `p:has(> a[href*="register"]) { display: none !important; }` (hiding only
  the `<a>` leaves the text). Any residual register/signup links are hidden
  too.
- **`custom2.js`** — the auth form is client-rendered AFTER `load`, so a
  plain load-time query matches nothing; a **MutationObserver** re-checks on
  every DOM change and hides the sign-up paragraph whenever React renders it.

There is also an `interface.theme` block in `librechat.yaml` — it is
**forward-compatible only**: the theme engine ships in LibreChat releases
AFTER 0.8.8 and the block is silently ignored today. When the appstore
package bundles a newer LibreChat, revisit whether the CSS overlay is still
needed.

## Feature notes (LibreChat v0.8.8)

- **Chat models**: custom endpoint "Inference Co-op" → gateway, 8 chat
  modelSpecs (emoji convention shared with models.md). `regolo/qwen-image`
  is deliberately NOT in the chat picker (image models are not chat models;
  chat-calling it was the source of the old 500s).
- **Image generation — two paths**:
  1. **MCP tool (default, no key ever)**: `inference-image` server (tool
     `generate_image`) — members select it in the composer's **+** palette
     and just ask. Per-member billing via `customUserVars.IC_API_KEY` (seeded
     from the portal) + `env: IC_API_KEY: "{{IC_API_KEY}}"`; a system-key
     fallback (`system-image-fallback`, $5/30d) covers the rare empty-value
     case. The `customUserVars` gate blocks a not-yet-seeded member until the
     next sync (≤10 min) — accepted as a short delay over billing everything
     to the system account.
  2. **Agent tool**: LibreChat's built-in "OpenAI Image Tools" (env
     `IMAGE_GEN_OAI_BASEURL`/`IMAGE_GEN_OAI_MODEL` → gateway; no admin key,
     members paste their own). Note LibreChat v0.8.8 renders the generated
     image only inside the agent conversation.
- **MCP**: `inference-docs` (tool `search_docs`, starts at boot) and
  `inference-image` (tool `generate_image`, `startup: false`). Members may
  add their own via the MCP Settings panel ("bring your own tools").
  Permissions seeded via `interface.mcpServers` (use/create true, share false).
- **Speech**: TTS `tinfoil/voxtral-tts` (voice `casual_female`), STT
  `tinfoil/whisper-large-v3-turbo`, both via the gateway with the capped key.
- **Web search**: SearXNG (self-hosted) + keyless Keenable scraper, no reranker.


## Per-member key auto-load (no API-key prompts)

Members never enter an API key. Chat and image generation resolve each
member's own portal-provisioned key automatically:

1. The portal provisions a per-member LiteLLM key at onboarding and stores it
   (`members.key_token`).
2. `GET /admin/librechat/keys` on the portal returns those keys for ACTIVE
   members — authenticated by the dedicated `X-Sync-Token` header
   (`LIBRECHAT_SYNC_TOKEN`, set on BOTH apps; not the admin token).
3. `sync-keys.js` (this repo, deployed to `/app/data/sync-keys.js`) fetches
   that feed and upserts, per member with a sandbox account:
   - `keys` collection: `{userId, name: "Inference Co-op", value: encrypt(JSON.stringify({apiKey}))}`
     — consumed by the chat endpoint (`apiKey: "user_provided"` in the yaml).
   - `pluginauths` collection: `{userId, pluginKey: "mcp_inference-image",
     authField: "IC_API_KEY", value: encrypt(key)}` — consumed by the MCP
     `customUserVars` config so image generation bills to the member's own
     team.
   Encryption uses LibreChat's own scheme (`CREDS_KEY`/`CREDS_IV`, AES-256-CBC
   via `@librechat/data-schemas` `encrypt()`).
4. Members who haven't logged in to the sandbox yet are skipped; the next run
   seeds them.

### Scheduling (in this app, since 2026-10-25)

`CloudronManifest.json` declares the task in `addons.scheduler`:

```json
"scheduler": {
  "librechat_key_sync": {
    "schedule": "*/10 * * * *",
    "command": "cd /app/code/librechat/api && node /app/code/sync-keys.js"
  }
}
```

The script is **baked into the image** (`COPY sync-keys.js /app/code/…`), so it
cannot be lost with a volume restore and cannot drift from this repo.

**Ordering matters on the first deploy of this change:** build the image → deploy
(pin the new tag in the manifest first — the manifest's repointed command ships
in the same `cloudron update`) → verify (`./release.sh verify` checks that the
command path exists in the running image) → only then delete the volume copy
`/app/data/sync-keys.js`, because rolling back to the previous image tag needs it
back. After that, a change to this script requires a rebuild, not a volume push.

Seeding therefore depends on nothing outside this app — it previously ran from a
steward cron on another machine, which made a member-facing function depend on
that machine's uptime and paths.

Two consequences of how the scheduler addon works, both learned the hard way:

- **Each task runs in its own container.** The app's process env is absent there,
  and `/proc/1/environ` belongs to the task launcher, not to LibreChat — so
  `CREDS_KEY`/`CREDS_IV` come from `/app/data/.librechat-creds` (root:cloudron
  `0640`), written from the live app env:

  ```bash
  cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec \
    --app 4023ccbf-d57a-4500-97df-14de9cf3516e \
    -- sh -c 'umask 077; tr "\0" "\n" < /proc/1/environ | grep -E "^CREDS_(KEY|IV)=" > /app/data/.librechat-creds; chown root:cloudron /app/data/.librechat-creds; chmod 640 /app/data/.librechat-creds'
  ```

  If the app's key is ever rotated, re-run that. A stale key does not corrupt
  anything — the guard below refuses to write.
- **It fails closed.** `sync-keys.js` writes nothing unless `CREDS_KEY` first
  decrypts an existing key row (proof it is the app's real key) and unless each
  fresh ciphertext round-trips back to the plaintext. A credential problem exits
  non-zero, logs `FATAL: ...`, and writes `/app/data/key-sync-diag.txt` (uid, cwd,
  which env sources were readable) instead of overwriting working member keys.

Manual run (unchanged, useful after an onboarding wave):

```bash
cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec \
  --app 4023ccbf-d57a-4500-97df-14de9cf3516e \
  -- bash -c "cd /app/code/librechat/api && node /app/data/sync-keys.js"
```

Idempotent. Verify: the app log shows the task's output
(`creds check: decrypt-ok …` / `portal feed: N active member key(s)` /
`done: N seeded, M skipped`), and `keys` / `pluginauths` counts should equal the
number of members who have logged in to the sandbox.

> **Output-redaction gotcha:** terminal output renders values after
> `apiKey:` as `***` even when the file contains `user_provided` or
> `${ENV_REF}`. Never conclude a file contains a literal placeholder from
> redacted output — probe lengths/counts instead
> (`tr -cd '*' < file | wc -c`, `awk -F'apiKey: ' '{print length($2)}'`).

## Known state / limitations

- The `[schedules]` Redis guard error is resolved by
  `SCHEDULES_SINGLE_PROCESS=true` (above).
- The `RAG API not reachable` warning is expected (no rag_api addon); it only
  affects LibreChat's own file-RAG, which we do not use.
- The `inference-image` MCP server logs `Tools: undefined` at boot —
  expected: with `startup: false` it registers at boot and spawns per-user
  on first use in a conversation.
- Members' chat usage bills per-member via their endpoint key; the MCP image
  path is currently keyless and bills to the gateway (not enforced
  per-member). Wiring per-member key injection into LibreChat chat is an
  open task.
- Repo ↔ live parity last verified 2026-10-19: `librechat.yaml`,
  `mcp/server.py`, `mcp/image_server.py` md5s match the live volume; overlay
  image `registry.inference.coop/co-op/librechat-custom:20261019-fc9b11e` is
  deployed (container asset md5s match this repo).

## PWA lessons (2026-10-22 — icon/band debugging)

- **Builder COPY layers can be stale**: after editing any asset, build with
  `--no-cache` and VERIFY the served file's md5 against the repo before
  reporting success. (The 20261022-df55d7f build shipped a stale maskable icon
  this way — the repo was right, the image wasn't.)
- **PWA icon URLs must carry a per-build `?v=`**: they are served with
  `max-age=172800`, and remove/re-add of the PWA does not clear the phone's
  HTTP cache — launchers keep rendering the previous build's icon bytes.
  patch_index.py injects the query into every manifest icon src per build.
- **Maskable icons**: logo must sit inside the safe zone — ~66% of the canvas
  on a solid brand background. Full-bleed logos get cropped ("zoomed in").
- **Firefox Android**: `crossorigin="use-credentials"` on the manifest link
  breaks icon resolution (blank launcher icon) — patch_index.py strips it.
- **Theme bars**: the manifest `theme_color` is one static value; Chromium's
  nav bar and all Firefox bars use it even when the meta says otherwise. The
  durable fix is `viewport-fit=cover` + standalone-mode safe-area padding
  (custom.css) so the bands show the theme-correct page background; custom2.js
  additionally syncs one meta `theme-color` to the app's resolved theme.