# LibreChat sandbox maintenance

The LibreChat evaluation instance at **librechat-test.inference.coop**
(Cloudron app `4023ccbf-d57a-4500-97df-14de9cf3516e`, appstore package
`ai.librechat.cloudronapp@1.4.1`, which bundles **LibreChat v0.8.8**).

The single source of truth for the deployment is the repo
**[co-op/librechat-custom](https://git.inference.coop/co-op/librechat-custom)**
— its README is the full runbook (config deploy, image build/deploy, rollback,
env vars, branding overlay, known state). This doc is the operator-facing
summary.

## Architecture: what lives where

- **App config** — `/app/data/librechat.yaml` (persistent volume, mirrored
  byte-identically in the repo). Holds the "Inference Co-op" custom endpoint
  (gateway), the 8-model chat picker, speech, web search, and the two MCP
  servers. Volume edits deploy via `cloudron exec` (base64), never via the
  image — the image never touches `/app/data`.
- **Env file** — `/app/data/env` (persistent volume). **`start.sh` sources it
  AFTER the process env, so it wins over `cloudron env`** — that is where
  `APP_TITLE="Inference Co-op"`, `ALLOW_EMAIL_LOGIN=false` (password login
  removed; Cloudron SSO is the only login path), and
  `SCHEDULES_SINGLE_PROCESS=true` must live.
- **Other env vars** — via `cloudron env`: `TTS_API_KEY`/`STT_API_KEY`
  (dedicated LiteLLM virtual key, alias `librechat-speech`, $5/30d, audio
  models only — never the gateway master key), `IMAGE_GEN_OAI_BASEURL` +
  `IMAGE_GEN_OAI_MODEL` (agent image tool → gateway), `SEARXNG_INSTANCE_URL`.
- **MCP servers** — `/app/data/mcp/server.py` (docs search, `search_docs`,
  starts at boot) and `/app/data/mcp/image_server.py` (image generation,
  `generate_image`, `startup: false`); both mirrored in the repo under
  `mcp/`, stdlib-only Python.
- **Branding overlay** (logo, favicons, PWA icons, `<title>`, `custom.css`,
  `custom2.js`) — baked into a pinned overlay image `FROM` the official
  appstore image. Deployed tag (last verified 2026-10-19):
  `registry.inference.coop/co-op/librechat-custom:20261019-fc9b11e`.
- **Branding CSS/JS** — `custom.css` overrides LibreChat's Tailwind theme
  variables on `html.light`/`html.dark` (cream surfaces, co-op green accents)
  and hides the register/sign-up prompt via
  `p:has(> a[href*="register"])`; `custom2.js` is a MutationObserver that
  hides the prompt when React renders it (the auth form renders after
  `load`). The `interface.theme` block in the yaml remains
  forward-compatible only (inert on v0.8.8).

## Image generation (two member-facing paths)

1. **MCP tool (default, no key ever)**: the `inference-image` MCP server
   exposes `generate_image` in the composer's **+** palette — no agent setup,
   no key prompt. The server uses the member's `IC_API_KEY` if LibreChat
   injected one, otherwise sends a **keyless request** (the gateway bills the
   request; per-member billing is NOT enforced on this path).
   `customUserVars` were removed from the live yaml on 2026-10-19 — members
   are never asked for a key.
2. **Agent tool**: LibreChat's built-in "OpenAI Image Tools"
   (`IMAGE_GEN_OAI_BASEURL`/`IMAGE_GEN_OAI_MODEL` → gateway; members paste
   their own key — v0.8.8 limitation).

## Update policy

- The app is **image-sourced** (the `--force` update moved it off the
  appstore source), so Cloudron will not auto-update it. This is deliberate —
  same lockdown model as LiteLLM.
- **Image deploys**: build `--no-cache`, write the new tag into
  `CloudronManifest.json.dockerImage` (the only editable field), commit, then
  `cloudron update --app <id> --force`. **`--image` alone is IGNORED** when
  the manifest sits in the deploy directory. **The remote builder's cache can
  serve stale COPY layers even when file content changed** (six consecutive
  "Build successful" runs did on 2026-10-19) — so after every image deploy,
  verify: the Cloudron API's `manifest.dockerImage`, container-file md5 vs
  repo md5, the served HTML, and the rendered DOM. Full checklist in the
  repo README.
- **When a new appstore package ships**: pull its manifest copy into the repo
  (`GET /api/v1/apps/<id>` → `.manifest`), repoint `dockerImage` to the
  rebuilt overlay, bump the Dockerfile `FROM` to the new `manifest.dockerImage`
  (pinned), rebuild, verify in the sandbox, then update.
- **Rollback**: repoint `CloudronManifest.json.dockerImage` to the previous
  tag and rerun `cloudron update --force` (or the original appstore image).
  Cloudron also snapshots the app before every image update.

## Hard-won pitfalls (do not re-learn these)

- `cloudron update` deploys **`manifest.dockerImage`** from the manifest in
  the deploy directory — the `--image` flag alone is ignored (and the update
  output still prints the `--image` value, so it *looks* right). The repo
  manifest is a byte-identical copy of the OFFICIAL appstore manifest except
  `dockerImage`; editing anything else in it (or hand-writing one that omits
  the addons) strips `postgresql`/`mongodb`/`oidc`/`sendmail`/`localstorage`
  from the app and crash-loops it (`mkdir: Read-only file system`). That is
  what broke the sandbox on 2026-10-18.
- The remote builder's layer cache can pin COPY steps to old file content
  even across "successful" rebuilds — use `--no-cache` after asset edits and
  confirm by comparing container-file md5 against the repo file.
- Deploy `librechat.yaml` only via base64 round-trip. A `sed` insert once
  broke the YAML indentation and LibreChat **silently fell back to defaults**
  (all custom branding/models/MCP gone) with a single log line.
- Reading large files from the container over `cloudron exec` with a plain
  `cat` resets the connection (`ECONNRESET`) — pipe through
  `gzip | base64 | fold` and decode locally.
- `cloudron env set` alone does NOT disable the password form —
  `ALLOW_EMAIL_LOGIN` must be flipped in `/app/data/env` (precedence above).
- Image generation in LibreChat is the **MCP tool** (composer **+** palette);
  the agent "OpenAI Image Tools" path also exists. Image models in the chat
  picker were the source of the old 500 errors and are deliberately excluded.
- No `branding:` block exists in LibreChat's config schema, and the login
  page has no `.login-page`/`.primary` classes to target — theming happens by
  overriding the Tailwind CSS variables on `html.light`/`html.dark`
  (`--surface-primary`, `--accent-primary`, …) from the overlay `custom.css`.


## Per-member key auto-load (no API-key prompts)

Members never enter an API key. The portal provisions each member's LiteLLM
key at onboarding; the sandbox seeds it into LibreChat's user-key store:

- `GET /admin/librechat/keys` on the portal returns active members' chat keys
  (dedicated `X-Sync-Token` auth — `LIBRECHAT_SYNC_TOKEN` on both apps).
- `/app/data/sync-keys.js` (in the LibreChat container, mirrored in the repo
  as `sync-keys.js`) fetches that feed and seeds the `keys` collection
  (chat endpoint, `apiKey: "user_provided"`) and `pluginauths`
  (`mcp:inference-image` → `IC_API_KEY`, injected per-user via
  `customUserVars` so image generation bills to the member's own team).
- Members without a sandbox account are skipped until they first log in; run
  the sync after onboarding waves and on a schedule (the command is in the
  repo README). Idempotent.
- Encryption matches LibreChat's own (`CREDS_KEY`/`CREDS_IV`, read from
  `/proc/1/environ` — exec sessions don't inherit the app env).

> Terminal output redacts values after `apiKey:` as `***` even for harmless
> strings (`user_provided`, `${TTS_API_KEY}`). Probe lengths/counts instead
> of trusting redacted output.

## Verification (after any change)

```bash
cloudron logs --app 4023ccbf-d57a-4500-97df-14de9cf3516e --lines 60 | \
  grep -iE 'custom config|error|invalid|MCP|schedules'
curl -s https://librechat-test.inference.coop/api/config   # appTitle, emailLoginEnabled
```

Expected: `Custom config file loaded`, `[MCP][inference-docs]` initialized
with tool `search_docs`, `inference-image` registered (its `Tools: undefined`
boot line is expected — per-user startup), `[schedules] engine started`,
`appTitle: Inference Co-op`, `emailLoginEnabled: false`. The login page shows
the co-op logo, the co-op favicon, a single "Login with Inference
Cooperative" (Cloudron SSO) path with no email/password form, **no
"Don't have an account?" line**, and the cream/green co-op color scheme.

After an IMAGE deploy, additionally run the four-step verification in the
repo README (API `manifest.dockerImage`, container md5 vs repo md5, served
HTML tags, rendered DOM colors).