`. Rebuilt and redeployed to change branding. The md5s of `librechat.yaml`, `mcp/server.py`, and `mcp/image_server.py` in this repo are kept byte-identical to the live volume files. Verify with: ```bash APP=4023ccbf-d57a-4500-97df-14de9cf3516e cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec --app $APP \ -- md5sum /app/data/librechat.yaml /app/data/mcp/server.py /app/data/mcp/image_server.py md5sum librechat.yaml mcp/server.py mcp/image_server.py ``` ## Architecture: what lives where | Layer | Location | Survives image update? | Repo mirror | |---|---|---|---| | App config | `/app/data/librechat.yaml` (volume) | yes | `librechat.yaml` | | Env vars (env-file wins!) | `/app/data/env` (volume, sourced LAST by start.sh) | yes | documented below | | Env vars (Cloudron-managed) | `cloudron env` (process env) | yes | documented below | | MCP servers | `/app/data/mcp/server.py` + `mcp/image_server.py` (volume) | yes | `mcp/` | | Per-member key seed | `sync-keys.js` baked into the image at `/app/code/sync-keys.js` (run by the manifest's `scheduler` task) | no — rebuilt per image | `sync-keys.js` | | Brand assets + branding CSS/JS | baked into image overlay | no — rebuilt per image | `assets/` | | `CloudronManifest.json` | repo only (controls what `cloudron update` deploys) | n/a | official appstore manifest **except `dockerImage`** | **Precedence gotcha:** `start.sh` sources `/run/librechat/env` AFTER the process env, and `/app/data/env` is appended into it — so a value defined in `/app/data/env` **overrides** the same `cloudron env` variable. That is why `ALLOW_EMAIL_LOGIN=false` must be set in `/app/data/env`, not (only) via `cloudron env`. ## Volume changes (librechat.yaml, mcp/*.py, env) > **Volume warning:** edits you make to the files in this repo do **not** > deploy with the image. The image never touches `/app/data`. Apply changes > to the live files first, then mirror them here. Deploy a config change (no image rebuild needed): ```bash B64=$(base64 -w0 librechat.yaml) cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec \ --app 4023ccbf-d57a-4500-97df-14de9cf3516e \ -- bash -c "echo '$B64' | base64 -d > /app/data/librechat.yaml" cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" restart \ --app 4023ccbf-d57a-4500-97df-14de9cf3516e --timeout 900 ``` - Always deploy via base64 — heredocs and `sed` inserts have corrupted this file twice (literal `$(...)` / bad indentation → LibreChat silently falls back to defaults with only one log line). - Always `diff` the repo file against the live one after deploying. - **Reading large live files back over `cloudron exec`** cannot be done with a plain `cat` (a single large stdout stream resets the connection, `ECONNRESET`). Pipe it small instead: ```bash cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec \ --app 4023ccbf-d57a-4500-97df-14de9cf3516e \ -- bash -c "gzip -c /app/data/librechat.yaml | base64 -w0 | fold -w 100" > /tmp/live.b64 tr -d '\n' < /tmp/live.b64 | base64 -d | gunzip > /tmp/live-librechat.yaml ``` Verify after every config change: ```bash cloudron logs --app 4023ccbf-d57a-4500-97df-14de9cf3516e --lines 60 | grep -iE 'custom config|error|invalid|MCP' curl -s https://librechat-test.inference.coop/api/config # appTitle, emailLoginEnabled ``` ## Build + deploy the image (branding only) > **Two traps have each silently eaten deploys of this overlay.** Both look > like success. Run the verification block below after every deploy. 1. **`cloudron update --image <tag>` is IGNORED when this directory contains `CloudronManifest.json`.** The CLI deploys `manifest.dockerImage` instead — while still printing your `--image` value in its output, so it *looks* right. The tag you build MUST be written into `CloudronManifest.json.dockerImage` before updating. 2. **The remote builder's cache can serve stale COPY layers even when file content changed** (observed 2026-10-19: six consecutive "Build successful" runs produced images with old assets). Build with `--no-cache` whenever anything under `assets/` changed. Full deploy sequence: ```bash cd /path/to/librechat-custom # the dir containing CloudronManifest.json # 1. edit assets/ … then: git add -A && git commit -m "…" && git push SHA=$(git rev-parse --short HEAD) # 2. build fresh layers (no-cache after asset edits) cloudron builder build --no-cache \ --repository registry.inference.coop/co-op/librechat-custom \ --tag 20261019-$SHA # 3. pin the manifest to the new tag — the ONLY field you may change # (edit "dockerImage" in CloudronManifest.json to :20261019-$SHA) git add CloudronManifest.json && git commit -m "Pin manifest to $SHA" && git push # 4. deploy — the manifest drives it cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" update \ --app 4023ccbf-d57a-4500-97df-14de9cf3516e --force ``` `./release.sh release` runs this same sequence with both traps handled — it refuses to build from a dirty tree, always passes `--no-cache`, writes the tag into the manifest before deploying, then runs the verification block. Builds are remote (the registry's BuildKit does the work), so the only requirement is a registry login on the host running the command; `verify`, `volume` and `pin`/`deploy` need only `CLOUDRON_TOKEN`. `./release.sh verify` is safe to run any time. If `build` reports no registry login, it prints both ways to bootstrap one (a narrow app password scoped to the registry app, or the CLI device flow). Rollback: repoint `CloudronManifest.json.dockerImage` to the previous tag (every build is tagged `20261019-<git-sha>`; the base image `cloudron/ai.librechat.cloudronapp:<tag>` is the pre-overlay fallback) and rerun step 4. ### Verify after every image deploy (all four, or it didn't happen) ```bash APP=4023ccbf-d57a-4500-97df-14de9cf3516e # 1. What is ACTUALLY running? (CLI update output is not evidence) curl -s -H "Authorization: Bearer $CLOUDRON_TOKEN" \ https://my.inference.coop/api/v1/apps/$APP \ | python3 -c "import json,sys; print(json.load(sys.stdin)['manifest']['dockerImage'])" # 2. Container files match the repo (catches stale builder cache) cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec --app $APP \ -- md5sum /app/code/librechat/client/dist/assets/custom.css md5sum assets/custom.css # 3. Served HTML carries the injection curl -s https://chat.inference.coop/login | grep -Eo 'assets/custom2?\.js|assets/custom\.css' # 4. Rendered DOM shows it (browser devtools or a browser tool). The DEPLOYED # palette is the GREEN WASH with white text areas — an earlier version of this # block asserted cream (rgb(250,248,245)), which would have led you to "fix" # a correct deploy: # computed background of div.bg-surface-primary == rgb(234, 243, 234); # chat text areas / composer == rgb(255, 255, 255); # dark scheme == rgb(34, 39, 31); # the "Don't have an account?" <p> computes display: none. # 5. The scheduler task's script exists in THIS image — a manifest repoint # deployed before its image lands leaves the key seed broken on schedule: cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec --app $APP \ -- test -f /app/data/sync-keys.js && echo "seed path present" ``` `./release.sh verify` runs checks 1-5 plus the seed task's own log lines and the running script's revision hash, and exits non-zero on any failure. ### Appstore pin bumps (base image) The overlay's `FROM` is pinned to the exact appstore image the app runs. Bumping it is a two-line change in this repo (`Dockerfile` `FROM` + the manifest's `version`/`changelog` copied from the official manifest) plus a rebuild, but it re-runs every build-time patch against a new base layer, so treat the overlay as the risk, not the bump. ```bash # what the appstore now offers (official manifest, authoritative) curl -s https://api.cloudron.io/api/v1/apps/ai.librechat.cloudronapp \ | python3 -c "import json,sys; m=json.load(sys.stdin)['data']['manifest']; print(m['version'], m.get('upstreamVersion'), m['changelog'].strip())" # diff it against our copy — every field must match except dockerImage (+ scheduler) # then: rebuild, pin, deploy, ./release.sh verify, and re-check the branding block above ``` Rollback is a one-line `dockerImage` repoint to the previous tag (built images stay in the registry), then `./release.sh deploy`. ### `CloudronManifest.json` rules - The **only** field you may change is `dockerImage` — it must point at the overlay tag you want deployed. - Everything else stays a byte-identical copy of the OFFICIAL appstore manifest. A hand-edited manifest that omits the addons (`postgresql`, `mongodb`, `oidc`, `sendmail`, `localstorage`) strips them from the running app and crash-loops it (`mkdir: Read-only file system`). That broke the sandbox on 2026-10-18 — the manifest, not the overlay, was the bug. - When the appstore package updates, refresh the file with a new copy from the Cloudron API (`GET /api/v1/apps/<id>` → `.manifest`), repoint `dockerImage` to your rebuilt overlay, and bump the `FROM` tag in the Dockerfile to the new appstore `dockerImage` (pinned — never a floating tag). ## Env vars (live state) Set in `/app/data/env` (volume — these WIN over cloudron env): - `APP_TITLE="Inference Co-op"` (was `"LibreChat"`) - `ALLOW_EMAIL_LOGIN=false` (was `true`) — password login removed; Cloudron SSO (OIDC) is the only login path - `SCHEDULES_SINGLE_PROCESS=true` — sandbox runs a single replica; silences the scheduler's Redis guard error Set via `cloudron env set` (process env): - `TTS_API_KEY` / `STT_API_KEY` — **dedicated LiteLLM virtual key** (alias `librechat-speech`, $5 / 30d budget, restricted to `tinfoil/voxtral-tts` + `tinfoil/whisper-large-v3-turbo`). Regenerate with the gateway master key: `POST /key/generate {"key_alias":"librechat-speech","models":[...],"max_budget":5,"budget_duration":"30d"}` — never put the master key here. - `IMAGE_GEN_OAI_BASEURL=https://gateway.inference.coop/v1` and `IMAGE_GEN_OAI_MODEL=regolo/qwen-image` — the built-in agent "OpenAI Image Tools" points at the co-op gateway. No admin key is configured; members using the AGENT path paste their own key in the tool UI (v0.8.8 limitation). The MCP path below needs no key at all. - `SEARXNG_INSTANCE_URL=https://search.inference.coop` — referenced from the YAML as `${SEARXNG_INSTANCE_URL}` (self-hosted search; keyless JSON API). ## Branding overlay (custom.css / custom2.js) Injected into the built SPA by the Dockerfile (`<link>` + `<script>` after the `<title>` sed). What it does: **Palette (final)**: light `--surface-primary` is `250 248 245` (#faf8f5 cream) — the dashboard's true page background (the green wash #eaf3ea was tried as the full background and rejected as too dark; on the dashboard the green is accent-only). Sidebar/composer box = white cards, tertiary = #f4f2ec (dashboard chips), hover = #eaf3ea green wash, border = #e4e0d8, chat text areas (`--surface-chat`, `--presentation`) = white. Green (#5b8c5a family) stays on accents: buttons, links, focus rings. Loading flash = cream. Dark mode: deep pine + green-light accents. Homepage variables remain the reference for other surfaces. **Fonts**: Figtree (the homepage's OFL font) is self-hosted from the design kit (`assets/fonts/figtree-*.woff2` + OFL.txt, served at `/assets/fonts/`) via `@font-face` in `custom.css` with a `font-family` override — LibreChat's built-in default is Inter. Never load fonts from a CDN. **Footer**: build-time text (no-JS fallback) + runtime link upgrade — `custom2.js` rewrites the footer to "[Inference Cooperative](https://inference.coop) · Our AI, governed together" with a `dataset` guard so the MutationObserver doesn't loop. **UI strings**: `patch_ui_strings.py` (build-time, idempotent, non-fatal) renames the "MCP Servers" button to **"Tools"** and replaces LibreChat's footer tagline ("Every AI for Everyone.") with the co-op footer ("Inference Cooperative · Our AI, governed together · inference.coop"), deleting stale `.gz`/`.br` siblings of edited files. Add new string replacements to the `REPLACEMENTS` list. **Palette source of truth**: the co-op homepage's own CSS variables — `--cream #faf8f5` (surfaces), `--wash #eef4ea` (panels/hover), `--green-deep #5b8c5a` (primary/submit), `--green-mid #6ba86b` (hover), `--green-light #7ab87a` (dark-mode accent), `--ink #2d3327` (text), `--muted #6b7a62` (secondary text), `--border #e8e2d8`. The `html.light` / `html.dark` blocks in `custom.css` map these onto LibreChat's Tailwind variables; keep them in sync when the homepage palette changes. **Logo & icons**: `assets/logo.svg` is the GREEN pine-dots vector (extracted from the homepage's inline SVG — keep the `xmlns` attribute, a standalone .svg is invalid without it). The earlier white-plate PNG variant was invisible on the cream background and only "flashed" against the dark loading screen. Favicons/PWA icons are the cream-plate + green-dots design rasterized from `design-assets/favicon.svg`. The Dockerfile also cache-busts the logo URL in the client bundle (`assets/logo.svg?v=3`) and deletes the stale precompressed `.gz`/`.br` siblings — bump `v=` whenever the logo changes, or browsers/service workers may keep serving an old cached copy. - **Colors** — LibreChat's Tailwind theme resolves every color through CSS variables set on `html.light` / `html.dark` (space-separated RGB): `--surface-primary`, `--surface-secondary/-tertiary/-hover/-submit/-submit-hover`, `--accent-primary/-hover`, `--ring-primary`, `--link`, `--text-primary/-secondary`, `--border-light`. `custom.css` overrides them (light: cream `250 248 245` surfaces + co-op green `91 140 90` accents; dark: matching co-op dark palette). **To rebrand, edit the `html.light` / `html.dark` blocks** — do not invent class names; `.login-page`, `.primary` etc. do not exist in the DOM. Read current values in devtools via `getComputedStyle(document.documentElement).getPropertyValue('--surface-primary')`. - **No sign-up prompt** — the "Don't have an account? Sign up" line is `<p><a href="/register">Sign up</a></p>`; hidden with `p:has(> a[href*="register"]) { display: none !important; }` (hiding only the `<a>` leaves the text). Any residual register/signup links are hidden too. - **`custom2.js`** — the auth form is client-rendered AFTER `load`, so a plain load-time query matches nothing; a **MutationObserver** re-checks on every DOM change and hides the sign-up paragraph whenever React renders it. There is also an `interface.theme` block in `librechat.yaml` — it is **forward-compatible only**: the theme engine ships in LibreChat releases AFTER 0.8.8 and the block is silently ignored today. When the appstore package bundles a newer LibreChat, revisit whether the CSS overlay is still needed. ## Feature notes (LibreChat v0.8.8) - **Chat models**: custom endpoint "Inference Co-op" → gateway, 8 chat modelSpecs (emoji convention shared with models.md). `regolo/qwen-image` is deliberately NOT in the chat picker (image models are not chat models; chat-calling it was the source of the old 500s). - **Image generation — two paths**: 1. **MCP tool (default, no key ever)**: `inference-image` server (tool `generate_image`) — members select it in the composer's **+** palette and just ask. Per-member billing via `customUserVars.IC_API_KEY` (seeded from the portal) + `env: IC_API_KEY: "{{IC_API_KEY}}"`; a system-key fallback (`system-image-fallback`, $5/30d) covers the rare empty-value case. The `customUserVars` gate blocks a not-yet-seeded member until the next sync (≤10 min) — accepted as a short delay over billing everything to the system account. 2. **Agent tool**: LibreChat's built-in "OpenAI Image Tools" (env `IMAGE_GEN_OAI_BASEURL`/`IMAGE_GEN_OAI_MODEL` → gateway; no admin key, members paste their own). Note LibreChat v0.8.8 renders the generated image only inside the agent conversation. - **MCP**: `inference-docs` (tool `search_docs`, starts at boot) and `inference-image` (tool `generate_image`, `startup: false`). Members may add their own via the MCP Settings panel ("bring your own tools"). Permissions seeded via `interface.mcpServers` (use/create true, share false). - **Speech**: TTS `tinfoil/voxtral-tts` (voice `casual_female`), STT `tinfoil/whisper-large-v3-turbo`, both via the gateway with the capped key. - **Web search**: SearXNG (self-hosted) + keyless Keenable scraper, no reranker. ## Per-member key auto-load (no API-key prompts) Members never enter an API key. Chat and image generation resolve each member's own portal-provisioned key automatically: 1. The portal provisions a per-member LiteLLM key at onboarding and stores it (`members.key_token`). 2. `GET /admin/librechat/keys` on the portal returns those keys for ACTIVE members — authenticated by the dedicated `X-Sync-Token` header (`LIBRECHAT_SYNC_TOKEN`, set on BOTH apps; not the admin token). 3. `sync-keys.js` (this repo, deployed to `/app/data/sync-keys.js`) fetches that feed and upserts, per member with a sandbox account: - `keys` collection: `{userId, name: "Inference Co-op", value: encrypt(JSON.stringify({apiKey}))}` — consumed by the chat endpoint (`apiKey: "user_provided"` in the yaml). - `pluginauths` collection: `{userId, pluginKey: "mcp_inference-image", authField: "IC_API_KEY", value: encrypt(key)}` — consumed by the MCP `customUserVars` config so image generation bills to the member's own team. Encryption uses LibreChat's own scheme (`CREDS_KEY`/`CREDS_IV`, AES-256-CBC via `@librechat/data-schemas` `encrypt()`). 4. Members who haven't logged in to the sandbox yet are skipped; the next run seeds them. ### Scheduling (in this app, since 2026-10-25) `CloudronManifest.json` declares the task in `addons.scheduler`: ```json "scheduler": { "librechat_key_sync": { "schedule": "*/10 * * * *", "command": "cd /app/code/librechat/api && node /app/code/sync-keys.js" } } ``` The script is **baked into the image** (`COPY sync-keys.js /app/code/…`), so it cannot be lost with a volume restore and cannot drift from this repo. **Ordering matters on the first deploy of this change:** build the image → deploy (pin the new tag in the manifest first — the manifest's repointed command ships in the same `cloudron update`) → verify (`./release.sh verify` checks that the command path exists in the running image) → only then delete the volume copy `/app/data/sync-keys.js`, because rolling back to the previous image tag needs it back. After that, a change to this script requires a rebuild, not a volume push. Seeding therefore depends on nothing outside this app — it previously ran from a steward cron on another machine, which made a member-facing function depend on that machine's uptime and paths. Two consequences of how the scheduler addon works, both learned the hard way: - **Each task runs in its own container.** The app's process env is absent there, and `/proc/1/environ` belongs to the task launcher, not to LibreChat — so `CREDS_KEY`/`CREDS_IV` come from `/app/data/.librechat-creds` (root:cloudron `0640`), written from the live app env: ```bash cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec \ --app 4023ccbf-d57a-4500-97df-14de9cf3516e \ -- sh -c 'umask 077; tr "\0" "\n" < /proc/1/environ | grep -E "^CREDS_(KEY|IV)=" > /app/data/.librechat-creds; chown root:cloudron /app/data/.librechat-creds; chmod 640 /app/data/.librechat-creds' ``` If the app's key is ever rotated, re-run that. A stale key does not corrupt anything — the guard below refuses to write. - **It fails closed.** `sync-keys.js` writes nothing unless `CREDS_KEY` first decrypts an existing key row (proof it is the app's real key) and unless each fresh ciphertext round-trips back to the plaintext. A credential problem exits non-zero, logs `FATAL: ...`, and writes `/app/data/key-sync-diag.txt` (uid, cwd, which env sources were readable) instead of overwriting working member keys. Manual run (unchanged, useful after an onboarding wave): ```bash cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec \ --app 4023ccbf-d57a-4500-97df-14de9cf3516e \ -- bash -c "cd /app/code/librechat/api && node /app/data/sync-keys.js" ``` Idempotent. Verify: the app log shows the task's output (`creds check: decrypt-ok …` / `portal feed: N active member key(s)` / `done: N seeded, M skipped`), and `keys` / `pluginauths` counts should equal the number of members who have logged in to the sandbox. > **Output-redaction gotcha:** terminal output renders values after > `apiKey:` as `***` even when the file contains `user_provided` or > `${ENV_REF}`. Never conclude a file contains a literal placeholder from > redacted output — probe lengths/counts instead > (`tr -cd '*' < file | wc -c`, `awk -F'apiKey: ' '{print length($2)}'`). ## Known state / limitations - The `[schedules]` Redis guard error is resolved by `SCHEDULES_SINGLE_PROCESS=true` (above). - The `RAG API not reachable` warning is expected (no rag_api addon); it only affects LibreChat's own file-RAG, which we do not use. - The `inference-image` MCP server logs `Tools: undefined` at boot — expected: with `startup: false` it registers at boot and spawns per-user on first use in a conversation. - Members' chat usage bills per-member via their endpoint key; the MCP image path is currently keyless and bills to the gateway (not enforced per-member). Wiring per-member key injection into LibreChat chat is an open task. - Repo ↔ live parity last verified 2026-10-19: `librechat.yaml`, `mcp/server.py`, `mcp/image_server.py` md5s match the live volume; overlay image `registry.inference.coop/co-op/librechat-custom:20261019-fc9b11e` is deployed (container asset md5s match this repo). ## PWA lessons (2026-10-22 — icon/band debugging) - **Builder COPY layers can be stale**: after editing any asset, build with `--no-cache` and VERIFY the served file's md5 against the repo before reporting success. (The 20261022-df55d7f build shipped a stale maskable icon this way — the repo was right, the image wasn't.) - **PWA icon URLs must carry a per-build `?v=`**: they are served with `max-age=172800`, and remove/re-add of the PWA does not clear the phone's HTTP cache — launchers keep rendering the previous build's icon bytes. patch_index.py injects the query into every manifest icon src per build. - **Maskable icons**: logo must sit inside the safe zone — ~66% of the canvas on a solid brand background. Full-bleed logos get cropped ("zoomed in"). - **Firefox Android**: `crossorigin="use-credentials"` on the manifest link breaks icon resolution (blank launcher icon) — patch_index.py strips it. - **Theme bars**: the manifest `theme_color` is one static value; Chromium's nav bar and all Firefox bars use it even when the meta says otherwise. The durable fix is `viewport-fit=cover` + standalone-mode safe-area padding (custom.css) so the bands show the theme-correct page background; custom2.js additionally syncs one meta `theme-color` to the app's resolved theme.

# LibreChat sandbox — Inference Co-op customization Source of truth for the co-op's chat instance at **chat.inference.coop** (Cloudron app `4023ccbf-d57a-4500-97df-14de9cf3516e`, appstore package `ai.librechat.cloudronapp@1.4.5` — upstream LibreChat is unchanged at **v0.8.8**; the bump is uv 0.13.1 → 0.13.5 — on base image `cloudron/ai.librechat.cloudronapp:202610230221180000`). `librechat-test.inference.coop` was the pre-swap hostname (the swap to `chat.` happened 2026-10-21; the test alias is still not provisioned). The deployment has **two independent layers** — know which one you are changing: 1. **Persistent volume** (`/app/data/…`) — behavior: `librechat.yaml`, `env`, `mcp/*.py`. Survives image updates. Edited via `cloudron exec`, then mirrored into this repo (see "Volume changes" below). 2. **Overlay image** (this repo's Dockerfile) — appearance only: logo, favicons, `custom.css`, `custom2.js`, ``. Rebuilt and redeployed to change branding. The md5s of `librechat.yaml`, `mcp/server.py`, and `mcp/image_server.py` in this repo are kept byte-identical to the live volume files. Verify with: ```bash APP=4023ccbf-d57a-4500-97df-14de9cf3516e cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec --app $APP \ -- md5sum /app/data/librechat.yaml /app/data/mcp/server.py /app/data/mcp/image_server.py md5sum librechat.yaml mcp/server.py mcp/image_server.py ``` ## Architecture: what lives where | Layer | Location | Survives image update? | Repo mirror | |---|---|---|---| | App config | `/app/data/librechat.yaml` (volume) | yes | `librechat.yaml` | | Env vars (env-file wins!) | `/app/data/env` (volume, sourced LAST by start.sh) | yes | documented below | | Env vars (Cloudron-managed) | `cloudron env` (process env) | yes | documented below | | MCP servers | `/app/data/mcp/server.py` + `mcp/image_server.py` (volume) | yes | `mcp/` | | Per-member key seed | `sync-keys.js` baked into the image at `/app/code/sync-keys.js` (run by the manifest's `scheduler` task) | no — rebuilt per image | `sync-keys.js` | | Brand assets + branding CSS/JS | baked into image overlay | no — rebuilt per image | `assets/` | | `CloudronManifest.json` | repo only (controls what `cloudron update` deploys) | n/a | official appstore manifest **except `dockerImage`** | **Precedence gotcha:** `start.sh` sources `/run/librechat/env` AFTER the process env, and `/app/data/env` is appended into it — so a value defined in `/app/data/env` **overrides** the same `cloudron env` variable. That is why `ALLOW_EMAIL_LOGIN=false` must be set in `/app/data/env`, not (only) via `cloudron env`. ## Volume changes (librechat.yaml, mcp/*.py, env) > **Volume warning:** edits you make to the files in this repo do **not** > deploy with the image. The image never touches `/app/data`. Apply changes > to the live files first, then mirror them here. Deploy a config change (no image rebuild needed): ```bash B64=$(base64 -w0 librechat.yaml) cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec \ --app 4023ccbf-d57a-4500-97df-14de9cf3516e \ -- bash -c "echo '$B64' | base64 -d > /app/data/librechat.yaml" cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" restart \ --app 4023ccbf-d57a-4500-97df-14de9cf3516e --timeout 900 ``` - Always deploy via base64 — heredocs and `sed` inserts have corrupted this file twice (literal `$(...)` / bad indentation → LibreChat silently falls back to defaults with only one log line). - Always `diff` the repo file against the live one after deploying. - **Reading large live files back over `cloudron exec`** cannot be done with a plain `cat` (a single large stdout stream resets the connection, `ECONNRESET`). Pipe it small instead: ```bash cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec \ --app 4023ccbf-d57a-4500-97df-14de9cf3516e \ -- bash -c "gzip -c /app/data/librechat.yaml | base64 -w0 | fold -w 100" > /tmp/live.b64 tr -d '\n' < /tmp/live.b64 | base64 -d | gunzip > /tmp/live-librechat.yaml ``` Verify after every config change: ```bash cloudron logs --app 4023ccbf-d57a-4500-97df-14de9cf3516e --lines 60 | grep -iE 'custom config|error|invalid|MCP' curl -s https://librechat-test.inference.coop/api/config # appTitle, emailLoginEnabled ``` ## Build + deploy the image (branding only) > **Two traps have each silently eaten deploys of this overlay.** Both look > like success. Run the verification block below after every deploy. 1. **`cloudron update --image <tag>` is IGNORED when this directory contains `CloudronManifest.json`.** The CLI deploys `manifest.dockerImage` instead — while still printing your `--image` value in its output, so it *looks* right. The tag you build MUST be written into `CloudronManifest.json.dockerImage` before updating. 2. **The remote builder's cache can serve stale COPY layers even when file content changed** (observed 2026-10-19: six consecutive "Build successful" runs produced images with old assets). Build with `--no-cache` whenever anything under `assets/` changed. Full deploy sequence: ```bash cd /path/to/librechat-custom # the dir containing CloudronManifest.json # 1. edit assets/ … then: git add -A && git commit -m "…" && git push SHA=$(git rev-parse --short HEAD) # 2. build fresh layers (no-cache after asset edits) cloudron builder build --no-cache \ --repository registry.inference.coop/co-op/librechat-custom \ --tag 20261019-$SHA # 3. pin the manifest to the new tag — the ONLY field you may change # (edit "dockerImage" in CloudronManifest.json to :20261019-$SHA) git add CloudronManifest.json && git commit -m "Pin manifest to $SHA" && git push # 4. deploy — the manifest drives it cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" update \ --app 4023ccbf-d57a-4500-97df-14de9cf3516e --force ``` `./release.sh release` runs this same sequence with both traps handled — it refuses to build from a dirty tree, always passes `--no-cache`, writes the tag into the manifest before deploying, then runs the verification block. Builds are remote (the registry's BuildKit does the work), so the only requirement is a registry login on the host running the command; `verify`, `volume` and `pin`/`deploy` need only `CLOUDRON_TOKEN`. `./release.sh verify` is safe to run any time. If `build` reports no registry login, it prints both ways to bootstrap one (a narrow app password scoped to the registry app, or the CLI device flow). Rollback: repoint `CloudronManifest.json.dockerImage` to the previous tag (every build is tagged `20261019-<git-sha>`; the base image `cloudron/ai.librechat.cloudronapp:<tag>` is the pre-overlay fallback) and rerun step 4. ### Verify after every image deploy (all four, or it didn't happen) ```bash APP=4023ccbf-d57a-4500-97df-14de9cf3516e # 1. What is ACTUALLY running? (CLI update output is not evidence) curl -s -H "Authorization: Bearer $CLOUDRON_TOKEN" \ https://my.inference.coop/api/v1/apps/$APP \ | python3 -c "import json,sys; print(json.load(sys.stdin)['manifest']['dockerImage'])" # 2. Container files match the repo (catches stale builder cache) cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec --app $APP \ -- md5sum /app/code/librechat/client/dist/assets/custom.css md5sum assets/custom.css # 3. Served HTML carries the injection curl -s https://chat.inference.coop/login | grep -Eo 'assets/custom2?\.js|assets/custom\.css' # 4. Rendered DOM shows it (browser devtools or a browser tool). The DEPLOYED # palette is the GREEN WASH with white text areas — an earlier version of this # block asserted cream (rgb(250,248,245)), which would have led you to "fix" # a correct deploy: # computed background of div.bg-surface-primary == rgb(234, 243, 234); # chat text areas / composer == rgb(255, 255, 255); # dark scheme == rgb(34, 39, 31); # the "Don't have an account?" <p> computes display: none. # 5. The scheduler task's script exists in THIS image — a manifest repoint # deployed before its image lands leaves the key seed broken on schedule: cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec --app $APP \ -- test -f /app/data/sync-keys.js && echo "seed path present" ``` `./release.sh verify` runs checks 1-5 plus the seed task's own log lines and the running script's revision hash, and exits non-zero on any failure. ### Appstore pin bumps (base image) The overlay's `FROM` is pinned to the exact appstore image the app runs. Bumping it is a two-line change in this repo (`Dockerfile` `FROM` + the manifest's `version`/`changelog` copied from the official manifest) plus a rebuild, but it re-runs every build-time patch against a new base layer, so treat the overlay as the risk, not the bump. ```bash # what the appstore now offers (official manifest, authoritative) curl -s https://api.cloudron.io/api/v1/apps/ai.librechat.cloudronapp \ | python3 -c "import json,sys; m=json.load(sys.stdin)['data']['manifest']; print(m['version'], m.get('upstreamVersion'), m['changelog'].strip())" # diff it against our copy — every field must match except dockerImage (+ scheduler) # then: rebuild, pin, deploy, ./release.sh verify, and re-check the branding block above ``` Rollback is a one-line `dockerImage` repoint to the previous tag (built images stay in the registry), then `./release.sh deploy`. ### `CloudronManifest.json` rules - The **only** field you may change is `dockerImage` — it must point at the overlay tag you want deployed. - Everything else stays a byte-identical copy of the OFFICIAL appstore manifest. A hand-edited manifest that omits the addons (`postgresql`, `mongodb`, `oidc`, `sendmail`, `localstorage`) strips them from the running app and crash-loops it (`mkdir: Read-only file system`). That broke the sandbox on 2026-10-18 — the manifest, not the overlay, was the bug. - When the appstore package updates, refresh the file with a new copy from the Cloudron API (`GET /api/v1/apps/<id>` → `.manifest`), repoint `dockerImage` to your rebuilt overlay, and bump the `FROM` tag in the Dockerfile to the new appstore `dockerImage` (pinned — never a floating tag). ## Env vars (live state) Set in `/app/data/env` (volume — these WIN over cloudron env): - `APP_TITLE="Inference Co-op"` (was `"LibreChat"`) - `ALLOW_EMAIL_LOGIN=false` (was `true`) — password login removed; Cloudron SSO (OIDC) is the only login path - `SCHEDULES_SINGLE_PROCESS=true` — sandbox runs a single replica; silences the scheduler's Redis guard error Set via `cloudron env set` (process env): - `TTS_API_KEY` / `STT_API_KEY` — **dedicated LiteLLM virtual key** (alias `librechat-speech`, $5 / 30d budget, restricted to `tinfoil/voxtral-tts` + `tinfoil/whisper-large-v3-turbo`). Regenerate with the gateway master key: `POST /key/generate {"key_alias":"librechat-speech","models":[...],"max_budget":5,"budget_duration":"30d"}` — never put the master key here. - `IMAGE_GEN_OAI_BASEURL=https://gateway.inference.coop/v1` and `IMAGE_GEN_OAI_MODEL=regolo/qwen-image` — the built-in agent "OpenAI Image Tools" points at the co-op gateway. No admin key is configured; members using the AGENT path paste their own key in the tool UI (v0.8.8 limitation). The MCP path below needs no key at all. - `SEARXNG_INSTANCE_URL=https://search.inference.coop` — referenced from the YAML as `${SEARXNG_INSTANCE_URL}` (self-hosted search; keyless JSON API). ## Branding overlay (custom.css / custom2.js) Injected into the built SPA by the Dockerfile (`<link>` + `<script>` after the `<title>` sed). What it does: **Palette (final)**: light `--surface-primary` is `250 248 245` (#faf8f5 cream) — the dashboard's true page background (the green wash #eaf3ea was tried as the full background and rejected as too dark; on the dashboard the green is accent-only). Sidebar/composer box = white cards, tertiary = #f4f2ec (dashboard chips), hover = #eaf3ea green wash, border = #e4e0d8, chat text areas (`--surface-chat`, `--presentation`) = white. Green (#5b8c5a family) stays on accents: buttons, links, focus rings. Loading flash = cream. Dark mode: deep pine + green-light accents. Homepage variables remain the reference for other surfaces. **Fonts**: Figtree (the homepage's OFL font) is self-hosted from the design kit (`assets/fonts/figtree-*.woff2` + OFL.txt, served at `/assets/fonts/`) via `@font-face` in `custom.css` with a `font-family` override — LibreChat's built-in default is Inter. Never load fonts from a CDN. **Footer**: build-time text (no-JS fallback) + runtime link upgrade — `custom2.js` rewrites the footer to "[Inference Cooperative](https://inference.coop) · Our AI, governed together" with a `dataset` guard so the MutationObserver doesn't loop. **UI strings**: `patch_ui_strings.py` (build-time, idempotent, non-fatal) renames the "MCP Servers" button to **"Tools"** and replaces LibreChat's footer tagline ("Every AI for Everyone.") with the co-op footer ("Inference Cooperative · Our AI, governed together · inference.coop"), deleting stale `.gz`/`.br` siblings of edited files. Add new string replacements to the `REPLACEMENTS` list. **Palette source of truth**: the co-op homepage's own CSS variables — `--cream #faf8f5` (surfaces), `--wash #eef4ea` (panels/hover), `--green-deep #5b8c5a` (primary/submit), `--green-mid #6ba86b` (hover), `--green-light #7ab87a` (dark-mode accent), `--ink #2d3327` (text), `--muted #6b7a62` (secondary text), `--border #e8e2d8`. The `html.light` / `html.dark` blocks in `custom.css` map these onto LibreChat's Tailwind variables; keep them in sync when the homepage palette changes. **Logo & icons**: `assets/logo.svg` is the GREEN pine-dots vector (extracted from the homepage's inline SVG — keep the `xmlns` attribute, a standalone .svg is invalid without it). The earlier white-plate PNG variant was invisible on the cream background and only "flashed" against the dark loading screen. Favicons/PWA icons are the cream-plate + green-dots design rasterized from `design-assets/favicon.svg`. The Dockerfile also cache-busts the logo URL in the client bundle (`assets/logo.svg?v=3`) and deletes the stale precompressed `.gz`/`.br` siblings — bump `v=` whenever the logo changes, or browsers/service workers may keep serving an old cached copy. - **Colors** — LibreChat's Tailwind theme resolves every color through CSS variables set on `html.light` / `html.dark` (space-separated RGB): `--surface-primary`, `--surface-secondary/-tertiary/-hover/-submit/-submit-hover`, `--accent-primary/-hover`, `--ring-primary`, `--link`, `--text-primary/-secondary`, `--border-light`. `custom.css` overrides them (light: cream `250 248 245` surfaces + co-op green `91 140 90` accents; dark: matching co-op dark palette). **To rebrand, edit the `html.light` / `html.dark` blocks** — do not invent class names; `.login-page`, `.primary` etc. do not exist in the DOM. Read current values in devtools via `getComputedStyle(document.documentElement).getPropertyValue('--surface-primary')`. - **No sign-up prompt** — the "Don't have an account? Sign up" line is `<p><a href="/register">Sign up</a></p>`; hidden with `p:has(> a[href*="register"]) { display: none !important; }` (hiding only the `<a>` leaves the text). Any residual register/signup links are hidden too. - **`custom2.js`** — the auth form is client-rendered AFTER `load`, so a plain load-time query matches nothing; a **MutationObserver** re-checks on every DOM change and hides the sign-up paragraph whenever React renders it. There is also an `interface.theme` block in `librechat.yaml` — it is **forward-compatible only**: the theme engine ships in LibreChat releases AFTER 0.8.8 and the block is silently ignored today. When the appstore package bundles a newer LibreChat, revisit whether the CSS overlay is still needed. ## Feature notes (LibreChat v0.8.8) - **Chat models**: custom endpoint "Inference Co-op" → gateway, 8 chat modelSpecs (emoji convention shared with models.md). `regolo/qwen-image` is deliberately NOT in the chat picker (image models are not chat models; chat-calling it was the source of the old 500s). - **Image generation — two paths**: 1. **MCP tool (default, no key ever)**: `inference-image` server (tool `generate_image`) — members select it in the composer's **+** palette and just ask. Per-member billing via `customUserVars.IC_API_KEY` (seeded from the portal) + `env: IC_API_KEY: "{{IC_API_KEY}}"`; a system-key fallback (`system-image-fallback`, $5/30d) covers the rare empty-value case. The `customUserVars` gate blocks a not-yet-seeded member until the next sync (≤10 min) — accepted as a short delay over billing everything to the system account. 2. **Agent tool**: LibreChat's built-in "OpenAI Image Tools" (env `IMAGE_GEN_OAI_BASEURL`/`IMAGE_GEN_OAI_MODEL` → gateway; no admin key, members paste their own). Note LibreChat v0.8.8 renders the generated image only inside the agent conversation. - **MCP**: `inference-docs` (tool `search_docs`, starts at boot) and `inference-image` (tool `generate_image`, `startup: false`). Members may add their own via the MCP Settings panel ("bring your own tools"). Permissions seeded via `interface.mcpServers` (use/create true, share false). - **Speech**: TTS `tinfoil/voxtral-tts` (voice `casual_female`), STT `tinfoil/whisper-large-v3-turbo`, both via the gateway with the capped key. - **Web search**: SearXNG (self-hosted) + keyless Keenable scraper, no reranker. ## Per-member key auto-load (no API-key prompts) Members never enter an API key. Chat and image generation resolve each member's own portal-provisioned key automatically: 1. The portal provisions a per-member LiteLLM key at onboarding and stores it (`members.key_token`). 2. `GET /admin/librechat/keys` on the portal returns those keys for ACTIVE members — authenticated by the dedicated `X-Sync-Token` header (`LIBRECHAT_SYNC_TOKEN`, set on BOTH apps; not the admin token). 3. `sync-keys.js` (this repo, deployed to `/app/data/sync-keys.js`) fetches that feed and upserts, per member with a sandbox account: - `keys` collection: `{userId, name: "Inference Co-op", value: encrypt(JSON.stringify({apiKey}))}` — consumed by the chat endpoint (`apiKey: "user_provided"` in the yaml). - `pluginauths` collection: `{userId, pluginKey: "mcp_inference-image", authField: "IC_API_KEY", value: encrypt(key)}` — consumed by the MCP `customUserVars` config so image generation bills to the member's own team. Encryption uses LibreChat's own scheme (`CREDS_KEY`/`CREDS_IV`, AES-256-CBC via `@librechat/data-schemas` `encrypt()`). 4. Members who haven't logged in to the sandbox yet are skipped; the next run seeds them. ### Scheduling (in this app, since 2026-10-25) `CloudronManifest.json` declares the task in `addons.scheduler`: ```json "scheduler": { "librechat_key_sync": { "schedule": "*/10 * * * *", "command": "cd /app/code/librechat/api && node /app/code/sync-keys.js" } } ``` The script is **baked into the image** (`COPY sync-keys.js /app/code/…`), so it cannot be lost with a volume restore and cannot drift from this repo. **Ordering matters on the first deploy of this change:** build the image → deploy (pin the new tag in the manifest first — the manifest's repointed command ships in the same `cloudron update`) → verify (`./release.sh verify` checks that the command path exists in the running image) → only then delete the volume copy `/app/data/sync-keys.js`, because rolling back to the previous image tag needs it back. After that, a change to this script requires a rebuild, not a volume push. Seeding therefore depends on nothing outside this app — it previously ran from a steward cron on another machine, which made a member-facing function depend on that machine's uptime and paths. Two consequences of how the scheduler addon works, both learned the hard way: - **Each task runs in its own container.** The app's process env is absent there, and `/proc/1/environ` belongs to the task launcher, not to LibreChat — so `CREDS_KEY`/`CREDS_IV` come from `/app/data/.librechat-creds` (root:cloudron `0640`), written from the live app env: ```bash cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec \ --app 4023ccbf-d57a-4500-97df-14de9cf3516e \ -- sh -c 'umask 077; tr "\0" "\n" < /proc/1/environ | grep -E "^CREDS_(KEY|IV)=" > /app/data/.librechat-creds; chown root:cloudron /app/data/.librechat-creds; chmod 640 /app/data/.librechat-creds' ``` If the app's key is ever rotated, re-run that. A stale key does not corrupt anything — the guard below refuses to write. - **It fails closed.** `sync-keys.js` writes nothing unless `CREDS_KEY` first decrypts an existing key row (proof it is the app's real key) and unless each fresh ciphertext round-trips back to the plaintext. A credential problem exits non-zero, logs `FATAL: ...`, and writes `/app/data/key-sync-diag.txt` (uid, cwd, which env sources were readable) instead of overwriting working member keys. Manual run (unchanged, useful after an onboarding wave): ```bash cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec \ --app 4023ccbf-d57a-4500-97df-14de9cf3516e \ -- bash -c "cd /app/code/librechat/api && node /app/data/sync-keys.js" ``` Idempotent. Verify: the app log shows the task's output (`creds check: decrypt-ok …` / `portal feed: N active member key(s)` / `done: N seeded, M skipped`), and `keys` / `pluginauths` counts should equal the number of members who have logged in to the sandbox. > **Output-redaction gotcha:** terminal output renders values after > `apiKey:` as `***` even when the file contains `user_provided` or > `${ENV_REF}`. Never conclude a file contains a literal placeholder from > redacted output — probe lengths/counts instead > (`tr -cd '*' < file | wc -c`, `awk -F'apiKey: ' '{print length($2)}'`). ## Known state / limitations - The `[schedules]` Redis guard error is resolved by `SCHEDULES_SINGLE_PROCESS=true` (above). - The `RAG API not reachable` warning is expected (no rag_api addon); it only affects LibreChat's own file-RAG, which we do not use. - The `inference-image` MCP server logs `Tools: undefined` at boot — expected: with `startup: false` it registers at boot and spawns per-user on first use in a conversation. - Members' chat usage bills per-member via their endpoint key; the MCP image path is currently keyless and bills to the gateway (not enforced per-member). Wiring per-member key injection into LibreChat chat is an open task. - Repo ↔ live parity last verified 2026-10-19: `librechat.yaml`, `mcp/server.py`, `mcp/image_server.py` md5s match the live volume; overlay image `registry.inference.coop/co-op/librechat-custom:20261019-fc9b11e` is deployed (container asset md5s match this repo). ## PWA lessons (2026-10-22 — icon/band debugging) - **Builder COPY layers can be stale**: after editing any asset, build with `--no-cache` and VERIFY the served file's md5 against the repo before reporting success. (The 20261022-df55d7f build shipped a stale maskable icon this way — the repo was right, the image wasn't.) - **PWA icon URLs must carry a per-build `?v=`**: they are served with `max-age=172800`, and remove/re-add of the PWA does not clear the phone's HTTP cache — launchers keep rendering the previous build's icon bytes. patch_index.py injects the query into every manifest icon src per build. - **Maskable icons**: logo must sit inside the safe zone — ~66% of the canvas on a solid brand background. Full-bleed logos get cropped ("zoomed in"). - **Firefox Android**: `crossorigin="use-credentials"` on the manifest link breaks icon resolution (blank launcher icon) — patch_index.py strips it. - **Theme bars**: the manifest `theme_color` is one static value; Chromium's nav bar and all Firefox bars use it even when the meta says otherwise. The durable fix is `viewport-fit=cover` + standalone-mode safe-area padding (custom.css) so the bands show the theme-correct page background; custom2.js additionally syncs one meta `theme-color` to the app's resolved theme.