inference-bot
d099d31f6a
Follow-up to the custom.css/custom2.js change: the Dockerfile seds the web
manifest theme_color/background_color and the index.html light meta to
#eaf3ea, and the README's verify block asserted the green wash as correct
("an earlier version of this block asserted cream ... would have led you to
fix a correct deploy"). Both are the same regression: the wash was rejected
earlier, restored by accident, and then documented as intended.
- Dockerfile: manifest + index meta -> #faf8f5 (cream), with an explicit
"keep these three in sync" note (this sed, custom.css html:not(.dark),
custom2.js COOP_BG.light).
- README verify block: asserts cream again, plus a new step for the
install-time colors no DOM check covers (manifest + light meta).
- README palette section: hover is #f0ede6 not the wash; --presentation is
cream; source of truth is the DEPLOYED CSS of dashboard/portal/panel, with
the read command; records the shared 4.32:1 / 3.93:1 accessibility caveat;
notes that cloudron exec silently swallows multi-line sh -c.
473 lines
25 KiB
Markdown
473 lines
25 KiB
Markdown
# LibreChat sandbox — Inference Co-op customization
|
|
|
|
Source of truth for the co-op's chat instance at
|
|
**chat.inference.coop** (Cloudron app `4023ccbf-d57a-4500-97df-14de9cf3516e`,
|
|
appstore package `ai.librechat.cloudronapp@1.4.5` — upstream LibreChat is
|
|
unchanged at **v0.8.8**; the bump is uv 0.13.1 → 0.13.5 — on base image
|
|
`cloudron/ai.librechat.cloudronapp:202610230221180000`).
|
|
`librechat-test.inference.coop` was the pre-swap hostname (the swap to
|
|
`chat.` happened 2026-10-21; the test alias is still not provisioned).
|
|
|
|
The deployment has **two independent layers** — know which one you are changing:
|
|
|
|
1. **Persistent volume** (`/app/data/…`) — behavior: `librechat.yaml`, `env`,
|
|
`mcp/*.py`. Survives image updates. Edited via `cloudron exec`, then
|
|
mirrored into this repo (see "Volume changes" below).
|
|
2. **Overlay image** (this repo's Dockerfile) — appearance only: logo,
|
|
favicons, `custom.css`, `custom2.js`, `<title>`. Rebuilt and redeployed to
|
|
change branding.
|
|
|
|
The md5s of `librechat.yaml`, `mcp/server.py`, and `mcp/image_server.py` in
|
|
this repo are kept byte-identical to the live volume files. Verify with:
|
|
|
|
```bash
|
|
APP=4023ccbf-d57a-4500-97df-14de9cf3516e
|
|
cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec --app $APP \
|
|
-- md5sum /app/data/librechat.yaml /app/data/mcp/server.py /app/data/mcp/image_server.py
|
|
md5sum librechat.yaml mcp/server.py mcp/image_server.py
|
|
```
|
|
|
|
## Architecture: what lives where
|
|
|
|
| Layer | Location | Survives image update? | Repo mirror |
|
|
|---|---|---|---|
|
|
| App config | `/app/data/librechat.yaml` (volume) | yes | `librechat.yaml` |
|
|
| Env vars (env-file wins!) | `/app/data/env` (volume, sourced LAST by start.sh) | yes | documented below |
|
|
| Env vars (Cloudron-managed) | `cloudron env` (process env) | yes | documented below |
|
|
| MCP servers | `/app/data/mcp/server.py` + `mcp/image_server.py` (volume) | yes | `mcp/` |
|
|
| Per-member key seed | `sync-keys.js` baked into the image at `/app/code/sync-keys.js` (run by the manifest's `scheduler` task) | no — rebuilt per image | `sync-keys.js` |
|
|
| Brand assets + branding CSS/JS | baked into image overlay | no — rebuilt per image | `assets/` |
|
|
| `CloudronManifest.json` | repo only (controls what `cloudron update` deploys) | n/a | official appstore manifest **except `dockerImage`** |
|
|
|
|
**Precedence gotcha:** `start.sh` sources `/run/librechat/env` AFTER the
|
|
process env, and `/app/data/env` is appended into it — so a value defined in
|
|
`/app/data/env` **overrides** the same `cloudron env` variable. That is why
|
|
`ALLOW_EMAIL_LOGIN=false` must be set in `/app/data/env`, not (only) via
|
|
`cloudron env`.
|
|
|
|
## Volume changes (librechat.yaml, mcp/*.py, env)
|
|
|
|
> **Volume warning:** edits you make to the files in this repo do **not**
|
|
> deploy with the image. The image never touches `/app/data`. Apply changes
|
|
> to the live files first, then mirror them here.
|
|
|
|
Deploy a config change (no image rebuild needed):
|
|
|
|
```bash
|
|
B64=$(base64 -w0 librechat.yaml)
|
|
cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec \
|
|
--app 4023ccbf-d57a-4500-97df-14de9cf3516e \
|
|
-- bash -c "echo '$B64' | base64 -d > /app/data/librechat.yaml"
|
|
cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" restart \
|
|
--app 4023ccbf-d57a-4500-97df-14de9cf3516e --timeout 900
|
|
```
|
|
|
|
- Always deploy via base64 — heredocs and `sed` inserts have corrupted this
|
|
file twice (literal `$(...)` / bad indentation → LibreChat silently falls
|
|
back to defaults with only one log line).
|
|
- Always `diff` the repo file against the live one after deploying.
|
|
- **Reading large live files back over `cloudron exec`** cannot be done with
|
|
a plain `cat` (a single large stdout stream resets the connection,
|
|
`ECONNRESET`). Pipe it small instead:
|
|
|
|
```bash
|
|
cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec \
|
|
--app 4023ccbf-d57a-4500-97df-14de9cf3516e \
|
|
-- bash -c "gzip -c /app/data/librechat.yaml | base64 -w0 | fold -w 100" > /tmp/live.b64
|
|
tr -d '\n' < /tmp/live.b64 | base64 -d | gunzip > /tmp/live-librechat.yaml
|
|
```
|
|
|
|
Verify after every config change:
|
|
|
|
```bash
|
|
cloudron logs --app 4023ccbf-d57a-4500-97df-14de9cf3516e --lines 60 | grep -iE 'custom config|error|invalid|MCP'
|
|
curl -s https://librechat-test.inference.coop/api/config # appTitle, emailLoginEnabled
|
|
```
|
|
|
|
## Build + deploy the image (branding only)
|
|
|
|
> **Two traps have each silently eaten deploys of this overlay.** Both look
|
|
> like success. Run the verification block below after every deploy.
|
|
|
|
1. **`cloudron update --image <tag>` is IGNORED when this directory contains
|
|
`CloudronManifest.json`.** The CLI deploys `manifest.dockerImage` instead —
|
|
while still printing your `--image` value in its output, so it *looks*
|
|
right. The tag you build MUST be written into
|
|
`CloudronManifest.json.dockerImage` before updating.
|
|
2. **The remote builder's cache can serve stale COPY layers even when file
|
|
content changed** (observed 2026-10-19: six consecutive "Build successful"
|
|
runs produced images with old assets). Build with `--no-cache` whenever
|
|
anything under `assets/` changed.
|
|
|
|
Full deploy sequence:
|
|
|
|
```bash
|
|
cd /path/to/librechat-custom # the dir containing CloudronManifest.json
|
|
# 1. edit assets/ … then:
|
|
git add -A && git commit -m "…" && git push
|
|
SHA=$(git rev-parse --short HEAD)
|
|
# 2. build fresh layers (no-cache after asset edits)
|
|
cloudron builder build --no-cache \
|
|
--repository registry.inference.coop/co-op/librechat-custom \
|
|
--tag 20261019-$SHA
|
|
# 3. pin the manifest to the new tag — the ONLY field you may change
|
|
# (edit "dockerImage" in CloudronManifest.json to :20261019-$SHA)
|
|
git add CloudronManifest.json && git commit -m "Pin manifest to $SHA" && git push
|
|
# 4. deploy — the manifest drives it
|
|
cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" update \
|
|
--app 4023ccbf-d57a-4500-97df-14de9cf3516e --force
|
|
```
|
|
|
|
`./release.sh release` runs this same sequence with both traps handled — it
|
|
refuses to build from a dirty tree, always passes `--no-cache`, writes the tag
|
|
into the manifest before deploying, then runs the verification block. Builds are
|
|
remote (the registry's BuildKit does the work), so the only requirement is a
|
|
registry login on the host running the command; `verify`, `volume` and
|
|
`pin`/`deploy` need only `CLOUDRON_TOKEN`. `./release.sh verify` is safe to run
|
|
any time. If `build` reports no registry login, it prints both ways to bootstrap
|
|
one (a narrow app password scoped to the registry app, or the CLI device flow).
|
|
|
|
Rollback: repoint `CloudronManifest.json.dockerImage` to the previous tag
|
|
(every build is tagged `20261019-<git-sha>`; the base image
|
|
`cloudron/ai.librechat.cloudronapp:<tag>` is the pre-overlay fallback) and
|
|
rerun step 4.
|
|
|
|
### Verify after every image deploy (all four, or it didn't happen)
|
|
|
|
```bash
|
|
APP=4023ccbf-d57a-4500-97df-14de9cf3516e
|
|
# 1. What is ACTUALLY running? (CLI update output is not evidence)
|
|
curl -s -H "Authorization: Bearer $CLOUDRON_TOKEN" \
|
|
https://my.inference.coop/api/v1/apps/$APP \
|
|
| python3 -c "import json,sys; print(json.load(sys.stdin)['manifest']['dockerImage'])"
|
|
# 2. Container files match the repo (catches stale builder cache)
|
|
cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec --app $APP \
|
|
-- md5sum /app/code/librechat/client/dist/assets/custom.css
|
|
md5sum assets/custom.css
|
|
# 3. Served HTML carries the injection
|
|
curl -s https://chat.inference.coop/login | grep -Eo 'assets/custom2?\.js|assets/custom\.css'
|
|
# 4. Rendered DOM shows it (browser devtools or a browser tool). The DEPLOYED
|
|
# palette is CREAM with white text areas — the dashboard's own look. (An
|
|
# intermediate version shipped the full green wash #eaf3ea as the page
|
|
# background and read as "too green" — cream is the intended value, so this
|
|
# block asserts cream; do not "fix" a cream deploy back to the wash):
|
|
# computed background of div.bg-surface-primary == rgb(250, 248, 245);
|
|
# chat text areas == rgb(255, 255, 255); composer band == cream;
|
|
# dark scheme == rgb(34, 39, 31);
|
|
# the "Don't have an account?" <p> computes display: none.
|
|
# 5. INSTALL-TIME colors, which no DOM check covers — the web manifest and the
|
|
# index.html light meta must be #faf8f5, not the wash:
|
|
curl -s https://chat.inference.coop/manifest.webmanifest | grep -o '"theme_color":"[^"]*"'
|
|
curl -s https://chat.inference.coop/ | grep -o 'media="(prefers-color-scheme: light)" content="[^"]*"'
|
|
# 6. The scheduler task's script exists in THIS image — a manifest repoint
|
|
# deployed before its image lands leaves the key seed broken on schedule:
|
|
cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec --app $APP \
|
|
-- test -f /app/data/sync-keys.js && echo "seed path present"
|
|
```
|
|
|
|
`./release.sh verify` runs checks 1-5 plus the seed task's own log lines and the
|
|
running script's revision hash, and exits non-zero on any failure.
|
|
|
|
### Appstore pin bumps (base image)
|
|
|
|
The overlay's `FROM` is pinned to the exact appstore image the app runs. Bumping
|
|
it is a two-line change in this repo (`Dockerfile` `FROM` + the manifest's
|
|
`version`/`changelog` copied from the official manifest) plus a rebuild, but it
|
|
re-runs every build-time patch against a new base layer, so treat the overlay as
|
|
the risk, not the bump.
|
|
|
|
```bash
|
|
# what the appstore now offers (official manifest, authoritative)
|
|
curl -s https://api.cloudron.io/api/v1/apps/ai.librechat.cloudronapp \
|
|
| python3 -c "import json,sys; m=json.load(sys.stdin)['data']['manifest']; print(m['version'], m.get('upstreamVersion'), m['changelog'].strip())"
|
|
# diff it against our copy — every field must match except dockerImage (+ scheduler)
|
|
# then: rebuild, pin, deploy, ./release.sh verify, and re-check the branding block above
|
|
```
|
|
|
|
Rollback is a one-line `dockerImage` repoint to the previous tag (built images
|
|
stay in the registry), then `./release.sh deploy`.
|
|
|
|
### `CloudronManifest.json` rules
|
|
|
|
- The **only** field you may change is `dockerImage` — it must point at the
|
|
overlay tag you want deployed.
|
|
- Everything else stays a byte-identical copy of the OFFICIAL appstore
|
|
manifest. A hand-edited manifest that omits the addons (`postgresql`,
|
|
`mongodb`, `oidc`, `sendmail`, `localstorage`) strips them from the running
|
|
app and crash-loops it (`mkdir: Read-only file system`). That broke the
|
|
sandbox on 2026-10-18 — the manifest, not the overlay, was the bug.
|
|
- When the appstore package updates, refresh the file with a new copy from
|
|
the Cloudron API (`GET /api/v1/apps/<id>` → `.manifest`), repoint
|
|
`dockerImage` to your rebuilt overlay, and bump the `FROM` tag in the
|
|
Dockerfile to the new appstore `dockerImage` (pinned — never a floating tag).
|
|
|
|
## Env vars (live state)
|
|
|
|
Set in `/app/data/env` (volume — these WIN over cloudron env):
|
|
|
|
- `APP_TITLE="Inference Co-op"` (was `"LibreChat"`)
|
|
- `ALLOW_EMAIL_LOGIN=false` (was `true`) — password login removed; Cloudron
|
|
SSO (OIDC) is the only login path
|
|
- `SCHEDULES_SINGLE_PROCESS=true` — sandbox runs a single replica; silences
|
|
the scheduler's Redis guard error
|
|
|
|
Set via `cloudron env set` (process env):
|
|
|
|
- `TTS_API_KEY` / `STT_API_KEY` — **dedicated LiteLLM virtual key** (alias
|
|
`librechat-speech`, $5 / 30d budget, restricted to `tinfoil/voxtral-tts` +
|
|
`tinfoil/whisper-large-v3-turbo`). Regenerate with the gateway master key:
|
|
`POST /key/generate {"key_alias":"librechat-speech","models":[...],"max_budget":5,"budget_duration":"30d"}`
|
|
— never put the master key here.
|
|
- `IMAGE_GEN_OAI_BASEURL=https://gateway.inference.coop/v1` and
|
|
`IMAGE_GEN_OAI_MODEL=regolo/qwen-image` — the built-in agent "OpenAI Image
|
|
Tools" points at the co-op gateway. No admin key is configured; members
|
|
using the AGENT path paste their own key in the tool UI (v0.8.8
|
|
limitation). The MCP path below needs no key at all.
|
|
- `SEARXNG_INSTANCE_URL=https://search.inference.coop` — referenced from the
|
|
YAML as `${SEARXNG_INSTANCE_URL}` (self-hosted search; keyless JSON API).
|
|
|
|
## Branding overlay (custom.css / custom2.js)
|
|
|
|
Injected into the built SPA by the Dockerfile (`<link>` + `<script>` after
|
|
the `<title>` sed). What it does:
|
|
|
|
**Palette (final, restored 2026-10-27)**: light `--surface-primary` is
|
|
`250 248 245` (#faf8f5 cream) — the dashboard's true page background. Sibling
|
|
surfaces: sidebar/composer box = white cards, tertiary = #f4f2ec (dashboard
|
|
chips), hover = #f0ede6 (warm), border = #e4e0d8, `--surface-chat` = white,
|
|
`--presentation` = cream (the composer band matches the page). Green
|
|
(#5b8c5a family) stays on ACCENTS only: buttons, links (#4e7a4d), focus rings.
|
|
Loading flash = cream. Dark mode: deep pine + green-light accents, unchanged.
|
|
The green wash #eaf3ea is NOT a surface value anywhere — on the dashboard it
|
|
appears only as badges/table headers. Applying it to `--surface-primary` is
|
|
the regression that reads as "too green"; it was tried, rejected, restored by
|
|
mistake in the Oct 21 rebuild, and removed again on Oct 27.
|
|
|
|
**Fonts**: Figtree (the homepage's OFL font) is self-hosted from the design
|
|
kit (`assets/fonts/figtree-*.woff2` + OFL.txt, served at `/assets/fonts/`)
|
|
via `@font-face` in `custom.css` with a `font-family` override — LibreChat's
|
|
built-in default is Inter. Never load fonts from a CDN.
|
|
|
|
**Footer**: build-time text (no-JS fallback) + runtime link upgrade —
|
|
`custom2.js` rewrites the footer to
|
|
"[Inference Cooperative](https://inference.coop) · Our AI, governed together"
|
|
with a `dataset` guard so the MutationObserver doesn't loop.
|
|
|
|
**UI strings**: `patch_ui_strings.py` (build-time, idempotent, non-fatal)
|
|
renames the "MCP Servers" button to **"Tools"** and replaces LibreChat's
|
|
footer tagline ("Every AI for Everyone.") with the co-op footer
|
|
("Inference Cooperative · Our AI, governed together · inference.coop"),
|
|
deleting stale `.gz`/`.br` siblings of edited files. Add new string
|
|
replacements to the `REPLACEMENTS` list.
|
|
|
|
**Palette source of truth**: the DEPLOYED CSS of the co-op's own interfaces —
|
|
dashboard, portal, panel — read from the running apps. All three are identical
|
|
because they share one design system: `--cream #faf8f5` (page background),
|
|
`#ffffff` (cards), `#f4f2ec`/`#f0ede6` (chips/hover), `--green-deep #5b8c5a`
|
|
(primary/submit), `--green-mid #6ba86b` (hover), `--green-light #7ab87a`
|
|
(dark-mode accent), `--ink #2d3327` (text), `--muted #6b7a62`, `--border
|
|
#e4e0d8`, `--danger #b3543a`. Read them from the apps rather than trusting
|
|
either this file or the homepage (which carries an older `--wash #eef4ea`):
|
|
|
|
cloudron exec --app <dashboard-id> -- sh -c \
|
|
"grep -rhoE -- '--[a-z0-9-]+: *[^;]+' /app/code | sort -u"
|
|
|
|
The `html.light` / `html.dark` blocks in `custom.css` map these onto
|
|
LibreChat's Tailwind variables; keep them in sync when the design system
|
|
changes. `cloudron exec` swallows a multi-line `sh -c` silently — base64 the
|
|
script or keep it one line. Shared accessibility caveat, all four interfaces
|
|
(not chat-specific): `--muted` on cream is 4.32:1, just under AA for body
|
|
text, and white-on-`green-deep` buttons are 3.93:1 — fix everywhere or
|
|
nowhere, so the interfaces stay consistent.
|
|
|
|
**Logo & icons**: `assets/logo.svg` is the GREEN pine-dots vector (extracted
|
|
from the homepage's inline SVG — keep the `xmlns` attribute, a standalone
|
|
.svg is invalid without it). The earlier white-plate PNG variant was
|
|
invisible on the cream background and only "flashed" against the dark
|
|
loading screen. Favicons/PWA icons are the cream-plate + green-dots design
|
|
rasterized from `design-assets/favicon.svg`. The Dockerfile also cache-busts
|
|
the logo URL in the client bundle (`assets/logo.svg?v=3`) and deletes the
|
|
stale precompressed `.gz`/`.br` siblings — bump `v=` whenever the logo
|
|
changes, or browsers/service workers may keep serving an old cached copy.
|
|
|
|
- **Colors** — LibreChat's Tailwind theme resolves every color through CSS
|
|
variables set on `html.light` / `html.dark` (space-separated RGB):
|
|
`--surface-primary`, `--surface-secondary/-tertiary/-hover/-submit/-submit-hover`,
|
|
`--accent-primary/-hover`, `--ring-primary`, `--link`,
|
|
`--text-primary/-secondary`, `--border-light`. `custom.css` overrides them
|
|
(light: cream `250 248 245` surfaces + co-op green `91 140 90` accents;
|
|
dark: matching co-op dark palette). **To rebrand, edit the `html.light` /
|
|
`html.dark` blocks** — do not invent class names; `.login-page`,
|
|
`.primary` etc. do not exist in the DOM. Read current values in devtools
|
|
via `getComputedStyle(document.documentElement).getPropertyValue('--surface-primary')`.
|
|
- **No sign-up prompt** — the "Don't have an account? Sign up" line is
|
|
`<p><a href="/register">Sign up</a></p>`; hidden with
|
|
`p:has(> a[href*="register"]) { display: none !important; }` (hiding only
|
|
the `<a>` leaves the text). Any residual register/signup links are hidden
|
|
too.
|
|
- **`custom2.js`** — the auth form is client-rendered AFTER `load`, so a
|
|
plain load-time query matches nothing; a **MutationObserver** re-checks on
|
|
every DOM change and hides the sign-up paragraph whenever React renders it.
|
|
|
|
There is also an `interface.theme` block in `librechat.yaml` — it is
|
|
**forward-compatible only**: the theme engine ships in LibreChat releases
|
|
AFTER 0.8.8 and the block is silently ignored today. When the appstore
|
|
package bundles a newer LibreChat, revisit whether the CSS overlay is still
|
|
needed.
|
|
|
|
## Feature notes (LibreChat v0.8.8)
|
|
|
|
- **Chat models**: custom endpoint "Inference Co-op" → gateway, 8 chat
|
|
modelSpecs (emoji convention shared with models.md). `regolo/qwen-image`
|
|
is deliberately NOT in the chat picker (image models are not chat models;
|
|
chat-calling it was the source of the old 500s).
|
|
- **Image generation — two paths**:
|
|
1. **MCP tool (default, no key ever)**: `inference-image` server (tool
|
|
`generate_image`) — members select it in the composer's **+** palette
|
|
and just ask. Per-member billing via `customUserVars.IC_API_KEY` (seeded
|
|
from the portal) + `env: IC_API_KEY: "{{IC_API_KEY}}"`; a system-key
|
|
fallback (`system-image-fallback`, $5/30d) covers the rare empty-value
|
|
case. The `customUserVars` gate blocks a not-yet-seeded member until the
|
|
next sync (≤10 min) — accepted as a short delay over billing everything
|
|
to the system account.
|
|
2. **Agent tool**: LibreChat's built-in "OpenAI Image Tools" (env
|
|
`IMAGE_GEN_OAI_BASEURL`/`IMAGE_GEN_OAI_MODEL` → gateway; no admin key,
|
|
members paste their own). Note LibreChat v0.8.8 renders the generated
|
|
image only inside the agent conversation.
|
|
- **MCP**: `inference-docs` (tool `search_docs`, starts at boot) and
|
|
`inference-image` (tool `generate_image`, `startup: false`). Members may
|
|
add their own via the MCP Settings panel ("bring your own tools").
|
|
Permissions seeded via `interface.mcpServers` (use/create true, share false).
|
|
- **Speech**: TTS `tinfoil/voxtral-tts` (voice `casual_female`), STT
|
|
`tinfoil/whisper-large-v3-turbo`, both via the gateway with the capped key.
|
|
- **Web search**: SearXNG (self-hosted) + keyless Keenable scraper, no reranker.
|
|
|
|
|
|
## Per-member key auto-load (no API-key prompts)
|
|
|
|
Members never enter an API key. Chat and image generation resolve each
|
|
member's own portal-provisioned key automatically:
|
|
|
|
1. The portal provisions a per-member LiteLLM key at onboarding and stores it
|
|
(`members.key_token`).
|
|
2. `GET /admin/librechat/keys` on the portal returns those keys for ACTIVE
|
|
members — authenticated by the dedicated `X-Sync-Token` header
|
|
(`LIBRECHAT_SYNC_TOKEN`, set on BOTH apps; not the admin token).
|
|
3. `sync-keys.js` (this repo, deployed to `/app/data/sync-keys.js`) fetches
|
|
that feed and upserts, per member with a sandbox account:
|
|
- `keys` collection: `{userId, name: "Inference Co-op", value: encrypt(JSON.stringify({apiKey}))}`
|
|
— consumed by the chat endpoint (`apiKey: "user_provided"` in the yaml).
|
|
- `pluginauths` collection: `{userId, pluginKey: "mcp_inference-image",
|
|
authField: "IC_API_KEY", value: encrypt(key)}` — consumed by the MCP
|
|
`customUserVars` config so image generation bills to the member's own
|
|
team.
|
|
Encryption uses LibreChat's own scheme (`CREDS_KEY`/`CREDS_IV`, AES-256-CBC
|
|
via `@librechat/data-schemas` `encrypt()`).
|
|
4. Members who haven't logged in to the sandbox yet are skipped; the next run
|
|
seeds them.
|
|
|
|
### Scheduling (in this app, since 2026-10-25)
|
|
|
|
`CloudronManifest.json` declares the task in `addons.scheduler`:
|
|
|
|
```json
|
|
"scheduler": {
|
|
"librechat_key_sync": {
|
|
"schedule": "*/10 * * * *",
|
|
"command": "cd /app/code/librechat/api && node /app/code/sync-keys.js"
|
|
}
|
|
}
|
|
```
|
|
|
|
The script is **baked into the image** (`COPY sync-keys.js /app/code/…`), so it
|
|
cannot be lost with a volume restore and cannot drift from this repo.
|
|
|
|
**Ordering matters on the first deploy of this change:** build the image → deploy
|
|
(pin the new tag in the manifest first — the manifest's repointed command ships
|
|
in the same `cloudron update`) → verify (`./release.sh verify` checks that the
|
|
command path exists in the running image) → only then delete the volume copy
|
|
`/app/data/sync-keys.js`, because rolling back to the previous image tag needs it
|
|
back. After that, a change to this script requires a rebuild, not a volume push.
|
|
|
|
Seeding therefore depends on nothing outside this app — it previously ran from a
|
|
steward cron on another machine, which made a member-facing function depend on
|
|
that machine's uptime and paths.
|
|
|
|
Two consequences of how the scheduler addon works, both learned the hard way:
|
|
|
|
- **Each task runs in its own container.** The app's process env is absent there,
|
|
and `/proc/1/environ` belongs to the task launcher, not to LibreChat — so
|
|
`CREDS_KEY`/`CREDS_IV` come from `/app/data/.librechat-creds` (root:cloudron
|
|
`0640`), written from the live app env:
|
|
|
|
```bash
|
|
cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec \
|
|
--app 4023ccbf-d57a-4500-97df-14de9cf3516e \
|
|
-- sh -c 'umask 077; tr "\0" "\n" < /proc/1/environ | grep -E "^CREDS_(KEY|IV)=" > /app/data/.librechat-creds; chown root:cloudron /app/data/.librechat-creds; chmod 640 /app/data/.librechat-creds'
|
|
```
|
|
|
|
If the app's key is ever rotated, re-run that. A stale key does not corrupt
|
|
anything — the guard below refuses to write.
|
|
- **It fails closed.** `sync-keys.js` writes nothing unless `CREDS_KEY` first
|
|
decrypts an existing key row (proof it is the app's real key) and unless each
|
|
fresh ciphertext round-trips back to the plaintext. A credential problem exits
|
|
non-zero, logs `FATAL: ...`, and writes `/app/data/key-sync-diag.txt` (uid, cwd,
|
|
which env sources were readable) instead of overwriting working member keys.
|
|
|
|
Manual run (unchanged, useful after an onboarding wave):
|
|
|
|
```bash
|
|
cloudron --server my.inference.coop --token "$CLOUDRON_TOKEN" exec \
|
|
--app 4023ccbf-d57a-4500-97df-14de9cf3516e \
|
|
-- bash -c "cd /app/code/librechat/api && node /app/data/sync-keys.js"
|
|
```
|
|
|
|
Idempotent. Verify: the app log shows the task's output
|
|
(`creds check: decrypt-ok …` / `portal feed: N active member key(s)` /
|
|
`done: N seeded, M skipped`), and `keys` / `pluginauths` counts should equal the
|
|
number of members who have logged in to the sandbox.
|
|
|
|
> **Output-redaction gotcha:** terminal output renders values after
|
|
> `apiKey:` as `***` even when the file contains `user_provided` or
|
|
> `${ENV_REF}`. Never conclude a file contains a literal placeholder from
|
|
> redacted output — probe lengths/counts instead
|
|
> (`tr -cd '*' < file | wc -c`, `awk -F'apiKey: ' '{print length($2)}'`).
|
|
|
|
## Known state / limitations
|
|
|
|
- The `[schedules]` Redis guard error is resolved by
|
|
`SCHEDULES_SINGLE_PROCESS=true` (above).
|
|
- The `RAG API not reachable` warning is expected (no rag_api addon); it only
|
|
affects LibreChat's own file-RAG, which we do not use.
|
|
- The `inference-image` MCP server logs `Tools: undefined` at boot —
|
|
expected: with `startup: false` it registers at boot and spawns per-user
|
|
on first use in a conversation.
|
|
- Members' chat usage bills per-member via their endpoint key; the MCP image
|
|
path is currently keyless and bills to the gateway (not enforced
|
|
per-member). Wiring per-member key injection into LibreChat chat is an
|
|
open task.
|
|
- Repo ↔ live parity last verified 2026-10-19: `librechat.yaml`,
|
|
`mcp/server.py`, `mcp/image_server.py` md5s match the live volume; overlay
|
|
image `registry.inference.coop/co-op/librechat-custom:20261019-fc9b11e` is
|
|
deployed (container asset md5s match this repo).
|
|
|
|
## PWA lessons (2026-10-22 — icon/band debugging)
|
|
|
|
- **Builder COPY layers can be stale**: after editing any asset, build with
|
|
`--no-cache` and VERIFY the served file's md5 against the repo before
|
|
reporting success. (The 20261022-df55d7f build shipped a stale maskable icon
|
|
this way — the repo was right, the image wasn't.)
|
|
- **PWA icon URLs must carry a per-build `?v=`**: they are served with
|
|
`max-age=172800`, and remove/re-add of the PWA does not clear the phone's
|
|
HTTP cache — launchers keep rendering the previous build's icon bytes.
|
|
patch_index.py injects the query into every manifest icon src per build.
|
|
- **Maskable icons**: logo must sit inside the safe zone — ~66% of the canvas
|
|
on a solid brand background. Full-bleed logos get cropped ("zoomed in").
|
|
- **Firefox Android**: `crossorigin="use-credentials"` on the manifest link
|
|
breaks icon resolution (blank launcher icon) — patch_index.py strips it.
|
|
- **Theme bars**: the manifest `theme_color` is one static value; Chromium's
|
|
nav bar and all Firefox bars use it even when the meta says otherwise. The
|
|
durable fix is `viewport-fit=cover` + standalone-mode safe-area padding
|
|
(custom.css) so the bands show the theme-correct page background; custom2.js
|
|
additionally syncs one meta `theme-color` to the app's resolved theme.
|