The sync seeds each member's gateway key into LibreChat's per-user key store, but it can only seed a user that already exists — and LibreChat creates that row on first login. Ten minutes therefore left a new member's very first message to fail against a key-less endpoint. At one minute the window is at most 60 seconds. Zero would mean patching LibreChat's login path, which is not worth a fork for this.
The key is auto-seeded within 10 minutes of first login by sync-keys.js, but a new user's first screen is the API-keys panel, which reads as mandatory. The welcome line is the surface we control; the settings dialog builds its strings from templates in a component chunk with no en locale bundle, so it is not cleanly patchable. Option 1 of three in the issue, approved by the GM.
LibreChat's PluginAuth schema keys on a string userId; Key (the chat key
collection) is ObjectId-typed and Mongoose casts for it. The seeder used
`user._id` (an ObjectId) for BOTH, so every MCP auth row it wrote was
invisible to PluginService.getUserPluginAuthValue():
string lookup -> 0 rows (what the app does)
ObjectId lookup-> 1 row
Consequence: userMCPAuthMap['mcp_inference-image'] was empty for every
member, ToolService skipped the server ("missing user configuration"), and
any agent expecting MCP tools failed with
AgentToolInitializationError: "The agent is configured to use MCP tools, but
none are available" — 7 times since Oct 19, most recently 2026-10-27 22:29.
The image-generation MCP server (generate_image) was unavailable to everyone.
The docs server was unaffected (it declares no customUserVars).
Fix: String(uid) in the two pluginauths upserts only. The keys[] upserts are
deliberately left as ObjectId — chat key lookup works and changing it would
break the working half.
Verified: 54 existing rows migrated ObjectId -> string (backup at
/app/data/backup-pluginauths-userid-2026-10-27T2239Z.json), all 55 image rows
now string-typed, the app-style string lookup returns a row, and a full
seeder run passes its own decrypt guard and encrypt/decrypt round-trip check
(54 seeded, 17 skipped).
With enforce: true (in place since at least Oct 23), every send whose body
carries no `spec` is rejected by api/server/middleware/buildEndpointOption.js
with "No model spec selected", surfaced to members as "Inference Co-op could
not complete this request."
400 of 547 conversations across 37 users had no `spec` — all created before
the spec system (newest Oct 20). They fail on the members' first send;
new chats were fine because the softDefault spec pre-selects.
enforce: false lets those conversations send with their own stored model
exactly as before Oct 20. prioritize stays true, so the spec picker is still
the primary chooser — it is just no longer mandatory. Verified: the change is
read at boot (no watcher in the image), so the app was restarted; the served
stylesheet md5 is unchanged by the restart; the mirror is byte-identical to
the deployed file (sha256 219dbdf2ca0b64f8…). Backup of the previous config
stays on the volume as /app/data/librechat.yaml.bak-20261027T221149Z.
If enforcement is ever restored, backfill `spec` on those conversations first,
mapping by each conversation's own `model` (only exact matches — a mismatch
triggers "Model spec mismatch" instead).
The release chain read the tag back from CloudronManifest.json BEFORE pinning
it, so t was always the previous tag: release built the new image, then
deployed the old one and printed success — the exact silent no-op this script
exists to prevent. Its usage text promises build -> pin -> deploy -> verify;
it did build -> deploy.
cmd_build now exports the tag it built and release pins THAT, then deploys and
verifies. Not exercised end-to-end yet (the theme deploy that found this bug
was run as separate build/pin/deploy steps and passed all checks); the edit is
syntax-checked only.
Follow-up to the custom.css/custom2.js change: the Dockerfile seds the web
manifest theme_color/background_color and the index.html light meta to
#eaf3ea, and the README's verify block asserted the green wash as correct
("an earlier version of this block asserted cream ... would have led you to
fix a correct deploy"). Both are the same regression: the wash was rejected
earlier, restored by accident, and then documented as intended.
- Dockerfile: manifest + index meta -> #faf8f5 (cream), with an explicit
"keep these three in sync" note (this sed, custom.css html:not(.dark),
custom2.js COOP_BG.light).
- README verify block: asserts cream again, plus a new step for the
install-time colors no DOM check covers (manifest + light meta).
- README palette section: hover is #f0ede6 not the wash; --presentation is
cream; source of truth is the DEPLOYED CSS of dashboard/portal/panel, with
the read command; records the shared 4.32:1 / 3.93:1 accessibility caveat;
notes that cloudron exec silently swallows multi-line sh -c.
v3 used the green wash #eaf3ea for --surface-primary, the page background,
which read as too green. Replaced with the palette read from the DEPLOYED
CSS of the dashboard, portal and panel (identical across all three):
--surface-primary #eaf3ea -> #faf8f5 (cream, page background)
--surface-tertiary green -> #f4f2ec (warm chip)
--surface-hover green -> #f0ede6 (warm hover)
--surface-composer-hover -> #f0ede6
--border-light #cde0cb -> #e4e0d8 (warm neutral border)
--presentation #eaf3ea -> #faf8f5 (composer band)
--link #4a7a4a -> #4e7a4d (deeper green, 4.70:1 on cream)
Accents are unchanged (green-deep #5b8c5a, green-mid #6ba86b) — they are
the earthy greens the co-op design system already uses. Dark mode is
untouched. Also fixed the two hard-coded washes: the #loading-container
flash and the bare html/body background, plus custom2.js COOP_BG.light,
which was still painting the browser/PWA chrome #eaf3ea while the
manifest said cream.
A fetch seconds after 'App is updated' can come back empty while the container
recreates, which false-negatived the x-coop-build check on the 1.4.5 deploy
(2026-10-26). http_body retries 3x before declaring failure.
REGISTRY_BUILD_TOKEN (+ optional REGISTRY_BUILD_USERNAME) lets a build run from
any host with the token in its environment, no ~/.cloudron.json builder config
required. The narrow credential (app password scoped to the registry app) is the
least-privileged way to hand a host build rights; with no token and no builder
config the script prints both bootstrap routes and exits instead of guessing.
I had it wrong: `cloudron builder build` uploads the build context and the
registry app's BuildKit builds it — the caller needs no docker, only a registry
login. The guard now checks for that login and prints both bootstrap routes
(narrow app password scoped to the registry app, or the CLI device flow) instead
of blaming a missing docker binary.
Not deployed. Diffed against the official 1.4.5 manifest: version, changelog and
the base-image tag were the only differences, so this copy now matches official
field-for-field except dockerImage (+ our scheduler block).
The risk is the overlay, not the bump: every build-time patch runs against a new
base layer, and they are all non-fatal by design, so a silent no-op is the failure
mode. Verify on the rendered DOM. Rollback = repoint dockerImage to
20261023-65a38bb.
Ready to build, not yet deployed — deploying the repointed scheduler command
without the new image would break the seed on schedule, which is why this sits on
a branch with the ordering written down.
- Dockerfile: COPY sync-keys.js /app/code/sync-keys.js; rule 3 in the header now
states the one exception (operational scripts that must not depend on a volume).
- CloudronManifest.json: scheduler command repointed to /app/code/sync-keys.js.
- README: ordering (build -> deploy -> verify -> delete the volume copy), and the
table row moved from the volume layer to the image layer.
After deploy: delete /app/data/sync-keys.js, but keep it until you are past the
rollback window — the previous image tag still runs the volume copy.
Three drifts between this repo and reality, found while moving the per-member key
seed into the app:
- The 'verify after every image deploy' block asserted the CREAM palette
(rgb(250,248,245)). The deployed palette has been the green wash
(rgb(234,243,234) with white text areas) for some time, so a correct deploy
would have read as a failure — and invited someone to 'fix' the CSS back.
- The block still verified against librechat-test.inference.coop; the swap to
chat.inference.coop happened 2026-10-21 and that alias is not provisioned.
- Nothing checked that the scheduler task's script exists in the running image,
which is exactly the failure mode a manifest repoint can introduce.
release.sh encodes the documented release flow (dirty-tree guard, always
--no-cache, pin-before-deploy) and its verify subcommand runs checks 1-5 plus the
seed task's log lines. sync-keys.js now prints its own md5 prefix on every run,
so a stale volume copy can never hide again — the repo copy's md5 is the
authority.
Verified: ./release.sh verify passes all six checks against the live app;
./release.sh volume sync-keys.js confirms md5 parity.
The first scheduled run failed closed (good) — Cloudron's scheduler addon runs
each task in its OWN container, so the app's process env is absent there and
/proc/1/environ belongs to the task launcher, not to LibreChat. CREDS_KEY/CREDS_IV
now come from /app/data/.librechat-creds (root:cloudron 0640), written from the
live app env; a stale key makes the decrypt-against-existing-rows guard refuse to
write rather than corrupt member keys. Credential failure now also writes
/app/data/key-sync-diag.txt so a future failure explains itself.
Verified: the 22:40 scheduled fire logged 'creds check: decrypt-ok (25 existing
row(s))' / 'done: 50 seeded, 15 skipped'.
The per-member key seed ran from a steward cron on the co-op agent VM, shelling
into this app over the Cloudron API. That made a member-facing function depend on
a separate machine's uptime and paths (it went silently stale once, exiting 127
on schedule while the job looked healthy).
- CloudronManifest.json: addons.scheduler gains librechat_key_sync
(cd /app/code/librechat/api && node /app/data/sync-keys.js, */10 * * * *).
Official addon set (postgresql, mongodb, localstorage, oidc, sendmail) and the
dockerImage pin are untouched.
- sync-keys.js: refuse to write unless CREDS_KEY can first decrypt an existing
key row, and unless each fresh ciphertext round-trips back to the plaintext.
Scheduled runs get a different environment than an interactive exec, so a wrong
or absent CREDS_KEY must fail loudly instead of overwriting working member keys.
A host-side daily job creates the next day's dated log files as root:root at
00:00 UTC; LibreChat (cloudron user) then EACCES-crashes on its first log
write of the day (2026-10-22 and 2026-10-23 incidents). The guard pre-creates
today's and tomorrow's files as cloudron and repairs strays every minute.
Defense-in-depth until the host rotation config is fixed.
- maskable-icon.png regenerated: logo at ~66% on the brand background —
inside the maskable safe zone, so Android no longer crops it ('zoomed in').
- index.html: viewport-fit=cover + manifest link without
crossorigin=use-credentials (Firefox Android fails to resolve manifest
icons on credentialed fetches — missing launcher icon).
- custom.css: standalone-mode safe-area padding so app content clears the
system bars; the bands show the theme-correct page background.
The Oct 19 theming (light = #eaf3ea dashboard green wash, white cards/surfaces,
per approval) was never committed - the Oct 21 pin-migration rebuild from this
repo silently reverted the deployed theme to the older cream iteration.
- custom.css: light block restored to the approved green-wash palette
(primary #eaf3ea, white secondary/chat cards, green hover/tertiary/border);
loading screen + bare page background match.
- Dockerfile: manifest theme_color/background_color and the index.html
theme-color media pair now follow the live palette (#eaf3ea light /
#22271f deep pine dark).
- custom2.js: sync ONE meta theme-color to the app's resolved theme
(html.dark class + prefers-color-scheme listener) so Android status/nav
bands and iOS safe areas track the theme at runtime; the static media
pair remains as the no-JS fallback.
inference-bot
inference-admin
Nathan Schneider