co-op/support
2
0 Fork 0
Code Issues 15 Pull Requests Actions Packages Projects Releases Wiki Activity

Security & bug audit — October 2026 (third pass): 1 critical, 2 high, sandbox debug-mode risk #24

New Issue
Closed
opened 2026-10-18 19:15:09 +00:00 by inference-bot · 1 comment
inference-bot commented 2026-10-18 19:15:09 +00:00
Owner
Copy link
Copy source

Security & bug audit — October 2026 (third pass)

Scope: all 17 Cloudron apps, the deploy pipeline, secrets hygiene, key
management, the LibreChat sandbox, and known-bug sweep. All findings verified
live against production (2026-10-18).

Findings by severity

CRITICAL (open)

A1. No off-box backups — STILL OPEN from the last two passes.
Backup provider = None. Nightly backups land on the same VPS disk as the
data they protect. The VPS expiry earlier this month proved the blast radius
is real: an expired hosting bill took down the service AND its backups
together. Fix: configure encrypted external backup storage (Infomaniak
Swiss Backup pairs naturally with the planned migration) and test a restore.

HIGH

A2. Nathan's chat key row has an empty hash in the portal DB.
member_api_keys row (n@nathanschneider.info, name=chat) has hash='' —
introduced during the manual rotation-fix session (a failed key generation
attempt updated the row with an empty hash before the successful retry
restored the fingerprint but not the hash). The key WORKS (it's stored
correctly in LibreChat and LiteLLM) but cannot be revoked from the
dashboard — the revoke flow resolves by hash. Impact: low (Nathan can
rotate it instead), but it's a data-integrity bug in a member-facing flow.
Fix: rotate the key (which now works post-fix) — the new row will carry its
hash.

A3. Orphaned LiteLLM resources from testing (cleanup needed):

  • member:libretest@inference.coop team — recreated when libretest was
    accidentally re-added to MANUAL_MEMBERS during the test-account churn.
    Zero spend, but it's a live team on the gateway.
  • 3 LiteLLM keys with key_alias IS NULL, team_id IS NULL (created
    2026-10-02, one with spend) — predate the alias-required enforcement;
    they authenticate against the gateway with no team budget cap and no
    revocation handle. These are unscoped, uncapped, irrevocable keys.
    Fix: delete by token via the DB, then re-check.
  • member:libretest2@inference.coop team + Cloudron user — intentional
    test account; keep or remove with the migration decision.

MEDIUM

A4. LibreChat sandbox runs in DEBUG MODE (writable rootfs, no memory
limit, repair-mode entrypoint). Required for the live-patched auto-key flow

  • branding. Risk: (a) an app container escape gains a writable rootfs,
    (b) any container recreate silently regresses the auto-key + branding
    patches. Mitigation: this is the documented stopgap until the package fork
    bakes the patches; the sandbox holds no member data beyond Nathan's + the
    test account's.

A5. Three alien-alias keys authenticate to the gateway with NO team.
(Same as A3's null-alias keys — called out separately because they bypass
the per-member budget enforcement that is our billing model.) If these keys
were leaked they'd spend against the master budget, not a member cap.

A6. enableAutomaticUpdate: True on the sandbox — if Cloudron
auto-updates the LibreChat package, the container recreates and the
debug-mode patches (auto-key, branding, greeting) silently vanish.
Fix: set automatic update OFF for the sandbox until the fork exists.

LOW / accepted

A7. Member keys plaintext in portal DB — unchanged from last pass
(accepted trade-off; per-request injection requires it).

A8. MANUAL_MEMBERS holds 9 founder emails — 3 of whom (per the
members table) may no longer be active. The env list grants provisioning
rights. Low risk (provisioning ≠ spending without OC membership), worth a
quarterly review.

A9. Security headers on the portal are sparse (no CSP/HSTS on the API
surface — Cloudron sets HSTS but no CSP). The portal is an API (no
user-content rendering), so CSP matters less than for the chat surfaces,
but a baseline CSP would be cheap defense-in-depth.

CLEAN (verified this pass)

  • Auth on all sensitive endpoints: portal admin/broker/rotate (401),
    gateway key/team/health/spend (401), registry API + /v2 (401) —
    all refuse unauthenticated requests.
  • Rate limiting: broker 30/min confirmed live (5 rapid bad-secret
    attempts → all 401, no lockout bypass visible).
  • CORS: no ACAO header on portal GET routes; preflight rejected.
  • Repos ↔ live config: gateway config.yaml byte-identical between
    repo and live (md5 cec0361fe946...). Portal runs the rebuilt image
    with the rotation fix. No drift detected.
  • No committed secrets in recent commits across all 9 repos
    (message-level scan; full-diff scan was clean last pass).
  • Security headers: main site + chat carry HSTS, nosniff,
    frame-deny/ancestors, referrer-policy.
  • Certificates: all current, expiring Dec 2026–Jan 2027 (auto-renew).
  • Test-account hygiene: sandboxtest + libretest deleted fully
    (Cloudron, portal, LiteLLM, LibreChat); libretest2 remains as the
    documented test account.
  • Cloudron users: 53 total; 5 never-accepted invites (jdietz, liz,
    ag, fawnmelissa, rfagen) — pending members, not stale.

Bug sweep (known open items)

Bug Status
LibreChat "scheduler NOT started" startup warning Present on every boot; benign single-replica assert (documented upstream); no functional impact observed
OWUI EPUB/DOCX spaCy fix Deployed + verified (issue #20 closed)
Portal key rotation 400 Fixed permanently (rebuilt image, verified live)
Nathan chat key empty hash NEW finding (A2) — needs one rotation to self-heal
LiteLLM spend-flush lag Async by design; documented quirk

Recommended action order

  1. Delete the 3 null-alias/null-team keys (A3/A5 — irrevocable, uncapped).
  2. Rotate Nathan's chat key (A2 — self-heals the hash).
  3. Delete the libretest member team (A3).
  4. Turn OFF auto-update on the sandbox (A6).
  5. Configure external backups (A1 — third pass carrying this; the
    Infomaniak migration is the natural moment, but the VPS expiry proved
    the risk is now, not later).
# Security & bug audit — October 2026 (third pass) Scope: all 17 Cloudron apps, the deploy pipeline, secrets hygiene, key management, the LibreChat sandbox, and known-bug sweep. All findings verified live against production (2026-10-18). ## Findings by severity ### CRITICAL (open) **A1. No off-box backups — STILL OPEN from the last two passes.** Backup provider = None. Nightly backups land on the same VPS disk as the data they protect. The VPS expiry earlier this month proved the blast radius is real: an expired hosting bill took down the service AND its backups together. **Fix: configure encrypted external backup storage (Infomaniak Swiss Backup pairs naturally with the planned migration) and test a restore.** ### HIGH **A2. Nathan's `chat` key row has an empty hash in the portal DB.** `member_api_keys` row (n@nathanschneider.info, name=chat) has `hash=''` — introduced during the manual rotation-fix session (a failed key generation attempt updated the row with an empty hash before the successful retry restored the fingerprint but not the hash). The key WORKS (it's stored correctly in LibreChat and LiteLLM) but **cannot be revoked from the dashboard** — the revoke flow resolves by hash. Impact: low (Nathan can rotate it instead), but it's a data-integrity bug in a member-facing flow. Fix: rotate the key (which now works post-fix) — the new row will carry its hash. **A3. Orphaned LiteLLM resources from testing (cleanup needed):** - `member:libretest@inference.coop` team — recreated when libretest was accidentally re-added to MANUAL_MEMBERS during the test-account churn. Zero spend, but it's a live team on the gateway. - 3 LiteLLM keys with `key_alias IS NULL`, `team_id IS NULL` (created 2026-10-02, one with spend) — predate the alias-required enforcement; they authenticate against the gateway with no team budget cap and no revocation handle. **These are unscoped, uncapped, irrevocable keys.** Fix: delete by token via the DB, then re-check. - `member:libretest2@inference.coop` team + Cloudron user — intentional test account; keep or remove with the migration decision. ### MEDIUM **A4. LibreChat sandbox runs in DEBUG MODE** (writable rootfs, no memory limit, repair-mode entrypoint). Required for the live-patched auto-key flow + branding. Risk: (a) an app container escape gains a writable rootfs, (b) any container recreate silently regresses the auto-key + branding patches. Mitigation: this is the documented stopgap until the package fork bakes the patches; the sandbox holds no member data beyond Nathan's + the test account's. **A5. Three alien-alias keys authenticate to the gateway with NO team.** (Same as A3's null-alias keys — called out separately because they bypass the per-member budget enforcement that is our billing model.) If these keys were leaked they'd spend against the master budget, not a member cap. **A6. `enableAutomaticUpdate: True` on the sandbox** — if Cloudron auto-updates the LibreChat package, the container recreates and the debug-mode patches (auto-key, branding, greeting) silently vanish. Fix: set automatic update OFF for the sandbox until the fork exists. ### LOW / accepted **A7. Member keys plaintext in portal DB** — unchanged from last pass (accepted trade-off; per-request injection requires it). **A8. `MANUAL_MEMBERS` holds 9 founder emails** — 3 of whom (per the members table) may no longer be active. The env list grants provisioning rights. Low risk (provisioning ≠ spending without OC membership), worth a quarterly review. **A9. Security headers on the portal are sparse** (no CSP/HSTS on the API surface — Cloudron sets HSTS but no CSP). The portal is an API (no user-content rendering), so CSP matters less than for the chat surfaces, but a baseline CSP would be cheap defense-in-depth. ### CLEAN (verified this pass) - **Auth on all sensitive endpoints**: portal admin/broker/rotate (401), gateway key/team/health/spend (401), registry API + /v2 (401) — all refuse unauthenticated requests. - **Rate limiting**: broker 30/min confirmed live (5 rapid bad-secret attempts → all 401, no lockout bypass visible). - **CORS**: no ACAO header on portal GET routes; preflight rejected. - **Repos ↔ live config**: gateway config.yaml byte-identical between repo and live (md5 cec0361fe946...). Portal runs the rebuilt image with the rotation fix. No drift detected. - **No committed secrets** in recent commits across all 9 repos (message-level scan; full-diff scan was clean last pass). - **Security headers**: main site + chat carry HSTS, nosniff, frame-deny/ancestors, referrer-policy. - **Certificates**: all current, expiring Dec 2026–Jan 2027 (auto-renew). - **Test-account hygiene**: sandboxtest + libretest deleted fully (Cloudron, portal, LiteLLM, LibreChat); libretest2 remains as the documented test account. - **Cloudron users**: 53 total; 5 never-accepted invites (jdietz, liz, ag, fawnmelissa, rfagen) — pending members, not stale. ## Bug sweep (known open items) | Bug | Status | |---|---| | LibreChat "scheduler NOT started" startup warning | Present on every boot; benign single-replica assert (documented upstream); no functional impact observed | | OWUI EPUB/DOCX spaCy fix | Deployed + verified (issue #20 closed) | | Portal key rotation 400 | **Fixed permanently** (rebuilt image, verified live) | | Nathan chat key empty hash | **NEW finding (A2)** — needs one rotation to self-heal | | LiteLLM spend-flush lag | Async by design; documented quirk | ## Recommended action order 1. **Delete the 3 null-alias/null-team keys** (A3/A5 — irrevocable, uncapped). 2. **Rotate Nathan's chat key** (A2 — self-heals the hash). 3. **Delete the libretest member team** (A3). 4. **Turn OFF auto-update on the sandbox** (A6). 5. **Configure external backups** (A1 — third pass carrying this; the Infomaniak migration is the natural moment, but the VPS expiry proved the risk is now, not later).
inference-admin commented 2026-10-18 22:44:31 +00:00
Owner
Copy link
Copy source

Resolved.

Resolved.
Sign in to join this conversation.
No Branch/Tag Specified
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
inference-admin (Admin) inference-bot (InferenceBot)
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: co-op/support#24