docs: clickable Key links, footer source link, move operator material to co-op/ops-docs
- README: render the Key links as real Markdown links (mailto: for the contact address); drop the 'lives in a public git repository' note now that the footer carries the source link. - mkdocs.yml: add a theme override (custom_dir: overrides) that puts a 'Documentation source' link in the footer of every page. - Move the operator runbook, maintenance notes and incident write-up out of the published site into the git-only co-op/ops-docs repository; remove them from the nav and repoint cross-references at Gitea.
This commit is contained in:
inference-bot committed
1 parent
c95bae161b
commit
e0b9c0cd7e
8 files changed
+42
-426
No files matched your search
@@ -20,13 +20,13 @@ funded through our [Open Collective](https://opencollective.com/inference-cooper
|
|||||||
|
|
||||||
## Key links
|
## Key links
|
||||||
|
|
||||||
- **Website:** https://inference.coop
|
- **Website:** [inference.coop](https://inference.coop)
|
||||||
- **AI Chat:** https://chat.inference.coop
|
- **AI Chat:** [chat.inference.coop](https://chat.inference.coop)
|
||||||
- **Matrix chat:** https://matrix.inference.coop
|
- **Matrix chat:** [matrix.inference.coop](https://matrix.inference.coop)
|
||||||
- **Member Dashboard:** https://dashboard.inference.coop
|
- **Member Dashboard:** [dashboard.inference.coop](https://dashboard.inference.coop)
|
||||||
- **Governance forum:** https://forum.inference.coop
|
- **Governance forum:** [forum.inference.coop](https://forum.inference.coop)
|
||||||
- **Source code:** https://git.inference.coop
|
- **Source code:** [git.inference.coop](https://git.inference.coop)
|
||||||
- **Contact:** info@inference.coop
|
- **Contact:** [info@inference.coop](mailto:info@inference.coop)
|
||||||
|
|
||||||
## Governance & legal
|
## Governance & legal
|
||||||
|
|
||||||
@@ -34,7 +34,3 @@ funded through our [Open Collective](https://opencollective.com/inference-cooper
|
|||||||
- [Open questions](open-questions.md) — decisions currently up for member discussion.
|
- [Open questions](open-questions.md) — decisions currently up for member discussion.
|
||||||
- [Terms of Service](terms-of-service.md)
|
- [Terms of Service](terms-of-service.md)
|
||||||
- [Privacy Policy](privacy-policy.md)
|
- [Privacy Policy](privacy-policy.md)
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
*This documentation lives in a public git repository. Contributions welcome.*
|
|
||||||
@@ -1,34 +0,0 @@
|
|||||||
# 2026-10-02 — gateway outage after a security-upgrade rebuild
|
|
||||||
|
|
||||||
*Written by InferenceBot on the `incident/2026-10-02-gateway-cve-upgrade`
|
|
||||||
branch on 2026-10-02 (05:35 UTC, a minute after service returned; the commit's local timestamp reads 2026-10-01 23:35 −06:00); moved to `main` with the substance unchanged so it's
|
|
||||||
findable.*
|
|
||||||
|
|
||||||
**Impact.** The gateway (chat and API) was down about 45 minutes,
|
|
||||||
04:50–05:34 UTC. No data loss.
|
|
||||||
|
|
||||||
**What happened.** A Cloudron update triggered by a member's CVE report
|
|
||||||
rebuilt the app image. The Dockerfile pinned the floating
|
|
||||||
`main-v1.74.0-stable` tag, which the registry had silently moved to newer
|
|
||||||
(v1.84.0-era) bits, so the rebuild changed the running version and kicked
|
|
||||||
off a full Prisma database migration that takes 10–15 minutes per boot with
|
|
||||||
internal timeouts and retries. The migration ran at Cloudron's default
|
|
||||||
256 MB memory cap and was OOM-killed; the app crash-looped on a missing
|
|
||||||
migrations ledger until the memory limit and `LITELLM_MIGRATION_DIR` were
|
|
||||||
fixed.
|
|
||||||
|
|
||||||
**Also during the incident.** The domain's authoritative nameservers
|
|
||||||
(yoursrs.com) were intermittently unreachable, briefly failing resolution of
|
|
||||||
some hostnames (git, portal). Transient, registrar-side, self-recovered; no
|
|
||||||
records were wrong. The upgrade had also silently reverted CORS to `*`.
|
|
||||||
|
|
||||||
**Repairs.** Version pinned to the exact tag in `code/litellm`;
|
|
||||||
`memoryLimit` set to 2 GB in the manifest; `LITELLM_MIGRATION_DIR` pointed
|
|
||||||
at the persistent volume; `LITELLM_CORS_ORIGINS` set to the chat origin;
|
|
||||||
packaging synced to mirror the live deployment. The upgrade rules learned
|
|
||||||
here are in the `code/litellm` README under "Updating LiteLLM".
|
|
||||||
|
|
||||||
**What it fixed on the security side.** CVE-2026-35029 (authorization bypass
|
|
||||||
on `/config/update`, fixed in 1.83.0) and CVE-2026-59822 (MCP session auth
|
|
||||||
bypass, fixed in 1.84.0, listed in CISA's Known Exploited Vulnerabilities
|
|
||||||
catalog). The gateway has since moved to v1.103.2.
|
|
||||||
+10
-3
@@ -299,9 +299,16 @@ through it; the other providers are called directly.
|
|||||||
|
|
||||||
Maintaining the co-op's infrastructure — rotating Cloudron tokens, editing the
|
Maintaining the co-op's infrastructure — rotating Cloudron tokens, editing the
|
||||||
portal's environment, and troubleshooting the provisioning pipeline — is
|
portal's environment, and troubleshooting the provisioning pipeline — is
|
||||||
documented in the [operator runbook](operator-token-runbook.md).
|
documented in the [operator
|
||||||
|
runbook](https://git.inference.coop/co-op/ops-docs/src/branch/main/operator-token-runbook.md).
|
||||||
|
All operator-facing material (that runbook, the maintenance notes, and incident
|
||||||
|
write-ups) lives in the git-only
|
||||||
|
[`co-op/ops-docs`](https://git.inference.coop/co-op/ops-docs) repository, which
|
||||||
|
is deliberately not published on this site.
|
||||||
|
|
||||||
## Incident log
|
## Incident log
|
||||||
|
|
||||||
Write-ups for outages live in [`incidents/`](incidents/). One so far:
|
Write-ups for outages live in the operator docs, in
|
||||||
[2026-10-02 — gateway outage after a security-upgrade rebuild](incidents/2026-10-02-gateway-cve-upgrade.md).
|
[`incidents/`](https://git.inference.coop/co-op/ops-docs/src/branch/main/incidents).
|
||||||
|
One so far:
|
||||||
|
[2026-10-02 — gateway outage after a security-upgrade rebuild](https://git.inference.coop/co-op/ops-docs/src/branch/main/incidents/2026-10-02-gateway-cve-upgrade.md).
|
||||||
@@ -1,27 +0,0 @@
|
|||||||
# Chat task models (titles, tags, suggestions)
|
|
||||||
|
|
||||||
Both chat front doors generate small helper outputs — conversation titles,
|
|
||||||
tags, retrieval queries — with a model separate from the one answering the
|
|
||||||
member. These tasks are pinned to a small, inexpensive model so they never
|
|
||||||
bill against the member's chosen (larger) model.
|
|
||||||
|
|
||||||
## Open WebUI (chat.inference.coop)
|
|
||||||
|
|
||||||
- Title generation and tag generation are **enabled**; follow-up reply
|
|
||||||
suggestions and input autocomplete are **disabled** (October 2026).
|
|
||||||
- The task model (`task.model.default` in the Postgres `config` table) is
|
|
||||||
pinned to `greenpt/glm-5.3-flash` — a small, inexpensive model — so titles
|
|
||||||
and tags always run on it regardless of which model the member is chatting
|
|
||||||
with. Before October 2026 it was empty, which meant helper tasks ran on the
|
|
||||||
member's selected model.
|
|
||||||
- Config lives in the app's Postgres database (`config` table, key/value rows
|
|
||||||
with JSON values) and is read at startup: change a row, then restart the
|
|
||||||
app (`cloudron restart --app <id>`; note `restart` takes no `--timeout`).
|
|
||||||
|
|
||||||
## LibreChat (librechat-test.inference.coop)
|
|
||||||
|
|
||||||
- Title generation is configured in `/app/data/librechat.yaml`:
|
|
||||||
`titleConvo: true`, `titleModel: greenpt/glm-5.3-flash`,
|
|
||||||
`titleMethod: completion` — the same small GreenPT model.
|
|
||||||
- LibreChat v0.8.8 has no reply-suggestion feature; there is nothing to
|
|
||||||
disable on that front.
|
|
||||||
@@ -1,156 +0,0 @@
|
|||||||
# LibreChat sandbox maintenance
|
|
||||||
|
|
||||||
The LibreChat evaluation instance at **librechat-test.inference.coop**
|
|
||||||
(Cloudron app `4023ccbf-d57a-4500-97df-14de9cf3516e`, appstore package
|
|
||||||
`ai.librechat.cloudronapp@1.4.1`, which bundles **LibreChat v0.8.8**).
|
|
||||||
|
|
||||||
The single source of truth for the deployment is the repo
|
|
||||||
**[co-op/librechat-custom](https://git.inference.coop/co-op/librechat-custom)**
|
|
||||||
— its README is the full runbook (config deploy, image build/deploy, rollback,
|
|
||||||
env vars, branding overlay, known state). This doc is the operator-facing
|
|
||||||
summary.
|
|
||||||
|
|
||||||
## Architecture: what lives where
|
|
||||||
|
|
||||||
- **App config** — `/app/data/librechat.yaml` (persistent volume, mirrored
|
|
||||||
byte-identically in the repo). Holds the "Inference Co-op" custom endpoint
|
|
||||||
(gateway), the 8-model chat picker, speech, web search, and the two MCP
|
|
||||||
servers. Volume edits deploy via `cloudron exec` (base64), never via the
|
|
||||||
image — the image never touches `/app/data`.
|
|
||||||
- **Env file** — `/app/data/env` (persistent volume). **`start.sh` sources it
|
|
||||||
AFTER the process env, so it wins over `cloudron env`** — that is where
|
|
||||||
`APP_TITLE="Inference Co-op"`, `ALLOW_EMAIL_LOGIN=false` (password login
|
|
||||||
removed; Cloudron SSO is the only login path), and
|
|
||||||
`SCHEDULES_SINGLE_PROCESS=true` must live.
|
|
||||||
- **Other env vars** — via `cloudron env`: `TTS_API_KEY`/`STT_API_KEY`
|
|
||||||
(dedicated LiteLLM virtual key, alias `librechat-speech`, $5/30d, audio
|
|
||||||
models only — never the gateway master key), `IMAGE_GEN_OAI_BASEURL` +
|
|
||||||
`IMAGE_GEN_OAI_MODEL` (agent image tool → gateway), `SEARXNG_INSTANCE_URL`.
|
|
||||||
- **MCP servers** — `/app/data/mcp/server.py` (docs search, `search_docs`,
|
|
||||||
starts at boot) and `/app/data/mcp/image_server.py` (image generation,
|
|
||||||
`generate_image`, `startup: false`); both mirrored in the repo under
|
|
||||||
`mcp/`, stdlib-only Python.
|
|
||||||
- **Branding overlay** (logo, favicons, PWA icons, `<title>`, `custom.css`,
|
|
||||||
`custom2.js`) — baked into a pinned overlay image `FROM` the official
|
|
||||||
appstore image. Deployed tag (last verified 2026-10-19):
|
|
||||||
`registry.inference.coop/co-op/librechat-custom:20261019-fc9b11e`.
|
|
||||||
- **Branding CSS/JS** — `custom.css` overrides LibreChat's Tailwind theme
|
|
||||||
variables on `html.light`/`html.dark` using the HOMEPAGE's canonical
|
|
||||||
palette (`--cream #faf8f5`, `--wash #eef4ea`, `--green-deep #5b8c5a`,
|
|
||||||
`--green-mid #6ba86b`, `--ink #2d3327`, `--muted #6b7a62`) and brands
|
|
||||||
the loading flash (`#loading-container`); the login logo is the green
|
|
||||||
pine-dots vector (white-plate variant was invisible on cream — the
|
|
||||||
'flash on load' glitch), cache-busted via `?v=3` in the client bundle.
|
|
||||||
and hides the register/sign-up prompt via
|
|
||||||
`p:has(> a[href*="register"])`; `custom2.js` is a MutationObserver that
|
|
||||||
hides the prompt when React renders it (the auth form renders after
|
|
||||||
`load`). The `interface.theme` block in the yaml remains
|
|
||||||
forward-compatible only (inert on v0.8.8).
|
|
||||||
|
|
||||||
## Image generation (two member-facing paths)
|
|
||||||
|
|
||||||
1. **MCP tool (default, no key ever)**: the `inference-image` MCP server
|
|
||||||
exposes `generate_image` in the composer's **+** palette — no agent setup,
|
|
||||||
no key prompt. Per-member billing via `customUserVars.IC_API_KEY` (seeded
|
|
||||||
from the portal) + `env: IC_API_KEY: "{{IC_API_KEY}}"`; a system-key
|
|
||||||
fallback (`system-image-fallback`, $5/30d) covers the rare empty-value
|
|
||||||
case. The `customUserVars` gate blocks a not-yet-seeded member until the
|
|
||||||
next sync (≤10 min) — accepted as a short delay over billing everything
|
|
||||||
to the system account.
|
|
||||||
2. **Agent tool**: LibreChat's built-in "OpenAI Image Tools"
|
|
||||||
(`IMAGE_GEN_OAI_BASEURL`/`IMAGE_GEN_OAI_MODEL` → gateway; members paste
|
|
||||||
their own key — v0.8.8 limitation).
|
|
||||||
|
|
||||||
## Update policy
|
|
||||||
|
|
||||||
- The app is **image-sourced** (the `--force` update moved it off the
|
|
||||||
appstore source), so Cloudron will not auto-update it. This is deliberate —
|
|
||||||
same lockdown model as LiteLLM.
|
|
||||||
- **Image deploys**: build `--no-cache`, write the new tag into
|
|
||||||
`CloudronManifest.json.dockerImage` (the only editable field), commit, then
|
|
||||||
`cloudron update --app <id> --force`. **`--image` alone is IGNORED** when
|
|
||||||
the manifest sits in the deploy directory. **The remote builder's cache can
|
|
||||||
serve stale COPY layers even when file content changed** (six consecutive
|
|
||||||
"Build successful" runs did on 2026-10-19) — so after every image deploy,
|
|
||||||
verify: the Cloudron API's `manifest.dockerImage`, container-file md5 vs
|
|
||||||
repo md5, the served HTML, and the rendered DOM. Full checklist in the
|
|
||||||
repo README.
|
|
||||||
- **When a new appstore package ships**: pull its manifest copy into the repo
|
|
||||||
(`GET /api/v1/apps/<id>` → `.manifest`), repoint `dockerImage` to the
|
|
||||||
rebuilt overlay, bump the Dockerfile `FROM` to the new `manifest.dockerImage`
|
|
||||||
(pinned), rebuild, verify in the sandbox, then update.
|
|
||||||
- **Rollback**: repoint `CloudronManifest.json.dockerImage` to the previous
|
|
||||||
tag and rerun `cloudron update --force` (or the original appstore image).
|
|
||||||
Cloudron also snapshots the app before every image update.
|
|
||||||
|
|
||||||
## Hard-won pitfalls (do not re-learn these)
|
|
||||||
|
|
||||||
- `cloudron update` deploys **`manifest.dockerImage`** from the manifest in
|
|
||||||
the deploy directory — the `--image` flag alone is ignored (and the update
|
|
||||||
output still prints the `--image` value, so it *looks* right). The repo
|
|
||||||
manifest is a byte-identical copy of the OFFICIAL appstore manifest except
|
|
||||||
`dockerImage`; editing anything else in it (or hand-writing one that omits
|
|
||||||
the addons) strips `postgresql`/`mongodb`/`oidc`/`sendmail`/`localstorage`
|
|
||||||
from the app and crash-loops it (`mkdir: Read-only file system`). That is
|
|
||||||
what broke the sandbox on 2026-10-18.
|
|
||||||
- The remote builder's layer cache can pin COPY steps to old file content
|
|
||||||
even across "successful" rebuilds — use `--no-cache` after asset edits and
|
|
||||||
confirm by comparing container-file md5 against the repo file.
|
|
||||||
- Deploy `librechat.yaml` only via base64 round-trip. A `sed` insert once
|
|
||||||
broke the YAML indentation and LibreChat **silently fell back to defaults**
|
|
||||||
(all custom branding/models/MCP gone) with a single log line.
|
|
||||||
- Reading large files from the container over `cloudron exec` with a plain
|
|
||||||
`cat` resets the connection (`ECONNRESET`) — pipe through
|
|
||||||
`gzip | base64 | fold` and decode locally.
|
|
||||||
- `cloudron env set` alone does NOT disable the password form —
|
|
||||||
`ALLOW_EMAIL_LOGIN` must be flipped in `/app/data/env` (precedence above).
|
|
||||||
- Image generation in LibreChat is the **MCP tool** (composer **+** palette);
|
|
||||||
the agent "OpenAI Image Tools" path also exists. Image models in the chat
|
|
||||||
picker were the source of the old 500 errors and are deliberately excluded.
|
|
||||||
- No `branding:` block exists in LibreChat's config schema, and the login
|
|
||||||
page has no `.login-page`/`.primary` classes to target — theming happens by
|
|
||||||
overriding the Tailwind CSS variables on `html.light`/`html.dark`
|
|
||||||
(`--surface-primary`, `--accent-primary`, …) from the overlay `custom.css`.
|
|
||||||
|
|
||||||
|
|
||||||
## Per-member key auto-load (no API-key prompts)
|
|
||||||
|
|
||||||
Members never enter an API key. The portal provisions each member's LiteLLM
|
|
||||||
key at onboarding; the sandbox seeds it into LibreChat's user-key store:
|
|
||||||
|
|
||||||
- `GET /admin/librechat/keys` on the portal returns active members' chat keys
|
|
||||||
(dedicated `X-Sync-Token` auth — `LIBRECHAT_SYNC_TOKEN` on both apps).
|
|
||||||
- `/app/data/sync-keys.js` (in the LibreChat container, mirrored in the repo
|
|
||||||
as `sync-keys.js`) fetches that feed and seeds the `keys` collection
|
|
||||||
(chat endpoint, `apiKey: "user_provided"`) and `pluginauths`
|
|
||||||
(`mcp:inference-image` → `IC_API_KEY`, injected per-user via
|
|
||||||
`customUserVars` so image generation bills to the member's own team).
|
|
||||||
- Members without a sandbox account are skipped until they first log in; run
|
|
||||||
the sync after onboarding waves and on a schedule (the command is in the
|
|
||||||
repo README). Idempotent.
|
|
||||||
- Encryption matches LibreChat's own (`CREDS_KEY`/`CREDS_IV`, read from
|
|
||||||
`/proc/1/environ` — exec sessions don't inherit the app env).
|
|
||||||
|
|
||||||
> Terminal output redacts values after `apiKey:` as `***` even for harmless
|
|
||||||
> strings (`user_provided`, `${TTS_API_KEY}`). Probe lengths/counts instead
|
|
||||||
> of trusting redacted output.
|
|
||||||
|
|
||||||
## Verification (after any change)
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cloudron logs --app 4023ccbf-d57a-4500-97df-14de9cf3516e --lines 60 | \
|
|
||||||
grep -iE 'custom config|error|invalid|MCP|schedules'
|
|
||||||
curl -s https://librechat-test.inference.coop/api/config # appTitle, emailLoginEnabled
|
|
||||||
```
|
|
||||||
|
|
||||||
Expected: `Custom config file loaded`, `[MCP][inference-docs]` initialized
|
|
||||||
with tool `search_docs`, `inference-image` registered (its `Tools: undefined`
|
|
||||||
boot line is expected — per-user startup), `[schedules] engine started`,
|
|
||||||
`appTitle: Inference Co-op`, `emailLoginEnabled: false`. The login page shows
|
|
||||||
the co-op logo, the co-op favicon, a single "Login with Inference
|
|
||||||
Cooperative" (Cloudron SSO) path with no email/password form, **no
|
|
||||||
"Don't have an account?" line**, and the cream/green co-op color scheme.
|
|
||||||
|
|
||||||
After an IMAGE deploy, additionally run the four-step verification in the
|
|
||||||
repo README (API `manifest.dockerImage`, container md5 vs repo md5, served
|
|
||||||
HTML tags, rendered DOM colors).
|
|
||||||
+8
-9
@@ -1,9 +1,9 @@
|
|||||||
# MkDocs configuration for the member-readable documentation site.
|
# MkDocs configuration for the member-readable documentation site.
|
||||||
#
|
#
|
||||||
# The documentation source stays exactly where it is — Markdown files at the
|
# The documentation source stays exactly where it is — Markdown files at the
|
||||||
# repository root (plus maintenance/ and incidents/). This is a *publishing*
|
# repository root. This is a *publishing* layer over that source, not an edit
|
||||||
# layer over that source, not an edit of it. docs_dir is the repo root, so no
|
# of it. docs_dir is the repo root, so no document has to move for the
|
||||||
# document has to move for the generator to find it.
|
# generator to find it.
|
||||||
#
|
#
|
||||||
# Build: see publishing/build.sh
|
# Build: see publishing/build.sh
|
||||||
# Deploy: see publishing/deploy.sh
|
# Deploy: see publishing/deploy.sh
|
||||||
@@ -30,6 +30,10 @@ use_directory_urls: true
|
|||||||
|
|
||||||
theme:
|
theme:
|
||||||
name: material
|
name: material
|
||||||
|
# Theme overrides. Used only to add a "documentation source" link to the
|
||||||
|
# footer on *every* page (overrides/partials/copyright.html) without putting
|
||||||
|
# it in the Markdown sources. No styling is changed here.
|
||||||
|
custom_dir: overrides
|
||||||
language: en
|
language: en
|
||||||
# Disable the theme's built-in Google Fonts loader. Without this, Material
|
# Disable the theme's built-in Google Fonts loader. Without this, Material
|
||||||
# requests Roboto from fonts.googleapis.com / fonts.gstatic.com.
|
# requests Roboto from fonts.googleapis.com / fonts.gstatic.com.
|
||||||
@@ -76,12 +80,6 @@ nav:
|
|||||||
- Open questions: open-questions.md
|
- Open questions: open-questions.md
|
||||||
- Terms of Service: terms-of-service.md
|
- Terms of Service: terms-of-service.md
|
||||||
- Privacy Policy: privacy-policy.md
|
- Privacy Policy: privacy-policy.md
|
||||||
- Operator runbook: operator-token-runbook.md
|
|
||||||
- Maintenance:
|
|
||||||
- Chat task models: maintenance/chat-task-models.md
|
|
||||||
- LibreChat sandbox: maintenance/librechat-sandbox.md
|
|
||||||
- Incidents:
|
|
||||||
- 2026-10-02 gateway CVE upgrade: incidents/2026-10-02-gateway-cve-upgrade.md
|
|
||||||
|
|
||||||
plugins:
|
plugins:
|
||||||
# Built-in, offline search — the index is a JSON file in the build output.
|
# Built-in, offline search — the index is a JSON file in the build output.
|
||||||
@@ -113,6 +111,7 @@ extra_css:
|
|||||||
exclude_docs: |
|
exclude_docs: |
|
||||||
.git/
|
.git/
|
||||||
*.sh
|
*.sh
|
||||||
|
overrides/
|
||||||
publishing/README.md
|
publishing/README.md
|
||||||
publishing/requirements.txt
|
publishing/requirements.txt
|
||||||
|
|
||||||
|
|||||||
@@ -1,186 +0,0 @@
|
|||||||
# Operator runbook — Cloudron tokens
|
|
||||||
|
|
||||||
How the co-op's Cloudron credentials are organized, and how to rotate them
|
|
||||||
without breaking the stack.
|
|
||||||
|
|
||||||
## The two-token split
|
|
||||||
|
|
||||||
There are **two separate** Cloudron API tokens, deliberately decoupled so
|
|
||||||
rotating one doesn't break the other:
|
|
||||||
|
|
||||||
| Token | Purpose | Where it lives |
|
|
||||||
|-------|---------|----------------|
|
|
||||||
| **Agent token** | Inferencebot's own Cloudron access | The agent's private `.env` (gitignored) |
|
|
||||||
| **Portal token** | The member portal's Cloudron access (create users, set groups, invite links) | Cloudron's env store, under the portal app's `CLOUDRON_TOKEN` |
|
|
||||||
|
|
||||||
The portal needs only a handful of Cloudron capabilities (user/group management,
|
|
||||||
invite links). It is the most exposed app (public webhook + broker endpoints),
|
|
||||||
so it must **not** hold the agent's token — a portal compromise shouldn't reach
|
|
||||||
the agent's wider Cloudron access.
|
|
||||||
|
|
||||||
## Rotating a token
|
|
||||||
|
|
||||||
### Portal token
|
|
||||||
|
|
||||||
1. In `my.inference.coop` → **Settings → API Tokens**, create a **"Read and Write"** token.
|
|
||||||
2. Set it as the portal's `CLOUDRON_TOKEN` env var (custom apps have no env-var
|
|
||||||
UI — use the Cloudron CLI or the API; see the next section).
|
|
||||||
3. Verify: the portal can reach `GET /api/v1/users` with the new token (HTTP 200).
|
|
||||||
|
|
||||||
### Agent token
|
|
||||||
|
|
||||||
1. Create a fresh token in `Settings → API Tokens`.
|
|
||||||
2. Put it in the agent's `.env` as `CLOUDRON_TOKEN`.
|
|
||||||
3. Verify against `/api/v1/cloudron/status`.
|
|
||||||
|
|
||||||
**The trap this split exists to prevent:** the portal keeps its *own copy* of a
|
|
||||||
Cloudron token. If the agent's token is rotated without also updating the portal
|
|
||||||
copy, the portal's Cloudron calls start returning `401` — which breaks
|
|
||||||
provisioning (new members can't be created) and the member dashboard (its
|
|
||||||
usage/keys data is resolved through the portal's Cloudron calls).
|
|
||||||
|
|
||||||
## Admin token vs webhook token (portal)
|
|
||||||
|
|
||||||
Two more, separate from the Cloudron tokens above:
|
|
||||||
|
|
||||||
| Token | Purpose | Where it lives |
|
|
||||||
|-------|---------|----------------|
|
|
||||||
| **`ADMIN_TOKEN`** | Authenticates `/admin/*`; sent ONLY as an `X-Admin-Token` header | Portal env; the **admin panel's own env has a separate copy**; `~/.hermes/.env` on the agent host |
|
|
||||||
| **`WEBHOOK_TOKEN`** | Authenticates Open Collective's inbound webhook | Portal env, and inside the webhook URL configured on the collective |
|
|
||||||
|
|
||||||
They were once one value, so anything that learned the webhook URL — OC admins,
|
|
||||||
OC's logs, our access logs, a Referer — got portal admin. Split 2026-10-27; the
|
|
||||||
cutover completed 2026-10-28 (`ADMIN_TOKEN_LEGACY=0`), verified on the wire:
|
|
||||||
`ADMIN_TOKEN` → HTTP 200, `WEBHOOK_TOKEN` → 403, no header → 403.
|
|
||||||
|
|
||||||
- **`WEBHOOK_TOKEN` is not retired and cannot be.** OC's generic webhooks cannot
|
|
||||||
send headers, so a secret in the URL is the standard authentication. Its only
|
|
||||||
power now is authenticating that one inbound route, which is the intended end
|
|
||||||
state.
|
|
||||||
- **Rotating `ADMIN_TOKEN` touches THREE places** — the portal env (shared by the
|
|
||||||
app and the scheduler addon), the admin panel's separate copy, and the agent's
|
|
||||||
`~/.hermes/.env`. Miss one and that caller 403s silently.
|
|
||||||
- **`WEBHOOK_TOKEN` was rotated on 2026-10-27**, as a precaution after a test
|
|
||||||
handled the URL. Rotating it *again* means updating BOTH the portal env and the
|
|
||||||
webhook URL configured on the collective, which needs Open Collective admin
|
|
||||||
access — the same blocker as moving `OC_PERSONAL_TOKEN` (below) to a
|
|
||||||
co-op-owned account.
|
|
||||||
- **Whether the collective's webhook URL matches the current token is still
|
|
||||||
unconfirmed** (as of 2026-10-28). The route compares the URL token against the
|
|
||||||
env var, so a mismatch means OC's deliveries 401 and new-member provisioning
|
|
||||||
waits for the nightly sweep — delayed, not lost.
|
|
||||||
- **To confirm it directly**, compare by hash rather than by reading the value
|
|
||||||
out: take the URL's last path segment in the collective's webhook settings and
|
|
||||||
run `printf %s '<token>' | sha256sum` LOCALLY, then compare its first 12 hex
|
|
||||||
characters against the same prefix of the portal's `WEBHOOK_TOKEN`. A hash
|
|
||||||
prefix of a live secret does not belong in this file or in chat.
|
|
||||||
- **The monitor already covers the failure case.** The `portal webhook delivery
|
|
||||||
watch` cron job alerts on ANY failed delivery (401/404 = OC is posting to a URL
|
|
||||||
the portal rejects) and stays silent on the happy path, which is why silence in
|
|
||||||
its output means "nothing to report" and not "blind". Verified 2026-10-28 by
|
|
||||||
posting a benign event with the current token: the portal answered 200 and the
|
|
||||||
job reported it with `--verbose`. Run it with `--verbose` to see accepted
|
|
||||||
deliveries; run it plainly to see only problems.
|
|
||||||
- **No caller may fall back from `ADMIN_TOKEN` to `WEBHOOK_TOKEN`.** The legacy
|
|
||||||
token 403s, so a fallback turns "unset" into a confusing mid-run failure
|
|
||||||
instead of a clear abort.
|
|
||||||
|
|
||||||
## Chat image: spaCy model for document parsing (EPUB/DOCX)
|
|
||||||
|
|
||||||
Open WebUI's document parser (`unstructured`) lazily downloads the spaCy model
|
|
||||||
`en_core_web_sm` on first EPUB/DOCX parse — but the app directory is read-only
|
|
||||||
under Cloudron, so that install always fails ("Errno 30 Read-only file
|
|
||||||
system"). Fixed 2026-10-14 (issue #20) by pre-installing the model into
|
|
||||||
persistent storage:
|
|
||||||
|
|
||||||
- `/app/data/spacy_model/` holds the unpacked wheel
|
|
||||||
(SHA-verified `en_core_web_sm-3.8.0-py3-none-any.whl` from the pinned URL in
|
|
||||||
`unstructured/nlp/tokenize.py`).
|
|
||||||
- App env has `PYTHONPATH=/app/data/spacy_model` so `import en_core_web_sm`
|
|
||||||
resolves and the lazy download never triggers.
|
|
||||||
|
|
||||||
**Restore caveat:** a backup restore from before 2026-10-14 will NOT contain
|
|
||||||
the model dir — re-create it (download the pinned wheel, unzip into
|
|
||||||
`/app/data/spacy_model/`, verify `spacy.load("en_core_web_sm")` works in the
|
|
||||||
container). The `PYTHONPATH` env survives restores (it's in app config).
|
|
||||||
|
|
||||||
## Scheduled tasks (scheduler addon)
|
|
||||||
|
|
||||||
The portal and gateway run their own housekeeping via the Cloudron scheduler
|
|
||||||
addon, declared in each app's `CloudronManifest.json` under `addons.scheduler`:
|
|
||||||
|
|
||||||
- **portal**: `membership_sweep` (nightly 3am, `/app/code/sweep.sh` →
|
|
||||||
`POST /admin/sweep` with the **`ADMIN_TOKEN`** header — NOT `WEBHOOK_TOKEN`,
|
|
||||||
which returns 403 on `/admin/*` since the 2026-10-28 cutover), `rag_refresh`
|
|
||||||
(nightly 3:30am, `/app/code/rag_refresh.py` — pulls docs from the Gitea raw
|
|
||||||
API and refreshes the chat knowledge base; needs `OWUI_API_KEY` +
|
|
||||||
`DOCS_KNOWLEDGE_ID` in the app env), and `metrics_snapshot` (nightly 3:15am,
|
|
||||||
`/app/code/metrics_snapshot.py` — writes the day's members / usage / energy /
|
|
||||||
carbon row into the portal's `metrics_daily` table; `--backfill` rebuilds
|
|
||||||
history and is idempotent).
|
|
||||||
- **gateway**: `audio_keepalive` (every 10 minutes, `/app/code/audio_keepalive.py`
|
|
||||||
— pings the Tinfoil audio models locally to avoid the ~24s TEE cold-start;
|
|
||||||
reads the master key from `/app/data/.master_key`, which `start.sh` writes —
|
|
||||||
the env var `LITELLM_MASTER_KEY` is only exported into the litellm process).
|
|
||||||
|
|
||||||
**After any manifest change, verify the scheduler survived the deploy:** the
|
|
||||||
build log must show `setupAddons: setting up addon scheduler with options
|
|
||||||
{...}` listing your task(s), and each script must succeed when run manually
|
|
||||||
inside the container (`cloudron exec --app <id> -- <command>`). The scheduler
|
|
||||||
has historically gone missing from the manifest during edits — a deploy without
|
|
||||||
the setupAddons line means nightly reconciliation is NOT running.
|
|
||||||
|
|
||||||
## Custom apps have no env-var UI
|
|
||||||
|
|
||||||
The member portal is a *custom* app (packaged by us), not an App Store app.
|
|
||||||
Custom apps do **not** expose environment variables in the Cloudron dashboard.
|
|
||||||
Env vars for custom apps are set via:
|
|
||||||
|
|
||||||
- **Cloudron CLI** (on your workstation, *not* the server): `cloudron env set --app portal.inference.coop CLOUDRON_TOKEN=...`
|
|
||||||
- **Cloudron API**: `POST /apps/:appId/configure/env`
|
|
||||||
|
|
||||||
## `configure/env` is a full replace
|
|
||||||
|
|
||||||
`POST /apps/:appId/configure/env` with body `{"env": {KEY: value, ...}}`
|
|
||||||
**replaces the entire environment** — it does not merge. Sending a partial map
|
|
||||||
(e.g. a single test variable) wipes every other variable.
|
|
||||||
|
|
||||||
- To change **one** variable, read the current env first (`GET /api/v1/apps/:id`
|
|
||||||
→ `.env`), merge in memory, and POST the full map back.
|
|
||||||
- Never probe the endpoint with a throwaway body against a live app.
|
|
||||||
- If the env is wiped, it is recoverable from a Cloudron app backup:
|
|
||||||
`GET /api/v1/apps/:appId/backups` → newest entry → `GET .../backups/:backupId/download`
|
|
||||||
(a `.tar.gz` containing `config.json`, whose `.env` field holds the original map).
|
|
||||||
|
|
||||||
## Reference
|
|
||||||
|
|
||||||
- Cloudron CLI: `npm install -g cloudron` (install on your workstation, not the server), then `cloudron login my.inference.coop`.
|
|
||||||
- Cloudron API docs: `docs.cloudron.io/api/`.
|
|
||||||
|
|
||||||
## Open Collective: the bot token needs the `email` scope (Sept 2026)
|
|
||||||
|
|
||||||
The portal identifies members by email, read via a **personal** OC token
|
|
||||||
(`OC_PERSONAL_TOKEN`) — currently Nathan's own rather than a co-op-owned
|
|
||||||
account's (open item F5); the fix is a co-op account plus a token minted there as `collective.members.nodes[].account.email` with the
|
|
||||||
`... on Individual { email }` inline fragment.
|
|
||||||
|
|
||||||
**The `email` scope is load-bearing.** The bot's personal token must be minted
|
|
||||||
with the `email` scope. Without it, the GraphQL `email` field returns `null` for
|
|
||||||
*every* member — including the bot's own `me { email }` — even though:
|
|
||||||
|
|
||||||
- the bot still authenticates and reports `isAdmin: true`, and
|
|
||||||
- the OC **admin CSV export still shows emails** (it's a separate, non-GraphQL
|
|
||||||
path; its `contributorEmail` column is not a GraphQL field).
|
|
||||||
|
|
||||||
**Symptoms when the scope is missing:** `fetch_members()` filters out empty
|
|
||||||
emails and returns `[]`, so the webhook can't provision anyone by email and the
|
|
||||||
nightly sweep goes into its empty-list fail-safe (skips, doesn't deactivate).
|
|
||||||
A new member's webhook fires cleanly but no "Sent email to …" log line follows,
|
|
||||||
and no Cloudron user is created. (This is easy to misread as a "guest with no
|
|
||||||
email" problem — it is not; verify the scope first.)
|
|
||||||
|
|
||||||
**Diagnosis:** run `me { email }` with the bot token. `null` here while
|
|
||||||
`isAdmin: true` → the token is missing the `email` scope (or the scope was
|
|
||||||
revoked). Fix: re-add the `email` scope to the token on Open Collective
|
|
||||||
(Settings → Developers → Personal Tokens). No token rotation, no re-granting of
|
|
||||||
admin — just the scope.
|
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
<div class="md-copyright">
|
||||||
|
{% if config.copyright %}
|
||||||
|
<div class="md-copyright__highlight">
|
||||||
|
{{ config.copyright }}
|
||||||
|
</div>
|
||||||
|
{% endif %}
|
||||||
|
<div class="md-copyright__highlight">
|
||||||
|
Documentation source:
|
||||||
|
<a href="https://git.inference.coop/co-op/docs" target="_blank" rel="noopener">co-op/docs on Gitea</a>
|
||||||
|
</div>
|
||||||
|
{% if not config.extra.generator == false %}
|
||||||
|
Made with
|
||||||
|
<a href="https://squidfunk.github.io/mkdocs-material/" target="_blank" rel="noopener">
|
||||||
|
Material for MkDocs
|
||||||
|
</a>
|
||||||
|
{% endif %}
|
||||||
|
</div>
|
||||||
Reference in new issue
Block a user