docs: clickable Key links, footer source link, move operator material to co-op/ops-docs

- README: render the Key links as real Markdown links (mailto: for the
  contact address); drop the 'lives in a public git repository' note now
  that the footer carries the source link.
- mkdocs.yml: add a theme override (custom_dir: overrides) that puts a
  'Documentation source' link in the footer of every page.
- Move the operator runbook, maintenance notes and incident write-up out
  of the published site into the git-only co-op/ops-docs repository;
  remove them from the nav and repoint cross-references at Gitea.
This commit is contained in:
inference-bot committed 2026-11-03 23:30:58 +00:00
1 parent c95bae161b
commit e0b9c0cd7e
8 files changed
+42 -426

No files matched your search

+7 -11
View File
@@ -20,13 +20,13 @@ funded through our [Open Collective](https://opencollective.com/inference-cooper
## Key links ## Key links
- **Website:** https://inference.coop - **Website:** [inference.coop](https://inference.coop)
- **AI Chat:** https://chat.inference.coop - **AI Chat:** [chat.inference.coop](https://chat.inference.coop)
- **Matrix chat:** https://matrix.inference.coop - **Matrix chat:** [matrix.inference.coop](https://matrix.inference.coop)
- **Member Dashboard:** https://dashboard.inference.coop - **Member Dashboard:** [dashboard.inference.coop](https://dashboard.inference.coop)
- **Governance forum:** https://forum.inference.coop - **Governance forum:** [forum.inference.coop](https://forum.inference.coop)
- **Source code:** https://git.inference.coop - **Source code:** [git.inference.coop](https://git.inference.coop)
- **Contact:** info@inference.coop - **Contact:** [info@inference.coop](mailto:info@inference.coop)
## Governance & legal ## Governance & legal
@@ -34,7 +34,3 @@ funded through our [Open Collective](https://opencollective.com/inference-cooper
- [Open questions](open-questions.md) — decisions currently up for member discussion. - [Open questions](open-questions.md) — decisions currently up for member discussion.
- [Terms of Service](terms-of-service.md) - [Terms of Service](terms-of-service.md)
- [Privacy Policy](privacy-policy.md) - [Privacy Policy](privacy-policy.md)
---
*This documentation lives in a public git repository. Contributions welcome.*
@@ -1,34 +0,0 @@
# 2026-10-02 — gateway outage after a security-upgrade rebuild
*Written by InferenceBot on the `incident/2026-10-02-gateway-cve-upgrade`
branch on 2026-10-02 (05:35 UTC, a minute after service returned; the commit's local timestamp reads 2026-10-01 23:35 −06:00); moved to `main` with the substance unchanged so it's
findable.*
**Impact.** The gateway (chat and API) was down about 45 minutes,
04:50–05:34 UTC. No data loss.
**What happened.** A Cloudron update triggered by a member's CVE report
rebuilt the app image. The Dockerfile pinned the floating
`main-v1.74.0-stable` tag, which the registry had silently moved to newer
(v1.84.0-era) bits, so the rebuild changed the running version and kicked
off a full Prisma database migration that takes 10–15 minutes per boot with
internal timeouts and retries. The migration ran at Cloudron's default
256 MB memory cap and was OOM-killed; the app crash-looped on a missing
migrations ledger until the memory limit and `LITELLM_MIGRATION_DIR` were
fixed.
**Also during the incident.** The domain's authoritative nameservers
(yoursrs.com) were intermittently unreachable, briefly failing resolution of
some hostnames (git, portal). Transient, registrar-side, self-recovered; no
records were wrong. The upgrade had also silently reverted CORS to `*`.
**Repairs.** Version pinned to the exact tag in `code/litellm`;
`memoryLimit` set to 2 GB in the manifest; `LITELLM_MIGRATION_DIR` pointed
at the persistent volume; `LITELLM_CORS_ORIGINS` set to the chat origin;
packaging synced to mirror the live deployment. The upgrade rules learned
here are in the `code/litellm` README under "Updating LiteLLM".
**What it fixed on the security side.** CVE-2026-35029 (authorization bypass
on `/config/update`, fixed in 1.83.0) and CVE-2026-59822 (MCP session auth
bypass, fixed in 1.84.0, listed in CISA's Known Exploited Vulnerabilities
catalog). The gateway has since moved to v1.103.2.
+10 -3
View File
@@ -299,9 +299,16 @@ through it; the other providers are called directly.
Maintaining the co-op's infrastructure — rotating Cloudron tokens, editing the Maintaining the co-op's infrastructure — rotating Cloudron tokens, editing the
portal's environment, and troubleshooting the provisioning pipeline — is portal's environment, and troubleshooting the provisioning pipeline — is
documented in the [operator runbook](operator-token-runbook.md). documented in the [operator
runbook](https://git.inference.coop/co-op/ops-docs/src/branch/main/operator-token-runbook.md).
All operator-facing material (that runbook, the maintenance notes, and incident
write-ups) lives in the git-only
[`co-op/ops-docs`](https://git.inference.coop/co-op/ops-docs) repository, which
is deliberately not published on this site.
## Incident log ## Incident log
Write-ups for outages live in [`incidents/`](incidents/). One so far: Write-ups for outages live in the operator docs, in
[2026-10-02 — gateway outage after a security-upgrade rebuild](incidents/2026-10-02-gateway-cve-upgrade.md). [`incidents/`](https://git.inference.coop/co-op/ops-docs/src/branch/main/incidents).
One so far:
[2026-10-02 — gateway outage after a security-upgrade rebuild](https://git.inference.coop/co-op/ops-docs/src/branch/main/incidents/2026-10-02-gateway-cve-upgrade.md).
-27
View File
@@ -1,27 +0,0 @@
# Chat task models (titles, tags, suggestions)
Both chat front doors generate small helper outputs — conversation titles,
tags, retrieval queries — with a model separate from the one answering the
member. These tasks are pinned to a small, inexpensive model so they never
bill against the member's chosen (larger) model.
## Open WebUI (chat.inference.coop)
- Title generation and tag generation are **enabled**; follow-up reply
suggestions and input autocomplete are **disabled** (October 2026).
- The task model (`task.model.default` in the Postgres `config` table) is
pinned to `greenpt/glm-5.3-flash` — a small, inexpensive model — so titles
and tags always run on it regardless of which model the member is chatting
with. Before October 2026 it was empty, which meant helper tasks ran on the
member's selected model.
- Config lives in the app's Postgres database (`config` table, key/value rows
with JSON values) and is read at startup: change a row, then restart the
app (`cloudron restart --app <id>`; note `restart` takes no `--timeout`).
## LibreChat (librechat-test.inference.coop)
- Title generation is configured in `/app/data/librechat.yaml`:
`titleConvo: true`, `titleModel: greenpt/glm-5.3-flash`,
`titleMethod: completion` — the same small GreenPT model.
- LibreChat v0.8.8 has no reply-suggestion feature; there is nothing to
disable on that front.
-156
View File
@@ -1,156 +0,0 @@
# LibreChat sandbox maintenance
The LibreChat evaluation instance at **librechat-test.inference.coop**
(Cloudron app `4023ccbf-d57a-4500-97df-14de9cf3516e`, appstore package
`ai.librechat.cloudronapp@1.4.1`, which bundles **LibreChat v0.8.8**).
The single source of truth for the deployment is the repo
**[co-op/librechat-custom](https://git.inference.coop/co-op/librechat-custom)**
— its README is the full runbook (config deploy, image build/deploy, rollback,
env vars, branding overlay, known state). This doc is the operator-facing
summary.
## Architecture: what lives where
- **App config** — `/app/data/librechat.yaml` (persistent volume, mirrored
byte-identically in the repo). Holds the "Inference Co-op" custom endpoint
(gateway), the 8-model chat picker, speech, web search, and the two MCP
servers. Volume edits deploy via `cloudron exec` (base64), never via the
image — the image never touches `/app/data`.
- **Env file** — `/app/data/env` (persistent volume). **`start.sh` sources it
AFTER the process env, so it wins over `cloudron env`** — that is where
`APP_TITLE="Inference Co-op"`, `ALLOW_EMAIL_LOGIN=false` (password login
removed; Cloudron SSO is the only login path), and
`SCHEDULES_SINGLE_PROCESS=true` must live.
- **Other env vars** — via `cloudron env`: `TTS_API_KEY`/`STT_API_KEY`
(dedicated LiteLLM virtual key, alias `librechat-speech`, $5/30d, audio
models only — never the gateway master key), `IMAGE_GEN_OAI_BASEURL` +
`IMAGE_GEN_OAI_MODEL` (agent image tool → gateway), `SEARXNG_INSTANCE_URL`.
- **MCP servers** — `/app/data/mcp/server.py` (docs search, `search_docs`,
starts at boot) and `/app/data/mcp/image_server.py` (image generation,
`generate_image`, `startup: false`); both mirrored in the repo under
`mcp/`, stdlib-only Python.
- **Branding overlay** (logo, favicons, PWA icons, `<title>`, `custom.css`,
`custom2.js`) — baked into a pinned overlay image `FROM` the official
appstore image. Deployed tag (last verified 2026-10-19):
`registry.inference.coop/co-op/librechat-custom:20261019-fc9b11e`.
- **Branding CSS/JS** — `custom.css` overrides LibreChat's Tailwind theme
variables on `html.light`/`html.dark` using the HOMEPAGE's canonical
palette (`--cream #faf8f5`, `--wash #eef4ea`, `--green-deep #5b8c5a`,
`--green-mid #6ba86b`, `--ink #2d3327`, `--muted #6b7a62`) and brands
the loading flash (`#loading-container`); the login logo is the green
pine-dots vector (white-plate variant was invisible on cream — the
'flash on load' glitch), cache-busted via `?v=3` in the client bundle.
and hides the register/sign-up prompt via
`p:has(> a[href*="register"])`; `custom2.js` is a MutationObserver that
hides the prompt when React renders it (the auth form renders after
`load`). The `interface.theme` block in the yaml remains
forward-compatible only (inert on v0.8.8).
## Image generation (two member-facing paths)
1. **MCP tool (default, no key ever)**: the `inference-image` MCP server
exposes `generate_image` in the composer's **+** palette — no agent setup,
no key prompt. Per-member billing via `customUserVars.IC_API_KEY` (seeded
from the portal) + `env: IC_API_KEY: "{{IC_API_KEY}}"`; a system-key
fallback (`system-image-fallback`, $5/30d) covers the rare empty-value
case. The `customUserVars` gate blocks a not-yet-seeded member until the
next sync (≤10 min) — accepted as a short delay over billing everything
to the system account.
2. **Agent tool**: LibreChat's built-in "OpenAI Image Tools"
(`IMAGE_GEN_OAI_BASEURL`/`IMAGE_GEN_OAI_MODEL` → gateway; members paste
their own key — v0.8.8 limitation).
## Update policy
- The app is **image-sourced** (the `--force` update moved it off the
appstore source), so Cloudron will not auto-update it. This is deliberate —
same lockdown model as LiteLLM.
- **Image deploys**: build `--no-cache`, write the new tag into
`CloudronManifest.json.dockerImage` (the only editable field), commit, then
`cloudron update --app <id> --force`. **`--image` alone is IGNORED** when
the manifest sits in the deploy directory. **The remote builder's cache can
serve stale COPY layers even when file content changed** (six consecutive
"Build successful" runs did on 2026-10-19) — so after every image deploy,
verify: the Cloudron API's `manifest.dockerImage`, container-file md5 vs
repo md5, the served HTML, and the rendered DOM. Full checklist in the
repo README.
- **When a new appstore package ships**: pull its manifest copy into the repo
(`GET /api/v1/apps/<id>` → `.manifest`), repoint `dockerImage` to the
rebuilt overlay, bump the Dockerfile `FROM` to the new `manifest.dockerImage`
(pinned), rebuild, verify in the sandbox, then update.
- **Rollback**: repoint `CloudronManifest.json.dockerImage` to the previous
tag and rerun `cloudron update --force` (or the original appstore image).
Cloudron also snapshots the app before every image update.
## Hard-won pitfalls (do not re-learn these)
- `cloudron update` deploys **`manifest.dockerImage`** from the manifest in
the deploy directory — the `--image` flag alone is ignored (and the update
output still prints the `--image` value, so it *looks* right). The repo
manifest is a byte-identical copy of the OFFICIAL appstore manifest except
`dockerImage`; editing anything else in it (or hand-writing one that omits
the addons) strips `postgresql`/`mongodb`/`oidc`/`sendmail`/`localstorage`
from the app and crash-loops it (`mkdir: Read-only file system`). That is
what broke the sandbox on 2026-10-18.
- The remote builder's layer cache can pin COPY steps to old file content
even across "successful" rebuilds — use `--no-cache` after asset edits and
confirm by comparing container-file md5 against the repo file.
- Deploy `librechat.yaml` only via base64 round-trip. A `sed` insert once
broke the YAML indentation and LibreChat **silently fell back to defaults**
(all custom branding/models/MCP gone) with a single log line.
- Reading large files from the container over `cloudron exec` with a plain
`cat` resets the connection (`ECONNRESET`) — pipe through
`gzip | base64 | fold` and decode locally.
- `cloudron env set` alone does NOT disable the password form —
`ALLOW_EMAIL_LOGIN` must be flipped in `/app/data/env` (precedence above).
- Image generation in LibreChat is the **MCP tool** (composer **+** palette);
the agent "OpenAI Image Tools" path also exists. Image models in the chat
picker were the source of the old 500 errors and are deliberately excluded.
- No `branding:` block exists in LibreChat's config schema, and the login
page has no `.login-page`/`.primary` classes to target — theming happens by
overriding the Tailwind CSS variables on `html.light`/`html.dark`
(`--surface-primary`, `--accent-primary`, …) from the overlay `custom.css`.
## Per-member key auto-load (no API-key prompts)
Members never enter an API key. The portal provisions each member's LiteLLM
key at onboarding; the sandbox seeds it into LibreChat's user-key store:
- `GET /admin/librechat/keys` on the portal returns active members' chat keys
(dedicated `X-Sync-Token` auth — `LIBRECHAT_SYNC_TOKEN` on both apps).
- `/app/data/sync-keys.js` (in the LibreChat container, mirrored in the repo
as `sync-keys.js`) fetches that feed and seeds the `keys` collection
(chat endpoint, `apiKey: "user_provided"`) and `pluginauths`
(`mcp:inference-image` → `IC_API_KEY`, injected per-user via
`customUserVars` so image generation bills to the member's own team).
- Members without a sandbox account are skipped until they first log in; run
the sync after onboarding waves and on a schedule (the command is in the
repo README). Idempotent.
- Encryption matches LibreChat's own (`CREDS_KEY`/`CREDS_IV`, read from
`/proc/1/environ` — exec sessions don't inherit the app env).
> Terminal output redacts values after `apiKey:` as `***` even for harmless
> strings (`user_provided`, `${TTS_API_KEY}`). Probe lengths/counts instead
> of trusting redacted output.
## Verification (after any change)
```bash
cloudron logs --app 4023ccbf-d57a-4500-97df-14de9cf3516e --lines 60 | \
grep -iE 'custom config|error|invalid|MCP|schedules'
curl -s https://librechat-test.inference.coop/api/config # appTitle, emailLoginEnabled
```
Expected: `Custom config file loaded`, `[MCP][inference-docs]` initialized
with tool `search_docs`, `inference-image` registered (its `Tools: undefined`
boot line is expected — per-user startup), `[schedules] engine started`,
`appTitle: Inference Co-op`, `emailLoginEnabled: false`. The login page shows
the co-op logo, the co-op favicon, a single "Login with Inference
Cooperative" (Cloudron SSO) path with no email/password form, **no
"Don't have an account?" line**, and the cream/green co-op color scheme.
After an IMAGE deploy, additionally run the four-step verification in the
repo README (API `manifest.dockerImage`, container md5 vs repo md5, served
HTML tags, rendered DOM colors).
+8 -9
View File
@@ -1,9 +1,9 @@
# MkDocs configuration for the member-readable documentation site. # MkDocs configuration for the member-readable documentation site.
# #
# The documentation source stays exactly where it is — Markdown files at the # The documentation source stays exactly where it is — Markdown files at the
# repository root (plus maintenance/ and incidents/). This is a *publishing* # repository root. This is a *publishing* layer over that source, not an edit
# layer over that source, not an edit of it. docs_dir is the repo root, so no # of it. docs_dir is the repo root, so no document has to move for the
# document has to move for the generator to find it. # generator to find it.
# #
# Build: see publishing/build.sh # Build: see publishing/build.sh
# Deploy: see publishing/deploy.sh # Deploy: see publishing/deploy.sh
@@ -30,6 +30,10 @@ use_directory_urls: true
theme: theme:
name: material name: material
# Theme overrides. Used only to add a "documentation source" link to the
# footer on *every* page (overrides/partials/copyright.html) without putting
# it in the Markdown sources. No styling is changed here.
custom_dir: overrides
language: en language: en
# Disable the theme's built-in Google Fonts loader. Without this, Material # Disable the theme's built-in Google Fonts loader. Without this, Material
# requests Roboto from fonts.googleapis.com / fonts.gstatic.com. # requests Roboto from fonts.googleapis.com / fonts.gstatic.com.
@@ -76,12 +80,6 @@ nav:
- Open questions: open-questions.md - Open questions: open-questions.md
- Terms of Service: terms-of-service.md - Terms of Service: terms-of-service.md
- Privacy Policy: privacy-policy.md - Privacy Policy: privacy-policy.md
- Operator runbook: operator-token-runbook.md
- Maintenance:
- Chat task models: maintenance/chat-task-models.md
- LibreChat sandbox: maintenance/librechat-sandbox.md
- Incidents:
- 2026-10-02 gateway CVE upgrade: incidents/2026-10-02-gateway-cve-upgrade.md
plugins: plugins:
# Built-in, offline search — the index is a JSON file in the build output. # Built-in, offline search — the index is a JSON file in the build output.
@@ -113,6 +111,7 @@ extra_css:
exclude_docs: | exclude_docs: |
.git/ .git/
*.sh *.sh
overrides/
publishing/README.md publishing/README.md
publishing/requirements.txt publishing/requirements.txt
-186
View File
@@ -1,186 +0,0 @@
# Operator runbook — Cloudron tokens
How the co-op's Cloudron credentials are organized, and how to rotate them
without breaking the stack.
## The two-token split
There are **two separate** Cloudron API tokens, deliberately decoupled so
rotating one doesn't break the other:
| Token | Purpose | Where it lives |
|-------|---------|----------------|
| **Agent token** | Inferencebot's own Cloudron access | The agent's private `.env` (gitignored) |
| **Portal token** | The member portal's Cloudron access (create users, set groups, invite links) | Cloudron's env store, under the portal app's `CLOUDRON_TOKEN` |
The portal needs only a handful of Cloudron capabilities (user/group management,
invite links). It is the most exposed app (public webhook + broker endpoints),
so it must **not** hold the agent's token — a portal compromise shouldn't reach
the agent's wider Cloudron access.
## Rotating a token
### Portal token
1. In `my.inference.coop` → **Settings → API Tokens**, create a **"Read and Write"** token.
2. Set it as the portal's `CLOUDRON_TOKEN` env var (custom apps have no env-var
UI — use the Cloudron CLI or the API; see the next section).
3. Verify: the portal can reach `GET /api/v1/users` with the new token (HTTP 200).
### Agent token
1. Create a fresh token in `Settings → API Tokens`.
2. Put it in the agent's `.env` as `CLOUDRON_TOKEN`.
3. Verify against `/api/v1/cloudron/status`.
**The trap this split exists to prevent:** the portal keeps its *own copy* of a
Cloudron token. If the agent's token is rotated without also updating the portal
copy, the portal's Cloudron calls start returning `401` — which breaks
provisioning (new members can't be created) and the member dashboard (its
usage/keys data is resolved through the portal's Cloudron calls).
## Admin token vs webhook token (portal)
Two more, separate from the Cloudron tokens above:
| Token | Purpose | Where it lives |
|-------|---------|----------------|
| **`ADMIN_TOKEN`** | Authenticates `/admin/*`; sent ONLY as an `X-Admin-Token` header | Portal env; the **admin panel's own env has a separate copy**; `~/.hermes/.env` on the agent host |
| **`WEBHOOK_TOKEN`** | Authenticates Open Collective's inbound webhook | Portal env, and inside the webhook URL configured on the collective |
They were once one value, so anything that learned the webhook URL — OC admins,
OC's logs, our access logs, a Referer — got portal admin. Split 2026-10-27; the
cutover completed 2026-10-28 (`ADMIN_TOKEN_LEGACY=0`), verified on the wire:
`ADMIN_TOKEN` → HTTP 200, `WEBHOOK_TOKEN` → 403, no header → 403.
- **`WEBHOOK_TOKEN` is not retired and cannot be.** OC's generic webhooks cannot
send headers, so a secret in the URL is the standard authentication. Its only
power now is authenticating that one inbound route, which is the intended end
state.
- **Rotating `ADMIN_TOKEN` touches THREE places** — the portal env (shared by the
app and the scheduler addon), the admin panel's separate copy, and the agent's
`~/.hermes/.env`. Miss one and that caller 403s silently.
- **`WEBHOOK_TOKEN` was rotated on 2026-10-27**, as a precaution after a test
handled the URL. Rotating it *again* means updating BOTH the portal env and the
webhook URL configured on the collective, which needs Open Collective admin
access — the same blocker as moving `OC_PERSONAL_TOKEN` (below) to a
co-op-owned account.
- **Whether the collective's webhook URL matches the current token is still
unconfirmed** (as of 2026-10-28). The route compares the URL token against the
env var, so a mismatch means OC's deliveries 401 and new-member provisioning
waits for the nightly sweep — delayed, not lost.
- **To confirm it directly**, compare by hash rather than by reading the value
out: take the URL's last path segment in the collective's webhook settings and
run `printf %s '<token>' | sha256sum` LOCALLY, then compare its first 12 hex
characters against the same prefix of the portal's `WEBHOOK_TOKEN`. A hash
prefix of a live secret does not belong in this file or in chat.
- **The monitor already covers the failure case.** The `portal webhook delivery
watch` cron job alerts on ANY failed delivery (401/404 = OC is posting to a URL
the portal rejects) and stays silent on the happy path, which is why silence in
its output means "nothing to report" and not "blind". Verified 2026-10-28 by
posting a benign event with the current token: the portal answered 200 and the
job reported it with `--verbose`. Run it with `--verbose` to see accepted
deliveries; run it plainly to see only problems.
- **No caller may fall back from `ADMIN_TOKEN` to `WEBHOOK_TOKEN`.** The legacy
token 403s, so a fallback turns "unset" into a confusing mid-run failure
instead of a clear abort.
## Chat image: spaCy model for document parsing (EPUB/DOCX)
Open WebUI's document parser (`unstructured`) lazily downloads the spaCy model
`en_core_web_sm` on first EPUB/DOCX parse — but the app directory is read-only
under Cloudron, so that install always fails ("Errno 30 Read-only file
system"). Fixed 2026-10-14 (issue #20) by pre-installing the model into
persistent storage:
- `/app/data/spacy_model/` holds the unpacked wheel
(SHA-verified `en_core_web_sm-3.8.0-py3-none-any.whl` from the pinned URL in
`unstructured/nlp/tokenize.py`).
- App env has `PYTHONPATH=/app/data/spacy_model` so `import en_core_web_sm`
resolves and the lazy download never triggers.
**Restore caveat:** a backup restore from before 2026-10-14 will NOT contain
the model dir — re-create it (download the pinned wheel, unzip into
`/app/data/spacy_model/`, verify `spacy.load("en_core_web_sm")` works in the
container). The `PYTHONPATH` env survives restores (it's in app config).
## Scheduled tasks (scheduler addon)
The portal and gateway run their own housekeeping via the Cloudron scheduler
addon, declared in each app's `CloudronManifest.json` under `addons.scheduler`:
- **portal**: `membership_sweep` (nightly 3am, `/app/code/sweep.sh` →
`POST /admin/sweep` with the **`ADMIN_TOKEN`** header — NOT `WEBHOOK_TOKEN`,
which returns 403 on `/admin/*` since the 2026-10-28 cutover), `rag_refresh`
(nightly 3:30am, `/app/code/rag_refresh.py` — pulls docs from the Gitea raw
API and refreshes the chat knowledge base; needs `OWUI_API_KEY` +
`DOCS_KNOWLEDGE_ID` in the app env), and `metrics_snapshot` (nightly 3:15am,
`/app/code/metrics_snapshot.py` — writes the day's members / usage / energy /
carbon row into the portal's `metrics_daily` table; `--backfill` rebuilds
history and is idempotent).
- **gateway**: `audio_keepalive` (every 10 minutes, `/app/code/audio_keepalive.py`
— pings the Tinfoil audio models locally to avoid the ~24s TEE cold-start;
reads the master key from `/app/data/.master_key`, which `start.sh` writes —
the env var `LITELLM_MASTER_KEY` is only exported into the litellm process).
**After any manifest change, verify the scheduler survived the deploy:** the
build log must show `setupAddons: setting up addon scheduler with options
{...}` listing your task(s), and each script must succeed when run manually
inside the container (`cloudron exec --app <id> -- <command>`). The scheduler
has historically gone missing from the manifest during edits — a deploy without
the setupAddons line means nightly reconciliation is NOT running.
## Custom apps have no env-var UI
The member portal is a *custom* app (packaged by us), not an App Store app.
Custom apps do **not** expose environment variables in the Cloudron dashboard.
Env vars for custom apps are set via:
- **Cloudron CLI** (on your workstation, *not* the server): `cloudron env set --app portal.inference.coop CLOUDRON_TOKEN=...`
- **Cloudron API**: `POST /apps/:appId/configure/env`
## `configure/env` is a full replace
`POST /apps/:appId/configure/env` with body `{"env": {KEY: value, ...}}`
**replaces the entire environment** — it does not merge. Sending a partial map
(e.g. a single test variable) wipes every other variable.
- To change **one** variable, read the current env first (`GET /api/v1/apps/:id`
→ `.env`), merge in memory, and POST the full map back.
- Never probe the endpoint with a throwaway body against a live app.
- If the env is wiped, it is recoverable from a Cloudron app backup:
`GET /api/v1/apps/:appId/backups` → newest entry → `GET .../backups/:backupId/download`
(a `.tar.gz` containing `config.json`, whose `.env` field holds the original map).
## Reference
- Cloudron CLI: `npm install -g cloudron` (install on your workstation, not the server), then `cloudron login my.inference.coop`.
- Cloudron API docs: `docs.cloudron.io/api/`.
## Open Collective: the bot token needs the `email` scope (Sept 2026)
The portal identifies members by email, read via a **personal** OC token
(`OC_PERSONAL_TOKEN`) — currently Nathan's own rather than a co-op-owned
account's (open item F5); the fix is a co-op account plus a token minted there as `collective.members.nodes[].account.email` with the
`... on Individual { email }` inline fragment.
**The `email` scope is load-bearing.** The bot's personal token must be minted
with the `email` scope. Without it, the GraphQL `email` field returns `null` for
*every* member — including the bot's own `me { email }` — even though:
- the bot still authenticates and reports `isAdmin: true`, and
- the OC **admin CSV export still shows emails** (it's a separate, non-GraphQL
path; its `contributorEmail` column is not a GraphQL field).
**Symptoms when the scope is missing:** `fetch_members()` filters out empty
emails and returns `[]`, so the webhook can't provision anyone by email and the
nightly sweep goes into its empty-list fail-safe (skips, doesn't deactivate).
A new member's webhook fires cleanly but no "Sent email to …" log line follows,
and no Cloudron user is created. (This is easy to misread as a "guest with no
email" problem — it is not; verify the scope first.)
**Diagnosis:** run `me { email }` with the bot token. `null` here while
`isAdmin: true` → the token is missing the `email` scope (or the scope was
revoked). Fix: re-add the `email` scope to the token on Open Collective
(Settings → Developers → Personal Tokens). No token rotation, no re-granting of
admin — just the scope.
+17
View File
@@ -0,0 +1,17 @@
<div class="md-copyright">
{% if config.copyright %}
<div class="md-copyright__highlight">
{{ config.copyright }}
</div>
{% endif %}
<div class="md-copyright__highlight">
Documentation source:
<a href="https://git.inference.coop/co-op/docs" target="_blank" rel="noopener">co-op/docs on Gitea</a>
</div>
{% if not config.extra.generator == false %}
Made with
<a href="https://squidfunk.github.io/mkdocs-material/" target="_blank" rel="noopener">
Material for MkDocs
</a>
{% endif %}
</div>