runbook: admin/webhook token split, metrics_snapshot task, correct sweep auth

operator-token-runbook.md

@@ -39,6 +39,34 @@ copy, the portal's Cloudron calls start returning `401` — which breaks
 provisioning (new members can't be created) and the member dashboard (its
 usage/keys data is resolved through the portal's Cloudron calls).
 
+## Admin token vs webhook token (portal)
+
+Two more, separate from the Cloudron tokens above:
+
+| Token | Purpose | Where it lives |
+|-------|---------|----------------|
+| **`ADMIN_TOKEN`** | Authenticates `/admin/*`; sent ONLY as an `X-Admin-Token` header | Portal env; the **admin panel's own env has a separate copy**; `~/.hermes/.env` on the agent host |
+| **`WEBHOOK_TOKEN`** | Authenticates Open Collective's inbound webhook | Portal env, and inside the webhook URL configured on the collective |
+
+They were once one value, so anything that learned the webhook URL — OC admins,
+OC's logs, our access logs, a Referer — got portal admin. Split 2026-10-27; the
+cutover completed 2026-10-28 (`ADMIN_TOKEN_LEGACY=0`), verified on the wire:
+`ADMIN_TOKEN` → HTTP 200, `WEBHOOK_TOKEN` → 403, no header → 403.
+
+- **`WEBHOOK_TOKEN` is not retired and cannot be.** OC's generic webhooks cannot
+  send headers, so a secret in the URL is the standard authentication. Its only
+  power now is authenticating that one inbound route, which is the intended end
+  state.
+- **Rotating `ADMIN_TOKEN` touches THREE places** — the portal env (shared by the
+  app and the scheduler addon), the admin panel's separate copy, and the agent's
+  `~/.hermes/.env`. Miss one and that caller 403s silently.
+- **Rotating `WEBHOOK_TOKEN` needs Open Collective admin access**, because the
+  value lives in the webhook URL on the collective — the same blocker as moving
+  `OC_PERSONAL_TOKEN` (below) to a co-op-owned account.
+- **No caller may fall back from `ADMIN_TOKEN` to `WEBHOOK_TOKEN`.** The legacy
+  token 403s, so a fallback turns "unset" into a confusing mid-run failure
+  instead of a clear abort.
+
 ## Chat image: spaCy model for document parsing (EPUB/DOCX)
 
 Open WebUI's document parser (`unstructured`) lazily downloads the spaCy model
@@ -64,10 +92,14 @@ The portal and gateway run their own housekeeping via the Cloudron scheduler
 addon, declared in each app's `CloudronManifest.json` under `addons.scheduler`:
 
 - **portal**: `membership_sweep` (nightly 3am, `/app/code/sweep.sh` →
-  `POST /admin/sweep` with the `WEBHOOK_TOKEN` header) and `rag_refresh`
+  `POST /admin/sweep` with the **`ADMIN_TOKEN`** header — NOT `WEBHOOK_TOKEN`,
+  which returns 403 on `/admin/*` since the 2026-10-28 cutover), `rag_refresh`
   (nightly 3:30am, `/app/code/rag_refresh.py` — pulls docs from the Gitea raw
   API and refreshes the chat knowledge base; needs `OWUI_API_KEY` +
-  `DOCS_KNOWLEDGE_ID` in the app env).
+  `DOCS_KNOWLEDGE_ID` in the app env), and `metrics_snapshot` (nightly 3:15am,
+  `/app/code/metrics_snapshot.py` — writes the day's members / usage / energy /
+  carbon row into the portal's `metrics_daily` table; `--backfill` rebuilds
+  history and is idempotent).
 - **gateway**: `audio_keepalive` (every 10 minutes, `/app/code/audio_keepalive.py`
   — pings the Tinfoil audio models locally to avoid the ~24s TEE cold-start;
   reads the master key from `/app/data/.master_key`, which `start.sh` writes —
@@ -109,8 +141,9 @@ Env vars for custom apps are set via:
 
 ## Open Collective: the bot token needs the `email` scope (Sept 2026)
 
-The portal identifies members by email, read via the bot's OC personal token
+The portal identifies members by email, read via a **personal** OC token
-(`OC_PERSONAL_TOKEN`) as `collective.members.nodes[].account.email` with the
+(`OC_PERSONAL_TOKEN`) — currently Nathan's own rather than a co-op-owned
+account's (open item F5); the fix is a co-op account plus a token minted there as `collective.members.nodes[].account.email` with the
 `... on Individual { email }` inline fragment.
 
 **The `email` scope is load-bearing.** The bot's personal token must be minted