diff --git a/AGENTS.md b/AGENTS.md index e33a6c8..9f3a7e3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -3,7 +3,7 @@ Applies to AI agents working in `capsite/` (goal tree + how-to-run: [capsite-checklist.md](capsite-checklist.md); background: [the capsite notes](capsite-notes.md); workspace-wide policy: -[AGENTS.md](../AGENTS.md) — commit trailers live there). +[AGENTS.md](../AGENTS.md) — points to the workspace skills, incl. commit trailers). ## Dev servers are the user's job @@ -13,13 +13,35 @@ terminals; the assistant observes via `.tmp/capsite.script` (the `script` capture; use `watchme`). One-shot codegen (`wrangler types`), linters, typechecks, and tests are fine. +## Durable state: be LOUD and conservative about breaking changes + +Persistence is a big deal here: DO ids (cache ids, webapp ids, public +ids) are headed toward representing real-world assets. A change that +leaves existing stored state unreadable, unreachable, or differently +interpreted is a **breaking change**, even in dev. + +- **Before** making it, say so, prominently: what stored state breaks, + why, and the options (migrate, stay compatible, or reset). Get the + user's go-ahead; don't discover it after the fact. +- Prefer compatibility: read old shapes, add fields rather than repurpose + them, migrate in place (schema steps like `SCHEMA_V1` → `V2`). +- No silent resets. "Wipe `.wrangler/state`" is a last resort the user + chooses, never the default advice. +- Cover it with a test: old-shape state in, current code reads it. + +(Prompted 2026-10-07: making `PublicCache` fetch through its own wiring +left public caches created by the earlier `ingest` code unwired — a 502, +and no way back through publish.) + ## Secrets discipline Worker secrets never in source or wrangler config. `.dev.vars` (gitignored) -locally; `wrangler secret put` deployed. As of the DO split, the cookie -lives only in the webapp DO's storage; `DIIGO_WEB_ID` is the one -diigo-dump secret. Error messages must not contain header or param values -(they cross RPC as exceptions and land in logs); URLs appear query-stripped. +locally; `wrangler secret put` deployed. Since the 1c cache evolution, no +worker secret is needed at all: the cookie lives only in the webapp DO's +storage; the webapp-instance id rides the cache DO's wiring; the cache id +is a user-held capability (password manager / per-POST paste). Error +messages must not contain header or param values (they cross RPC as +exceptions and land in logs); URLs appear query-stripped. ## Keep config types generated diff --git a/capsite-checklist.md b/capsite-checklist.md index d970022..ab24169 100644 --- a/capsite-checklist.md +++ b/capsite-checklist.md @@ -12,50 +12,69 @@ narrative. [the capsite notes](capsite-notes.md); after step 2, expands to [bib sync #195](https://github.com/dckc/madmode-blog/issues/195)) - - [x] Credential discipline established: cookies/keys/secrets live in - Worker secrets or DO storage, never in source, config, or the - plugin. (`.dev.vars` gitignored: only `DIIGO_WEB_ID`; - cookie lives in the webapp DO's storage since the 1b split.) - - [x] **1. Cookie-path dump worker** (`diigo-dump/`): port of - mm-diigo's diigo-dump.mjs. GET inert, dump only on POST. - Verified 2026-10-06: 200, `total: 1674`. (Scrubbed: static - API-key ping worker 401'd — the cookie path was the one that - worked; see the API-key subgoal below for where that left us.) - - [x] **1b. webapp DO factory** (`webapp/`): credentialed authority - in a Durable Object; ids are bearer capabilities (not logged); - factory worker is site-agnostic; consumer holds only the id. - Dev wiring verified: `pnpm dev` boots both, home + /_connect - answer on :8787. - - [ ] 1b-i. Mint through the running dev server end-to-end: - POST /_mint (or /_connect page) with a fresh cookie → - id → dump page 1 through the DO → 200/items. - (Everything up to mint is verified; this closes the loop.) - - [ ] 1b-ii. Prod mint + handoff: deploy both, mint against - prod, `wrangler secret put DIIGO_WEB_ID`, dump a page. + - [x] Credential discipline: no worker secrets; the cookie lives only + in webapp's DO storage, ids are held by the user + ([AGENTS.md](AGENTS.md), secrets section). + - [x] **1. Cookie-path dump worker**: port of mm-diigo's diigo-dump.mjs + (verified 2026-10-06: 200, `total: 1674`); evolved into 1c. + - [x] **1b. webapp DO factory** ([webapp/README.md](webapp/README.md)); + mint verified end-to-end in dev 2026-10-07 (via 1c). Prod mint is + goal 5. + + - [x] **1c. diigo-dump evolves into a bookmark cache** + ([diigo-cache/README.md](diigo-cache/README.md); design in the + JSDoc of `src/cache.ts`, `src/index.ts`, `src/pm-save.ts`): + - [x] 1c-i. Two independent page caches, private and public + (`PagedCache`, `DiigoCache`, `PublicCache`). + - [x] 1c-ii. Create flow on one port: `/_connect` → `/_mint` + → `/_wire` (`mintPage`, `proxyMint`, `wire`). + - [x] 1c-iii. Ids saved by explicit user action under the + holder's petname (`pmSave`); pages show a fingerprint + (`nick`). + - [x] 1c-iv. Verified against live diigo in dev, 2026-10-07: + create, dump, publish; `/public/dckc-madmode?page=4` → + `cache: HIT`. + - [x] 1c-v. Tests: `pnpm lint && pnpm test` (the real worker + + DOs against [a fake webapp](diigo-cache/tests/fake-webapp.js)). + + - [ ] **1d. Date windows and search on the public page**, toward the + weekly PESOS roundup of [#23](https://github.com/dckc/madmode-blog/issues/23) + (a week's permalink: `in=2026-W41`). One filter, two spellings: + devs pass `{ days: 7, q: "#weekly" }`, URLs the same fields; + `q` takes gmail/Slack/GitHub operators. UTC, ISO weeks, half-open + windows; a fetch budget per request. Design: the API in + `diigo-cache/src/query.ts` and the skipped suites in + `tests/query.test.ts` and `tests/caches.test.ts` ("public page + filters"); done when they're unskipped and green. + + - [ ] **1e. Fill job: a button that grabs every page** into a cache + (owner-only `POST /_fill`, cache id in the body; DO alarms, a + few pages per step; progress in storage; skips cached pages; + stop; fails loudly). Makes old `by=created` windows cheap. Design: + `startFill`/`fillStatus`/`stopFill` in `diigo-cache/src/cache.ts` + and the skipped suites in `tests/fill.test.ts`; done when they're + unskipped and green. - [ ] **2. madmode collects real bookmarks via the webapp cookie path** - (API-key path is DEAD — verified in session - ses_eec320fe4ffehaMAn44U1nTKJU: both known real keys, all - documented schemes (key= param, key+user=, HTTP Basic), both - /api/v2/bookmarks and /api/v2/notes → 401 unconditionally, with - 2024–2025 historical proof the same creds once downloaded 13.5k - bookmarks. Diigo deprecated/revoked v2 key auth; the session- - cookie /interact_api channel is the one that works.) - - [ ] 2-i. Plugin rides the webapp DO: plugin's sync calls - diigo-dump's DiigoBridge (or the instance id directly); - cookie + host gate stay host-side. + (the API-key path is dead: [evidence in the notes](capsite-notes.md#where-the-diigo-credentials-go)) + - [ ] 2-i. Plugin rides the cache: plugin's sync calls + diigo-cache's DiigoBridge (cache id by wiring, not + secret); cookie + host gate stay host-side. - [ ] 2-ii. Plugin "Sync now" succeeds: status 200, bookmarks stored (admin at /_emdash/admin → Diigo Sync). - [ ] 2-iii. Cookie-expiry story for cron: rotation is manual - (re-mint via /_connect) unless we automate it later. + (re-create via the home page form; the cache id rotates + with it) unless we automate it later. - [ ] **3. `/bookmarks` renders real data** after a sync — titles/tags at http://localhost:4321/bookmarks. - [ ] **4. Cron sync** works locally without errors (`0 */6 * * *` in `_emdash_cron_tasks`). - [ ] **5. Deployed end-to-end** (order matters; webapp first): - [ ] 5-i. webapp: `cd webapp && npx wrangler deploy` - - [ ] 5-ii. diigo-dump: deploy + prod mint + `wrangler secret - put DIIGO_WEB_ID` (id from x-diigo-web-id header) + - [ ] 5-ii. diigo-cache: deploy + first create against prod + (ids → password manager; no worker secret put needed); + rate-limit the public routes first (diigo-cache README, + "Deployment concern"). - [ ] 5-iii. madmode: `pnpm wrangler deploy`, secrets via `wrangler secret put`; plugin "Sync now" against prod - [ ] **6. bib sync #195**: repeat the pattern for the next @@ -68,29 +87,41 @@ narrative. ## How to run (continuation prompts) -Dev (diigo-dump + webapp in one workerd): +Dev (diigo-cache + webapp in one workerd): - cd ~/oros-bots/capsite/diigo-dump + cd ~/oros-bots/capsite/diigo-cache pnpm dev # → http://localhost:8787 # webapp rides along (binding-only) -First connect (or re-connect after cookie expiry) — browser flow, primary: +First connect (or re-connect after cookie expiry) — browser flow, two steps: - open http://localhost:8787/_connect - # paste the Cookie header from diigo.com devtools; mint - # success page shows a copy button for the id - # add DIIGO_WEB_ID="" to diigo-dump/.dev.vars (only the id lives - # on disk; the cookie lives in the DO's storage) - # restart wrangler (pnpm dev) to pick it up + open http://localhost:8787/ + # 1. "mint the diigo session at webapp" → webapp's own mint form + # (diigo-prefilled, served on :8787); paste the Cookie header from + # diigo.com devtools → mint (the cookie transits diigo-cache as an + # opaque relay into webapp's DO only) + # 2. the wire step: webapp id pre-filled, choose public|private → + # cacheId; copy the ids somewhere you control (password manager), + # paste the cacheId into the dump form + # (no .dev.vars, no restarts: ids ride the form POSTs / your paste) -Terminal flow (superseded by the page; needs a display): +Terminal flow (the same shape, scripted): - pnpm -s diigo-login >> .dev.vars # askpass prompt; appends the id line - # (.dev.vars: pnpm stdout noise can - # interleave; grep ^DIIGO_ to clean) + wid=$(curl -s -X POST http://localhost:8787/_mint --data-urlencode \ + 'url=https://www.diigo.com/interact_api/' --data-urlencode \ + 'headers={"user-agent":"Mozilla/5.0","x-requested-with":"XMLHttpRequest","cookie":""}' \ + --data-urlencode 'msPerCall=500' -D - -o /dev/null | grep -i x-webapp-id) + curl -s -X POST http://localhost:8787/_wire -H 'content-type: application/json' \ + -d "{\"webappId\":\"\",\"mode\":\"private\"}" -Then: enter a page number on the home page, dump → JSON items. GET / -is inert (no accidental fetches; dumps happen only on POST submits). +Then: enter a page number, dump → JSON rows with a `cache: HIT/MISS` +line (a repeat page is answered from DO sqlite storage). GET / is inert +(dumps happen only on POST submits). Public pages: +/public/?page=n (each page fills from diigo on first visit). +A private-only cache can be published later from its created page +(`POST /_publish`, cache id in the body). + +Checks: `cd diigo-cache && pnpm lint && pnpm test`. madmode (EmDash site; separate worker, separate concerns): @@ -101,18 +132,22 @@ madmode (EmDash site; separate worker, separate concerns): ## Working rules Agent working rules are in [AGENTS.md](AGENTS.md) — dev servers are the -user's job; secrets discipline; keep config types generated. They apply +user's job; durable state (loud about breaking changes); secrets +discipline; keep config types generated. They apply to every goal in this tree. ## Pointers - [the capsite notes](capsite-notes.md) — the original goal and the EmDash native-plugin wiring details +- [docs/webkeys.md](docs/webkeys.md) — the Waterken URL-capability + pattern, abstracted from the E-on-Java sources (bearer secrets, + YURL32, secret≠address); the reason ids never ride URLs - [the mm-diigo diigo-bak scripts](https://github.com/dckc/mm-diigo) — provenance of the fluent client and dump generator (`projects/diigo-bak/`: WebApp.js, diigo-dump.mjs, diigo-interact.mjs; mm-diigo commit 7dbb367, Oct 6 2026) - [webapp/README.md](webapp/README.md) — HTTP + RPC surfaces of the factory worker; open TODO on the mint-form placement -- [diigo-dump/README.md](diigo-dump/README.md) — consumer wiring and - run/deploy details \ No newline at end of file +- [diigo-cache/README.md](diigo-cache/README.md) — the cache worker: + create flow, the two DOs, run/deploy details \ No newline at end of file diff --git a/capsite-notes.md b/capsite-notes.md index 18e28f9..6cb4136 100644 --- a/capsite-notes.md +++ b/capsite-notes.md @@ -17,7 +17,9 @@ If it works out well, we should be able to expand to [bib sync #195](https://git - `webapp/` — site-agnostic Durable-Object factory for attenuated credentialed web clients (fluent client, ids are bearer capabilities); [webapp/README.md](webapp/README.md) -- `diigo-dump/` — the webapp consumer: its mint call carries all diigo +- `diigo-cache/` — the webapp consumer, evolved from the diigo-dump worker: + bookmark dumps cached in DO SQLite storage (private cache by unique id, + public-only cache by username-in-path); the mint call carries all diigo knowledge; the cookie lives in the DO's storage (see [the checklist's goal tree](capsite-checklist.md) for the verified/not-yet split) @@ -74,7 +76,10 @@ authoritative. Current shape: the session cookie lives only in the webapp DO's storage; `madmode/.dev.vars` still holds placeholder `DIIGO_API_KEY` / `DIIGO_USERNAME` for the (dead) API-key path — both documented schemes return 401 unconditionally, verified against both known -real keys (see checklist goal 2). Deployed secrets: `cd madmode && pnpm +real keys (opencode session ses_eec320fe4ffehaMAn44U1nTKJU: `key=` param, +`key`+`user=`, and HTTP Basic, on both /api/v2/bookmarks and /api/v2/notes, +though the same creds downloaded 13.5k bookmarks in 2024–2025; diigo +evidently revoked v2 key auth). Deployed secrets: `cd madmode && pnpm wrangler secret put `; the sync rides the cookie path since the API path died.