docs: goal tree and rules track the 1c cache
- checklist: 1c (diigo-dump evolves into a bookmark cache) with its create-flow, id-wiring, and verification subgoals - notes: diigo-cache replaces diigo-dump in What-we-built - AGENTS: no worker secret since 1c; durable state — be LOUD and conservative about breaking changes to stored DB state; pointer to the workspace skills for commit trailers Generated-by: opencode (GLM) Generated-by: Claude Opus 5.5
This commit is contained in:
dckc committed
1 parent
6f409fc83a
commit
d0a9b899f2
3 files changed
+120
-58
No files matched your search
@@ -3,7 +3,7 @@
|
||||
Applies to AI agents working in `capsite/` (goal tree + how-to-run:
|
||||
[capsite-checklist.md](capsite-checklist.md); background:
|
||||
[the capsite notes](capsite-notes.md); workspace-wide policy:
|
||||
[AGENTS.md](../AGENTS.md) — commit trailers live there).
|
||||
[AGENTS.md](../AGENTS.md) — points to the workspace skills, incl. commit trailers).
|
||||
|
||||
## Dev servers are the user's job
|
||||
|
||||
@@ -13,13 +13,35 @@ terminals; the assistant observes via `.tmp/capsite.script` (the `script`
|
||||
capture; use `watchme`). One-shot codegen (`wrangler types`), linters,
|
||||
typechecks, and tests are fine.
|
||||
|
||||
## Durable state: be LOUD and conservative about breaking changes
|
||||
|
||||
Persistence is a big deal here: DO ids (cache ids, webapp ids, public
|
||||
ids) are headed toward representing real-world assets. A change that
|
||||
leaves existing stored state unreadable, unreachable, or differently
|
||||
interpreted is a **breaking change**, even in dev.
|
||||
|
||||
- **Before** making it, say so, prominently: what stored state breaks,
|
||||
why, and the options (migrate, stay compatible, or reset). Get the
|
||||
user's go-ahead; don't discover it after the fact.
|
||||
- Prefer compatibility: read old shapes, add fields rather than repurpose
|
||||
them, migrate in place (schema steps like `SCHEMA_V1` → `V2`).
|
||||
- No silent resets. "Wipe `.wrangler/state`" is a last resort the user
|
||||
chooses, never the default advice.
|
||||
- Cover it with a test: old-shape state in, current code reads it.
|
||||
|
||||
(Prompted 2026-10-07: making `PublicCache` fetch through its own wiring
|
||||
left public caches created by the earlier `ingest` code unwired — a 502,
|
||||
and no way back through publish.)
|
||||
|
||||
## Secrets discipline
|
||||
|
||||
Worker secrets never in source or wrangler config. `.dev.vars` (gitignored)
|
||||
locally; `wrangler secret put` deployed. As of the DO split, the cookie
|
||||
lives only in the webapp DO's storage; `DIIGO_WEB_ID` is the one
|
||||
diigo-dump secret. Error messages must not contain header or param values
|
||||
(they cross RPC as exceptions and land in logs); URLs appear query-stripped.
|
||||
locally; `wrangler secret put` deployed. Since the 1c cache evolution, no
|
||||
worker secret is needed at all: the cookie lives only in the webapp DO's
|
||||
storage; the webapp-instance id rides the cache DO's wiring; the cache id
|
||||
is a user-held capability (password manager / per-POST paste). Error
|
||||
messages must not contain header or param values (they cross RPC as
|
||||
exceptions and land in logs); URLs appear query-stripped.
|
||||
|
||||
## Keep config types generated
|
||||
|
||||
|
||||
+86
-51
@@ -12,50 +12,69 @@ narrative.
|
||||
[the capsite notes](capsite-notes.md); after step 2, expands to
|
||||
[bib sync #195](https://github.com/dckc/madmode-blog/issues/195))
|
||||
|
||||
- [x] Credential discipline established: cookies/keys/secrets live in
|
||||
Worker secrets or DO storage, never in source, config, or the
|
||||
plugin. (`.dev.vars` gitignored: only `DIIGO_WEB_ID`;
|
||||
cookie lives in the webapp DO's storage since the 1b split.)
|
||||
- [x] **1. Cookie-path dump worker** (`diigo-dump/`): port of
|
||||
mm-diigo's diigo-dump.mjs. GET inert, dump only on POST.
|
||||
Verified 2026-10-06: 200, `total: 1674`. (Scrubbed: static
|
||||
API-key ping worker 401'd — the cookie path was the one that
|
||||
worked; see the API-key subgoal below for where that left us.)
|
||||
- [x] **1b. webapp DO factory** (`webapp/`): credentialed authority
|
||||
in a Durable Object; ids are bearer capabilities (not logged);
|
||||
factory worker is site-agnostic; consumer holds only the id.
|
||||
Dev wiring verified: `pnpm dev` boots both, home + /_connect
|
||||
answer on :8787.
|
||||
- [ ] 1b-i. Mint through the running dev server end-to-end:
|
||||
POST /_mint (or /_connect page) with a fresh cookie →
|
||||
id → dump page 1 through the DO → 200/items.
|
||||
(Everything up to mint is verified; this closes the loop.)
|
||||
- [ ] 1b-ii. Prod mint + handoff: deploy both, mint against
|
||||
prod, `wrangler secret put DIIGO_WEB_ID`, dump a page.
|
||||
- [x] Credential discipline: no worker secrets; the cookie lives only
|
||||
in webapp's DO storage, ids are held by the user
|
||||
([AGENTS.md](AGENTS.md), secrets section).
|
||||
- [x] **1. Cookie-path dump worker**: port of mm-diigo's diigo-dump.mjs
|
||||
(verified 2026-10-06: 200, `total: 1674`); evolved into 1c.
|
||||
- [x] **1b. webapp DO factory** ([webapp/README.md](webapp/README.md));
|
||||
mint verified end-to-end in dev 2026-10-07 (via 1c). Prod mint is
|
||||
goal 5.
|
||||
|
||||
- [x] **1c. diigo-dump evolves into a bookmark cache**
|
||||
([diigo-cache/README.md](diigo-cache/README.md); design in the
|
||||
JSDoc of `src/cache.ts`, `src/index.ts`, `src/pm-save.ts`):
|
||||
- [x] 1c-i. Two independent page caches, private and public
|
||||
(`PagedCache`, `DiigoCache`, `PublicCache`).
|
||||
- [x] 1c-ii. Create flow on one port: `/_connect` → `/_mint`
|
||||
→ `/_wire` (`mintPage`, `proxyMint`, `wire`).
|
||||
- [x] 1c-iii. Ids saved by explicit user action under the
|
||||
holder's petname (`pmSave`); pages show a fingerprint
|
||||
(`nick`).
|
||||
- [x] 1c-iv. Verified against live diigo in dev, 2026-10-07:
|
||||
create, dump, publish; `/public/dckc-madmode?page=4` →
|
||||
`cache: HIT`.
|
||||
- [x] 1c-v. Tests: `pnpm lint && pnpm test` (the real worker +
|
||||
DOs against [a fake webapp](diigo-cache/tests/fake-webapp.js)).
|
||||
|
||||
- [ ] **1d. Date windows and search on the public page**, toward the
|
||||
weekly PESOS roundup of [#23](https://github.com/dckc/madmode-blog/issues/23)
|
||||
(a week's permalink: `in=2026-W41`). One filter, two spellings:
|
||||
devs pass `{ days: 7, q: "#weekly" }`, URLs the same fields;
|
||||
`q` takes gmail/Slack/GitHub operators. UTC, ISO weeks, half-open
|
||||
windows; a fetch budget per request. Design: the API in
|
||||
`diigo-cache/src/query.ts` and the skipped suites in
|
||||
`tests/query.test.ts` and `tests/caches.test.ts` ("public page
|
||||
filters"); done when they're unskipped and green.
|
||||
|
||||
- [ ] **1e. Fill job: a button that grabs every page** into a cache
|
||||
(owner-only `POST /_fill`, cache id in the body; DO alarms, a
|
||||
few pages per step; progress in storage; skips cached pages;
|
||||
stop; fails loudly). Makes old `by=created` windows cheap. Design:
|
||||
`startFill`/`fillStatus`/`stopFill` in `diigo-cache/src/cache.ts`
|
||||
and the skipped suites in `tests/fill.test.ts`; done when they're
|
||||
unskipped and green.
|
||||
|
||||
- [ ] **2. madmode collects real bookmarks via the webapp cookie path**
|
||||
(API-key path is DEAD — verified in session
|
||||
ses_eec320fe4ffehaMAn44U1nTKJU: both known real keys, all
|
||||
documented schemes (key= param, key+user=, HTTP Basic), both
|
||||
/api/v2/bookmarks and /api/v2/notes → 401 unconditionally, with
|
||||
2024–2025 historical proof the same creds once downloaded 13.5k
|
||||
bookmarks. Diigo deprecated/revoked v2 key auth; the session-
|
||||
cookie /interact_api channel is the one that works.)
|
||||
- [ ] 2-i. Plugin rides the webapp DO: plugin's sync calls
|
||||
diigo-dump's DiigoBridge (or the instance id directly);
|
||||
cookie + host gate stay host-side.
|
||||
(the API-key path is dead: [evidence in the notes](capsite-notes.md#where-the-diigo-credentials-go))
|
||||
- [ ] 2-i. Plugin rides the cache: plugin's sync calls
|
||||
diigo-cache's DiigoBridge (cache id by wiring, not
|
||||
secret); cookie + host gate stay host-side.
|
||||
- [ ] 2-ii. Plugin "Sync now" succeeds: status 200, bookmarks
|
||||
stored (admin at /_emdash/admin → Diigo Sync).
|
||||
- [ ] 2-iii. Cookie-expiry story for cron: rotation is manual
|
||||
(re-mint via /_connect) unless we automate it later.
|
||||
(re-create via the home page form; the cache id rotates
|
||||
with it) unless we automate it later.
|
||||
- [ ] **3. `/bookmarks` renders real data** after a sync — titles/tags at
|
||||
http://localhost:4321/bookmarks.
|
||||
- [ ] **4. Cron sync** works locally without errors (`0 */6 * * *` in
|
||||
`_emdash_cron_tasks`).
|
||||
- [ ] **5. Deployed end-to-end** (order matters; webapp first):
|
||||
- [ ] 5-i. webapp: `cd webapp && npx wrangler deploy`
|
||||
- [ ] 5-ii. diigo-dump: deploy + prod mint + `wrangler secret
|
||||
put DIIGO_WEB_ID` (id from x-diigo-web-id header)
|
||||
- [ ] 5-ii. diigo-cache: deploy + first create against prod
|
||||
(ids → password manager; no worker secret put needed);
|
||||
rate-limit the public routes first (diigo-cache README,
|
||||
"Deployment concern").
|
||||
- [ ] 5-iii. madmode: `pnpm wrangler deploy`, secrets via
|
||||
`wrangler secret put`; plugin "Sync now" against prod
|
||||
- [ ] **6. bib sync #195**: repeat the pattern for the next
|
||||
@@ -68,29 +87,41 @@ narrative.
|
||||
|
||||
## How to run (continuation prompts)
|
||||
|
||||
Dev (diigo-dump + webapp in one workerd):
|
||||
Dev (diigo-cache + webapp in one workerd):
|
||||
|
||||
cd ~/oros-bots/capsite/diigo-dump
|
||||
cd ~/oros-bots/capsite/diigo-cache
|
||||
pnpm dev # → http://localhost:8787
|
||||
# webapp rides along (binding-only)
|
||||
|
||||
First connect (or re-connect after cookie expiry) — browser flow, primary:
|
||||
First connect (or re-connect after cookie expiry) — browser flow, two steps:
|
||||
|
||||
open http://localhost:8787/_connect
|
||||
# paste the Cookie header from diigo.com devtools; mint
|
||||
# success page shows a copy button for the id
|
||||
# add DIIGO_WEB_ID="<id>" to diigo-dump/.dev.vars (only the id lives
|
||||
# on disk; the cookie lives in the DO's storage)
|
||||
# restart wrangler (pnpm dev) to pick it up
|
||||
open http://localhost:8787/
|
||||
# 1. "mint the diigo session at webapp" → webapp's own mint form
|
||||
# (diigo-prefilled, served on :8787); paste the Cookie header from
|
||||
# diigo.com devtools → mint (the cookie transits diigo-cache as an
|
||||
# opaque relay into webapp's DO only)
|
||||
# 2. the wire step: webapp id pre-filled, choose public|private →
|
||||
# cacheId; copy the ids somewhere you control (password manager),
|
||||
# paste the cacheId into the dump form
|
||||
# (no .dev.vars, no restarts: ids ride the form POSTs / your paste)
|
||||
|
||||
Terminal flow (superseded by the page; needs a display):
|
||||
Terminal flow (the same shape, scripted):
|
||||
|
||||
pnpm -s diigo-login >> .dev.vars # askpass prompt; appends the id line
|
||||
# (.dev.vars: pnpm stdout noise can
|
||||
# interleave; grep ^DIIGO_ to clean)
|
||||
wid=$(curl -s -X POST http://localhost:8787/_mint --data-urlencode \
|
||||
'url=https://www.diigo.com/interact_api/' --data-urlencode \
|
||||
'headers={"user-agent":"Mozilla/5.0","x-requested-with":"XMLHttpRequest","cookie":"<paste>"}' \
|
||||
--data-urlencode 'msPerCall=500' -D - -o /dev/null | grep -i x-webapp-id)
|
||||
curl -s -X POST http://localhost:8787/_wire -H 'content-type: application/json' \
|
||||
-d "{\"webappId\":\"<from $wid>\",\"mode\":\"private\"}"
|
||||
|
||||
Then: enter a page number on the home page, dump → JSON items. GET /
|
||||
is inert (no accidental fetches; dumps happen only on POST submits).
|
||||
Then: enter a page number, dump → JSON rows with a `cache: HIT/MISS`
|
||||
line (a repeat page is answered from DO sqlite storage). GET / is inert
|
||||
(dumps happen only on POST submits). Public pages:
|
||||
/public/<username>?page=n (each page fills from diigo on first visit).
|
||||
A private-only cache can be published later from its created page
|
||||
(`POST /_publish`, cache id in the body).
|
||||
|
||||
Checks: `cd diigo-cache && pnpm lint && pnpm test`.
|
||||
|
||||
madmode (EmDash site; separate worker, separate concerns):
|
||||
|
||||
@@ -101,18 +132,22 @@ madmode (EmDash site; separate worker, separate concerns):
|
||||
## Working rules
|
||||
|
||||
Agent working rules are in [AGENTS.md](AGENTS.md) — dev servers are the
|
||||
user's job; secrets discipline; keep config types generated. They apply
|
||||
user's job; durable state (loud about breaking changes); secrets
|
||||
discipline; keep config types generated. They apply
|
||||
to every goal in this tree.
|
||||
|
||||
## Pointers
|
||||
|
||||
- [the capsite notes](capsite-notes.md) — the original goal and the EmDash
|
||||
native-plugin wiring details
|
||||
- [docs/webkeys.md](docs/webkeys.md) — the Waterken URL-capability
|
||||
pattern, abstracted from the E-on-Java sources (bearer secrets,
|
||||
YURL32, secret≠address); the reason ids never ride URLs
|
||||
- [the mm-diigo diigo-bak scripts](https://github.com/dckc/mm-diigo) —
|
||||
provenance of the fluent client and dump generator (`projects/diigo-bak/`:
|
||||
WebApp.js, diigo-dump.mjs, diigo-interact.mjs; mm-diigo commit 7dbb367,
|
||||
Oct 6 2026)
|
||||
- [webapp/README.md](webapp/README.md) — HTTP + RPC surfaces of the
|
||||
factory worker; open TODO on the mint-form placement
|
||||
- [diigo-dump/README.md](diigo-dump/README.md) — consumer wiring and
|
||||
run/deploy details
|
||||
- [diigo-cache/README.md](diigo-cache/README.md) — the cache worker:
|
||||
create flow, the two DOs, run/deploy details
|
||||
+7
-2
@@ -17,7 +17,9 @@ If it works out well, we should be able to expand to [bib sync #195](https://git
|
||||
- `webapp/` — site-agnostic Durable-Object factory for attenuated credentialed
|
||||
web clients (fluent client, ids are bearer capabilities);
|
||||
[webapp/README.md](webapp/README.md)
|
||||
- `diigo-dump/` — the webapp consumer: its mint call carries all diigo
|
||||
- `diigo-cache/` — the webapp consumer, evolved from the diigo-dump worker:
|
||||
bookmark dumps cached in DO SQLite storage (private cache by unique id,
|
||||
public-only cache by username-in-path); the mint call carries all diigo
|
||||
knowledge; the cookie lives in the DO's storage (see
|
||||
[the checklist's goal tree](capsite-checklist.md) for the verified/not-yet
|
||||
split)
|
||||
@@ -74,7 +76,10 @@ authoritative. Current shape: the session cookie lives only in the webapp
|
||||
DO's storage; `madmode/.dev.vars` still holds placeholder
|
||||
`DIIGO_API_KEY` / `DIIGO_USERNAME` for the (dead) API-key path — both
|
||||
documented schemes return 401 unconditionally, verified against both known
|
||||
real keys (see checklist goal 2). Deployed secrets: `cd madmode && pnpm
|
||||
real keys (opencode session ses_eec320fe4ffehaMAn44U1nTKJU: `key=` param,
|
||||
`key`+`user=`, and HTTP Basic, on both /api/v2/bookmarks and /api/v2/notes,
|
||||
though the same creds downloaded 13.5k bookmarks in 2024–2025; diigo
|
||||
evidently revoked v2 key auth). Deployed secrets: `cd madmode && pnpm
|
||||
wrangler secret put <NAME>`; the sync rides the cookie path since the API
|
||||
path died.
|
||||
|
||||
|
||||
Reference in new issue
Block a user