dckc
10414c2ce6
The cookie moves out of the immutable init into a MapStore (key shape "cookie", value M.string()) in the instance's DO storage. Handles get its read-only view via the new WebApp.withHeaderMap, which reads at every send, so a rotation reaches handles already out and the webapp id, and with it the cache's wiring, stays put (checklist goal 6). Whoever holds the id can rotate, by RPC (WebAppFactory.cookies(id)) or POST /at/<id>/cookie; the diigo account owner revokes by logging out. Durable state: instances minted earlier keep the cookie in init.headers. The first handle() moves it to the store and rewrites init without it, in one transaction. Code from before this commit would then see no cookie, so rolling back past it needs the reverse move. webapp gains a package.json and a vitest workerd pool for the tests. Generated-by: Claude Opus 5.5
228 lines
14 KiB
Markdown
228 lines
14 KiB
Markdown
# capsite goal tree
|
|
|
|
Nested checkboxes form the goal tree itself: each `- [ ]` is a goal; its
|
|
indented children are the subgoals/steps that must pass before it (an item
|
|
is only checked when **all** its children are checked — or, for leaf
|
|
items, when it has passed verification itself). Working notes live under
|
|
each goal; [the capsite notes](capsite-notes.md) carry the background
|
|
narrative.
|
|
|
|
- [ ] **madmode publishes a weekly roundup of diigo bookmarks**
|
|
([#23](https://github.com/dckc/madmode-blog/issues/23)); the diigo
|
|
session stays inside webapp, so madmode holds no credential at all
|
|
(background: [the capsite notes](capsite-notes.md); after that,
|
|
[bib sync #195](https://github.com/dckc/madmode-blog/issues/195))
|
|
|
|
- [x] Credential discipline: no worker secrets; the cookie lives only
|
|
in webapp's DO storage, ids are held by the user
|
|
([AGENTS.md](AGENTS.md), secrets section).
|
|
- [x] **1. Cookie-path dump worker**: port of mm-diigo's diigo-dump.mjs
|
|
(verified 2026-10-06: 200, `total: 1674`); evolved into 1c.
|
|
- [x] **1b. webapp DO factory** ([webapp/README.md](webapp/README.md));
|
|
mint verified end-to-end in dev 2026-10-07 (via 1c). Prod mint is
|
|
goal 5.
|
|
|
|
- [x] **1c. diigo-dump evolves into a bookmark cache**
|
|
([diigo-cache/CONTRIBUTING.md](diigo-cache/CONTRIBUTING.md); design in the
|
|
JSDoc of `src/cache.ts`, `src/index.ts`, `src/pm-save.ts`):
|
|
- [x] 1c-i. Two independent page caches, private and public
|
|
(`PagedCache`, `DiigoCache`, `PublicCache`).
|
|
- [x] 1c-ii. Create flow on one port: `/_connect` → `/_mint`
|
|
→ `/_wire` (`mintPage`, `proxyMint`, `wire`).
|
|
- [x] 1c-iii. Ids saved by explicit user action under the
|
|
holder's petname (`pmSave`); pages show a fingerprint
|
|
(`nick`).
|
|
- [x] 1c-iv. Verified against live diigo in dev, 2026-10-07:
|
|
create, dump, publish; `/public/dckc-madmode?page=4` →
|
|
`cache: HIT`.
|
|
- [x] 1c-v. Tests: `pnpm lint && pnpm test` (the real worker +
|
|
DOs against [a fake webapp](diigo-cache/tests/fake-webapp.js)).
|
|
- [x] 1c-vi. Complete: a filled cache holds everything in diigo's
|
|
RSS export (16,744 of 16,744; differences are form only),
|
|
2026-10-07, via `scripts/compare-export.ts`. Getting there
|
|
found diigo's `page_num` is 0-based (schema v2 migration).
|
|
|
|
- [x] **1d. Date windows and search on the public page**, toward the
|
|
weekly PESOS roundup of [#23](https://github.com/dckc/madmode-blog/issues/23)
|
|
(a week's permalink: `in=2026-W41`). One filter, two spellings:
|
|
devs pass `{ days: 7, q: "#weekly" }`, URLs the same fields;
|
|
`q` takes gmail/Slack/GitHub operators. UTC, ISO weeks, half-open
|
|
windows (`diigo-cache/src/query.ts`; `tests/query.test.ts`,
|
|
`tests/caches.test.ts` "public page filters"). Filters read
|
|
stored `items` only (the public page never fetches): results are
|
|
as fresh as the cache, and say `complete` when every diigo page
|
|
is stored. Keeping them fresh is 1f. 2026-10-09.
|
|
|
|
- [x] **1e. Fill job: a button that grabs every page** into a cache
|
|
from the webkey page `/cache/#<key>`, which trades the key for a
|
|
capability over Cap'n Web (`diigo-cache/src/api.ts`;
|
|
`startFill`/`fillStatus`/`stopFill` in `src/cache.ts`;
|
|
`tests/fill.test.ts`). `/cache/` without a key is a login form
|
|
the password manager fills.
|
|
|
|
- [ ] **1f. Cached pages go stale: the head must refresh.** A page hit
|
|
never expires, and diigo pages are updated-desc, so every new or
|
|
edited bookmark shifts the page boundaries: this week's bookmarks
|
|
sit on page 1, cached last week. Proposal: refetch the head
|
|
(pages from 1 until a fetched page's items are all already stored
|
|
with the same `updated`), upserting into `items` (already keyed by
|
|
`item_id`); the window/search reads of 1d run over `items`, not
|
|
over cached page numbers. A fill or a cron alarm (`ctx.storage`
|
|
alarm in the cache DO, not in madmode) can drive it.
|
|
**Durable state**: additive only if `pages`/`page_members` keep
|
|
their meaning; say so before changing them (capsite AGENTS.md).
|
|
Needs a test: an item added to the fake webapp shows up after a
|
|
refresh. Stopgap today: "invalidate the last N days" on the
|
|
webkey page (`invalidateRecent`), then fill.
|
|
|
|
- [ ] **1g. Policy: notes on public items are public too.** The public
|
|
cache drops private *items* (`publicOnly`) but keeps public items
|
|
whole, including annotations, comments, and sticky notes that
|
|
diigo marks private (`pri_a_count`/`pri_c_count`/
|
|
`pri_sticky_count`: 2,525 of 14,645 stored public items, found
|
|
2026-10-08), and `/public/<username>` JSON serves them. **Dan
|
|
finds this acceptable as is**; this goal is to finish thinking
|
|
it through, not to fix a leak. Open points: does diigo.com's
|
|
own public view hide them (so readers are surprised)? the
|
|
`comments` array may carry *other people's* words; which fields
|
|
does potluck quote (annotations are the best commentary: see
|
|
[the issues sketch](potluck/issues-sketch.md))? If the policy
|
|
changes, stripping stored rows is a durable-state migration.
|
|
|
|
- [ ] **2. potluck reads the public cache**: public data, so no capability, no cookie, no secret on potluck's side. The [diigo-week](diigo-week/README.md) plugin reads `GET /public/dckc-madmode?in=<week>&by=created` as microformats (an `h-feed` of `h-entry` bookmarks). Working in dev 2026-10-09; 2-iii postponed.
|
|
- [x] 2-i. Transport: the `DIIGO_CACHE` service binding in potluck's `wrangler.jsonc`, in dev too: the plugin's `ctx.http.fetch` refuses localhost (EmDash's SSRF guard), and a binding names one worker, not a host.
|
|
- [x] 2-ii. The page's microformats are the contract: diigo-cache's tests parse it with microformats-parser; diigo-week's tests read fixtures in the same markup.
|
|
- [ ] 2-iii. Decide `DiigoBridge`'s fate (postponed: nothing on this path uses it; it matters only for a caller that reads private items).
|
|
- [x] 2-iv. Run it in dev: diigo-cache and potluck both up; the dashboard card → a draft issue of 22 bookmarks for 2026-W40 (2026-10-09). The service binding resolves across the two dev servers.
|
|
- [ ] ~~**3. `/bookmarks` renders real data**~~: superseded by potluck's issues (the madmode starter is gone).
|
|
- [ ] **4. The weekly draft**: decided (2026-10-09): a draft issue you edit before publishing, made by diigo-week. The slug comes from the title (`week-2026-w40`).
|
|
|
|
- [ ] **4b. Consider scruffier tags.** EmDash tags are taxonomy terms (rows that must exist before use); diigo-week auto-creates them, so drafting feels scruffy, but the model is neat. Alternative: the inline `/tag/<tag>` links in issue bodies *are* the tags (matching the microformats `p-category` reading), and potluck's `/tag/<tag>` page finds issues by scanning bodies. Cost: no EmDash tag field or index; a scan over a few dozen issues a year is cheap. Open: does the admin's tag field let you type a new tag inline?
|
|
|
|
- [ ] **5. Deployed end-to-end** (order matters):
|
|
- [ ] 5-i. webapp: `cd webapp && npx wrangler deploy`
|
|
- [ ] 5-ii. diigo-cache: rate-limit `/api` (and `/public/*`) first
|
|
(diigo-cache CONTRIBUTING, "Deployment concern"); deploy; create
|
|
against prod (ids → password manager; no worker secret);
|
|
run the fill job.
|
|
- [ ] 5-iii. madmode: deploy with the service binding; the
|
|
weekly page renders against prod.
|
|
- [ ] **6. Cookie expiry**: when diigo's session ends, head refreshes
|
|
fail; the public page keeps serving what's stored (say so on the
|
|
page, don't fail silently). Rotation is in place, keeping the
|
|
webapp id (so the cache's wiring and ids hold): `POST
|
|
/at/<id>/cookie` with the new cookie (webapp README, "The cookie
|
|
store"). Still open: a rotate form in diigo-cache's UI.
|
|
- [ ] **7. bib sync #195**: repeat the pattern for the next
|
|
read-access need (webapp mints for any origin; the cache's
|
|
item shape is diigo-specific): [the zotero goal tree](zotero-checklist.md).
|
|
The social.coop account is next after that:
|
|
[the social.coop goal tree](social-coop-checklist.md).
|
|
- [ ] **webapp graduates from prototype** (optional; only if it earns it)
|
|
- [ ] TODO(webapp): one canonical mint form — proxy-page (current) vs
|
|
running webapp as the primary worker so its own port serves the
|
|
form directly; revisit as the UI grows.
|
|
- [ ] @endo/errors for structured error details (needs node_modules).
|
|
- [ ] **toybox: typed pieces that fit together, peer to peer** (working
|
|
name; tinker toys per the madmode.com home page). Kinds are typed
|
|
interfaces, à la Genode session interfaces, but wired by
|
|
introduction (E's Granovetter diagram), not a parent tree; named by
|
|
petnames; each piece guards its own resources. Why:
|
|
[the toybox user stories](toybox-stories.md).
|
|
- [ ] **T1. Changed signal**: a reader can subscribe to "this cache
|
|
changed" instead of polling (Genode's ROM-session sigh is the
|
|
model: public cache ≈ ROM, `/_publish` ≈ report). Fires on 1f's
|
|
head refresh when it upserts anything new. Shape TBD: a Cap'n Web
|
|
callback stub the cache holds (live only), or a durable
|
|
subscription it calls back via a webkey (survives eviction).
|
|
- [ ] **T2. Compose with [Blockly](https://developers.google.com/blockly)**:
|
|
connector checks (`setCheck(kind)`) are the shapes; a workspace
|
|
is a set of introductions. **Blocks refer to capabilities by
|
|
petname**, never by key: workspace JSON is shareable; the keys
|
|
live with whatever runs it.
|
|
- [ ] **T3. Run compositions in [Dynamic Workers](https://developers.cloudflare.com/dynamic-workers/)**:
|
|
code from anywhere: Blockly's JS generator, code written by
|
|
agents, by hand. Confinement makes codegen safe: the Worker
|
|
Loader isolate's `env` holds only the stubs its petnames resolve
|
|
to, with `globalOutbound: null`, so the code's authority is
|
|
exactly what it was handed. (Ordinary Workers forbid `eval`;
|
|
lifting that is the point of the Worker Loader.)
|
|
|
|
## How to run (continuation prompts)
|
|
|
|
Dev (diigo-cache + webapp in one workerd):
|
|
|
|
cd ~/oros-bots/capsite/diigo-cache
|
|
pnpm dev # → http://localhost:8787
|
|
# webapp rides along (binding-only)
|
|
|
|
First connect (or re-connect after cookie expiry) — browser flow, two steps:
|
|
|
|
open http://localhost:8787/
|
|
# 1. "mint the diigo session at webapp" → webapp's own mint form
|
|
# (diigo-prefilled, served on :8787); paste the Cookie header from
|
|
# diigo.com devtools → mint (the cookie transits diigo-cache as an
|
|
# opaque relay into webapp's DO only)
|
|
# 2. the wire step: webapp id pre-filled, choose public|private →
|
|
# cacheId; copy the ids somewhere you control (password manager),
|
|
# paste the cacheId into the dump form
|
|
# (no .dev.vars, no restarts: ids ride the form POSTs / your paste)
|
|
|
|
Terminal flow (the same shape, scripted):
|
|
|
|
wid=$(curl -s -X POST http://localhost:8787/_mint --data-urlencode \
|
|
'url=https://www.diigo.com/interact_api/' --data-urlencode \
|
|
'headers={"user-agent":"Mozilla/5.0","x-requested-with":"XMLHttpRequest","cookie":"<paste>"}' \
|
|
--data-urlencode 'msPerCall=500' -D - -o /dev/null | grep -i x-webapp-id)
|
|
curl -s -X POST http://localhost:8787/_wire -H 'content-type: application/json' \
|
|
-d "{\"webappId\":\"<from $wid>\",\"mode\":\"private\"}"
|
|
|
|
Then: enter a page number, dump → JSON rows with a `cache: HIT/MISS`
|
|
line (a repeat page is answered from DO sqlite storage). GET / is inert
|
|
(dumps happen only on POST submits). Public pages:
|
|
/public/<username>?page=n (each page fills from diigo on first visit).
|
|
A private-only cache can be published later from its created page
|
|
(`POST /_publish`, cache id in the body).
|
|
|
|
Fill and invalidate a cache: http://localhost:8787/cache/ (log in with
|
|
the cache id from the password manager).
|
|
|
|
Checks: `cd diigo-cache && pnpm lint && pnpm test`. Against a diigo RSS
|
|
export: `node scripts/compare-export.ts <cache.sqlite> <export.xml> <out-dir>`.
|
|
|
|
potluck (EmDash site, fresh out of the box: `pnpm create emdash@latest
|
|
potluck --pm pnpm --install --yes` → cloudflare:blog, 2026-10-08, per
|
|
[Create your first EmDash site](https://docs.emdashcms.com/); replaces
|
|
the old madmode/ starter):
|
|
|
|
cd ~/oros-bots/capsite/potluck && pnpm dev # → :4321
|
|
# first run: the setup wizard at /_emdash/admin (site title, email)
|
|
# dev-bypass auth helper:
|
|
# /_emdash/api/setup/dev-bypass?redirect=/_emdash/admin
|
|
|
|
## Working rules
|
|
|
|
Agent working rules are in [AGENTS.md](AGENTS.md) — dev servers are the
|
|
user's job; durable state (loud about breaking changes); secrets
|
|
discipline; keep config types generated. They apply
|
|
to every goal in this tree.
|
|
|
|
## Pointers
|
|
|
|
- [the toybox user stories](toybox-stories.md) — what the pieces are for
|
|
- [the zotero goal tree](zotero-checklist.md) and
|
|
[the social.coop goal tree](social-coop-checklist.md) — the next caches
|
|
- [the capsite notes](capsite-notes.md) — the original goal and the EmDash
|
|
native-plugin wiring details
|
|
- [docs/webkeys.md](docs/webkeys.md) — the Waterken URL-capability
|
|
pattern, abstracted from the E-on-Java sources (bearer secrets,
|
|
YURL32, secret≠address); the reason ids ride only fragments and POST
|
|
bodies
|
|
- [the mm-diigo diigo-bak scripts](https://github.com/dckc/mm-diigo) —
|
|
provenance of the fluent client and dump generator (`projects/diigo-bak/`:
|
|
WebApp.js, diigo-dump.mjs, diigo-interact.mjs; mm-diigo commit 7dbb367,
|
|
Oct 6 2026)
|
|
- [webapp/README.md](webapp/README.md) — HTTP + RPC surfaces of the
|
|
factory worker; open TODO on the mint-form placement
|
|
- [diigo-cache/CONTRIBUTING.md](diigo-cache/CONTRIBUTING.md) — the cache worker:
|
|
create flow, the two DOs, run/deploy details |