228 lines
14 KiB
Markdown
228 lines
14 KiB
Markdown
# capsite goal tree
|
|
|
|
Nested checkboxes form the goal tree itself: each `- [ ]` is a goal; its
|
|
indented children are the subgoals/steps that must pass before it (an item
|
|
is only checked when **all** its children are checked — or, for leaf
|
|
items, when it has passed verification itself). Working notes live under
|
|
each goal; [the capsite notes](capsite-notes.md) carry the background
|
|
narrative.
|
|
|
|
- [ ] **madmode publishes a weekly roundup of diigo bookmarks**
|
|
([#23](https://github.com/dckc/madmode-blog/issues/23)); the diigo
|
|
session stays inside webapp, so madmode holds no credential at all
|
|
(background: [the capsite notes](capsite-notes.md); after that,
|
|
[bib sync #195](https://github.com/dckc/madmode-blog/issues/195))
|
|
|
|
- [x] Credential discipline: no worker secrets; the cookie lives only
|
|
in webapp's DO storage, ids are held by the user
|
|
([AGENTS.md](AGENTS.md), secrets section).
|
|
- [x] **1. Cookie-path dump worker**: port of mm-diigo's diigo-dump.mjs
|
|
(verified 2026-10-06: 200, `total: 1674`); evolved into 1c.
|
|
- [x] **1b. webapp DO factory** ([webapp/README.md](webapp/README.md));
|
|
mint verified end-to-end in dev 2026-10-07 (via 1c). Prod mint is
|
|
goal 5.
|
|
|
|
- [x] **1c. diigo-dump evolves into a bookmark cache**
|
|
([diigo-cache/CONTRIBUTING.md](diigo-cache/CONTRIBUTING.md); design in the
|
|
JSDoc of `src/cache.ts`, `src/index.ts`, `src/pm-save.ts`):
|
|
- [x] 1c-i. Two independent page caches, private and public
|
|
(`PagedCache`, `DiigoCache`, `PublicCache`).
|
|
- [x] 1c-ii. Create flow on one port: `/_connect` → `/_mint`
|
|
→ `/_wire` (`mintPage`, `proxyMint`, `wire`).
|
|
- [x] 1c-iii. Ids saved by explicit user action under the
|
|
holder's petname (`pmSave`); pages show a fingerprint
|
|
(`nick`).
|
|
- [x] 1c-iv. Verified against live diigo in dev, 2026-10-07:
|
|
create, dump, publish; `/public/dckc-madmode?page=4` →
|
|
`cache: HIT`.
|
|
- [x] 1c-v. Tests: `pnpm lint && pnpm test` (the real worker +
|
|
DOs against [a fake webapp](diigo-cache/tests/fake-webapp.js)).
|
|
- [x] 1c-vi. Complete: a filled cache holds everything in diigo's
|
|
RSS export (16,744 of 16,744; differences are form only),
|
|
2026-10-07, via `scripts/compare-export.ts`. Getting there
|
|
found diigo's `page_num` is 0-based (schema v2 migration).
|
|
|
|
- [x] **1d. Date windows and search on the public page**, toward the
|
|
weekly PESOS roundup of [#23](https://github.com/dckc/madmode-blog/issues/23)
|
|
(a week's permalink: `in=2026-W41`). One filter, two spellings:
|
|
devs pass `{ days: 7, q: "#weekly" }`, URLs the same fields;
|
|
`q` takes gmail/Slack/GitHub operators. UTC, ISO weeks, half-open
|
|
windows (`diigo-cache/src/query.ts`; `tests/query.test.ts`,
|
|
`tests/caches.test.ts` "public page filters"). Filters read
|
|
stored `items` only (the public page never fetches): results are
|
|
as fresh as the cache, and say `complete` when every diigo page
|
|
is stored. Keeping them fresh is 1f. 2026-10-09.
|
|
|
|
- [x] **1e. Fill job: a button that grabs every page** into a cache
|
|
from the webkey page `/cache/#<key>`, which trades the key for a
|
|
capability over Cap'n Web (`diigo-cache/src/api.ts`;
|
|
`startFill`/`fillStatus`/`stopFill` in `src/cache.ts`;
|
|
`tests/fill.test.ts`). `/cache/` without a key is a login form
|
|
the password manager fills.
|
|
|
|
- [ ] **1f. Cached pages go stale: the head must refresh.** A page hit
|
|
never expires, and diigo pages are updated-desc, so every new or
|
|
edited bookmark shifts the page boundaries: this week's bookmarks
|
|
sit on page 1, cached last week. Proposal: refetch the head
|
|
(pages from 1 until a fetched page's items are all already stored
|
|
with the same `updated`), upserting into `items` (already keyed by
|
|
`item_id`); the window/search reads of 1d run over `items`, not
|
|
over cached page numbers. A fill or a cron alarm (`ctx.storage`
|
|
alarm in the cache DO, not in madmode) can drive it.
|
|
**Durable state**: additive only if `pages`/`page_members` keep
|
|
their meaning; say so before changing them (capsite AGENTS.md).
|
|
Needs a test: an item added to the fake webapp shows up after a
|
|
refresh. Stopgap today: "invalidate the last N days" on the
|
|
webkey page (`invalidateRecent`), then fill.
|
|
|
|
- [ ] **1g. Policy: notes on public items are public too.** The public
|
|
cache drops private *items* (`publicOnly`) but keeps public items
|
|
whole, including annotations, comments, and sticky notes that
|
|
diigo marks private (`pri_a_count`/`pri_c_count`/
|
|
`pri_sticky_count`: 2,525 of 14,645 stored public items, found
|
|
2026-10-08), and `/public/<username>` JSON serves them. **Dan
|
|
finds this acceptable as is**; this goal is to finish thinking
|
|
it through, not to fix a leak. Open points: does diigo.com's
|
|
own public view hide them (so readers are surprised)? the
|
|
`comments` array may carry *other people's* words; which fields
|
|
does potluck quote (annotations are the best commentary: see
|
|
[the issues sketch](potluck/issues-sketch.md))? If the policy
|
|
changes, stripping stored rows is a durable-state migration.
|
|
|
|
- [ ] **2. potluck reads the public cache**: public data, so no capability, no cookie, no secret on potluck's side. The [diigo-week](diigo-week/README.md) plugin reads `GET /public/dckc-madmode?in=<week>&by=created` as microformats (an `h-feed` of `h-entry` bookmarks). Working in dev 2026-10-09; 2-iii postponed.
|
|
- [x] 2-i. Transport: the `DIIGO_CACHE` service binding in potluck's `wrangler.jsonc`, in dev too: the plugin's `ctx.http.fetch` refuses localhost (EmDash's SSRF guard), and a binding names one worker, not a host.
|
|
- [x] 2-ii. The page's microformats are the contract: diigo-cache's tests parse it with microformats-parser; diigo-week's tests read fixtures in the same markup.
|
|
- [ ] 2-iii. Decide `DiigoBridge`'s fate (postponed: nothing on this path uses it; it matters only for a caller that reads private items).
|
|
- [x] 2-iv. Run it in dev: diigo-cache and potluck both up; the dashboard card → a draft issue of 22 bookmarks for 2026-W40 (2026-10-09). The service binding resolves across the two dev servers.
|
|
- [ ] ~~**3. `/bookmarks` renders real data**~~: superseded by potluck's issues (the madmode starter is gone).
|
|
- [ ] **4. The weekly draft**: decided (2026-10-09): a draft issue you edit before publishing, made by diigo-week. The slug comes from the title (`week-2026-w40`).
|
|
|
|
- [ ] **4b. Consider scruffier tags.** EmDash tags are taxonomy terms (rows that must exist before use); diigo-week auto-creates them, so drafting feels scruffy, but the model is neat. Alternative: the inline `/tag/<tag>` links in issue bodies *are* the tags (matching the microformats `p-category` reading), and potluck's `/tag/<tag>` page finds issues by scanning bodies. Cost: no EmDash tag field or index; a scan over a few dozen issues a year is cheap. Open: does the admin's tag field let you type a new tag inline?
|
|
|
|
- [ ] **5. Deployed end-to-end** (order matters):
|
|
- [ ] 5-i. webapp: `cd webapp && npx wrangler deploy`
|
|
- [ ] 5-ii. diigo-cache: rate-limit `/api` (and `/public/*`) first
|
|
(diigo-cache CONTRIBUTING, "Deployment concern"); deploy; create
|
|
against prod (ids → password manager; no worker secret);
|
|
run the fill job.
|
|
- [ ] 5-iii. madmode: deploy with the service binding; the
|
|
weekly page renders against prod.
|
|
- [ ] **6. Cookie expiry**: when diigo's session ends, head refreshes
|
|
fail; the public page keeps serving what's stored (say so on the
|
|
page, don't fail silently). Rotation is manual: re-mint via
|
|
`/_connect`. Open question: can an existing cache be re-wired to
|
|
a new webapp id (keeping its cache id and public username), or
|
|
does re-mint mean a new cache? Prefer re-wire — durable ids.
|
|
- [ ] **7. bib sync #195**: repeat the pattern for the next
|
|
read-access need (webapp mints for any origin; the cache's
|
|
item shape is diigo-specific): [the zotero goal tree](zotero-checklist.md).
|
|
The social.coop account is next after that:
|
|
[the social.coop goal tree](social-coop-checklist.md).
|
|
- [ ] **webapp graduates from prototype** (optional; only if it earns it)
|
|
- [ ] TODO(webapp): one canonical mint form — proxy-page (current) vs
|
|
running webapp as the primary worker so its own port serves the
|
|
form directly; revisit as the UI grows.
|
|
- [ ] @endo/errors for structured error details (needs node_modules).
|
|
- [ ] **toybox: typed pieces that fit together, peer to peer** (working
|
|
name; tinker toys per the madmode.com home page). Kinds are typed
|
|
interfaces, à la Genode session interfaces, but wired by
|
|
introduction (E's Granovetter diagram), not a parent tree; named by
|
|
petnames; each piece guards its own resources. Why:
|
|
[the toybox user stories](toybox-stories.md).
|
|
- [ ] **T1. Changed signal**: a reader can subscribe to "this cache
|
|
changed" instead of polling (Genode's ROM-session sigh is the
|
|
model: public cache ≈ ROM, `/_publish` ≈ report). Fires on 1f's
|
|
head refresh when it upserts anything new. Shape TBD: a Cap'n Web
|
|
callback stub the cache holds (live only), or a durable
|
|
subscription it calls back via a webkey (survives eviction).
|
|
- [ ] **T2. Compose with [Blockly](https://developers.google.com/blockly)**:
|
|
connector checks (`setCheck(kind)`) are the shapes; a workspace
|
|
is a set of introductions. **Blocks refer to capabilities by
|
|
petname**, never by key: workspace JSON is shareable; the keys
|
|
live with whatever runs it.
|
|
- [ ] **T3. Run compositions in [Dynamic Workers](https://developers.cloudflare.com/dynamic-workers/)**:
|
|
code from anywhere: Blockly's JS generator, code written by
|
|
agents, by hand. Confinement makes codegen safe: the Worker
|
|
Loader isolate's `env` holds only the stubs its petnames resolve
|
|
to, with `globalOutbound: null`, so the code's authority is
|
|
exactly what it was handed. (Ordinary Workers forbid `eval`;
|
|
lifting that is the point of the Worker Loader.)
|
|
|
|
## How to run (continuation prompts)
|
|
|
|
Dev (diigo-cache + webapp in one workerd):
|
|
|
|
cd ~/oros-bots/capsite/diigo-cache
|
|
pnpm dev # → http://localhost:8787
|
|
# webapp rides along (binding-only)
|
|
|
|
First connect (or re-connect after cookie expiry) — browser flow, two steps:
|
|
|
|
open http://localhost:8787/
|
|
# 1. "mint the diigo session at webapp" → webapp's own mint form
|
|
# (diigo-prefilled, served on :8787); paste the Cookie header from
|
|
# diigo.com devtools → mint (the cookie transits diigo-cache as an
|
|
# opaque relay into webapp's DO only)
|
|
# 2. the wire step: webapp id pre-filled, choose public|private →
|
|
# cacheId; copy the ids somewhere you control (password manager),
|
|
# paste the cacheId into the dump form
|
|
# (no .dev.vars, no restarts: ids ride the form POSTs / your paste)
|
|
|
|
Terminal flow (the same shape, scripted):
|
|
|
|
wid=$(curl -s -X POST http://localhost:8787/_mint --data-urlencode \
|
|
'url=https://www.diigo.com/interact_api/' --data-urlencode \
|
|
'headers={"user-agent":"Mozilla/5.0","x-requested-with":"XMLHttpRequest","cookie":"<paste>"}' \
|
|
--data-urlencode 'msPerCall=500' -D - -o /dev/null | grep -i x-webapp-id)
|
|
curl -s -X POST http://localhost:8787/_wire -H 'content-type: application/json' \
|
|
-d "{\"webappId\":\"<from $wid>\",\"mode\":\"private\"}"
|
|
|
|
Then: enter a page number, dump → JSON rows with a `cache: HIT/MISS`
|
|
line (a repeat page is answered from DO sqlite storage). GET / is inert
|
|
(dumps happen only on POST submits). Public pages:
|
|
/public/<username>?page=n (each page fills from diigo on first visit).
|
|
A private-only cache can be published later from its created page
|
|
(`POST /_publish`, cache id in the body).
|
|
|
|
Fill and invalidate a cache: http://localhost:8787/cache/ (log in with
|
|
the cache id from the password manager).
|
|
|
|
Checks: `cd diigo-cache && pnpm lint && pnpm test`. Against a diigo RSS
|
|
export: `node scripts/compare-export.ts <cache.sqlite> <export.xml> <out-dir>`.
|
|
|
|
potluck (EmDash site, fresh out of the box: `pnpm create emdash@latest
|
|
potluck --pm pnpm --install --yes` → cloudflare:blog, 2026-10-08, per
|
|
[Create your first EmDash site](https://docs.emdashcms.com/); replaces
|
|
the old madmode/ starter):
|
|
|
|
cd ~/oros-bots/capsite/potluck && pnpm dev # → :4321
|
|
# first run: the setup wizard at /_emdash/admin (site title, email)
|
|
# dev-bypass auth helper:
|
|
# /_emdash/api/setup/dev-bypass?redirect=/_emdash/admin
|
|
|
|
## Working rules
|
|
|
|
Agent working rules are in [AGENTS.md](AGENTS.md) — dev servers are the
|
|
user's job; durable state (loud about breaking changes); secrets
|
|
discipline; keep config types generated. They apply
|
|
to every goal in this tree.
|
|
|
|
## Pointers
|
|
|
|
- [the toybox user stories](toybox-stories.md) — what the pieces are for
|
|
- [the zotero goal tree](zotero-checklist.md) and
|
|
[the social.coop goal tree](social-coop-checklist.md) — the next caches
|
|
- [the capsite notes](capsite-notes.md) — the original goal and the EmDash
|
|
native-plugin wiring details
|
|
- [docs/webkeys.md](docs/webkeys.md) — the Waterken URL-capability
|
|
pattern, abstracted from the E-on-Java sources (bearer secrets,
|
|
YURL32, secret≠address); the reason ids ride only fragments and POST
|
|
bodies
|
|
- [the mm-diigo diigo-bak scripts](https://github.com/dckc/mm-diigo) —
|
|
provenance of the fluent client and dump generator (`projects/diigo-bak/`:
|
|
WebApp.js, diigo-dump.mjs, diigo-interact.mjs; mm-diigo commit 7dbb367,
|
|
Oct 6 2026)
|
|
- [webapp/README.md](webapp/README.md) — HTTP + RPC surfaces of the
|
|
factory worker; open TODO on the mint-form placement
|
|
- [diigo-cache/CONTRIBUTING.md](diigo-cache/CONTRIBUTING.md) — the cache worker:
|
|
create flow, the two DOs, run/deploy details |