dckc
0af7a3ea6c
New README: two sentences, the dev run through connect, fill, and a filtered public page, then features. CONTRIBUTING gains the 1d filters and its own run and deploy sections instead of pointing at the checklist; links to the old README now point at CONTRIBUTING. Generated-by: Claude Opus 5.5
143 lines
6.8 KiB
Markdown
143 lines
6.8 KiB
Markdown
# Contributing to diigo-cache
|
|
|
|
diigo-cache is a Cloudflare Worker cache over [the webapp factory](../webapp/README.md):
|
|
session-cookie reads of diigo bookmarks, cached in Durable-Object SQLite
|
|
storage. Evolved from the diigo-dump worker (provenance:
|
|
[mm-diigo `projects/diigo-bak/`](https://github.com/dckc/mm-diigo)); the
|
|
static API-key path is dead (see [the capsite checklist](../capsite-checklist.md),
|
|
goal 2 preamble).
|
|
|
|
## What it holds
|
|
|
|
- **The create flow, on one port**: `/_connect` renders webapp's own
|
|
mint form (diigo-prefilled, cookie blank) — wrangler multi-config dev
|
|
doesn't serve aux workers' HTTP, so `/_mint` relays the form opaquely
|
|
into the WEBAPP binding (the cookie crosses; nothing on this side opens
|
|
or logs it). Mint → `/_wire` (webapp id pre-filled, mode
|
|
public|private) → the created page, where the user saves the ids.
|
|
- **Two independent page caches** (`src/cache.ts`), one behavior: a
|
|
page hit is served from storage with no diigo call; a miss asks the
|
|
webapp instance in the cache's wiring and etches the page.
|
|
- `DiigoCache` (binding `CACHE_DO`): every item. Addressed by its
|
|
cache id.
|
|
- `PublicCache` (binding `PUBLIC_DO`): shared items only — private
|
|
rows are dropped *before* etching (`publicOnly`), so its storage
|
|
holds nothing private. Same cookie webapp as the private cache.
|
|
Public page n is diigo page n, filtered (may show fewer than
|
|
`PAGE_SIZE`, 100: diigo's cap).
|
|
`/public/<username>` resolves through the `idFromName("index")`
|
|
registry, and reads through `viewer()`, a read-only facet: it shows
|
|
stored pages and fill progress, and can't fetch from diigo or write.
|
|
- **Storage**: SQL lives in `src/cache-read.sql` (selects only) and
|
|
`src/cache-write.sql`; `src/store.ts` turns them into a reader and a
|
|
writer facet, so the cache logic holds no SQL text.
|
|
- **Pages**: home's dump form (cache id + page number, POST) shows raw
|
|
dump rows; `/public/<username>?page=n` shows an HTML list with a page
|
|
form (JSON for `Accept: application/json`). Both say `cache: HIT` or
|
|
`MISS`; every page read also logs `{event: "cache", kind, page,
|
|
outcome}`.
|
|
- **Filters on the public page** (`src/query.ts`): `days`, `after`,
|
|
`before`, `in` (`2026`, `2026-03`, `2026-Q1`, `2026-W41`),
|
|
`by=updated|created`, and `q` (words, `"phrases"`, `#tag`, `-`
|
|
negation, gmail/Slack/GitHub date operators). UTC, half-open windows.
|
|
The same fields work as a dev-side spec (`toQuery({ days: 7 })`). A
|
|
filter queries the stored `items` table through `viewer()`, never
|
|
diigo: results are as fresh as the cache, and JSON says `complete`
|
|
when every diigo page is stored.
|
|
- **Publish later**: `POST /_publish` (cache id in the body) wires a
|
|
public companion to a private-only cache.
|
|
- **The webkey page and Cap'n Web**: `/cache/#<cache id>` is a skeleton
|
|
page, the same for every key; its script reads the key from the
|
|
fragment and trades it in-band for a capability over
|
|
[Cap'n Web](https://github.com/cloudflare/capnweb) at `/api`
|
|
(`CacheApi.cache(key)` → a `CacheCap` facet: fill, status, the public
|
|
companion; `src/api.ts`). The worker serves capnweb's browser build
|
|
itself at `/_capnweb.js`.
|
|
- **Fill job**: walks every diigo page into a cache in the background —
|
|
DO alarms, a few pages per step, progress stored in the DO
|
|
(`startFill`/`fillStatus`/`stopFill`), driven from the webkey page;
|
|
the public page shows progress. "Invalidate the last N days" there
|
|
drops the cached pages holding anything updated that recently
|
|
(`invalidateRecent`), so the next fill refetches them.
|
|
- The `DiigoBridge` entrypoint for other workers (its cache-id source is
|
|
the goal-2 wiring TODO).
|
|
|
|
## State surface
|
|
|
|
Ids are **webkeys** ([the pattern](../docs/webkeys.md)): 256-bit bearer
|
|
secrets that never ride a query string or path. The webkey page carries
|
|
the cache id in the URL fragment (never sent to the server) and passes
|
|
it in-band over Cap'n Web; forms carry ids in POST bodies; logs and pages
|
|
show only a fingerprint (a prefix of the id's one-way hash).
|
|
|
|
- **The cache id**: the cache's read capability (bearer; not logged).
|
|
Held by the user — saved to the password manager under a petname they
|
|
choose (the username field; never sent to the server) — and passed
|
|
per POST.
|
|
- **The webapp instance id**: the diigo-read capability; lives in each
|
|
cache DO's wiring, arrives from the mint step.
|
|
- **The cookie**: lives **only in webapp's DO storage**; crosses the
|
|
wire once, at mint time, as an opaque relay.
|
|
- **The public cache**: no capability — `/public/<username>` is public
|
|
by design (a guessable URL; hiding the link doesn't hide the page).
|
|
|
|
Stored state is meant to be durable: see the breaking-change rule in
|
|
[capsite AGENTS.md](../AGENTS.md).
|
|
|
|
## Deployment concern: rate-limit `/api`
|
|
|
|
Public pages only read storage, so strangers can't make the cache spend
|
|
your diigo session. `/api` is keyed (a cache id is needed), but Cap'n
|
|
Web pipelining makes calls cheap to send. Before exposing a deployment:
|
|
|
|
- rate-limit `/api` (and `/public/*`, which reads storage) — on
|
|
Cloudflare, the
|
|
[Workers rate-limit binding](https://developers.cloudflare.com/workers/runtime-apis/bindings/rate-limit/)
|
|
(keyed by client and/or username); on other workerd hosts, a rate
|
|
limit in the proxy in front;
|
|
- run the fill job (goal 1e): public pages show only what's stored.
|
|
|
|
## RPC use (from another worker, same account)
|
|
|
|
```jsonc
|
|
// wrangler.jsonc of the caller
|
|
"services": [{ "binding": "DIIGO", "service": "diigo-cache", "entrypoint": "DiigoBridge" }]
|
|
```
|
|
|
|
```ts
|
|
// cache ids ride goal-2's wiring; see the capsite checklist
|
|
const ndjson = await env.DIIGO.dumpNdjson(4); // 4 pages of PAGE_SIZE
|
|
```
|
|
|
|
## Test
|
|
|
|
pnpm types # regenerate worker-configuration.d.ts (after wrangler.jsonc changes)
|
|
pnpm lint # fails if those types are stale; tsc over src/ and tests/
|
|
pnpm test # vitest
|
|
|
|
Tests run the real worker and both DOs in the vitest pool, with
|
|
`WEBAPP` bound to [a fake webapp](tests/fake-webapp.js) serving
|
|
diigo-shaped items (`wrangler.tests.jsonc`).
|
|
|
|
## Run
|
|
|
|
[The README](README.md) gets a cache going in dev. The same create flow
|
|
from a terminal:
|
|
|
|
wid=$(curl -s -X POST http://localhost:8787/_mint --data-urlencode \
|
|
'url=https://www.diigo.com/interact_api/' --data-urlencode \
|
|
'headers={"user-agent":"Mozilla/5.0","x-requested-with":"XMLHttpRequest","cookie":"<paste>"}' \
|
|
--data-urlencode 'msPerCall=500' -D - -o /dev/null | grep -i x-webapp-id)
|
|
curl -s -X POST http://localhost:8787/_wire -H 'content-type: application/json' \
|
|
-d "{\"webappId\":\"<from $wid>\",\"mode\":\"private\"}"
|
|
|
|
Check a filled cache against a diigo RSS export:
|
|
`node scripts/compare-export.ts <cache.sqlite> <export.xml> <out-dir>`.
|
|
|
|
## Deploy
|
|
|
|
Not deployed yet; the order matters (goal 5 in
|
|
[the capsite checklist](../capsite-checklist.md)): webapp first
|
|
(`cd ../webapp && npx wrangler deploy`), then diigo-cache, after
|
|
rate-limiting `/api` (above).
|