diigo-cache: bookmark cache in Durable Objects (goal 1c)
The dump worker becomes a page cache: a hit is served from DO SQLite storage with no diigo call; a miss asks the webapp instance in the cache's wiring and etches the page. - two independent caches, one behavior (PagedCache): DiigoCache keeps every item; PublicCache keeps shared items only, dropping private rows before they are stored; served at /public/<username> (HTML page form, or JSON) - create flow on one port: /_connect relays webapp's mint form, /_mint proxies the mint, /_wire wires a private or public cache; /_publish adds a public cache later - ids are webkeys: kept out of URLs (POST bodies), shown and logged only as a hash fingerprint; saved to the password manager under the holder's petname (never sent to the server) - items keyed on diigo's link_id; username from u_name; unkeyable items and every failure path are logged; each page read logs hit/miss - tests run the real worker + DOs in the vitest pool against a fake webapp (tests/fake-webapp.js) Generated-by: opencode (GLM) Generated-by: Claude Opus 5.5
This commit is contained in:
1 parent
c5e433cc9c
commit
6f409fc83a
15 files changed
+1664
-211
No files matched your search
+81
-37
@@ -1,57 +1,101 @@
|
||||
# diigo-dump
|
||||
# diigo-cache
|
||||
|
||||
Cloudflare Worker consumer of [the webapp factory](../webapp/README.md):
|
||||
the site-side of dumping [Diigo bookmarks from the mm-diigo provenance
|
||||
scripts](https://github.com/dckc/mm-diigo) (`projects/diigo-bak/`) —
|
||||
session-cookie access, most-recently-updated first. The static API-key
|
||||
path is dead (both known keys, all documented schemes → 401; see
|
||||
[the capsite checklist](../capsite-checklist.md), goal 2 preamble).
|
||||
Cloudflare Worker cache over [the webapp factory](../webapp/README.md):
|
||||
session-cookie reads of diigo bookmarks, cached in Durable-Object SQLite
|
||||
storage. Evolved from the diigo-dump worker (provenance:
|
||||
[mm-diigo `projects/diigo-bak/`](https://github.com/dckc/mm-diigo)); the
|
||||
static API-key path is dead (see [the capsite checklist](../capsite-checklist.md),
|
||||
goal 2 preamble).
|
||||
|
||||
## What it holds
|
||||
|
||||
**Only the diigo flavoring** — everything generic lives in webapp:
|
||||
|
||||
- the mint call: `url` = `https://www.diigo.com/interact_api/`, seed
|
||||
headers `{user-agent, x-requested-with}` (+ the cookie at mint time),
|
||||
`msPerCall: 500`
|
||||
- the dump page: page-number form → `WEBAPP.at(id)` → pipelined
|
||||
`pathjoin("load_user_items").query({sort, count, page_num}).get()`
|
||||
- the `DiigoBridge` entrypoint (`open()`, `dumpNdjson()`) for other
|
||||
workers; `dumpBookmarks` (the generator from mm-diigo's
|
||||
diigo-dump.mjs) is in `src/diigo.ts`
|
||||
|
||||
GET `/` is inert; dumps happen only on POST submits.
|
||||
- **The create flow, on one port**: `/_connect` renders webapp's own
|
||||
mint form (diigo-prefilled, cookie blank) — wrangler multi-config dev
|
||||
doesn't serve aux workers' HTTP, so `/_mint` relays the form opaquely
|
||||
into the WEBAPP binding (the cookie crosses; nothing on this side opens
|
||||
or logs it). Mint → `/_wire` (webapp id pre-filled, mode
|
||||
public|private) → the created page, where the user saves the ids.
|
||||
- **Two independent page caches** (`src/cache.ts`), one behavior: a
|
||||
page hit is served from storage with no diigo call; a miss asks the
|
||||
webapp instance in the cache's wiring and etches the page.
|
||||
- `DiigoCache` (binding `CACHE_DO`): every item. Addressed by its
|
||||
cache id.
|
||||
- `PublicCache` (binding `PUBLIC_DO`): shared items only — private
|
||||
rows are dropped *before* etching (`publicOnly`), so its storage
|
||||
holds nothing private. Same cookie webapp as the private cache.
|
||||
Public page n is diigo page n, filtered (may show fewer than 24).
|
||||
`/public/<username>` resolves through the `idFromName("index")`
|
||||
registry.
|
||||
- **Pages**: home's dump form (cache id + page number, POST) shows raw
|
||||
dump rows; `/public/<username>?page=n` shows an HTML list with a page
|
||||
form (JSON for `Accept: application/json`). Both say `cache: HIT` or
|
||||
`MISS`; every page read also logs `{event: "cache", kind, page,
|
||||
outcome}`.
|
||||
- **Publish later**: `POST /_publish` (cache id in the body) wires a
|
||||
public companion to a private-only cache.
|
||||
- The `DiigoBridge` entrypoint for other workers (its cache-id source is
|
||||
the goal-2 wiring TODO).
|
||||
|
||||
## State surface
|
||||
|
||||
- The instance id (`DIIGO_WEB_ID`): the one secret (bearer capability;
|
||||
possession is authority). Not logged.
|
||||
- The cookie: lives **only in the webapp DO's storage**, crosses the wire
|
||||
exactly once, at mint time. Nothing on disk holds it.
|
||||
- `/_connect`: state-aware — pre-mint it's the paste-cookie form;
|
||||
post-mint it health-checks the instance (count=1 probe) and offers
|
||||
re-mint only for stale cookies.
|
||||
Ids are **webkeys** ([the pattern](../docs/webkeys.md)): 256-bit bearer
|
||||
secrets that never ride URLs — not query strings, not path segments.
|
||||
Forms carry them in POST bodies; logs and pages show only a fingerprint
|
||||
(a prefix of the id's one-way hash).
|
||||
|
||||
- **The cache id**: the cache's read capability (bearer; not logged).
|
||||
Held by the user — saved to the password manager under a petname they
|
||||
choose (the username field; never sent to the server) — and passed
|
||||
per POST.
|
||||
- **The webapp instance id**: the diigo-read capability; lives in each
|
||||
cache DO's wiring, arrives from the mint step.
|
||||
- **The cookie**: lives **only in webapp's DO storage**; crosses the
|
||||
wire once, at mint time, as an opaque relay.
|
||||
- **The public cache**: no capability — `/public/<username>` is public
|
||||
by design (a guessable URL; hiding the link doesn't hide the page).
|
||||
|
||||
Stored state is meant to be durable: see the breaking-change rule in
|
||||
[capsite AGENTS.md](../AGENTS.md).
|
||||
|
||||
## Deployment concern: public pages spend your diigo session
|
||||
|
||||
Anyone who can reach `/public/<username>` can make the cache fetch from
|
||||
diigo through *your* cookie session: each request may spend up to
|
||||
`FETCH_BUDGET` (40) diigo fetches on uncached pages (goal 1d), paced at
|
||||
`MS_PER_CALL`. Cached pages cost nothing. Before exposing a deployment:
|
||||
|
||||
- rate-limit the public routes — on Cloudflare, the
|
||||
[Workers rate-limit binding](https://developers.cloudflare.com/workers/runtime-apis/bindings/rate-limit/)
|
||||
(keyed by client and/or username); on other workerd hosts, a rate
|
||||
limit in the proxy in front;
|
||||
- run the fill job (goal 1e) so most pages are cached before strangers
|
||||
arrive.
|
||||
|
||||
## RPC use (from another worker, same account)
|
||||
|
||||
```jsonc
|
||||
// wrangler.jsonc of the caller
|
||||
"services": [{ "binding": "DIIGO", "service": "diigo-dump", "entrypoint": "DiigoBridge" }]
|
||||
"services": [{ "binding": "DIIGO", "service": "diigo-cache", "entrypoint": "DiigoBridge" }]
|
||||
```
|
||||
|
||||
```ts
|
||||
const stub = await env.DIIGO.open();
|
||||
// promise pipelining: whole chain = one round trip; cookie never leaves the DO
|
||||
const first = JSON.parse(await stub.pathjoin("load_user_items")
|
||||
.query({ sort: "updated", count: 24, page_num: 1 }).get());
|
||||
// cache ids ride goal-2's wiring; see the capsite checklist
|
||||
const ndjson = await env.DIIGO.dumpNdjson(4, 24);
|
||||
```
|
||||
|
||||
## Test
|
||||
|
||||
pnpm test
|
||||
|
||||
Runs the real worker and both DOs in the vitest pool, with `WEBAPP`
|
||||
bound to [a fake webapp](tests/fake-webapp.js) serving diigo-shaped
|
||||
items (`wrangler.tests.jsonc`). vitest is borrowed from
|
||||
`../diigo-bookmarks/node_modules` (see `vitest.config.ts`).
|
||||
|
||||
## Run, deploy
|
||||
|
||||
[How to run / connect / deploy is maintained in the capsite checklist]
|
||||
(../capsite-checklist.md) (single source; this README keeps only
|
||||
worker-level facts).
|
||||
[How to run / connect / deploy is maintained in the capsite checklist](../capsite-checklist.md)
|
||||
(single source; this README keeps only worker-level facts).
|
||||
|
||||
Status: dev wiring verified (boots, pages answer); mint → dump through
|
||||
the DO not yet exercised end-to-end (checklist 1b-i). Earlier flat
|
||||
(non-DO) build passed manual test: 200, `total: 1674` (2026-10-06).
|
||||
Status: create, dump, publish, and public pages exercised against live
|
||||
diigo in dev (2026-10-07); not yet deployed.
|
||||
Reference in new issue
Block a user