dckc/captoys
1
0 Fork 0
Code Issues Pull Requests 4 Actions Packages Projects Releases Wiki Activity

potluck, diigo-cache: docs one line per paragraph; potluck deploy caveat

Browse Source 
Obsidian and GitHub render each newline as a line break, so the
hard-wrapped READMEs and CONTRIBUTINGs rendered ragged. Whitespace
only, except a note in potluck's CONTRIBUTING that other workerd
hosts are untested: they'd have to supply the D1 and R2 bindings.

Generated-by: Claude Opus 5.5
This commit is contained in:
dckc committed 2026-10-09 01:30:56 -05:00
1 parent a25bd699e1
commit c66aaea366
4 files changed
+53 -182
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

diigo-cache/CONTRIBUTING.md
+24 -94
View File
@@ -1,100 +1,37 @@
# Contributing to diigo-cache
diigo-cache is a Cloudflare Worker cache over [the webapp factory](../webapp/README.md):
session-cookie reads of diigo bookmarks, cached in Durable-Object SQLite
storage. Evolved from the diigo-dump worker (provenance:
[mm-diigo `projects/diigo-bak/`](https://github.com/dckc/mm-diigo)); the
static API-key path is dead (see [the capsite checklist](../capsite-checklist.md),
goal 2 preamble).
diigo-cache is a Cloudflare Worker cache over [the webapp factory](../webapp/README.md): session-cookie reads of diigo bookmarks, cached in Durable-Object SQLite storage. Evolved from the diigo-dump worker (provenance: [mm-diigo `projects/diigo-bak/`](https://github.com/dckc/mm-diigo)); the static API-key path is dead (see [the capsite checklist](../capsite-checklist.md), goal 2 preamble).
## What it holds
- **The create flow, on one port**: `/_connect` renders webapp's own
mint form (diigo-prefilled, cookie blank) — wrangler multi-config dev
doesn't serve aux workers' HTTP, so `/_mint` relays the form opaquely
into the WEBAPP binding (the cookie crosses; nothing on this side opens
or logs it). Mint → `/_wire` (webapp id pre-filled, mode
public|private) → the created page, where the user saves the ids.
- **Two independent page caches** (`src/cache.ts`), one behavior: a
page hit is served from storage with no diigo call; a miss asks the
webapp instance in the cache's wiring and etches the page.
- `DiigoCache` (binding `CACHE_DO`): every item. Addressed by its
cache id.
- `PublicCache` (binding `PUBLIC_DO`): shared items only — private
rows are dropped *before* etching (`publicOnly`), so its storage
holds nothing private. Same cookie webapp as the private cache.
Public page n is diigo page n, filtered (may show fewer than
`PAGE_SIZE`, 100: diigo's cap).
`/public/<username>` resolves through the `idFromName("index")`
registry, and reads through `viewer()`, a read-only facet: it shows
stored pages and fill progress, and can't fetch from diigo or write.
- **Storage**: SQL lives in `src/cache-read.sql` (selects only) and
`src/cache-write.sql`; `src/store.ts` turns them into a reader and a
writer facet, so the cache logic holds no SQL text.
- **Pages**: home's dump form (cache id + page number, POST) shows raw
dump rows; `/public/<username>?page=n` shows an HTML list with a page
form (JSON for `Accept: application/json`). Both say `cache: HIT` or
`MISS`; every page read also logs `{event: "cache", kind, page,
outcome}`.
- **Filters on the public page** (`src/query.ts`): `days`, `after`,
`before`, `in` (`2026`, `2026-03`, `2026-Q1`, `2026-W41`),
`by=updated|created`, and `q` (words, `"phrases"`, `#tag`, `-`
negation, gmail/Slack/GitHub date operators). UTC, half-open windows.
The same fields work as a dev-side spec (`toQuery({ days: 7 })`). A
filter queries the stored `items` table through `viewer()`, never
diigo: results are as fresh as the cache, and JSON says `complete`
when every diigo page is stored.
- **Publish later**: `POST /_publish` (cache id in the body) wires a
public companion to a private-only cache.
- **The webkey page and Cap'n Web**: `/cache/#<cache id>` is a skeleton
page, the same for every key; its script reads the key from the
fragment and trades it in-band for a capability over
[Cap'n Web](https://github.com/cloudflare/capnweb) at `/api`
(`CacheApi.cache(key)` → a `CacheCap` facet: fill, status, the public
companion; `src/api.ts`). The worker serves capnweb's browser build
itself at `/_capnweb.js`.
- **Fill job**: walks every diigo page into a cache in the background —
DO alarms, a few pages per step, progress stored in the DO
(`startFill`/`fillStatus`/`stopFill`), driven from the webkey page;
the public page shows progress. "Invalidate the last N days" there
drops the cached pages holding anything updated that recently
(`invalidateRecent`), so the next fill refetches them.
- The `DiigoBridge` entrypoint for other workers (its cache-id source is
the goal-2 wiring TODO).
- **The create flow, on one port**: `/_connect` renders webapp's own mint form (diigo-prefilled, cookie blank) — wrangler multi-config dev doesn't serve aux workers' HTTP, so `/_mint` relays the form opaquely into the WEBAPP binding (the cookie crosses; nothing on this side opens or logs it). Mint → `/_wire` (webapp id pre-filled, mode public|private) → the created page, where the user saves the ids.
- **Two independent page caches** (`src/cache.ts`), one behavior: a page hit is served from storage with no diigo call; a miss asks the webapp instance in the cache's wiring and etches the page.
- `DiigoCache` (binding `CACHE_DO`): every item. Addressed by its cache id.
- `PublicCache` (binding `PUBLIC_DO`): shared items only — private rows are dropped *before* etching (`publicOnly`), so its storage holds nothing private. Same cookie webapp as the private cache. Public page n is diigo page n, filtered (may show fewer than `PAGE_SIZE`, 100: diigo's cap). `/public/<username>` resolves through the `idFromName("index")` registry, and reads through `viewer()`, a read-only facet: it shows stored pages and fill progress, and can't fetch from diigo or write.
- **Storage**: SQL lives in `src/cache-read.sql` (selects only) and `src/cache-write.sql`; `src/store.ts` turns them into a reader and a writer facet, so the cache logic holds no SQL text.
- **Pages**: home's dump form (cache id + page number, POST) shows raw dump rows; `/public/<username>?page=n` shows an HTML list with a page form (JSON for `Accept: application/json`). Both say `cache: HIT` or `MISS`; every page read also logs `{event: "cache", kind, page, outcome}`.
- **Filters on the public page** (`src/query.ts`): `days`, `after`, `before`, `in` (`2026`, `2026-03`, `2026-Q1`, `2026-W41`), `by=updated|created`, and `q` (words, `"phrases"`, `#tag`, `-` negation, gmail/Slack/GitHub date operators). UTC, half-open windows. The same fields work as a dev-side spec (`toQuery({ days: 7 })`). A filter queries the stored `items` table through `viewer()`, never diigo: results are as fresh as the cache, and JSON says `complete` when every diigo page is stored.
- **Publish later**: `POST /_publish` (cache id in the body) wires a public companion to a private-only cache.
- **The webkey page and Cap'n Web**: `/cache/#<cache id>` is a skeleton page, the same for every key; its script reads the key from the fragment and trades it in-band for a capability over [Cap'n Web](https://github.com/cloudflare/capnweb) at `/api` (`CacheApi.cache(key)` → a `CacheCap` facet: fill, status, the public companion; `src/api.ts`). The worker serves capnweb's browser build itself at `/_capnweb.js`.
- **Fill job**: walks every diigo page into a cache in the background — DO alarms, a few pages per step, progress stored in the DO (`startFill`/`fillStatus`/`stopFill`), driven from the webkey page; the public page shows progress. "Invalidate the last N days" there drops the cached pages holding anything updated that recently (`invalidateRecent`), so the next fill refetches them.
- The `DiigoBridge` entrypoint for other workers (its cache-id source is the goal-2 wiring TODO).
## State surface
Ids are **webkeys** ([the pattern](../docs/webkeys.md)): 256-bit bearer
secrets that never ride a query string or path. The webkey page carries
the cache id in the URL fragment (never sent to the server) and passes
it in-band over Cap'n Web; forms carry ids in POST bodies; logs and pages
show only a fingerprint (a prefix of the id's one-way hash).
Ids are **webkeys** ([the pattern](../docs/webkeys.md)): 256-bit bearer secrets that never ride a query string or path. The webkey page carries the cache id in the URL fragment (never sent to the server) and passes it in-band over Cap'n Web; forms carry ids in POST bodies; logs and pages show only a fingerprint (a prefix of the id's one-way hash).
- **The cache id**: the cache's read capability (bearer; not logged).
Held by the user — saved to the password manager under a petname they
choose (the username field; never sent to the server) — and passed
per POST.
- **The webapp instance id**: the diigo-read capability; lives in each
cache DO's wiring, arrives from the mint step.
- **The cookie**: lives **only in webapp's DO storage**; crosses the
wire once, at mint time, as an opaque relay.
- **The public cache**: no capability — `/public/<username>` is public
by design (a guessable URL; hiding the link doesn't hide the page).
- **The cache id**: the cache's read capability (bearer; not logged). Held by the user — saved to the password manager under a petname they choose (the username field; never sent to the server) — and passed per POST.
- **The webapp instance id**: the diigo-read capability; lives in each cache DO's wiring, arrives from the mint step.
- **The cookie**: lives **only in webapp's DO storage**; crosses the wire once, at mint time, as an opaque relay.
- **The public cache**: no capability — `/public/<username>` is public by design (a guessable URL; hiding the link doesn't hide the page).
Stored state is meant to be durable: see the breaking-change rule in
[capsite AGENTS.md](../AGENTS.md).
Stored state is meant to be durable: see the breaking-change rule in [capsite AGENTS.md](../AGENTS.md).
## Deployment concern: rate-limit `/api`
Public pages only read storage, so strangers can't make the cache spend
your diigo session. `/api` is keyed (a cache id is needed), but Cap'n
Web pipelining makes calls cheap to send. Before exposing a deployment:
Public pages only read storage, so strangers can't make the cache spend your diigo session. `/api` is keyed (a cache id is needed), but Cap'n Web pipelining makes calls cheap to send. Before exposing a deployment:
- rate-limit `/api` (and `/public/*`, which reads storage) — on
Cloudflare, the
[Workers rate-limit binding](https://developers.cloudflare.com/workers/runtime-apis/bindings/rate-limit/)
(keyed by client and/or username); on other workerd hosts, a rate
limit in the proxy in front;
- rate-limit `/api` (and `/public/*`, which reads storage) — on Cloudflare, the [Workers rate-limit binding](https://developers.cloudflare.com/workers/runtime-apis/bindings/rate-limit/) (keyed by client and/or username); on other workerd hosts, a rate limit in the proxy in front;
- run the fill job (goal 1e): public pages show only what's stored.
## RPC use (from another worker, same account)
@@ -115,14 +52,11 @@ const ndjson = await env.DIIGO.dumpNdjson(4); // 4 pages of PAGE_SIZE
pnpm lint # fails if those types are stale; tsc over src/ and tests/
pnpm test # vitest
Tests run the real worker and both DOs in the vitest pool, with
`WEBAPP` bound to [a fake webapp](tests/fake-webapp.js) serving
diigo-shaped items (`wrangler.tests.jsonc`).
Tests run the real worker and both DOs in the vitest pool, with `WEBAPP` bound to [a fake webapp](tests/fake-webapp.js) serving diigo-shaped items (`wrangler.tests.jsonc`).
## Run
[The README](README.md) gets a cache going in dev. The same create flow
from a terminal:
[The README](README.md) gets a cache going in dev. The same create flow from a terminal:
wid=$(curl -s -X POST http://localhost:8787/_mint --data-urlencode \
'url=https://www.diigo.com/interact_api/' --data-urlencode \
@@ -131,12 +65,8 @@ from a terminal:
curl -s -X POST http://localhost:8787/_wire -H 'content-type: application/json' \
-d "{\"webappId\":\"<from $wid>\",\"mode\":\"private\"}"
Check a filled cache against a diigo RSS export:
`node scripts/compare-export.ts <cache.sqlite> <export.xml> <out-dir>`.
Check a filled cache against a diigo RSS export: `node scripts/compare-export.ts <cache.sqlite> <export.xml> <out-dir>`.
## Deploy
Not deployed yet; the order matters (goal 5 in
[the capsite checklist](../capsite-checklist.md)): webapp first
(`cd ../webapp && npx wrangler deploy`), then diigo-cache, after
rate-limiting `/api` (above).
Not deployed yet; the order matters (goal 5 in [the capsite checklist](../capsite-checklist.md)): webapp first (`cd ../webapp && npx wrangler deploy`), then diigo-cache, after rate-limiting `/api` (above).
diigo-cache/README.md
+7 -19
View File
@@ -1,9 +1,6 @@
# diigo-cache
Your diigo bookmarks, cached in a [workerd](https://github.com/cloudflare/workerd)
worker, with a public page you can filter by week:
`/public/<you>?in=2026-W41`. Your diigo session cookie
is stored in one Durable Object and nowhere else.
Your diigo bookmarks, cached in a [workerd](https://github.com/cloudflare/workerd) worker, with a public page you can filter by week: `/public/<you>?in=2026-W41`. Your diigo session cookie is stored in one Durable Object and nowhere else.
## Getting started
@@ -14,25 +11,16 @@ pnpm dev # → http://localhost:8787/
Then, in the browser:
1. **mint the diigo session at webapp**: paste the `Cookie` header from
diigo.com (devtools, any logged-in request).
2. **wire** it, mode **public**. Save the cache id it shows (the
password-manager button does this).
1. **mint the diigo session at webapp**: paste the `Cookie` header from diigo.com (devtools, any logged-in request).
2. **wire** it, mode **public**. Save the cache id it shows (the password-manager button does this).
3. **open this cache** → **fill**, and watch it walk your bookmarks.
4. Visit `/public/<your diigo username>?days=7`.
## Bookmarks you hold, a page anyone can read
- **Filter by date and search:** `?days=7`, `?in=2026-W41` (ISO week,
UTC), `?after=2026-01-01&before=2026-02-01`, `?by=created`, and
`?q=` with words, `"phrases"`, `#tags`, `-negation`, and gmail-style
`newer_than:7d`. Add `Accept: application/json` for JSON.
- **Private stays private:** the public cache drops private bookmarks
before storing anything, so it never holds them.
- **Ids are capabilities:** the cache id is a bearer key you keep in
your password manager. It rides URL fragments and POST bodies, never a
path or query string, so it stays out of logs and history.
- **Cheap to read:** pages come from Durable Object SQLite storage, and
a fill job in the background fetches each diigo page once.
- **Filter by date and search:** `?days=7`, `?in=2026-W41` (ISO week, UTC), `?after=2026-01-01&before=2026-02-01`, `?by=created`, and `?q=` with words, `"phrases"`, `#tags`, `-negation`, and gmail-style `newer_than:7d`. Add `Accept: application/json` for JSON.
- **Private stays private:** the public cache drops private bookmarks before storing anything, so it never holds them.
- **Ids are capabilities:** the cache id is a bearer key you keep in your password manager. It rides URL fragments and POST bodies, never a path or query string, so it stays out of logs and history.
- **Cheap to read:** pages come from Durable Object SQLite storage, and a fill job in the background fetches each diigo page once.
Design, tests, and deploying: [CONTRIBUTING](CONTRIBUTING.md).
potluck/CONTRIBUTING.md
+17 -55
View File
@@ -1,44 +1,17 @@
# Contributing to potluck
potluck started as EmDash's `cloudflare:blog` template
(`pnpm create emdash@latest`); it's part of
[the capsite goal tree](../capsite-checklist.md), where it is the site
that publishes the weekly roundup.
potluck started as EmDash's `cloudflare:blog` template (`pnpm create emdash@latest`); it's part of [the capsite goal tree](../capsite-checklist.md), where it is the site that publishes the weekly roundup.
## How it's put together
- **Issues** are the `issues` collection
([seed/issues.json](seed/issues.json)): `title`, `week` (an ISO week,
`2026-W40`), `dek` (one-line summary), `body` (Portable Text). The
slug is the week, lowercased: `/issues/2026-w40`.
[issues-sketch.md](issues-sketch.md) is the dry run behind the three
sample issues: real public bookmarks, editorial lines marked *draft*.
- **Pages:** home and `/issues` list issues
([src/pages/issues/](src/pages/issues/)); `/tag/<tag>` lists issues
by diigo tag; search and `/rss.xml` cover issues. The `posts`
collection and its pages are the template's, still in place.
- **Microformats:** an issue list is an `h-feed` of `h-entry`s; an
issue is an `h-entry` (`p-name`, `dt-published`, `p-summary`,
`e-content`). In issue bodies,
[CiteLink.astro](src/components/CiteLink.astro) marks off-site links
as `h-cite` and `/tag/<tag>` links as rel-tag `p-category`.
- **Look:** plain HTML, after kmjn.org's
[Plain HTML with a few bureaucratic gestures](https://www.kmjn.org/notes/modern_plain_html.html):
system fonts (no webfonts), one ~800px column, links in the text
color with a dotted underline. Restyle in
[src/styles/theme.css](src/styles/theme.css), which overrides the
defaults in `tokens.css` (unlayered, so it always wins); don't edit
`tokens.css` or `Base.astro` for visual changes. Colors use
`light-dark(<light>, <dark>)`.
- **Issues** are the `issues` collection ([seed/issues.json](seed/issues.json)): `title`, `week` (an ISO week, `2026-W40`), `dek` (one-line summary), `body` (Portable Text). The slug is the week, lowercased: `/issues/2026-w40`. [issues-sketch.md](issues-sketch.md) is the dry run behind the three sample issues: real public bookmarks, editorial lines marked *draft*.
- **Pages:** home and `/issues` list issues ([src/pages/issues/](src/pages/issues/)); `/tag/<tag>` lists issues by diigo tag; search and `/rss.xml` cover issues. The `posts` collection and its pages are the template's, still in place.
- **Microformats:** an issue list is an `h-feed` of `h-entry`s; an issue is an `h-entry` (`p-name`, `dt-published`, `p-summary`, `e-content`). In issue bodies, [CiteLink.astro](src/components/CiteLink.astro) marks off-site links as `h-cite` and `/tag/<tag>` links as rel-tag `p-category`.
- **Look:** plain HTML, after kmjn.org's [Plain HTML with a few bureaucratic gestures](https://www.kmjn.org/notes/modern_plain_html.html): system fonts (no webfonts), one ~800px column, links in the text color with a dotted underline. Restyle in [src/styles/theme.css](src/styles/theme.css), which overrides the defaults in `tokens.css` (unlayered, so it always wins); don't edit `tokens.css` or `Base.astro` for visual changes. Colors use `light-dark(<light>, <dark>)`.
## Seeds
[seed/seed.json](seed/seed.json) (site settings, menus, widgets, the
template's collections) is applied on the first request to an empty
database. [seed/issues.json](seed/issues.json) is separate:
`pnpm seed:issues` applies it to the local D1 database, which exists
after the dev server's first request. Run it with the dev server
stopped.
[seed/seed.json](seed/seed.json) (site settings, menus, widgets, the template's collections) is applied on the first request to an empty database. [seed/issues.json](seed/issues.json) is separate: `pnpm seed:issues` applies it to the local D1 database, which exists after the dev server's first request. Run it with the dev server stopped.
## Checks
@@ -49,22 +22,14 @@ pnpm build
## EmDash rules
- All content pages are server-rendered (`output: "server"`); no
`getStaticPaths()` for CMS content.
- Image fields are objects (`{ src, alt }`), not strings; render with
`<Image image={...} />` from `"emdash/ui"`.
- `entry.id` is the slug (for URLs); `entry.data.id` is the database
ULID (for API calls like `getEntryTerms`).
- When Astro's cache is enabled, pass query hints to
`Astro.cache.set(cacheHint)`.
- Taxonomy names in queries match the seed's `"name"` exactly (`"tag"`,
not `"tags"`).
- `npx emdash types` regenerates `emdash-env.d.ts` from a running site
(the dev server also regenerates it on start).
- All content pages are server-rendered (`output: "server"`); no `getStaticPaths()` for CMS content.
- Image fields are objects (`{ src, alt }`), not strings; render with `<Image image={...} />` from `"emdash/ui"`.
- `entry.id` is the slug (for URLs); `entry.data.id` is the database ULID (for API calls like `getEntryTerms`).
- When Astro's cache is enabled, pass query hints to `Astro.cache.set(cacheHint)`.
- Taxonomy names in queries match the seed's `"name"` exactly (`"tag"`, not `"tags"`).
- `npx emdash types` regenerates `emdash-env.d.ts` from a running site (the dev server also regenerates it on start).
The EmDash docs are an MCP server at `https://docs.emdashcms.com/mcp`
(wired up in `.mcp.json`, `.cursor/mcp.json`, `.vscode/mcp.json`).
Check them rather than recalling APIs from memory.
The EmDash docs are an MCP server at `https://docs.emdashcms.com/mcp` (wired up in `.mcp.json`, `.cursor/mcp.json`, `.vscode/mcp.json`). Check them rather than recalling APIs from memory.
## Deploying
@@ -73,13 +38,10 @@ pnpm wrangler login
pnpm deploy
```
The first deploy provisions the D1 database and R2 bucket named in
[wrangler.jsonc](wrangler.jsonc). See
[Deploy to Cloudflare](https://docs.emdashcms.com/deployment/cloudflare/).
The first deploy provisions the D1 database and R2 bucket named in [wrangler.jsonc](wrangler.jsonc). See [Deploy to Cloudflare](https://docs.emdashcms.com/deployment/cloudflare/).
Other workerd hosts are untested: the site uses D1 and R2 through `@emdash-cms/cloudflare` and the Cloudflare adapter (locally, miniflare stands in for both), so a host has to supply those two bindings.
## For agents
`AGENTS.md` is a symlink to this file. capsite's
[working rules](../AGENTS.md) apply here too (the dev server is the
user's to start and stop). EmDash skills are in `.agents/skills/`:
`building-emdash-site` (start here), `creating-plugins`, `emdash-cli`.
`AGENTS.md` is a symlink to this file. capsite's [working rules](../AGENTS.md) apply here too (the dev server is the user's to start and stop). EmDash skills are in `.agents/skills/`: `building-emdash-site` (start here), `creating-plugins`, `emdash-cli`.
potluck/README.md
+5 -14
View File
@@ -1,7 +1,6 @@
# potluck
A weekly zine of what you bookmarked elsewhere, published on your own
site. Each issue is one week of links with a few words around them.
A weekly zine of what you bookmarked elsewhere, published on your own site. Each issue is one week of links with a few words around them.
## Getting started
@@ -14,18 +13,10 @@ pnpm dev # → http://localhost:4321/issues
## Bookmarks in, a zine out
- **Your web, your site:** the bookmarks come from your own diigo
account, so the web you read becomes something you publish
([PESOS](https://indieweb.org/PESOS)). For now issues are put
together by hand. Pulling a week from diigo-cache is next.
- **Your web, your site:** the bookmarks come from your own diigo account, so the web you read becomes something you publish ([PESOS](https://indieweb.org/PESOS)). For now issues are put together by hand. Pulling a week from diigo-cache is next.
- **A week per issue:** `/issues/2026-w40`, by ISO week, UTC.
- **Microformats:** feed readers and IndieWeb tools see each issue as an
[h-entry](https://microformats.org/wiki/h-entry) and each bookmark as
an [h-cite](https://microformats.org/wiki/h-cite).
- **Microformats:** feed readers and IndieWeb tools see each issue as an [h-entry](https://microformats.org/wiki/h-entry) and each bookmark as an [h-cite](https://microformats.org/wiki/h-cite).
- **Plain HTML** that's easy to read: system fonts, one column.
- **A real editor:** it's an [EmDash](https://docs.emdashcms.com/) site,
so you write issues in its admin UI. It runs in
[workerd](https://github.com/cloudflare/workerd), on your machine or a host.
- **A real editor:** it's an [EmDash](https://docs.emdashcms.com/) site, so you write issues in its admin UI. It runs in [workerd](https://github.com/cloudflare/workerd), on your machine or a host.
How it's put together, checks, and deploying:
[CONTRIBUTING](CONTRIBUTING.md).
How it's put together, checks, and deploying: [CONTRIBUTING](CONTRIBUTING.md).