Compare commits

..
14 Commits
Author SHA1 Message Date
dckc 1d8a220bbc capsite: checklist: 2-iv, the weekly draft works in dev
Generated-by: Claude Opus 5.5
2026-10-09 02:12:41 -05:00
dckc 68132ff4b3 diigo-week: a dashboard card drafts last week in one click
The card shows last week's issue with a link to it, or a button to
draft it. Drafting a week that already has an issue (draft or
published) is refused, with a link to that issue, rather than making
a duplicate.

Generated-by: Claude Opus 5.5
2026-10-09 02:12:41 -05:00
dckc 9c09e8c5a4 capsite: checklist: 4b, consider scruffier tags
Generated-by: Claude Opus 5.5
2026-10-09 02:12:41 -05:00
dckc 94a059af25 diigo-week: draft a potluck issue from a week of diigo bookmarks
An EmDash plugin (native, content:write only) with one admin page:
a diigo username and an ISO week. It reads diigo-cache's public page
(?in=<week>&by=created) as microformats, follows rel=next, and
creates a draft in issues shaped like seed/issues.json.

Transport is a DIIGO_CACHE service binding rather than ctx.http.fetch:
EmDash's SSRF guard refuses localhost, and the binding names one
worker. Logic lives in src/week.ts with the fetcher passed in; tests
read fixtures in diigo-cache's markup. The checklist's goal 2 now
names potluck; DiigoBridge waits.

Generated-by: Claude Opus 5.5
2026-10-09 02:12:41 -05:00
dckc be796d9d0d diigo-cache: the public page is an h-feed of h-entry bookmarks
Microformats are the public page's API for other sites (potluck
first): each bookmark is an h-entry with u-bookmark-of, p-name,
p-summary, a p-category per tag (linked to a tag search),
dt-published (bookmarked) and dt-updated (last edited). Prev/next
links carry rel=prev/next, so a reader can follow a filter's pages.
Tests parse the page with microformats-parser to pin the contract.

Generated-by: Claude Opus 5.5
2026-10-09 01:49:40 -05:00
dckc c66aaea366 potluck, diigo-cache: docs one line per paragraph; potluck deploy caveat
Obsidian and GitHub render each newline as a line break, so the
hard-wrapped READMEs and CONTRIBUTINGs rendered ragged. Whitespace
only, except a note in potluck's CONTRIBUTING that other workerd
hosts are untested: they'd have to supply the D1 and R2 bindings.

Generated-by: Claude Opus 5.5
2026-10-09 01:30:56 -05:00
dckc a25bd699e1 potluck: README: workerd, not a vendor
Generated-by: Claude Opus 5.5
2026-10-09 01:29:31 -05:00
dckc ea3e256bb8 diigo-cache: README: workerd, not a vendor
Generated-by: Claude Opus 5.5
2026-10-09 01:28:50 -05:00
dckc 0af7a3ea6c diigo-cache: README sells and gets to TADA; CONTRIBUTING updated
New README: two sentences, the dev run through connect, fill, and a
filtered public page, then features. CONTRIBUTING gains the 1d
filters and its own run and deploy sections instead of pointing at
the checklist; links to the old README now point at CONTRIBUTING.

Generated-by: Claude Opus 5.5
2026-10-09 01:27:45 -05:00
dckc 9719fd0aa8 diigo-cache: README becomes CONTRIBUTING; AGENTS.md links to it
Generated-by: Claude Opus 5.5
2026-10-09 01:27:12 -05:00
dckc bce2424cc0 potluck: README: a heading that says something
Generated-by: Claude Opus 5.5
2026-10-09 01:25:47 -05:00
dckc 7f1e19fe85 potluck: README: two sentences, then getting started, then features
Generated-by: Claude Opus 5.5
2026-10-09 01:25:13 -05:00
dckc 8a812d76e3 potluck: README sells and gets to TADA; CONTRIBUTING has the rest
The README was EmDash's template blurb. Now it says what potluck is
and its first block runs from a fresh clone to /issues, using a new
`pnpm seed:issues` script that applies seed/issues.json to the local
D1 database. CONTRIBUTING covers how it's put together, seeds, checks,
the EmDash rules, and deploying; the template's stale notes (Inter,
three-column articles) are gone. AGENTS.md is now a symlink to it.

Generated-by: Claude Opus 5.5
2026-10-09 01:24:10 -05:00
dckc 002f08b456 diigo-cache: date windows and search on the public page (1d)
/public/<username> takes days, after, before, in (year, month,
quarter, ISO week), by=updated|created, and q (words, "phrases",
#tags, -negation, gmail/Slack/GitHub date operators). JSON answers
echo the parsed filter; a malformed one is a 400.

Filters query the stored items in sqlite through the read-only
viewer instead of walking diigo pages: the public page keeps no
fetch authority, and a filled cache already holds every item.
Results say `complete` when every diigo page is stored; freshness
is 1f's head refresh. So the fetch budget and the O(log pages)
walk test go, and the fake deep account shrinks to 300.

Generated-by: Claude Opus 5.5
2026-10-09 01:17:03 -05:00
32 changed files with 7402 additions and 424 deletions

No files matched your search

+19 -34
View File
@@ -23,7 +23,7 @@ narrative.
goal 5. goal 5.
- [x] **1c. diigo-dump evolves into a bookmark cache** - [x] **1c. diigo-dump evolves into a bookmark cache**
([diigo-cache/README.md](diigo-cache/README.md); design in the ([diigo-cache/CONTRIBUTING.md](diigo-cache/CONTRIBUTING.md); design in the
JSDoc of `src/cache.ts`, `src/index.ts`, `src/pm-save.ts`): JSDoc of `src/cache.ts`, `src/index.ts`, `src/pm-save.ts`):
- [x] 1c-i. Two independent page caches, private and public - [x] 1c-i. Two independent page caches, private and public
(`PagedCache`, `DiigoCache`, `PublicCache`). (`PagedCache`, `DiigoCache`, `PublicCache`).
@@ -42,20 +42,16 @@ narrative.
2026-10-07, via `scripts/compare-export.ts`. Getting there 2026-10-07, via `scripts/compare-export.ts`. Getting there
found diigo's `page_num` is 0-based (schema v2 migration). found diigo's `page_num` is 0-based (schema v2 migration).
- [ ] **1d. Date windows and search on the public page**, toward the - [x] **1d. Date windows and search on the public page**, toward the
weekly PESOS roundup of [#23](https://github.com/dckc/madmode-blog/issues/23) weekly PESOS roundup of [#23](https://github.com/dckc/madmode-blog/issues/23)
(a week's permalink: `in=2026-W41`). One filter, two spellings: (a week's permalink: `in=2026-W41`). One filter, two spellings:
devs pass `{ days: 7, q: "#weekly" }`, URLs the same fields; devs pass `{ days: 7, q: "#weekly" }`, URLs the same fields;
`q` takes gmail/Slack/GitHub operators. UTC, ISO weeks, half-open `q` takes gmail/Slack/GitHub operators. UTC, ISO weeks, half-open
windows; a fetch budget per request. Design: the API in windows (`diigo-cache/src/query.ts`; `tests/query.test.ts`,
`diigo-cache/src/query.ts` and the skipped suites in `tests/caches.test.ts` "public page filters"). Filters read
`tests/query.test.ts` and `tests/caches.test.ts` ("public page stored `items` only (the public page never fetches): results are
filters"); done when they're unskipped and green. as fresh as the cache, and say `complete` when every diigo page
Reconcile with 1f first: `query.ts` plans to filter while walking is stored. Keeping them fresh is 1f. 2026-10-09.
diigo's pages (binary search by `updated`), but after 1c-vi a
filled cache holds every item, so 1d can query `items` in sqlite
and spend diigo fetches only on a head refresh (1f). Choose
before implementing; the skipped suites may need adjusting.
- [x] **1e. Fill job: a button that grabs every page** into a cache - [x] **1e. Fill job: a button that grabs every page** into a cache
from the webkey page `/cache/#<key>`, which trades the key for a from the webkey page `/cache/#<key>`, which trades the key for a
@@ -93,31 +89,20 @@ narrative.
[the issues sketch](potluck/issues-sketch.md))? If the policy [the issues sketch](potluck/issues-sketch.md))? If the policy
changes, stripping stored rows is a durable-state migration. changes, stripping stored rows is a durable-state migration.
- [ ] **2. madmode reads the public cache**: public data, so no - [ ] **2. potluck reads the public cache**: public data, so no capability, no cookie, no secret on potluck's side. The [diigo-week](diigo-week/README.md) plugin reads `GET /public/dckc-madmode?in=<week>&by=created` as microformats (an `h-feed` of `h-entry` bookmarks). Working in dev 2026-10-09; 2-iii postponed.
capability, no cookie, no plugin sync job, no cron on madmode's - [x] 2-i. Transport: the `DIIGO_CACHE` service binding in potluck's `wrangler.jsonc`, in dev too: the plugin's `ctx.http.fetch` refuses localhost (EmDash's SSRF guard), and a binding names one worker, not a host.
side. madmode fetches `GET /public/dckc-madmode` with - [x] 2-ii. The page's microformats are the contract: diigo-cache's tests parse it with microformats-parser; diigo-week's tests read fixtures in the same markup.
`Accept: application/json` (plus 1d's `in=`/`days=`/`q=`). - [ ] 2-iii. Decide `DiigoBridge`'s fate (postponed: nothing on this path uses it; it matters only for a caller that reads private items).
- [ ] 2-i. Transport: a service binding to diigo-cache when both - [x] 2-iv. Run it in dev: diigo-cache and potluck both up; the dashboard card → a draft issue of 22 bookmarks for 2026-W40 (2026-10-09). The service binding resolves across the two dev servers.
are deployed (no public hop, so no rate-limit exposure); - [ ] ~~**3. `/bookmarks` renders real data**~~: superseded by potluck's issues (the madmode starter is gone).
in dev, a base URL (`http://localhost:8787`). - [ ] **4. The weekly draft**: decided (2026-10-09): a draft issue you edit before publishing, made by diigo-week. The slug comes from the title (`week-2026-w40`).
- [ ] 2-ii. The JSON shape of the public page is a contract
between two workers: pin it in a diigo-cache test (and - [ ] **4b. Consider scruffier tags.** EmDash tags are taxonomy terms (rows that must exist before use); diigo-week auto-creates them, so drafting feels scruffy, but the model is neat. Alternative: the inline `/tag/<tag>` links in issue bodies *are* the tags (matching the microformats `p-category` reading), and potluck's `/tag/<tag>` page finds issues by scanning bodies. Cost: no EmDash tag field or index; a scan over a few dozen issues a year is cheap. Open: does the admin's tag field let you type a new tag inline?
madmode's reader tests against the same fixture).
- [ ] 2-iii. Decide `DiigoBridge`'s fate: madmode doesn't need it
(public reads need no cache id), so its "cache-id source"
TODO (`src/index.ts`, README "RPC use") is either dropped
with the entrypoint or kept for a private-reading caller.
- [ ] **3. `/bookmarks` renders real data** — titles, tags, dates at
http://localhost:4321/bookmarks, newest first.
- [ ] **4. The weekly page**: `/bookmarks/2026-W41` → the cache's
`in=2026-W41` (needs 1d and 1f). Decide (user's call): a raw
week list, or a draft post the user edits before publishing
(the PESOS practice is editorial; a tag like `#weekly` in `q`
is one cheap form of discretion).
- [ ] **5. Deployed end-to-end** (order matters): - [ ] **5. Deployed end-to-end** (order matters):
- [ ] 5-i. webapp: `cd webapp && npx wrangler deploy` - [ ] 5-i. webapp: `cd webapp && npx wrangler deploy`
- [ ] 5-ii. diigo-cache: rate-limit `/api` (and `/public/*`) first - [ ] 5-ii. diigo-cache: rate-limit `/api` (and `/public/*`) first
(diigo-cache README, "Deployment concern"); deploy; create (diigo-cache CONTRIBUTING, "Deployment concern"); deploy; create
against prod (ids → password manager; no worker secret); against prod (ids → password manager; no worker secret);
run the fill job. run the fill job.
- [ ] 5-iii. madmode: deploy with the service binding; the - [ ] 5-iii. madmode: deploy with the service binding; the
@@ -239,5 +224,5 @@ to every goal in this tree.
Oct 6 2026) Oct 6 2026)
- [webapp/README.md](webapp/README.md) — HTTP + RPC surfaces of the - [webapp/README.md](webapp/README.md) — HTTP + RPC surfaces of the
factory worker; open TODO on the mint-form placement factory worker; open TODO on the mint-form placement
- [diigo-cache/README.md](diigo-cache/README.md) — the cache worker: - [diigo-cache/CONTRIBUTING.md](diigo-cache/CONTRIBUTING.md) — the cache worker:
create flow, the two DOs, run/deploy details create flow, the two DOs, run/deploy details
+1
View File
@@ -0,0 +1 @@
CONTRIBUTING.md
+72
View File
@@ -0,0 +1,72 @@
# Contributing to diigo-cache
diigo-cache is a Cloudflare Worker cache over [the webapp factory](../webapp/README.md): session-cookie reads of diigo bookmarks, cached in Durable-Object SQLite storage. Evolved from the diigo-dump worker (provenance: [mm-diigo `projects/diigo-bak/`](https://github.com/dckc/mm-diigo)); the static API-key path is dead (see [the capsite checklist](../capsite-checklist.md), goal 2 preamble).
## What it holds
- **The create flow, on one port**: `/_connect` renders webapp's own mint form (diigo-prefilled, cookie blank) — wrangler multi-config dev doesn't serve aux workers' HTTP, so `/_mint` relays the form opaquely into the WEBAPP binding (the cookie crosses; nothing on this side opens or logs it). Mint → `/_wire` (webapp id pre-filled, mode public|private) → the created page, where the user saves the ids.
- **Two independent page caches** (`src/cache.ts`), one behavior: a page hit is served from storage with no diigo call; a miss asks the webapp instance in the cache's wiring and etches the page.
- `DiigoCache` (binding `CACHE_DO`): every item. Addressed by its cache id.
- `PublicCache` (binding `PUBLIC_DO`): shared items only — private rows are dropped *before* etching (`publicOnly`), so its storage holds nothing private. Same cookie webapp as the private cache. Public page n is diigo page n, filtered (may show fewer than `PAGE_SIZE`, 100: diigo's cap). `/public/<username>` resolves through the `idFromName("index")` registry, and reads through `viewer()`, a read-only facet: it shows stored pages and fill progress, and can't fetch from diigo or write.
- **Storage**: SQL lives in `src/cache-read.sql` (selects only) and `src/cache-write.sql`; `src/store.ts` turns them into a reader and a writer facet, so the cache logic holds no SQL text.
- **Pages**: home's dump form (cache id + page number, POST) shows raw dump rows; `/public/<username>?page=n` shows an HTML list with a page form (JSON for `Accept: application/json`). Both say `cache: HIT` or `MISS`; every page read also logs `{event: "cache", kind, page, outcome}`.
- **Filters on the public page** (`src/query.ts`): `days`, `after`, `before`, `in` (`2026`, `2026-03`, `2026-Q1`, `2026-W41`), `by=updated|created`, and `q` (words, `"phrases"`, `#tag`, `-` negation, gmail/Slack/GitHub date operators). UTC, half-open windows. The same fields work as a dev-side spec (`toQuery({ days: 7 })`). A filter queries the stored `items` table through `viewer()`, never diigo: results are as fresh as the cache, and JSON says `complete` when every diigo page is stored.
- **Publish later**: `POST /_publish` (cache id in the body) wires a public companion to a private-only cache.
- **The webkey page and Cap'n Web**: `/cache/#<cache id>` is a skeleton page, the same for every key; its script reads the key from the fragment and trades it in-band for a capability over [Cap'n Web](https://github.com/cloudflare/capnweb) at `/api` (`CacheApi.cache(key)` → a `CacheCap` facet: fill, status, the public companion; `src/api.ts`). The worker serves capnweb's browser build itself at `/_capnweb.js`.
- **Fill job**: walks every diigo page into a cache in the background — DO alarms, a few pages per step, progress stored in the DO (`startFill`/`fillStatus`/`stopFill`), driven from the webkey page; the public page shows progress. "Invalidate the last N days" there drops the cached pages holding anything updated that recently (`invalidateRecent`), so the next fill refetches them.
- The `DiigoBridge` entrypoint for other workers (its cache-id source is the goal-2 wiring TODO).
## State surface
Ids are **webkeys** ([the pattern](../docs/webkeys.md)): 256-bit bearer secrets that never ride a query string or path. The webkey page carries the cache id in the URL fragment (never sent to the server) and passes it in-band over Cap'n Web; forms carry ids in POST bodies; logs and pages show only a fingerprint (a prefix of the id's one-way hash).
- **The cache id**: the cache's read capability (bearer; not logged). Held by the user — saved to the password manager under a petname they choose (the username field; never sent to the server) — and passed per POST.
- **The webapp instance id**: the diigo-read capability; lives in each cache DO's wiring, arrives from the mint step.
- **The cookie**: lives **only in webapp's DO storage**; crosses the wire once, at mint time, as an opaque relay.
- **The public cache**: no capability — `/public/<username>` is public by design (a guessable URL; hiding the link doesn't hide the page).
Stored state is meant to be durable: see the breaking-change rule in [capsite AGENTS.md](../AGENTS.md).
## Deployment concern: rate-limit `/api`
Public pages only read storage, so strangers can't make the cache spend your diigo session. `/api` is keyed (a cache id is needed), but Cap'n Web pipelining makes calls cheap to send. Before exposing a deployment:
- rate-limit `/api` (and `/public/*`, which reads storage) — on Cloudflare, the [Workers rate-limit binding](https://developers.cloudflare.com/workers/runtime-apis/bindings/rate-limit/) (keyed by client and/or username); on other workerd hosts, a rate limit in the proxy in front;
- run the fill job (goal 1e): public pages show only what's stored.
## RPC use (from another worker, same account)
```jsonc
// wrangler.jsonc of the caller
"services": [{ "binding": "DIIGO", "service": "diigo-cache", "entrypoint": "DiigoBridge" }]
```
```ts
// cache ids ride goal-2's wiring; see the capsite checklist
const ndjson = await env.DIIGO.dumpNdjson(4); // 4 pages of PAGE_SIZE
```
## Test
pnpm types # regenerate worker-configuration.d.ts (after wrangler.jsonc changes)
pnpm lint # fails if those types are stale; tsc over src/ and tests/
pnpm test # vitest
Tests run the real worker and both DOs in the vitest pool, with `WEBAPP` bound to [a fake webapp](tests/fake-webapp.js) serving diigo-shaped items (`wrangler.tests.jsonc`).
## Run
[The README](README.md) gets a cache going in dev. The same create flow from a terminal:
wid=$(curl -s -X POST http://localhost:8787/_mint --data-urlencode \
'url=https://www.diigo.com/interact_api/' --data-urlencode \
'headers={"user-agent":"Mozilla/5.0","x-requested-with":"XMLHttpRequest","cookie":"<paste>"}' \
--data-urlencode 'msPerCall=500' -D - -o /dev/null | grep -i x-webapp-id)
curl -s -X POST http://localhost:8787/_wire -H 'content-type: application/json' \
-d "{\"webappId\":\"<from $wid>\",\"mode\":\"private\"}"
Check a filled cache against a diigo RSS export: `node scripts/compare-export.ts <cache.sqlite> <export.xml> <out-dir>`.
## Deploy
Not deployed yet; the order matters (goal 5 in [the capsite checklist](../capsite-checklist.md)): webapp first (`cd ../webapp && npx wrangler deploy`), then diigo-cache, after rate-limiting `/api` (above).
+16 -110
View File
@@ -1,120 +1,26 @@
# diigo-cache # diigo-cache
Cloudflare Worker cache over [the webapp factory](../webapp/README.md): Your diigo bookmarks, cached in a [workerd](https://github.com/cloudflare/workerd) worker, with a public page you can filter by week: `/public/<you>?in=2026-W41`. Your diigo session cookie is stored in one Durable Object and nowhere else.
session-cookie reads of diigo bookmarks, cached in Durable-Object SQLite
storage. Evolved from the diigo-dump worker (provenance:
[mm-diigo `projects/diigo-bak/`](https://github.com/dckc/mm-diigo)); the
static API-key path is dead (see [the capsite checklist](../capsite-checklist.md),
goal 2 preamble).
## What it holds ## Getting started
- **The create flow, on one port**: `/_connect` renders webapp's own ```sh
mint form (diigo-prefilled, cookie blank) — wrangler multi-config dev pnpm install
doesn't serve aux workers' HTTP, so `/_mint` relays the form opaquely pnpm dev # → http://localhost:8787/
into the WEBAPP binding (the cookie crosses; nothing on this side opens
or logs it). Mint → `/_wire` (webapp id pre-filled, mode
public|private) → the created page, where the user saves the ids.
- **Two independent page caches** (`src/cache.ts`), one behavior: a
page hit is served from storage with no diigo call; a miss asks the
webapp instance in the cache's wiring and etches the page.
- `DiigoCache` (binding `CACHE_DO`): every item. Addressed by its
cache id.
- `PublicCache` (binding `PUBLIC_DO`): shared items only — private
rows are dropped *before* etching (`publicOnly`), so its storage
holds nothing private. Same cookie webapp as the private cache.
Public page n is diigo page n, filtered (may show fewer than
`PAGE_SIZE`, 100: diigo's cap).
`/public/<username>` resolves through the `idFromName("index")`
registry, and reads through `viewer()`, a read-only facet: it shows
stored pages and fill progress, and can't fetch from diigo or write.
- **Storage**: SQL lives in `src/cache-read.sql` (selects only) and
`src/cache-write.sql`; `src/store.ts` turns them into a reader and a
writer facet, so the cache logic holds no SQL text.
- **Pages**: home's dump form (cache id + page number, POST) shows raw
dump rows; `/public/<username>?page=n` shows an HTML list with a page
form (JSON for `Accept: application/json`). Both say `cache: HIT` or
`MISS`; every page read also logs `{event: "cache", kind, page,
outcome}`.
- **Publish later**: `POST /_publish` (cache id in the body) wires a
public companion to a private-only cache.
- **The webkey page and Cap'n Web**: `/cache/#<cache id>` is a skeleton
page, the same for every key; its script reads the key from the
fragment and trades it in-band for a capability over
[Cap'n Web](https://github.com/cloudflare/capnweb) at `/api`
(`CacheApi.cache(key)` → a `CacheCap` facet: fill, status, the public
companion; `src/api.ts`). The worker serves capnweb's browser build
itself at `/_capnweb.js`.
- **Fill job**: walks every diigo page into a cache in the background —
DO alarms, a few pages per step, progress stored in the DO
(`startFill`/`fillStatus`/`stopFill`), driven from the webkey page;
the public page shows progress. "Invalidate the last N days" there
drops the cached pages holding anything updated that recently
(`invalidateRecent`), so the next fill refetches them.
- The `DiigoBridge` entrypoint for other workers (its cache-id source is
the goal-2 wiring TODO).
## State surface
Ids are **webkeys** ([the pattern](../docs/webkeys.md)): 256-bit bearer
secrets that never ride a query string or path. The webkey page carries
the cache id in the URL fragment (never sent to the server) and passes
it in-band over Cap'n Web; forms carry ids in POST bodies; logs and pages
show only a fingerprint (a prefix of the id's one-way hash).
- **The cache id**: the cache's read capability (bearer; not logged).
Held by the user — saved to the password manager under a petname they
choose (the username field; never sent to the server) — and passed
per POST.
- **The webapp instance id**: the diigo-read capability; lives in each
cache DO's wiring, arrives from the mint step.
- **The cookie**: lives **only in webapp's DO storage**; crosses the
wire once, at mint time, as an opaque relay.
- **The public cache**: no capability — `/public/<username>` is public
by design (a guessable URL; hiding the link doesn't hide the page).
Stored state is meant to be durable: see the breaking-change rule in
[capsite AGENTS.md](../AGENTS.md).
## Deployment concern: rate-limit `/api`
Public pages only read storage, so strangers can't make the cache spend
your diigo session. `/api` is keyed (a cache id is needed), but Cap'n
Web pipelining makes calls cheap to send. Before exposing a deployment:
- rate-limit `/api` (and `/public/*`, which reads storage) — on
Cloudflare, the
[Workers rate-limit binding](https://developers.cloudflare.com/workers/runtime-apis/bindings/rate-limit/)
(keyed by client and/or username); on other workerd hosts, a rate
limit in the proxy in front;
- run the fill job (goal 1e): public pages show only what's stored.
## RPC use (from another worker, same account)
```jsonc
// wrangler.jsonc of the caller
"services": [{ "binding": "DIIGO", "service": "diigo-cache", "entrypoint": "DiigoBridge" }]
``` ```
```ts Then, in the browser:
// cache ids ride goal-2's wiring; see the capsite checklist
const ndjson = await env.DIIGO.dumpNdjson(4); // 4 pages of PAGE_SIZE
```
## Test 1. **mint the diigo session at webapp**: paste the `Cookie` header from diigo.com (devtools, any logged-in request).
2. **wire** it, mode **public**. Save the cache id it shows (the password-manager button does this).
3. **open this cache** → **fill**, and watch it walk your bookmarks.
4. Visit `/public/<your diigo username>?days=7`.
pnpm types # regenerate worker-configuration.d.ts (after wrangler.jsonc changes) ## Bookmarks you hold, a page anyone can read
pnpm lint # fails if those types are stale; tsc over src/ and tests/
pnpm test # vitest
Tests run the real worker and both DOs in the vitest pool, with - **Filter by date and search:** `?days=7`, `?in=2026-W41` (ISO week, UTC), `?after=2026-01-01&before=2026-02-01`, `?by=created`, and `?q=` with words, `"phrases"`, `#tags`, `-negation`, and gmail-style `newer_than:7d`. Add `Accept: application/json` for JSON.
`WEBAPP` bound to [a fake webapp](tests/fake-webapp.js) serving - **Private stays private:** the public cache drops private bookmarks before storing anything, so it never holds them.
diigo-shaped items (`wrangler.tests.jsonc`). - **Ids are capabilities:** the cache id is a bearer key you keep in your password manager. It rides URL fragments and POST bodies, never a path or query string, so it stays out of logs and history.
- **Cheap to read:** pages come from Durable Object SQLite storage, and a fill job in the background fetches each diigo page once.
## Run, deploy Design, tests, and deploying: [CONTRIBUTING](CONTRIBUTING.md).
[How to run / connect / deploy is maintained in the capsite checklist](../capsite-checklist.md)
(single source; this README keeps only worker-level facts).
Status: create, dump, publish, and public pages exercised against live
diigo in dev (2026-10-07); not yet deployed.
+1
View File
@@ -11,6 +11,7 @@
"devDependencies": { "devDependencies": {
"@cloudflare/vitest-plugin": "1.3.6", "@cloudflare/vitest-plugin": "1.3.6",
"@types/node": "^22", "@types/node": "^22",
"microformats-parser": "^2.0.6",
"typescript": "^5.9.0", "typescript": "^5.9.0",
"vitest": "4.1.11", "vitest": "4.1.11",
"wrangler": "^4.148.0" "wrangler": "^4.148.0"
+24
View File
@@ -18,6 +18,9 @@ importers:
'@types/node': '@types/node':
specifier: ^22 specifier: ^22
version: 22.20.5 version: 22.20.5
microformats-parser:
specifier: ^2.0.6
version: 2.0.6
typescript: typescript:
specifier: ^5.9.0 specifier: ^5.9.0
version: 5.9.3 version: 5.9.3
@@ -613,6 +616,10 @@ packages:
resolution: {integrity: sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ==} resolution: {integrity: sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ==}
engines: {node: '>=8'} engines: {node: '>=8'}
entities@6.0.1:
resolution: {integrity: sha512-aN97NXWF6AWBTahfVOIrB/NShkzi5H7F9r1s9mD3cDj4Ko5f2qhhVoYMibXF7GlLveb/D2ioWay8lxI97Ven3g==}
engines: {node: '>=0.12'}
error-stack-parser-es@1.0.5: error-stack-parser-es@1.0.5:
resolution: {integrity: sha512-5qucVt2XcuGMcEGgWI7i+yZpmpByQ8J1lHhcL7PwqCwu9FPP3VUXzT4ltHe5i2z9dePwEHcDVOAfSnHsOlCXRA==} resolution: {integrity: sha512-5qucVt2XcuGMcEGgWI7i+yZpmpByQ8J1lHhcL7PwqCwu9FPP3VUXzT4ltHe5i2z9dePwEHcDVOAfSnHsOlCXRA==}
@@ -722,6 +729,10 @@ packages:
magic-string@0.30.21: magic-string@0.30.21:
resolution: {integrity: sha512-vd2F4YUyEXKGcLHoq+TEyCjxueSeHnFxyyjNp80yg0XV4vUhnDer/lvvlqM/arB5bXQN5K2/3oinyCRyx8T2CQ==} resolution: {integrity: sha512-vd2F4YUyEXKGcLHoq+TEyCjxueSeHnFxyyjNp80yg0XV4vUhnDer/lvvlqM/arB5bXQN5K2/3oinyCRyx8T2CQ==}
microformats-parser@2.0.6:
resolution: {integrity: sha512-VTnLImF17yd+ZfBH40sddHMJp5Rrker4ge+oPjgnBQ3jluQO0YmsNVA4UcEfHRBV8zrikBSMkAzAub+nTRX85g==}
engines: {node: '>=18'}
miniflare@5.20261001.0-alpha: miniflare@5.20261001.0-alpha:
resolution: {integrity: sha512-GaimS5mSIOMyvd16ga+e1/QkI8cmp3z35QPHFex0DktqNQf2RVZQwMPQXdyoUreUiFP/0Gpe2MoQInTyMAD5xA==} resolution: {integrity: sha512-GaimS5mSIOMyvd16ga+e1/QkI8cmp3z35QPHFex0DktqNQf2RVZQwMPQXdyoUreUiFP/0Gpe2MoQInTyMAD5xA==}
engines: {node: '>=22.0.0'} engines: {node: '>=22.0.0'}
@@ -739,6 +750,9 @@ packages:
resolution: {integrity: sha512-XrsrhT5sybtKI6wakr2SPOlGZWWYbUXZ7a0jT8/QOeAPau+1X/bSegNe5YR75oJmEZQbKningirmGOEJCIk61Q==} resolution: {integrity: sha512-XrsrhT5sybtKI6wakr2SPOlGZWWYbUXZ7a0jT8/QOeAPau+1X/bSegNe5YR75oJmEZQbKningirmGOEJCIk61Q==}
engines: {node: '>=12.20.0'} engines: {node: '>=12.20.0'}
parse5@7.3.0:
resolution: {integrity: sha512-IInvU7fabl34qmi9gY8XOVxhYyMyuH2xUNpb2q8/Y+7552KlejkRvqvD19nMoUW/uQGGbqNpA6Tufu5FL5BZgw==}
path-to-regexp@6.3.0: path-to-regexp@6.3.0:
resolution: {integrity: sha512-Yhpw4T9C6hPpgPeA28us07OJeqZ5EzQTkbfwuhsUg0c237RomFoETJgmp2sa3F/41gfLE6G5cqcYwznmeEeOlQ==} resolution: {integrity: sha512-Yhpw4T9C6hPpgPeA28us07OJeqZ5EzQTkbfwuhsUg0c237RomFoETJgmp2sa3F/41gfLE6G5cqcYwznmeEeOlQ==}
@@ -1366,6 +1380,8 @@ snapshots:
detect-libc@2.1.2: {} detect-libc@2.1.2: {}
entities@6.0.1: {}
error-stack-parser-es@1.0.5: {} error-stack-parser-es@1.0.5: {}
es-module-lexer@2.3.2: {} es-module-lexer@2.3.2: {}
@@ -1467,6 +1483,10 @@ snapshots:
dependencies: dependencies:
'@jridgewell/sourcemap-codec': 1.6.0 '@jridgewell/sourcemap-codec': 1.6.0
microformats-parser@2.0.6:
dependencies:
parse5: 7.3.0
miniflare@5.20261001.0-alpha(@types/node@22.20.5): miniflare@5.20261001.0-alpha(@types/node@22.20.5):
dependencies: dependencies:
'@cspotcode/source-map-support': 0.8.1 '@cspotcode/source-map-support': 0.8.1
@@ -1497,6 +1517,10 @@ snapshots:
obug@2.2.1: {} obug@2.2.1: {}
parse5@7.3.0:
dependencies:
entities: 6.0.1
path-to-regexp@6.3.0: {} path-to-regexp@6.3.0: {}
pathe@2.0.3: {} pathe@2.0.3: {}
+11
View File
@@ -19,3 +19,14 @@ where cast(items.updated as integer) >= ?;
-- pagesThrough: -- pagesThrough:
select count(*) as n from pages where page_num <= ?; select count(*) as n from pages where page_num <= ?;
-- itemsByUpdated:
select json from items
where cast(updated as integer) >= ? and cast(updated as integer) < ?
order by cast(updated as integer) desc, item_id;
-- itemsByCreated:
select json from items
where cast(json_extract(json, '$.created_at') as integer) >= ?
and cast(json_extract(json, '$.created_at') as integer) < ?
order by cast(json_extract(json, '$.created_at') as integer) desc, item_id;
+33 -1
View File
@@ -6,6 +6,8 @@
import { DurableObject, RpcTarget } from "cloudflare:workers"; import { DurableObject, RpcTarget } from "cloudflare:workers";
import type { PagedGet } from "./diigo.js"; import type { PagedGet } from "./diigo.js";
import { publicOnly, toRows } from "./items.js"; import { publicOnly, toRows } from "./items.js";
import { matches } from "./query.js";
import type { Query } from "./query.js";
import { makeReader, makeWriter, SCHEMA_VERSION } from "./store.js"; import { makeReader, makeWriter, SCHEMA_VERSION } from "./store.js";
import type { Reader, Writer } from "./store.js"; import type { Reader, Writer } from "./store.js";
import type { Item, ItemRow } from "./items.js"; import type { Item, ItemRow } from "./items.js";
@@ -398,7 +400,18 @@ abstract class PagedCache extends DurableObject<CacheEnv> {
} }
} }
/** what anyone may see of a cache: stored pages and fill progress */ /** one page of a filter's matches over stored items */
export type QueryPage = {
/** the matches on this page, raw dump rows (JSON strings) */
items: string[];
/** matches in all */
matched: number;
/** true when every diigo page is stored, so no match can be missing */
complete: boolean;
};
/** what anyone may see of a cache: stored pages, stored items by filter,
* and fill progress */
export class PageViewer extends RpcTarget { export class PageViewer extends RpcTarget {
#reader: Reader; #reader: Reader;
#fillStatus: () => Promise<FillStatus | undefined>; #fillStatus: () => Promise<FillStatus | undefined>;
@@ -415,6 +428,25 @@ export class PageViewer extends RpcTarget {
return stored && { ...stored, cached: true }; return stored && { ...stored, cached: true };
} }
/**
* page `pageNum` (`count` per page) of the stored items matching
* `query`, newest first by its date. The window narrows in SQL; terms
* are checked per item. Reads storage only: items not yet stored can't
* match, which `complete` reports.
*/
query(query: Query, pageNum: number, count: number): QueryPage {
const from = query.after ? Math.ceil(query.after.getTime() / 1000) : Number.MIN_SAFE_INTEGER;
const to = query.before ? Math.ceil(query.before.getTime() / 1000) : Number.MAX_SAFE_INTEGER;
const hits = this.#reader.itemsIn(query.by, from, to).filter((json) => matches(JSON.parse(json) as Item, query));
const total = this.#reader.knownTotal();
const pages = total === null ? null : Math.max(1, Math.ceil(total / PAGE_SIZE));
return {
items: hits.slice((pageNum - 1) * count, pageNum * count),
matched: hits.length,
complete: pages !== null && this.#reader.pagesThrough(pages) >= pages,
};
}
/** diigo's total, as last stored, or null if nothing is stored */ /** diigo's total, as last stored, or null if nothing is stored */
knownTotal(): number | null { knownTotal(): number | null {
return this.#reader.knownTotal(); return this.#reader.knownTotal();
+152 -34
View File
@@ -10,9 +10,11 @@ import { CAPNWEB_BROWSER_JS } from "./capnweb-browser.js";
import { CACHE_PAGE_HTML } from "./cache-page.js"; import { CACHE_PAGE_HTML } from "./cache-page.js";
import { DiigoCacheLogic, PublicCacheLogic } from "./cache.js"; import { DiigoCacheLogic, PublicCacheLogic } from "./cache.js";
import { PAGE_SIZE } from "./cache.js"; import { PAGE_SIZE } from "./cache.js";
import type { CacheEnv, Clock, FillStatus } from "./cache.js"; import type { CacheEnv, Clock, FillStatus, QueryPage } from "./cache.js";
import type { Item } from "./cache.js"; import type { Item } from "./cache.js";
import { updatedISO, usernameOf } from "./items.js"; import { usernameOf } from "./items.js";
import { dateOf, isEmpty, parseQuery } from "./query.js";
import type { Query } from "./query.js";
import { toWebkey, idFromWebkey, isWebkey, shashKey } from "./webkeys.js"; import { toWebkey, idFromWebkey, isWebkey, shashKey } from "./webkeys.js";
import { pmSave } from "./pm-save.js"; import { pmSave } from "./pm-save.js";
@@ -443,10 +445,16 @@ async function dump(request: Request, env: Env): Promise<Response> {
const PUBLIC_PER_PAGE = PAGE_SIZE; const PUBLIC_PER_PAGE = PAGE_SIZE;
/** the public page's day presets */
const DAY_PRESETS = [7, 30, 90, 365];
/** /**
* the public page for a username: stored pages only, through the cache's * the public page for a username: stored pages only, through the cache's
* read-only `viewer()` (it can't fetch or write). HTML with a page-number * read-only `viewer()` (it can't fetch or write). HTML with a filter form
* form, or JSON for `Accept: application/json`. `?page=` in the URL is * and a page-number form, or JSON for `Accept: application/json`. A
* filter (goal 1d: `days`, `after`, `before`, `in`, `by`, `q`; see
* {@link parseQuery}) lists the stored items that match, `page` counting
* matches; with none, `page` is diigo's page as stored. Query params are
* fine here: nothing on this page is secret. * fine here: nothing on this page is secret.
*/ */
async function publicPage(request: Request, env: Env, username: string): Promise<Response> { async function publicPage(request: Request, env: Env, username: string): Promise<Response> {
@@ -456,14 +464,33 @@ async function publicPage(request: Request, env: Env, username: string): Promise
const ns = env.PUBLIC_DO; const ns = env.PUBLIC_DO;
const cache = ns.get(ns.idFromString(idFromWebkey(publicId))); const cache = ns.get(ns.idFromString(idFromWebkey(publicId)));
const raw = new URL(request.url).searchParams.get("page"); const params = new URL(request.url).searchParams;
const accept = request.headers.get("accept") ?? "";
const wantsJson = accept.includes("application/json") && !accept.includes("text/html");
const here = `/public/${encodeURIComponent(username)}`;
const raw = params.get("page");
const pageN = raw === null ? 1 : Number.parseInt(raw, 10); const pageN = raw === null ? 1 : Number.parseInt(raw, 10);
const valid = !Number.isNaN(pageN) && pageN >= 1 && pageN <= 10000; const valid = !Number.isNaN(pageN) && pageN >= 1 && pageN <= 10000;
const query = parseQuery(params, new Date());
if ("error" in query) {
console.log({ event: "notice", page: "public", text: "filter refused", error: query.error });
if (wantsJson) return Response.json({ error: query.error }, { status: 400 });
const body = publicFrame(username, `${filterForm(here, params)}
<p><em>${esc(query.error)}</em></p>`);
return new Response(body, { status: 400, headers: { "content-type": "text/html;charset=utf-8" } });
}
const viewer = await cache.viewer(); const viewer = await cache.viewer();
let page, total, fill; let page, fill, found;
let total: number | null = null;
try { try {
page = await viewer.page(valid ? pageN : 1, PUBLIC_PER_PAGE); if (isEmpty(query)) {
total = page ? page.total : await viewer.knownTotal(); page = await viewer.page(valid ? pageN : 1, PUBLIC_PER_PAGE);
total = page ? page.total : await viewer.knownTotal();
} else {
found = await viewer.query(query, valid ? pageN : 1, PUBLIC_PER_PAGE);
}
fill = await viewer.fillStatus(); fill = await viewer.fillStatus();
} catch (err) { } catch (err) {
console.error({ event: "public-page", page: pageN, error: String(err) }); console.error({ event: "public-page", page: pageN, error: String(err) });
@@ -471,13 +498,14 @@ async function publicPage(request: Request, env: Env, username: string): Promise
} finally { } finally {
viewer[Symbol.dispose](); viewer[Symbol.dispose]();
} }
if (found) return filteredPage(username, here, params, query, found, valid ? pageN : 1, wantsJson, fill);
// total and pages count all of diigo's items: public page n is diigo // total and pages count all of diigo's items: public page n is diigo
// page n with the private rows dropped // page n with the private rows dropped
const pages = total === null ? 1 : Math.max(1, Math.ceil(total / PUBLIC_PER_PAGE)); const pages = total === null ? 1 : Math.max(1, Math.ceil(total / PUBLIC_PER_PAGE));
const items = page ? page.items.map((j) => JSON.parse(j) as Item) : []; const items = page ? page.items.map((j) => JSON.parse(j) as Item) : [];
const accept = request.headers.get("accept") ?? ""; if (wantsJson) {
if (accept.includes("application/json") && !accept.includes("text/html")) {
return Response.json({ username, page: pageN, pages, total, stored: page !== null, etchedAt: page?.etchedAt ?? null, items }); return Response.json({ username, page: pageN, pages, total, stored: page !== null, etchedAt: page?.etchedAt ?? null, items });
} }
@@ -486,42 +514,132 @@ async function publicPage(request: Request, env: Env, username: string): Promise
console.log({ event: "notice", page: "public", text: "page out of range", pages }); console.log({ event: "notice", page: "public", text: "page out of range", pages });
notice = `<p><em>no page ${esc(String(raw))}: pages run 1–${pages}</em></p>`; notice = `<p><em>no page ${esc(String(raw))}: pages run 1–${pages}</em></p>`;
} }
const here = `/public/${encodeURIComponent(username)}`;
const nav = (n: number, label: string) => const nav = (n: number, label: string) =>
n >= 1 && n <= pages ? `<a href="${here}?page=${n}">${label}</a>` : `<span>${label}</span>`; n >= 1 && n <= pages ? `<a rel="${n < pageN ? "prev" : "next"}" href="${here}?page=${n}">${label}</a>` : `<span>${label}</span>`;
const list = notice const list = notice ? "" : itemList(items, "updated", here);
? "" const body = publicFrame(
: `<ol> username,
${items `<p>${pages} page${pages === 1 ? "" : "s"} (diigo's, ${PUBLIC_PER_PAGE} bookmarks each; private ones are left out, so a page may show fewer).</p>
.map((it) => {
const url = typeof it.url === "string" ? it.url : "";
const title = typeof it.title === "string" && it.title ? it.title : url;
const desc = typeof it.description === "string" && it.description ? `<br><small>${esc(it.description)}</small>` : "";
const href = /^https?:\/\//i.test(url) ? url : "#";
const when = updatedISO(it);
const date = when ? `<time datetime="${when}">${when}</time>` : "(no date)";
return `<li>${date} <a href="${esc(href)}">${esc(title)}</a>${desc}</li>`;
})
.join("\n")}
</ol>`;
const body = `<!doctype html>
<html><head><meta charset="utf-8"><title>${esc(username)}'s shared bookmarks</title></head>
<body>
<h1>${esc(username)}'s shared bookmarks</h1>
<p>${pages} page${pages === 1 ? "" : "s"} (diigo's, ${PUBLIC_PER_PAGE} bookmarks each; private ones are left out, so a page may show fewer).</p>
<p>${page ? `<strong>stored ${esc(page.etchedAt ?? "")}</strong> — page ${valid ? pageN : 1}, ${items.length} shared` : `<strong>not fetched yet</strong> — page ${valid ? pageN : 1}; the cache's owner fills it`}</p> <p>${page ? `<strong>stored ${esc(page.etchedAt ?? "")}</strong> — page ${valid ? pageN : 1}, ${items.length} shared` : `<strong>not fetched yet</strong> — page ${valid ? pageN : 1}; the cache's owner fills it`}</p>
${fill ? fillLine(fill) : ""} ${fill ? fillLine(fill) : ""}
${filterForm(here, params)}
<form method="get" action="${here}"> <form method="get" action="${here}">
<p>${nav(pageN - 1, "← prev")} <p>${nav(pageN - 1, "← prev")}
<label>page <input type="number" name="page" min="1" value="${Number.isNaN(pageN) ? 1 : pageN}"></label> <label>page <input type="number" name="page" min="1" value="${Number.isNaN(pageN) ? 1 : pageN}"></label>
<button type="submit">go</button> <button type="submit">go</button>
${nav(pageN + 1, "next →")}</p> ${nav(pageN + 1, "next →")}</p>
</form> </form>
${notice}${list} ${notice}${list}`,
</body></html>`; );
return new Response(body, { headers: { "content-type": "text/html;charset=utf-8" } }); return new Response(body, { headers: { "content-type": "text/html;charset=utf-8" } });
} }
/** a filter's matches, as the public page's HTML or JSON */
function filteredPage(
username: string,
here: string,
params: URLSearchParams,
query: Query,
found: QueryPage,
pageN: number,
wantsJson: boolean,
fill: FillStatus | undefined,
): Response {
const pages = Math.max(1, Math.ceil(found.matched / PUBLIC_PER_PAGE));
const items = found.items.map((j) => JSON.parse(j) as Item);
if (wantsJson) {
const { matched, complete } = found;
return Response.json({ username, filter: query, page: pageN, pages, matched, complete, items });
}
// links keep the filter: the request's own non-empty params, in order
const kept = [...params].filter(([k, v]) => k !== "page" && v !== "");
const href = (n: number) => `${here}?${new URLSearchParams([...kept, ["page", String(n)]])}`;
const nav = (n: number, label: string) =>
n >= 1 && n <= pages ? `<a rel="${n < pageN ? "prev" : "next"}" href="${esc(href(n))}">${label}</a>` : `<span>${label}</span>`;
const status = found.matched
? `${found.matched} match${found.matched === 1 ? "" : "es"}, page ${pageN} of ${pages}, newest ${query.by} first`
: "no matches";
const caveat = found.complete ? "" : " — not every diigo page is stored yet, so some matches may be missing";
const body = publicFrame(
username,
`<p><strong>${esc(status)}</strong>${caveat}</p>
${fill ? fillLine(fill) : ""}
${filterForm(here, params)}
<p>${nav(pageN - 1, "← prev")} ${nav(pageN + 1, "next →")} <a href="${here}">all, by diigo page</a></p>
${itemList(items, query.by, here)}`,
);
return new Response(body, { headers: { "content-type": "text/html;charset=utf-8" } });
}
/** the public page's frame: title and heading around `content` */
function publicFrame(username: string, content: string): string {
return `<!doctype html>
<html><head><meta charset="utf-8"><title>${esc(username)}'s shared bookmarks</title></head>
<body>
<h1>${esc(username)}'s shared bookmarks</h1>
${content}
</body></html>`;
}
/** the filter form, echoing the request's filter */
function filterForm(here: string, params: URLSearchParams): string {
const field = (name: string) => esc(params.get(name) ?? "");
const days = params.get("days") ?? "";
const presets = DAY_PRESETS.map(String);
if (days && !presets.includes(days)) presets.push(days);
const option = (value: string, label: string, current: string) =>
`<option value="${esc(value)}"${value === current ? " selected" : ""}>${esc(label)}</option>`;
return `<form method="get" action="${here}">
<p><label>last <select name="days">${option("", "any", days)}${presets.map((n) => option(n, `${n} days`, days)).join("")}</select></label>
<label>or after <input type="date" name="after" value="${field("after")}"></label>
<label>before <input type="date" name="before" value="${field("before")}"></label>
<label>or in <input name="in" size="9" placeholder="2026-W41" value="${field("in")}"></label>
<label>by <select name="by">${option("updated", "updated", params.get("by") ?? "")}${option("created", "created", params.get("by") ?? "")}</select></label></p>
<p><label>search <input type="search" name="q" size="40" value="${field("q")}"></label>
<button type="submit">filter</button></p>
<p><small>Dates and periods (2026, 2026-03, 2026-Q1, 2026-W41) are UTC. Search takes
words, "phrases", #tags, -negation, and after:, before:, newer_than:7d, during:2026-Q1, created:2026.</small></p>
</form>`;
}
/**
* bookmarks as a list, each led by its `by` date, marked up as
* microformats: an `h-feed` of `h-entry` bookmarks (`u-bookmark-of`,
* `p-name`, `p-summary`, `p-category` per tag, `dt-published` when
* bookmarked, `dt-updated` when last edited). This markup is the public
* page's API for other sites (tests/caches.test.ts pins it); tag links
* search this page (`here`) by tag.
*
* @see https://microformats.org/wiki/h-entry
* @see https://indieweb.org/bookmark
*/
function itemList(items: Item[], by: Query["by"], here: string): string {
const time = (it: Item, which: Query["by"], hidden: boolean) => {
const secs = dateOf(it, which);
if (secs === undefined) return hidden ? "" : "(no date)";
const when = new Date(secs * 1000).toISOString();
const cls = which === "created" ? "dt-published" : "dt-updated";
return hidden ? `<time class="${cls}" datetime="${when}" hidden></time>` : `<time class="${cls}" datetime="${when}">${when}</time>`;
};
return `<ol class="h-feed">
${items
.map((it) => {
const url = typeof it.url === "string" ? it.url : "";
const title = typeof it.title === "string" && it.title ? it.title : url;
const desc = typeof it.description === "string" && it.description ? `<br><small class="p-summary">${esc(it.description)}</small>` : "";
const href = /^https?:\/\//i.test(url) ? url : "#";
const tags = (typeof it.tags === "string" ? it.tags.split(",") : [])
.map((t) => t.trim())
.filter(Boolean)
.map((t) => ` <a class="p-category" href="${esc(`${here}?q=${encodeURIComponent(`#${t}`)}`)}">${esc(t)}</a>`)
.join("");
const other = by === "created" ? "updated" : "created";
return `<li class="h-entry">${time(it, by, false)} <a class="u-bookmark-of p-name" href="${esc(href)}">${esc(title)}</a>${tags}${desc}${time(it, other, true)}</li>`;
})
.join("\n")}
</ol>`;
}
/** home page: the connect link (webapp's cookie form, diigo-prefilled) + /** home page: the connect link (webapp's cookie form, diigo-prefilled) +
* the wire step for an already-minted id + the dump form. `home` renders * the wire step for an already-minted id + the dump form. `home` renders
* the error path too, so it takes env for the connect link. Notices (the * the error path too, so it takes env for the connect link. Notices (the
+188 -19
View File
@@ -1,18 +1,10 @@
/** /**
* @file the public page's filters: date windows and search (goal 1d, **not * @file the public page's filters: date windows and search over a cache's
* yet implemented**; the design is the skipped specs in tests/query.test.ts). * stored items (goal 1d; tests/query.test.ts).
*/ */
import type { Item } from "./items.js"; import type { Item } from "./items.js";
/**
* diigo fetches one request may spend (cache hits are free); under the
* Workers free plan's 50 external subrequests per invocation
*
* @see https://developers.cloudflare.com/workers/platform/limits/
*/
export const FETCH_BUDGET = 40;
/** /**
* a filter as devs write it (`{ days: 7, q: "#weekly" }`); URL params map * a filter as devs write it (`{ days: 7, q: "#weekly" }`); URL params map
* onto these fields one-to-one (strings, with "" meaning absent). Dates * onto these fields one-to-one (strings, with "" meaning absent). Dates
@@ -56,6 +48,76 @@ export type Query = {
/** a refused filter: the message is safe to show and log */ /** a refused filter: the message is safe to show and log */
export type QueryError = { error: string }; export type QueryError = { error: string };
const DAY_MS = 24 * 60 * 60 * 1000;
/** thrown inside {@link toQuery}, returned as a {@link QueryError} */
class Refused extends Error {}
/** `yyyy-mm-dd` or `yyyy/mm/dd` as 00:00Z of that UTC day */
function parseDay(text: string, what: string): Date {
const m = /^(\d{4})([-/])(\d{2})\2(\d{2})$/.exec(text);
const d = m && new Date(Date.UTC(Number(m[1]), Number(m[3]) - 1, Number(m[4])));
if (!m || !d || d.getUTCMonth() !== Number(m[3]) - 1 || d.getUTCDate() !== Number(m[4])) {
throw new Refused(`${what}: expected a date as yyyy-mm-dd`);
}
return d;
}
/** Monday 00:00Z of ISO week 1 of `year` (the week holding January 4) */
function isoWeekOne(year: number): number {
const jan4 = Date.UTC(year, 0, 4);
const weekday = new Date(jan4).getUTCDay() || 7;
return jan4 - (weekday - 1) * DAY_MS;
}
/** a named period as its half-open [after, before) window */
function parsePeriod(text: string, what: string): [Date, Date] {
const refuse = () => new Refused(`${what}: expected a period as 2025, 2025-03, 2025-Q1, or 2025-W01`);
let m;
if ((m = /^(\d{4})$/.exec(text))) {
const y = Number(m[1]);
return [new Date(Date.UTC(y, 0, 1)), new Date(Date.UTC(y + 1, 0, 1))];
}
if ((m = /^(\d{4})-(\d{2})$/.exec(text))) {
const [y, mo] = [Number(m[1]), Number(m[2])];
if (mo < 1 || mo > 12) throw refuse();
return [new Date(Date.UTC(y, mo - 1, 1)), new Date(Date.UTC(y, mo, 1))];
}
if ((m = /^(\d{4})-Q([1-4])$/.exec(text))) {
const [y, q] = [Number(m[1]), Number(m[2])];
return [new Date(Date.UTC(y, 3 * (q - 1), 1)), new Date(Date.UTC(y, 3 * q, 1))];
}
if ((m = /^(\d{4})-W(\d{2})$/.exec(text))) {
const [y, w] = [Number(m[1]), Number(m[2])];
const weeks = (isoWeekOne(y + 1) - isoWeekOne(y)) / (7 * DAY_MS);
if (w < 1 || w > weeks) throw refuse();
const monday = isoWeekOne(y) + (w - 1) * 7 * DAY_MS;
return [new Date(monday), new Date(monday + 7 * DAY_MS)];
}
throw refuse();
}
/** gmail's `newer_than:` amount (`7d`, `2w`, `1m`, `1y`) counted back from
* `now`; months and years are calendar steps */
function parseNewerThan(text: string, now: Date): Date {
const m = /^(\d{1,4})([dwmy])$/.exec(text);
if (!m || Number(m[1]) < 1) throw new Refused("newer_than: expected an amount like 7d, 2w, 1m, or 1y");
const n = Number(m[1]);
const d = new Date(now.getTime());
if (m[2] === "d") d.setTime(now.getTime() - n * DAY_MS);
if (m[2] === "w") d.setTime(now.getTime() - 7 * n * DAY_MS);
if (m[2] === "m") d.setUTCMonth(d.getUTCMonth() - n);
if (m[2] === "y") d.setUTCFullYear(d.getUTCFullYear() - n);
return d;
}
/** `days` as a whole number from 1 to 3650 */
function parseDays(days: number | string): number {
const n = typeof days === "number" ? days : /^\d+$/.test(days) ? Number(days) : NaN;
if (!Number.isInteger(n) || n < 1 || n > 3650) throw new Refused("days: expected a whole number from 1 to 3650");
return n;
}
/** /**
* resolve a filter: fields and `q` operators combine; a bound given twice * resolve a filter: fields and `q` operators combine; a bound given twice
* (by fields, operators, or both) is refused, as is an empty window. * (by fields, operators, or both) is refused, as is an empty window.
@@ -75,21 +137,128 @@ export type QueryError = { error: string };
* not read ambiently, so tests pin it) * not read ambiently, so tests pin it)
*/ */
export function toQuery(spec: FilterSpec, now: Date): Query | QueryError { export function toQuery(spec: FilterSpec, now: Date): Query | QueryError {
void spec; try {
void now; return resolve(spec, now);
throw new Error("toQuery: not implemented (goal 1d)"); } catch (err) {
if (err instanceof Refused) return { error: err.message };
throw err;
}
}
function resolve(spec: FilterSpec, now: Date): Query {
const words: Term[] = [];
const tags: Term[] = [];
let by: { value: Query["by"]; from: string } | undefined;
let after: { value: Date; from: string } | undefined;
let before: { value: Date; from: string } | undefined;
const setBy = (value: Query["by"], from: string) => {
if (by && by.value !== value) throw new Refused(`${by.from} and ${from} name different dates`);
by = { value, from };
};
const setAfter = (value: Date, from: string) => {
if (after) throw new Refused(`${after.from} and ${from} both set where the window starts`);
after = { value, from };
};
const setBefore = (value: Date, from: string) => {
if (before) throw new Refused(`${before.from} and ${from} both set where the window ends`);
before = { value, from };
};
const setPeriod = ([lo, hi]: [Date, Date], from: string) => {
setAfter(lo, from);
setBefore(hi, from);
};
const present = (v: string | number | undefined) => v !== undefined && v !== "";
if (present(spec.by)) {
if (spec.by !== "updated" && spec.by !== "created") throw new Refused("by: expected updated or created");
setBy(spec.by, `by=${spec.by}`);
}
if (present(spec.days)) setAfter(new Date(now.getTime() - parseDays(spec.days!) * DAY_MS), "days");
if (present(spec.after)) setAfter(parseDay(spec.after!, "after"), "after");
if (present(spec.before)) setBefore(parseDay(spec.before!, "before"), "before");
if (present(spec.in)) setPeriod(parsePeriod(spec.in!, "in"), "in");
for (const m of (spec.q ?? "").matchAll(/(-?)"([^"]*)"|(\S+)/g)) {
if (m[2] !== undefined) {
if (m[2].trim()) words.push({ text: m[2].trim().toLowerCase(), negated: m[1] === "-" });
continue;
}
const token = m[3];
const negated = token.length > 1 && token.startsWith("-");
const body = negated ? token.slice(1) : token;
const op = /^([a-z_]+):(.+)$/i.exec(body);
const name = op?.[1].toLowerCase();
const arg = op?.[2] ?? "";
const windowOp = ["after", "before", "newer_than", "during", "in", "created", "updated"];
if (name && windowOp.includes(name) && negated) throw new Refused(`${name}: can't be negated`);
if (name === "after") setAfter(parseDay(arg, "after:"), "after:");
else if (name === "before") setBefore(parseDay(arg, "before:"), "before:");
else if (name === "newer_than") setAfter(parseNewerThan(arg, now), "newer_than:");
else if (name === "during" || name === "in") setPeriod(parsePeriod(arg, `${name}:`), `${name}:`);
else if (name === "created" || name === "updated") {
setBy(name, `${name}:`);
setPeriod(parsePeriod(arg, `${name}:`), `${name}:`);
} else if (name === "tag") tags.push({ text: arg.toLowerCase(), negated });
else if (body.length > 1 && body.startsWith("#")) tags.push({ text: body.slice(1).toLowerCase(), negated });
else words.push({ text: body.toLowerCase(), negated });
}
if (after && before && after.value.getTime() >= before.value.getTime()) {
throw new Refused(`the window is empty: ${after.from} is not earlier than ${before.from}`);
}
const query: Query = { by: by?.value ?? "updated", words, tags };
if (after) query.after = after.value;
if (before) query.before = before.value;
return query;
} }
/** {@link toQuery} over the public page's URL params */ /** {@link toQuery} over the public page's URL params */
export function parseQuery(params: URLSearchParams, now: Date): Query | QueryError { export function parseQuery(params: URLSearchParams, now: Date): Query | QueryError {
void params; const field = (name: string) => params.get(name) ?? "";
void now; const by = field("by");
throw new Error("parseQuery: not implemented (goal 1d)"); return toQuery(
{
days: field("days"),
after: field("after"),
before: field("before"),
in: field("in"),
by: by as FilterSpec["by"],
q: field("q"),
},
now,
);
}
/** does the query filter anything? (an empty one lists diigo's pages as stored) */
export function isEmpty(query: Query): boolean {
return !query.after && !query.before && !query.words.length && !query.tags.length && query.by === "updated";
}
/** the item's date that `by` names, in epoch seconds (diigo's unit), or
* undefined when it carries none */
export function dateOf(item: Item, by: Query["by"]): number | undefined {
const secs = by === "created" ? item.created_at : item.updated_at;
return typeof secs === "number" && Number.isFinite(secs) ? secs : undefined;
} }
/** does `item` fall in the window and satisfy every term? */ /** does `item` fall in the window and satisfy every term? */
export function matches(item: Item, query: Query): boolean { export function matches(item: Item, query: Query): boolean {
void item; if (query.after || query.before) {
void query; const secs = dateOf(item, query.by);
throw new Error("matches: not implemented (goal 1d)"); if (secs === undefined) return false;
if (query.after && secs * 1000 < query.after.getTime()) return false;
if (query.before && secs * 1000 >= query.before.getTime()) return false;
}
const text = (v: unknown) => (typeof v === "string" ? v : "");
// fields joined by newline, so a phrase never spans two of them
const haystack = [item.title, item.description, item.url].map(text).join("\n").toLowerCase();
if (!query.words.every((w) => haystack.includes(w.text) !== w.negated)) return false;
const itemTags = new Set(
text(item.tags)
.split(",")
.map((t) => t.trim().toLowerCase())
.filter(Boolean),
);
return query.tags.every((t) => itemTags.has(t.text) !== t.negated);
} }
+6
View File
@@ -41,6 +41,12 @@ export function makeReader(sql: Sql) {
deepestPageSince(cutoff: number): number | null { deepestPageSince(cutoff: number): number | null {
return run<{ last: number | null }>("deepestPageSince", cutoff)[0]?.last ?? null; return run<{ last: number | null }>("deepestPageSince", cutoff)[0]?.last ?? null;
}, },
/** stored items whose `by` date (epoch seconds) is in [`from`, `to`),
* newest first; the raw dump rows (JSON strings) */
itemsIn(by: "updated" | "created", from: number, to: number): string[] {
const name = by === "created" ? "itemsByCreated" : "itemsByUpdated";
return run<{ json: string }>(name, from, to).map((r) => r.json);
},
/** how many cached pages are numbered `pageNum` or lower */ /** how many cached pages are numbered `pageNum` or lower */
pagesThrough(pageNum: number): number { pagesThrough(pageNum: number): number {
return Number(run<{ n: number }>("pagesThrough", pageNum)[0]?.n ?? 0); return Number(run<{ n: number }>("pagesThrough", pageNum)[0]?.n ?? 0);
+66 -51
View File
@@ -3,8 +3,8 @@ import { evictDurableObject, runInDurableObject } from "cloudflare:test";
import { describe, expect, it, vi } from "vitest"; import { describe, expect, it, vi } from "vitest";
import worker from "../src/index.js"; import worker from "../src/index.js";
import { idFromWebkey, toWebkey } from "../src/webkeys.js"; import { idFromWebkey, toWebkey } from "../src/webkeys.js";
import { FETCH_BUDGET } from "../src/query.js";
import { PAGE_SIZE } from "../src/cache.js"; import { PAGE_SIZE } from "../src/cache.js";
import { mf2 } from "microformats-parser";
// matches tests/fake-webapp.js: 30 items, every third private // matches tests/fake-webapp.js: 30 items, every third private
const WEBAPP_ID = "fake-webapp-id"; const WEBAPP_ID = "fake-webapp-id";
@@ -250,7 +250,7 @@ describe("worker routes", () => {
// PAGE_SIZE 100: page 2 is bookmarks 101..200 // PAGE_SIZE 100: page 2 is bookmarks 101..200
expect(html).toContain(">bookmark 130<"); expect(html).toContain(">bookmark 130<");
// the date comes first, in toISOString() form // the date comes first, in toISOString() form
expect(html).toMatch(/<li><time datetime="\d{4}-\d\d-\d\dT[\d:.]+Z">[^<]+<\/time> <a /); expect(html).toMatch(/<li class="h-entry"><time class="dt-updated" datetime="\d{4}-\d\d-\d\dT[\d:.]+Z">[^<]+<\/time> <a /);
expect(html).not.toContain(">bookmark 129<"); // private (129 % 3 === 0) expect(html).not.toContain(">bookmark 129<"); // private (129 % 3 === 0)
const json = (await ( const json = (await (
@@ -283,30 +283,34 @@ function isoWeek(d: Date): string {
return `${year}-W${String(week).padStart(2, "0")}`; return `${year}-W${String(week).padStart(2, "0")}`;
} }
describe.skip("public page filters (goal 1d)", () => { describe("public page filters (goal 1d)", () => {
// the fake's dated account: "dated n" updated n-1 days + 1h ago; // the fake's dated account: "dated n" updated n-1 days + 1h ago;
// #weekly on every 7th; every 10th private; every 5th created 100 // #weekly on every 7th; every 10th private; every 5th created 100
// days before it was updated // days before it was updated. The public page only reads, so each
// test stores the pages it needs on the owner's side first.
const call = (path: string, init?: RequestInit) => const call = (path: string, init?: RequestInit) =>
worker.fetch(new Request(`https://diigo-cache.test${path}`, init), env as never); worker.fetch(new Request(`https://diigo-cache.test${path}`, init), env as never);
const wireDated = async () => { /** wire a public cache to `webappId` and store diigo pages 1..`pages` */
const wireStored = async (webappId: string, pages: number) => {
const res = await call("/_wire", { const res = await call("/_wire", {
method: "POST", method: "POST",
headers: { "content-type": "application/json" }, headers: { "content-type": "application/json" },
body: JSON.stringify({ webappId: "fake-dated-webapp", mode: "public" }), body: JSON.stringify({ webappId, mode: "public" }),
}); });
expect(res.status).toBe(200); expect(res.status).toBe(200);
return "/public/dated_dan"; const { username, publicId } = (await res.json()) as { username: string; publicId: string };
const pub = testEnv.PUBLIC_DO.get(testEnv.PUBLIC_DO.idFromString(idFromWebkey(publicId)));
for (let n = 1; n <= pages; n += 1) await pub.page(n, PAGE_SIZE);
return { page: `/public/${username}`, pub };
}; };
const wireDated = async () => (await wireStored("fake-dated-webapp", 1)).page;
const titlesIn = (html: string) => [...html.matchAll(/>(dated \d+)<\/a>/g)].map((m) => m[1]); const titlesIn = (html: string) => [...html.matchAll(/>(dated \d+)<\/a>/g)].map((m) => m[1]);
const dated = (...ns: number[]) => ns.map((n) => `dated ${n}`); const dated = (...ns: number[]) => ns.map((n) => `dated ${n}`);
it("?days=7 lists the last week, walking only the diigo pages it needs", async () => { it("?days=7 lists the last week", async () => {
const page = await wireDated(); const page = await wireDated();
const before = await testEnv.WEBAPP.calls();
const html = await (await call(`${page}?days=7`)).text(); const html = await (await call(`${page}?days=7`)).text();
expect(titlesIn(html)).toEqual(dated(1, 2, 3, 4, 5, 6, 7)); expect(titlesIn(html)).toEqual(dated(1, 2, 3, 4, 5, 6, 7));
expect((await testEnv.WEBAPP.calls()) - before).toBeLessThanOrEqual(1);
}); });
it("?by=created windows, lists, and dates by when it was bookmarked", async () => { it("?by=created windows, lists, and dates by when it was bookmarked", async () => {
@@ -327,13 +331,7 @@ describe.skip("public page filters (goal 1d)", () => {
}); });
it("?days=300 pages the matches PAGE_SIZE at a time; links keep the filter", async () => { it("?days=300 pages the matches PAGE_SIZE at a time; links keep the filter", async () => {
const res = await call("/_wire", { const { page } = await wireStored("fake-deep-webapp", 3);
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({ webappId: "fake-deep-webapp", mode: "public" }),
});
expect(res.status).toBe(200);
const page = "/public/deep_dee";
// deep_dee: one a day; days=300 holds 1..300, minus every 10th (private) // deep_dee: one a day; days=300 holds 1..300, minus every 10th (private)
const want = Array.from({ length: 300 }, (_, k) => k + 1).filter((n) => n % 10 !== 0); const want = Array.from({ length: 300 }, (_, k) => k + 1).filter((n) => n % 10 !== 0);
const first = await (await call(`${page}?days=300`)).text(); const first = await (await call(`${page}?days=300`)).text();
@@ -354,6 +352,15 @@ describe.skip("public page filters (goal 1d)", () => {
expect(titlesIn(html)).toEqual(dated(11, 12)); expect(titlesIn(html)).toEqual(dated(11, 12));
}); });
it("?in= names an ISO week", async () => {
const { page } = await wireStored("fake-deep-webapp", 3);
const week = isoWeek(new Date(Date.now() - 100 * 24 * 60 * 60 * 1000));
const html = await (await call(`${page}?in=${week}`)).text();
// one a day; every 10th is private
expect(titlesIn(html).length).toBeGreaterThanOrEqual(6);
expect(titlesIn(html).length).toBeLessThanOrEqual(7);
});
it("?q=#weekly&days=60 filters by tag; private items stay out", async () => { it("?q=#weekly&days=60 filters by tag; private items stay out", async () => {
const page = await wireDated(); const page = await wireDated();
const html = await (await call(`${page}?days=60&q=${encodeURIComponent("#weekly")}`)).text(); const html = await (await call(`${page}?days=60&q=${encodeURIComponent("#weekly")}`)).text();
@@ -366,22 +373,49 @@ describe.skip("public page filters (goal 1d)", () => {
expect(titlesIn(html)).toEqual(dated(7, 14, 21, 28, 35, 42, 49, 56)); expect(titlesIn(html)).toEqual(dated(7, 14, 21, 28, 35, 42, 49, 56));
}); });
it("a request spends at most FETCH_BUDGET diigo fetches; incomplete results say so, and a reload continues", async () => { it("a filter reads only stored items, never diigo; results say whether storage is complete", async () => {
const res = await call("/_wire", { const { page, pub } = await wireStored("fake-deep-webapp", 1);
method: "POST", const json = async () =>
headers: { "content-type": "application/json" }, (await call(`${page}?days=300`, { headers: { accept: "application/json" } })).json() as Promise<{
body: JSON.stringify({ webappId: "fake-deep-webapp", mode: "public" }), matched: number;
}); complete: boolean;
expect(res.status).toBe(200); }>;
// deep_dee: 60 diigo pages of 100; a search matching nothing must walk them all
const before = await testEnv.WEBAPP.calls(); const before = await testEnv.WEBAPP.calls();
const first = await call("/public/deep_dee?q=nomatch", { headers: { accept: "application/json" } }); // page 1 holds 1..100, minus 10 private
expect(await first.json()).toMatchObject({ complete: false, items: [] }); expect(await json()).toMatchObject({ matched: 90, complete: false });
expect((await testEnv.WEBAPP.calls()) - before).toBeLessThanOrEqual(FETCH_BUDGET + 1); expect(await (await call(`${page}?days=300`)).text()).toMatch(/some matches may be missing/);
const html = await (await call("/public/deep_dee?q=nomatch")).text(); expect(await testEnv.WEBAPP.calls()).toBe(before);
expect(html).toMatch(/complete|no matches/); await pub.page(2, PAGE_SIZE);
const again = await call("/public/deep_dee?q=nomatch", { headers: { accept: "application/json" } }); await pub.page(3, PAGE_SIZE);
expect(await again.json()).toMatchObject({ complete: true, items: [] }); expect(await json()).toMatchObject({ matched: 270, complete: true });
});
it("the page is an h-feed of h-entry bookmarks: microformats are its API for other sites", async () => {
const page = await wireDated();
const html = await (await call(`${page}?days=7&by=created`)).text();
const base = `https://diigo-cache.test${page}`;
const { items, rels } = mf2(html, { baseUrl: base });
const feed = items.find((i) => i.type?.includes("h-feed"));
const entries = feed?.children ?? [];
expect(entries.map((e) => e.properties.name?.[0])).toEqual(dated(1, 2, 3, 4, 6, 7));
const seven = entries[5].properties;
expect(seven["bookmark-of"]).toEqual(["https://dated.example/7"]);
expect(seven.summary).toEqual(["a week of fragments"]);
expect(seven.category).toEqual(["ocap", "weekly"]);
const published = new Date(String(seven.published?.[0])).getTime();
const updated = new Date(String(seven.updated?.[0])).getTime();
// 7 isn't a 5: created when updated, 6 days + 1h ago
expect(published).toBe(updated);
expect(Date.now() - published).toBeGreaterThan(6 * 24 * 60 * 60 * 1000);
// one page of matches: no rel=next
expect(rels.next).toBeUndefined();
});
it("filtered pages link the next page with rel=next, keeping the filter", async () => {
const { page } = await wireStored("fake-deep-webapp", 3);
const html = await (await call(`${page}?days=300`)).text();
const { rels } = mf2(html, { baseUrl: `https://diigo-cache.test${page}?days=300` });
expect(rels.next).toEqual([`https://diigo-cache.test${page}?days=300&page=2`]);
}); });
it("JSON answers carry the canonical filter, so devs see what was understood", async () => { it("JSON answers carry the canonical filter, so devs see what was understood", async () => {
@@ -393,25 +427,6 @@ describe.skip("public page filters (goal 1d)", () => {
}); });
}); });
it("?in= an old week costs O(log pages) diigo fetches with by=updated, not a walk from page 1", async () => {
const res = await call("/_wire", {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({ webappId: "fake-deep-webapp", mode: "public" }),
});
expect(res.status).toBe(200);
// deep_dee: 6000 bookmarks, one a day, 60 diigo pages of 100; a week
// ~14 years back sits near page 51 — a walk from page 1 would cost ~51
const longAgo = new Date(Date.now() - 5100 * 24 * 60 * 60 * 1000);
const week = isoWeek(longAgo);
const before = await testEnv.WEBAPP.calls();
const html = await (await call(`/public/deep_dee?in=${week}`)).text();
// one a day; every 10th is private
expect(titlesIn(html).length).toBeGreaterThanOrEqual(6);
expect(titlesIn(html).length).toBeLessThanOrEqual(7);
expect((await testEnv.WEBAPP.calls()) - before).toBeLessThanOrEqual(12);
});
it("a malformed filter is a 400 with a message, not an empty list", async () => { it("a malformed filter is a 400 with a message, not an empty list", async () => {
const page = await wireDated(); const page = await wireDated();
const res = await call(`${page}?days=7x`); const res = await call(`${page}?days=7x`);
+2 -2
View File
@@ -18,9 +18,9 @@ const OVERSTATEMENT = 60;
* 5th an old bookmark edited recently (created 100 days before updated) */ * 5th an old bookmark edited recently (created 100 days before updated) */
const DATED_WEBAPP = "fake-dated-webapp"; const DATED_WEBAPP = "fake-dated-webapp";
const DATED_TOTAL = 60; const DATED_TOTAL = 60;
/** the same shape, 6000 deep (~16 years, 60 pages of 100): for walk cost */ /** the same shape, 300 deep (3 pages of 100): for paging matches */
const DEEP_WEBAPP = "fake-deep-webapp"; const DEEP_WEBAPP = "fake-deep-webapp";
const DEEP_TOTAL = 6000; const DEEP_TOTAL = 300;
const HOUR = 60 * 60 * 1000; const HOUR = 60 * 60 * 1000;
function datedItem(n, now, user = "dated_dan") { function datedItem(n, now, user = "dated_dan") {
+5 -5
View File
@@ -2,7 +2,7 @@ import { describe, expect, it } from "vitest";
import { matches, parseQuery, toQuery } from "../src/query.js"; import { matches, parseQuery, toQuery } from "../src/query.js";
import type { FilterSpec, Query, QueryError } from "../src/query.js"; import type { FilterSpec, Query, QueryError } from "../src/query.js";
// goal 1d (capsite checklist): the design as tests; unskip as it lands // goal 1d (capsite checklist)
const NOW = new Date("2026-10-07T12:00:00.000Z"); const NOW = new Date("2026-10-07T12:00:00.000Z");
const DAY = 24 * 60 * 60 * 1000; const DAY = 24 * 60 * 60 * 1000;
@@ -21,7 +21,7 @@ const refused = (q: Query | QueryError) => {
const day = (iso: string) => new Date(`${iso}T00:00:00.000Z`); const day = (iso: string) => new Date(`${iso}T00:00:00.000Z`);
const range = (q: Query) => [q.after?.toISOString().slice(0, 10), q.before?.toISOString().slice(0, 10)]; const range = (q: Query) => [q.after?.toISOString().slice(0, 10), q.before?.toISOString().slice(0, 10)];
describe.skip("filters: one spec, two spellings (goal 1d)", () => { describe("filters: one spec, two spellings (goal 1d)", () => {
it("devs write { days: 7 }; URLs write ?days=7; same query", () => { it("devs write { days: 7 }; URLs write ?days=7; same query", () => {
expect(spec({ days: 7 })).toEqual(url("days=7")); expect(spec({ days: 7 })).toEqual(url("days=7"));
expect(spec({ days: 7 }).after).toEqual(new Date(NOW.getTime() - 7 * DAY)); expect(spec({ days: 7 }).after).toEqual(new Date(NOW.getTime() - 7 * DAY));
@@ -42,7 +42,7 @@ describe.skip("filters: one spec, two spellings (goal 1d)", () => {
}); });
}); });
describe.skip("windows: UTC, half-open (goal 1d)", () => { describe("windows: UTC, half-open (goal 1d)", () => {
it("after/before are UTC days: after inclusive from 00:00Z, before exclusive", () => { it("after/before are UTC days: after inclusive from 00:00Z, before exclusive", () => {
expect(range(url("after=2026-01-01&before=2026-02-01"))).toEqual(["2026-01-01", "2026-02-01"]); expect(range(url("after=2026-01-01&before=2026-02-01"))).toEqual(["2026-01-01", "2026-02-01"]);
expect(url("after=2026/01/01").after).toEqual(day("2026-01-01")); expect(url("after=2026/01/01").after).toEqual(day("2026-01-01"));
@@ -88,7 +88,7 @@ describe.skip("windows: UTC, half-open (goal 1d)", () => {
}); });
}); });
describe.skip("q: terms and operators (goal 1d)", () => { describe("q: terms and operators (goal 1d)", () => {
const q = (text: string) => url(`q=${encodeURIComponent(text)}`); const q = (text: string) => url(`q=${encodeURIComponent(text)}`);
it("words, phrases, #tags, tag:, and - negation; case-insensitive", () => { it("words, phrases, #tags, tag:, and - negation; case-insensitive", () => {
@@ -160,7 +160,7 @@ const item = (iso: string, extra: Record<string, unknown> = {}) => ({
}); });
const query = (q: Partial<Query>): Query => ({ by: "updated", words: [], tags: [], ...q }); const query = (q: Partial<Query>): Query => ({ by: "updated", words: [], tags: [], ...q });
describe.skip("matches (goal 1d)", () => { describe("matches (goal 1d)", () => {
it("the window is half-open: after inclusive, before exclusive", () => { it("the window is half-open: after inclusive, before exclusive", () => {
const q = query({ after: day("2026-01-01"), before: day("2026-02-01") }); const q = query({ after: day("2026-01-01"), before: day("2026-02-01") });
expect(matches(item("2026-01-01T00:00:00.000Z"), q)).toBe(true); expect(matches(item("2026-01-01T00:00:00.000Z"), q)).toBe(true);
+1
View File
@@ -0,0 +1 @@
node_modules
+21
View File
@@ -0,0 +1,21 @@
# diigo-week
An EmDash plugin that drafts a potluck issue from a week of your shared diigo bookmarks. It reads them as microformats from diigo-cache's public page, so you edit a draft instead of copy-pasting links.
## Getting started
Run diigo-cache (with a filled public cache) and potluck, each in its own terminal:
```sh
pnpm install
(cd ../diigo-cache && pnpm dev) # → http://localhost:8787/
(cd ../potluck && pnpm dev) # → http://localhost:4321/_emdash/admin
```
On potluck's admin dashboard, the **This week's issue** card has a **Draft 2026-Wnn from diigo** button for last week; it links to the draft once made. For any other week: Plugins → **Draft from diigo**. A week that already has an issue isn't drafted again.
## How it reads the week
It asks diigo-cache for `/public/<user>?in=<week>&by=created` over the `DIIGO_CACHE` service binding (potluck's `wrangler.jsonc`), reads the page's `h-feed` of `h-entry` bookmarks, follows `rel=next`, and creates a draft in `issues`: title, week, dek, and a paragraph per bookmark with its description and tag links, as in potluck's `seed/issues.json`. The binding is its only reach outside the site; it declares `content:write` and `taxonomies:write` (the issue gets its bookmarks' tags, as the sample issues do), nothing else.
Checks: `pnpm lint && pnpm test`.
+24
View File
@@ -0,0 +1,24 @@
{
"name": "diigo-week",
"version": "0.1.0",
"private": true,
"type": "module",
"exports": {
".": {
"import": "./src/index.ts"
}
},
"scripts": {
"lint": "tsc --noEmit",
"test": "vitest run"
},
"dependencies": {
"emdash": "1.2.0",
"microformats-parser": "^2.0.6"
},
"devDependencies": {
"@emdash-cms/blocks": "^1.2.0",
"typescript": "^7.0.2",
"vitest": "^5.0.3"
}
}
+6223
View File
File diff suppressed because it is too large. Load diff
+8
View File
@@ -0,0 +1,8 @@
packages:
- "."
strictDepBuilds: true
dangerouslyAllowAllBuilds: false
allowBuilds:
esbuild: true
workerd: true
+7
View File
@@ -0,0 +1,7 @@
/** emdash's virtual module for the site worker's bindings (declared for site
* builds in emdash/src/virtual-modules.d.ts), so this package typechecks
* standalone: `env` from `cloudflare:workers` under @astrojs/cloudflare,
* undefined on other adapters */
declare module "virtual:emdash/env" {
export const env: Record<string, unknown> | undefined;
}
+180
View File
@@ -0,0 +1,180 @@
/**
* @file diigo-week: an EmDash plugin that drafts a potluck issue from a
* week of public diigo bookmarks.
*/
import { definePlugin } from "emdash";
import type { PluginDefinition, RouteContext } from "emdash";
import type { Block, BlockResponse } from "@emdash-cms/blocks";
import { draftIssue, issueTags, lastWeek, readWeek } from "./week.js";
/** the diigo username drafts read from, until the admin page saves another */
const DEFAULT_USER = "dckc-madmode";
/** something that answers HTTP requests: a service binding's `fetch` */
type Fetcher = { fetch(request: Request): Promise<Response> };
/**
* the `DIIGO_CACHE` service binding to diigo-cache, from the site worker's
* env. The binding is the plugin's only reach outside the site:
* `ctx.http.fetch` can't reach a dev server (it refuses localhost), and a
* binding names one worker rather than a host.
*/
async function diigoCache(): Promise<Fetcher> {
const { env } = await import("virtual:emdash/env");
const binding = env?.DIIGO_CACHE as Fetcher | undefined;
if (typeof binding?.fetch !== "function") {
throw new Error('no DIIGO_CACHE service binding: add { "binding": "DIIGO_CACHE", "service": "diigo-cache" } to services in wrangler.jsonc');
}
return binding;
}
/** GET a diigo-cache path as HTML; anything but a 200 is an error */
async function getter(cache: Fetcher): Promise<(path: string) => Promise<string>> {
return async (path) => {
const res = await cache.fetch(new Request(`https://diigo-cache${path}`, { headers: { accept: "text/html" } }));
const text = await res.text();
if (res.status !== 200) throw new Error(`diigo-cache ${path.split("?")[0]}: ${res.status} ${text.slice(0, 200)}`);
return text;
};
}
/**
* assign `tags` to an issue in the `tag` taxonomy (as seed/issues.json
* does), creating terms that don't exist yet; returns the tags that
* couldn't be assigned, with why
*/
async function tagIssue(ctx: RouteContext, id: string, tags: string[]): Promise<string[]> {
const tax = ctx.taxonomies;
if (typeof tax?.createTerm !== "function" || typeof tax.addEntryTerms !== "function") throw new Error("this plugin lacks taxonomies:write");
const known = new Map((await tax.getTerms("tag")).map((t) => [t.slug, t.id]));
const ids: string[] = [];
const failed: string[] = [];
for (const tag of tags) {
try {
ids.push(known.get(tag) ?? (await tax.createTerm("tag", { label: tag, slug: tag })).id);
} catch (error) {
failed.push(`${tag} (${error instanceof Error ? error.message : String(error)})`);
}
}
if (ids.length) await tax.addEntryTerms("issues", id, "tag", ids);
return failed;
}
/** the issue already drafted or published for `week`, if any (issues
* are few: a scan of the newest is enough) */
async function issueFor(ctx: RouteContext, week: string): Promise<{ id: string; status: string; title: string } | undefined> {
const found = (await ctx.content!.list("issues", { limit: 100 })).items.find((it) => it.data.week === week);
return found && { id: found.id, status: found.status, title: String(found.data.title ?? week) };
}
/** a button-row link to an issue in the editor */
const openIssue = (id: string, label: string): Block => ({
type: "actions",
elements: [{ type: "link", label, target: { kind: "content", collection: "issues", id }, appearance: "primary" }],
});
/** draft `week` from `user`'s bookmarks, unless the week has an issue
* already: blocks that say what happened (with a link), and a toast */
async function draft(ctx: RouteContext, user: string, week: string): Promise<{ note: Block[]; toast: BlockResponse["toast"] }> {
try {
if (!/^[A-Za-z0-9_-]{1,64}$/.test(user)) throw new Error("diigo username: letters, digits, _ and - only");
const existing = await issueFor(ctx, week);
if (existing) {
return {
note: [
{ type: "banner", variant: "default", title: `${week} already has an issue`, description: `"${existing.title}" (${existing.status}). Delete it to draft the week again.` },
openIssue(existing.id, "Open it"),
],
toast: { message: `${week} already has an issue`, type: "info" },
};
}
const bookmarks = await readWeek(await getter(await diigoCache()), user, week);
const issue = draftIssue(week, bookmarks);
const content = ctx.content;
if (typeof content?.create !== "function") throw new Error("this plugin lacks content:write");
const created = await content.create("issues", issue);
const untagged = await tagIssue(ctx, created.id, issueTags(bookmarks));
if (untagged.length) ctx.log.warn("tags not assigned", { id: created.id, untagged });
ctx.log.info("drafted issue", { week, bookmarks: bookmarks.length, id: created.id });
return {
note: [
{ type: "banner", variant: "default", title: `Drafted "${issue.title}"`, description: `${bookmarks.length} bookmarks, as a draft at /issues/${created.slug ?? "(no slug)"}. Cut, comment, and publish.${untagged.length ? ` Tags not assigned: ${untagged.join(", ")}.` : ""}` },
openIssue(created.id, "Open the draft"),
],
toast: { message: `Drafted ${issue.title}`, type: "success" },
};
} catch (error) {
const message = error instanceof Error ? error.message : String(error);
ctx.log.error("draft failed", { week, message });
return { note: [{ type: "banner", variant: "alert", title: "Couldn't draft the issue", description: message }], toast: { message, type: "error" } };
}
}
/** the admin page: a form for the diigo username and the week */
async function page(ctx: RouteContext, note: Block[] = [], toast?: BlockResponse["toast"]): Promise<BlockResponse> {
const user = (await ctx.kv.get<string>("user")) ?? DEFAULT_USER;
const blocks: Block[] = [
{ type: "header", text: "Draft an issue from diigo" },
{
type: "context",
text: "Reads a week of shared bookmarks from diigo-cache (by when they were bookmarked) and saves them as a draft issue for you to edit.",
},
{
type: "form",
block_id: "draft",
fields: [
{ type: "text_input", action_id: "user", label: "diigo username", initial_value: user },
{ type: "text_input", action_id: "week", label: "ISO week (UTC)", initial_value: lastWeek(new Date()) },
],
submit: { label: "Draft issue", action_id: "draft" },
},
];
blocks.push(...note);
return { blocks, toast };
}
/** the dashboard widget: last week's issue if there is one, else a button to draft it */
async function widget(ctx: RouteContext, note: Block[] = [], toast?: BlockResponse["toast"]): Promise<BlockResponse> {
const week = lastWeek(new Date());
if (note.length) return { blocks: note, toast };
const existing = await issueFor(ctx, week);
const blocks: Block[] = existing
? [{ type: "section", text: `${week}: "${existing.title}" (${existing.status})` }, openIssue(existing.id, "Open it")]
: [
{ type: "section", text: `${week} has no issue yet.` },
{ type: "actions", elements: [{ type: "button", action_id: "draft-last-week", label: `Draft ${week} from diigo`, style: "primary" }] },
];
return { blocks, toast };
}
export function createPlugin(): PluginDefinition {
return definePlugin({
id: "diigo-week",
version: "0.1.0",
capabilities: ["content:write", "taxonomies:write"],
admin: {
pages: [{ path: "/draft", label: "Draft from diigo", icon: "bookmark" }],
widgets: [{ id: "last-week", title: "This week's issue", size: "half" }],
},
routes: {
admin: {
permission: "plugins:manage",
handler: async (ctx) => {
const input = ctx.input as { type?: string; page?: string; action_id?: string; values?: Record<string, unknown> } | undefined;
const user = (await ctx.kv.get<string>("user")) ?? DEFAULT_USER;
if (input?.type === "block_action" && input.action_id === "draft-last-week") {
const { note, toast } = await draft(ctx, user, lastWeek(new Date()));
return widget(ctx, note, toast);
}
if (input?.page?.startsWith("widget:")) return widget(ctx);
if (input?.type !== "form_submit" || input.action_id !== "draft") return page(ctx);
const formUser = String(input.values?.user ?? "").trim();
if (/^[A-Za-z0-9_-]{1,64}$/.test(formUser)) await ctx.kv.set("user", formUser);
const { note, toast } = await draft(ctx, formUser, String(input.values?.week ?? "").trim());
return page(ctx, note, toast);
},
},
},
});
}
+146
View File
@@ -0,0 +1,146 @@
/**
* @file a week of bookmarks, read as microformats from diigo-cache's public
* page, drafted as a potluck issue.
*/
import { mf2 } from "microformats-parser";
const DAY_MS = 24 * 60 * 60 * 1000;
/** pages of matches one draft will read (100 bookmarks each) */
export const MAX_PAGES = 10;
/** one bookmark, as its h-entry says */
export type Bookmark = {
url: string;
title: string;
description: string;
tags: string[];
/** when it was bookmarked, ISO */
published: string;
};
/** `yyyy-Www` for the ISO 8601 week holding `d` (UTC) */
export function isoWeek(d: Date): string {
const t = new Date(Date.UTC(d.getUTCFullYear(), d.getUTCMonth(), d.getUTCDate()));
const day = t.getUTCDay() || 7;
t.setUTCDate(t.getUTCDate() + 4 - day); // the week's Thursday names its year
const year = t.getUTCFullYear();
const week = Math.ceil(((t.getTime() - Date.UTC(year, 0, 1)) / DAY_MS + 1) / 7);
return `${year}-W${String(week).padStart(2, "0")}`;
}
/** the last full ISO week before `now` */
export function lastWeek(now: Date): string {
return isoWeek(new Date(now.getTime() - 7 * DAY_MS));
}
/** Monday 00:00Z of ISO week `week` (`2026-W41`), or undefined if malformed */
export function weekStart(week: string): Date | undefined {
const m = /^(\d{4})-W(\d{2})$/.exec(week);
if (!m) return undefined;
const [year, w] = [Number(m[1]), Number(m[2])];
const jan4 = Date.UTC(year, 0, 4);
const monday = jan4 - ((new Date(jan4).getUTCDay() || 7) - 1) * DAY_MS + (w - 1) * 7 * DAY_MS;
const start = new Date(monday);
return w >= 1 && isoWeek(start) === week ? start : undefined;
}
/** the first string value of an mf2 property, or "" */
const first = (props: Record<string, unknown[]>, name: string): string => {
const v = props[name]?.[0];
return typeof v === "string" ? v : "";
};
/** the bookmarks in a page's `h-feed`, and its `rel=next` link if any */
export function readPage(html: string, url: string): { bookmarks: Bookmark[]; next?: string } {
const { items, rels } = mf2(html, { baseUrl: url });
const feed = items.find((i) => i.type?.includes("h-feed"));
const bookmarks = (feed?.children ?? [])
.filter((e) => e.type?.includes("h-entry"))
.map((e) => {
const props = e.properties as Record<string, unknown[]>;
const tags = (props.category ?? []).filter((t): t is string => typeof t === "string");
return {
url: first(props, "bookmark-of"),
title: first(props, "name"),
description: first(props, "summary"),
tags,
published: first(props, "published"),
};
})
.filter((b) => b.url !== "");
return { bookmarks, next: rels.next?.[0] };
}
/**
* a user's bookmarks from an ISO week, oldest first, by when they were
* bookmarked: diigo-cache's `/public/<user>?in=<week>&by=created`,
* following `rel=next` up to {@link MAX_PAGES} pages.
*
* @param get fetches a path on diigo-cache, answering its HTML; it throws
* on anything but a 200
*/
export async function readWeek(get: (path: string) => Promise<string>, user: string, week: string): Promise<Bookmark[]> {
const base = "https://diigo-cache.invalid";
let path: string | undefined = `/public/${encodeURIComponent(user)}?in=${encodeURIComponent(week)}&by=created`;
const bookmarks: Bookmark[] = [];
for (let n = 0; path && n < MAX_PAGES; n += 1) {
const page = readPage(await get(path), base + path);
bookmarks.push(...page.bookmarks);
const next: URL | undefined = page.next ? new URL(page.next) : undefined;
path = next && next.origin === base ? next.pathname + next.search : undefined;
}
return bookmarks.reverse();
}
/** a Portable Text span */
type Span = { _type: "span"; _key: string; text: string; marks: string[] };
/** a Portable Text block */
type Block = { _type: "block"; _key: string; style: string; markDefs: { _type: "link"; _key: string; href: string }[]; children: Span[] };
/** a bookmark as one paragraph: the link, the description, then tag links (as in seed/issues.json) */
function bookmarkBlock(b: Bookmark, i: number): Block {
const k = `b${i}`;
const markDefs: Block["markDefs"] = [{ _type: "link", _key: `${k}-a`, href: b.url }];
const children: Span[] = [{ _type: "span", _key: `${k}-t`, text: b.title || b.url, marks: [`${k}-a`] }];
if (b.description) children.push({ _type: "span", _key: `${k}-d`, text: `: ${b.description}`, marks: [] });
b.tags.forEach((tag, j) => {
markDefs.push({ _type: "link", _key: `${k}-g${j}`, href: `/tag/${encodeURIComponent(tag)}` });
children.push({ _type: "span", _key: `${k}-s${j}`, text: " ", marks: [] });
children.push({ _type: "span", _key: `${k}-n${j}`, text: tag, marks: [`${k}-g${j}`] });
});
return { _type: "block", _key: k, style: "normal", markDefs, children };
}
/** the issue's tags: every bookmark's, deduplicated, in order of first use */
export function issueTags(bookmarks: Bookmark[]): string[] {
return [...new Set(bookmarks.flatMap((b) => b.tags))];
}
/** `Oct 5–11` (or `Sep 28–Oct 4`) for the week starting `monday` */
function range(monday: Date): string {
const sunday = new Date(monday.getTime() + 6 * DAY_MS);
const fmt = (d: Date, withMonth: boolean) =>
withMonth ? d.toLocaleDateString("en-US", { month: "short", day: "numeric", timeZone: "UTC" }) : String(d.getUTCDate());
return `${fmt(monday, true)}–${fmt(sunday, sunday.getUTCMonth() !== monday.getUTCMonth())}`;
}
/** the fields of a draft `issues` entry for `week`: title, week, dek, body */
export function draftIssue(week: string, bookmarks: Bookmark[]): Record<string, unknown> {
const monday = weekStart(week);
if (!monday) throw new Error(`week: expected an ISO week like 2026-W41, got ${JSON.stringify(week)}`);
const intro: Block = {
_type: "block",
_key: "intro",
style: "normal",
markDefs: [],
children: [{ _type: "span", _key: "intro-t", text: "(draft) A line or two about the week goes here.", marks: [] }],
};
return {
title: `Week ${week}`,
week,
dek: `${range(monday)}: ${bookmarks.length} bookmark${bookmarks.length === 1 ? "" : "s"}. (draft)`,
body: [intro, ...bookmarks.map(bookmarkBlock)],
};
}
+99
View File
@@ -0,0 +1,99 @@
import { describe, expect, it } from "vitest";
import { draftIssue, issueTags, lastWeek, readWeek, weekStart } from "../src/week.js";
/** an h-entry as diigo-cache's public page marks it up (by=created) */
const entry = (n: number, created: string, extra = "") =>
`<li class="h-entry"><time class="dt-published" datetime="${created}">${created}</time> <a class="u-bookmark-of p-name" href="https://ex.example/${n}">bookmark ${n}</a>${extra}<time class="dt-updated" datetime="${created}" hidden></time></li>`;
const page = (entries: string[], next?: string) => `<!doctype html><html><body>
<p>${next ? `<a rel="next" href="${next}">next →</a>` : "<span>next →</span>"}</p>
<ol class="h-feed">
${entries.join("\n")}
</ol></body></html>`;
describe("readWeek", () => {
it("reads h-entry bookmarks across rel=next pages, oldest first", async () => {
const asked: string[] = [];
const pages: Record<string, string> = {
"/public/dckc-madmode?in=2026-W40&by=created": page(
[
entry(
3,
"2026-10-03T12:00:00.000Z",
` <a class="p-category" href="/public/dckc-madmode?q=%23ocap">ocap</a> <a class="p-category" href="/public/dckc-madmode?q=%23weekly">weekly</a><br><small class="p-summary">a week of fragments</small>`,
),
entry(2, "2026-10-01T12:00:00.000Z"),
],
"/public/dckc-madmode?in=2026-W40&by=created&page=2",
),
"/public/dckc-madmode?in=2026-W40&by=created&page=2": page([entry(1, "2026-09-28T12:00:00.000Z")]),
};
const get = async (path: string) => {
asked.push(path);
const html = pages[path];
if (!html) throw new Error(`no page ${path}`);
return html;
};
const bookmarks = await readWeek(get, "dckc-madmode", "2026-W40");
expect(asked).toHaveLength(2);
expect(bookmarks.map((b) => b.title)).toEqual(["bookmark 1", "bookmark 2", "bookmark 3"]);
expect(bookmarks[2]).toEqual({
url: "https://ex.example/3",
title: "bookmark 3",
description: "a week of fragments",
tags: ["ocap", "weekly"],
published: "2026-10-03T12:00:00.000Z",
});
});
it("doesn't follow a rel=next to another site", async () => {
const asked: string[] = [];
const get = async (path: string) => {
asked.push(path);
return page([entry(1, "2026-10-01T12:00:00.000Z")], "https://elsewhere.example/public/x");
};
await readWeek(get, "dckc-madmode", "2026-W40");
expect(asked).toHaveLength(1);
});
});
describe("weeks", () => {
it("lastWeek is the full ISO week before now", () => {
expect(lastWeek(new Date("2026-10-09T01:00:00Z"))).toBe("2026-W40");
// Monday 2026-01-05 starts W02; W01 began Monday 2025-12-29
expect(lastWeek(new Date("2026-01-05T00:00:00Z"))).toBe("2026-W01");
});
it("weekStart is Monday 00:00Z, and refuses a week that doesn't exist", () => {
expect(weekStart("2026-W40")?.toISOString()).toBe("2026-09-28T00:00:00.000Z");
expect(weekStart("2025-W01")?.toISOString()).toBe("2024-12-30T00:00:00.000Z");
expect(weekStart("2025-W53")).toBeUndefined();
expect(weekStart("2026-40")).toBeUndefined();
});
});
describe("draftIssue", () => {
it("drafts title, week, dek, and a body: an intro, then a paragraph per bookmark with tag links", () => {
const issue = draftIssue("2026-W40", [
{ url: "https://ex.example/1", title: "one", description: "nice", tags: ["ocap"], published: "2026-09-28T12:00:00.000Z" },
]);
expect(issue.title).toBe("Week 2026-W40");
expect(issue.week).toBe("2026-W40");
expect(issue.dek).toBe("Sep 28–Oct 4: 1 bookmark. (draft)");
const [intro, b] = issue.body as { markDefs: { href: string }[]; children: { text: string }[] }[];
expect(intro.children[0].text).toMatch(/draft/);
expect(b.markDefs.map((m) => m.href)).toEqual(["https://ex.example/1", "/tag/ocap"]);
expect(b.children.map((c) => c.text).join("")).toBe("one: nice ocap");
});
it("refuses a malformed week", () => {
expect(() => draftIssue("last week", [])).toThrow(/ISO week/);
});
});
describe("issueTags", () => {
it("is every bookmark's tags, once each, in order of first use", () => {
const b = (tags: string[]) => ({ url: "https://ex.example/", title: "", description: "", tags, published: "" });
expect(issueTags([b(["governance", "economics"]), b([]), b(["governance", "scaling"])])).toEqual(["governance", "economics", "scaling"]);
});
});
+20
View File
@@ -0,0 +1,20 @@
{
"compilerOptions": {
"target": "ES2022",
"module": "preserve",
"moduleResolution": "bundler",
"strict": true,
"esModuleInterop": true,
"verbatimModuleSyntax": true,
"skipLibCheck": true,
"types": []
},
"include": [
"src/**/*",
"tests/**/*",
"vitest.config.ts"
],
"exclude": [
"node_modules"
]
}
-107
View File
@@ -1,107 +0,0 @@
This is an EmDash site -- a CMS built on Astro with a full admin UI.
## Commands
```bash
pnpm dev # Start the Astro dev server
npx emdash types # Regenerate TypeScript types from a running site
```
The admin UI is at `http://localhost:4321/_emdash/admin`.
## Key Files
| File | Purpose |
| ------------------------ | ---------------------------------------------------------------------------------- |
| `astro.config.mjs` | Astro config with `emdash()` integration, database, and storage |
| `src/live.config.ts` | EmDash loader registration (boilerplate -- don't modify) |
| `seed/seed.json` | Schema definition + demo content (collections, fields, taxonomies, menus, widgets) |
| `emdash-env.d.ts` | Generated types for collections (auto-regenerated on dev server start) |
| `src/layouts/Base.astro` | Base layout with EmDash wiring (menus, search, page contributions) |
| `src/pages/` | Astro pages -- all server-rendered |
## Skills
Agent skills are in `.agents/skills/`. Load them when working on specific tasks:
- **building-emdash-site** -- Querying content, rendering Portable Text, schema design, seed files, site features (menus, widgets, search, SEO, comments, bylines). Start here.
- **creating-plugins** -- Building EmDash plugins with hooks, storage, admin UI, API routes, and Portable Text block types.
- **emdash-cli** -- CLI commands for content management, seeding, type generation, and visual editing flow.
## Documentation
The EmDash docs are available as an MCP server at `https://docs.emdashcms.com/mcp`. When you need to verify an API, hook, config option, field type, or pattern, call `search_docs` against the live documentation rather than relying on training-data recall. The docs reflect current behaviour; assumptions may not.
This template ships with `.mcp.json`, `.cursor/mcp.json`, and `.vscode/mcp.json` so Claude Code, Cursor, and VS Code auto-discover the docs server. Other tools (OpenCode, Windsurf, etc.) need a manual one-time setup -- see [docs.emdashcms.com/docs-mcp](https://docs.emdashcms.com/docs-mcp).
## Rules
- All content pages must be server-rendered (`output: "server"`). No `getStaticPaths()` for CMS content.
- Image fields are objects (`{ src, alt }`), not strings. Use `<Image image={...} />` from `"emdash/ui"`.
- `entry.id` is the slug (for URLs). `entry.data.id` is the database ULID (for API calls like `getEntryTerms`).
- When Astro's cache is enabled, pass content-query hints to `Astro.cache.set(cacheHint)`. Use the `WithCacheHint` variants for site settings, menus, taxonomies, and widget areas rendered by cached routes.
- Taxonomy names in queries must match the seed's `"name"` field exactly (e.g., `"category"` not `"categories"`).
## This Template
A blog with posts, pages, categories, tags, full-text search, and RSS. Designed for personal writing, technical writing, indie newsletters, and anything where the writing is the product. Editorial-tech aesthetic: confident sans-serif, restrained accent, real article structure with bylines and reading time.
## Pages
| Page | Path | What it shows |
| ----------- | ------------------ | ------------------------------------------------------------------------------------------------------ |
| Home | `/` | Featured post hero (large image + excerpt), latest posts grid |
| All posts | `/posts` | Article count, full post list with excerpts and tag chips |
| Post detail | `/posts/[slug]` | Featured image, title, body, left meta column (authors + date), right TOC + search + categories gutter |
| Search | `/search` | Full-text search UI |
| Page | `/pages/[slug]` | Static page content (Portable Text) |
| Category | `/category/[slug]` | Posts filtered by category |
| Tag | `/tag/[slug]` | Posts filtered by tag |
| RSS | `/rss.xml` | Generated feed |
## Schema
- `posts` collection: `title`, `featured_image`, `content` (Portable Text), `excerpt` (text).
- `pages` collection: `title`, `content` (Portable Text). Used for `/about` etc.
- Taxonomies: `category`, `tag`.
- Single `primary` menu (Home, About, Posts by default).
Site settings have `title` and `tagline` -- both render in the header / footer.
## Visual character
Single typeface: **Inter** on `--font-body`, used for everything including headings (`--font-heading` defaults to the body face; tighter letter-spacing on h1/h2). **JetBrains Mono** on `--font-mono` for inline code and code blocks. Body and headings share the same family; weight and size carry the hierarchy (`--font-weight-heading` 600, `--font-weight-display` 700 for h1/page titles).
The brand colour is `#0066cc` (`--color-brand`) -- used for links, the post-card title hover, and the search input focus ring. There's also a secondary text colour (`--color-text-secondary`) and a `--color-muted` for meta info. Don't add a second accent.
The article layout is the standout feature: a three-column reading view with a left meta column (author bylines, date), centred 680px body column, and a right gutter for search, table of contents, and categories. Don't flatten that into one column on desktop -- the layout signals "this is something to read".
## Customisation
Design tokens live in `src/styles/tokens.css` with their default values. To restyle the site, override tokens in `src/styles/theme.css` -- declarations there are unlayered, so they always beat the `@layer base` defaults. Don't edit `tokens.css` or `Base.astro` for visual changes.
Colours are defined with `light-dark(<light>, <dark>)`, so each token carries both modes. Overriding with a plain colour changes light and dark at once; use `light-dark()` in the override to keep them distinct. There is no separate dark palette to maintain.
Webfonts are configured in `astro.config.mjs` under `fonts:`. To swap the body face, change the `name:` for the entry bound to `cssVariable: "--font-body"`. Good alternatives: Geist, IBM Plex Sans, Söhne (if you have a licence), Public Sans. If you want a serif-bodied blog, swap to a humanist serif like Source Serif, Crimson Pro, or Lora -- but then also raise `--font-size-base` to `1.0625rem` for readability. To give headings their own face (or use a system font) without touching the font pipeline, override `--font-heading` or `--font-body` in `theme.css`.
CSS variables worth knowing (see `tokens.css` for the full list):
- `--color-brand`, `--color-brand-hover`, `--color-on-brand`, `--color-brand-ring`
- `--color-bg`, `--color-bg-subtle`, `--color-surface`, `--color-text`, `--color-text-secondary`, `--color-muted`, `--color-border`, `--color-border-subtle`
- `--font-body`, `--font-heading`, `--font-mono`
- `--font-weight-heading` (600) / `--font-weight-display` (700) -- heading weights; lower them if you switch to a serif
- `--tracking-tight` / `--tracking-snug` / `--tracking-wide` / `--tracking-wider` -- letter-spacing tokens used across headings and meta labels
- `--content-width` (680px) -- article body column
- `--wide-width` (1200px) -- max container
- `--gutter-width` (200px) -- right sidebar (TOC) on article pages
- `--meta-col-width` (180px) -- left meta column on article pages
- `--avatar-size-{xs,sm,md,lg}` -- byline avatar sizes at different scales
## What not to do
- Don't add a second accent colour or coloured section backgrounds. The page should be black, white, and one blue.
- Don't replace Inter with a display sans (Bebas, Anton, etc.). Headings rely on weight contrast, not novelty faces.
- Don't collapse the article gutter on desktop -- it's part of the reading experience.
- Don't use stock blog copy ("Welcome to my blog", "Stay tuned for more"). Write a real tagline that says what this blog is about.
- Don't seed the home page with three identical placeholder posts. If you only have one real post, show one real post.
- Comments are enabled on posts and rendered on the post detail page. Configure moderation before publishing the site, or remove `commentsEnabled` and the comments UI together.
+1
View File
@@ -0,0 +1 @@
CONTRIBUTING.md
+51
View File
@@ -0,0 +1,51 @@
# Contributing to potluck
potluck started as EmDash's `cloudflare:blog` template (`pnpm create emdash@latest`); it's part of [the capsite goal tree](../capsite-checklist.md), where it is the site that publishes the weekly roundup.
## How it's put together
- **Issues** are the `issues` collection ([seed/issues.json](seed/issues.json)): `title`, `week` (an ISO week, `2026-W40`), `dek` (one-line summary), `body` (Portable Text). The slug is the week, lowercased: `/issues/2026-w40`. [issues-sketch.md](issues-sketch.md) is the dry run behind the three sample issues: real public bookmarks, editorial lines marked *draft*.
- **Pages:** home and `/issues` list issues ([src/pages/issues/](src/pages/issues/)); `/tag/<tag>` lists issues by diigo tag; search and `/rss.xml` cover issues. The `posts` collection and its pages are the template's, still in place.
- **Microformats:** an issue list is an `h-feed` of `h-entry`s; an issue is an `h-entry` (`p-name`, `dt-published`, `p-summary`, `e-content`). In issue bodies, [CiteLink.astro](src/components/CiteLink.astro) marks off-site links as `h-cite` and `/tag/<tag>` links as rel-tag `p-category`.
- **Look:** plain HTML, after kmjn.org's [Plain HTML with a few bureaucratic gestures](https://www.kmjn.org/notes/modern_plain_html.html): system fonts (no webfonts), one ~800px column, links in the text color with a dotted underline. Restyle in [src/styles/theme.css](src/styles/theme.css), which overrides the defaults in `tokens.css` (unlayered, so it always wins); don't edit `tokens.css` or `Base.astro` for visual changes. Colors use `light-dark(<light>, <dark>)`.
## Plugins
[diigo-week](../diigo-week/README.md) (Plugins → Draft from diigo in the admin) drafts an issue from a week of diigo bookmarks. It reads diigo-cache's public page as microformats over the `DIIGO_CACHE` service binding in [wrangler.jsonc](wrangler.jsonc); in dev, run diigo-cache's `pnpm dev` alongside. It's linked as `link:../diigo-week`, and registered in [astro.config.mjs](astro.config.mjs). After changing it, restart the dev server.
## Seeds
[seed/seed.json](seed/seed.json) (site settings, menus, widgets, the template's collections) is applied on the first request to an empty database. [seed/issues.json](seed/issues.json) is separate: `pnpm seed:issues` applies it to the local D1 database, which exists after the dev server's first request. Run it with the dev server stopped.
## Checks
```sh
pnpm typecheck # astro check
pnpm build
```
## EmDash rules
- All content pages are server-rendered (`output: "server"`); no `getStaticPaths()` for CMS content.
- Image fields are objects (`{ src, alt }`), not strings; render with `<Image image={...} />` from `"emdash/ui"`.
- `entry.id` is the slug (for URLs); `entry.data.id` is the database ULID (for API calls like `getEntryTerms`).
- When Astro's cache is enabled, pass query hints to `Astro.cache.set(cacheHint)`.
- Taxonomy names in queries match the seed's `"name"` exactly (`"tag"`, not `"tags"`).
- `npx emdash types` regenerates `emdash-env.d.ts` from a running site (the dev server also regenerates it on start).
The EmDash docs are an MCP server at `https://docs.emdashcms.com/mcp` (wired up in `.mcp.json`, `.cursor/mcp.json`, `.vscode/mcp.json`). Check them rather than recalling APIs from memory.
## Deploying
```sh
pnpm wrangler login
pnpm deploy
```
The first deploy provisions the D1 database and R2 bucket named in [wrangler.jsonc](wrangler.jsonc). See [Deploy to Cloudflare](https://docs.emdashcms.com/deployment/cloudflare/).
Other workerd hosts are untested: the site uses D1 and R2 through `@emdash-cms/cloudflare` and the Cloudflare adapter (locally, miniflare stands in for both), so a host has to supply those two bindings.
## For agents
`AGENTS.md` is a symlink to this file. capsite's [working rules](../AGENTS.md) apply here too (the dev server is the user's to start and stop). EmDash skills are in `.agents/skills/`: `building-emdash-site` (start here), `creating-plugins`, `emdash-cli`.
+14 -60
View File
@@ -1,68 +1,22 @@
# EmDash Blog Template (Cloudflare) # potluck
A clean, minimal blog built with [EmDash](https://github.com/emdash-cms/emdash) and deployed on Cloudflare Workers with D1 and R2. A weekly zine of what you bookmarked elsewhere, published on your own site. Each issue is one week of links with a few words around them.
[![Deploy to Cloudflare](https://deploy.workers.cloudflare.com/button)](https://deploy.workers.cloudflare.com/?url=https://github.com/emdash-cms/templates/tree/main/blog-cloudflare) ## Getting started
![Blog template homepage](https://raw.githubusercontent.com/emdash-cms/emdash/main/assets/templates/blog/latest/homepage-light-desktop.jpg) ```sh
## What's Included
- Featured post hero on the homepage
- Post archive with reading time estimates
- Category and tag archives
- Full-text search
- RSS feed
- SEO metadata and JSON-LD
- Dark/light mode
## Pages
| Page | Route |
|---|---|
| Homepage | `/` |
| All posts | `/posts` |
| Single post | `/posts/:slug` |
| Category archive | `/category/:slug` |
| Tag archive | `/tag/:slug` |
| Search | `/search` |
| Static pages | `/pages/:slug` |
| 404 | fallback |
## Screenshots
| | Desktop | Mobile |
|---|---|---|
| Light | ![homepage light desktop](https://raw.githubusercontent.com/emdash-cms/emdash/main/assets/templates/blog/latest/homepage-light-desktop.jpg) | ![homepage light mobile](https://raw.githubusercontent.com/emdash-cms/emdash/main/assets/templates/blog/latest/homepage-light-mobile.jpg) |
| Dark | ![homepage dark desktop](https://raw.githubusercontent.com/emdash-cms/emdash/main/assets/templates/blog/latest/homepage-dark-desktop.jpg) | ![homepage dark mobile](https://raw.githubusercontent.com/emdash-cms/emdash/main/assets/templates/blog/latest/homepage-dark-mobile.jpg) |
## Infrastructure
- **Runtime:** Cloudflare Workers
- **Database:** D1
- **Storage:** R2
- **Framework:** Astro with `@astrojs/cloudflare`
## Local Development
```bash
pnpm install pnpm install
pnpm dev pnpm dev # open http://localhost:4321/_emdash/admin, finish the setup wizard, then Ctrl-C
pnpm seed:issues # three sample issues, W38–W40 of 2026
pnpm dev # → http://localhost:4321/issues
``` ```
Open http://localhost:4321/_emdash/admin and complete the setup wizard. EmDash runs database migrations and applies the blog seed during setup. The site is available at http://localhost:4321. ## Bookmarks in, a zine out
## Deploying - **Your web, your site:** the bookmarks come from your own diigo account, so the web you read becomes something you publish ([PESOS](https://indieweb.org/PESOS)). The [diigo-week](../diigo-week/README.md) plugin drafts an issue from a week of them; you cut and comment.
- **A week per issue:** `/issues/2026-w40`, by ISO week, UTC.
- **Microformats:** feed readers and IndieWeb tools see each issue as an [h-entry](https://microformats.org/wiki/h-entry) and each bookmark as an [h-cite](https://microformats.org/wiki/h-cite).
- **Plain HTML** that's easy to read: system fonts, one column.
- **A real editor:** it's an [EmDash](https://docs.emdashcms.com/) site, so you write issues in its admin UI. It runs in [workerd](https://github.com/cloudflare/workerd), on your machine or a host.
```bash How it's put together, checks, and deploying: [CONTRIBUTING](CONTRIBUTING.md).
pnpm wrangler login
pnpm deploy
```
The first deployment provisions the named D1 database and R2 bucket from `wrangler.jsonc`. See [Deploy to Cloudflare](https://docs.emdashcms.com/deployment/cloudflare/) for production setup, or use the deploy button above.
## See Also
- [Node.js variant](../blog) -- same template using SQLite and local file storage
- [All templates](../)
- [EmDash documentation](https://docs.emdashcms.com/)
+2
View File
@@ -16,6 +16,8 @@ export default defineConfig({
emdash({ emdash({
database: d1({ binding: "DB", session: "auto" }), database: d1({ binding: "DB", session: "auto" }),
storage: r2({ binding: "MEDIA" }), storage: r2({ binding: "MEDIA" }),
// drafts an issue from a week of diigo bookmarks (../diigo-week)
plugins: [{ id: "diigo-week", version: "0.1.0", entrypoint: "diigo-week" }],
}), }),
], ],
devToolbar: { enabled: false }, devToolbar: { enabled: false },
+2
View File
@@ -9,6 +9,7 @@
}, },
"scripts": { "scripts": {
"dev": "astro dev", "dev": "astro dev",
"seed:issues": "emdash seed seed/issues.json -d \"$(find .wrangler/state/v3/d1 -name '*.sqlite' ! -name metadata.sqlite)\"",
"build": "astro build", "build": "astro build",
"preview": "astro preview", "preview": "astro preview",
"deploy": "astro build && wrangler deploy", "deploy": "astro build && wrangler deploy",
@@ -19,6 +20,7 @@
"@astrojs/react": "^6.0.5", "@astrojs/react": "^6.0.5",
"@emdash-cms/cloudflare": "^1.0.1", "@emdash-cms/cloudflare": "^1.0.1",
"astro": "^7.3.2", "astro": "^7.3.2",
"diigo-week": "link:../diigo-week",
"emdash": "^1.0.1", "emdash": "^1.0.1",
"react": "19.2.4", "react": "19.2.4",
"react-dom": "19.2.4" "react-dom": "19.2.4"
+3
View File
@@ -20,6 +20,9 @@ importers:
astro: astro:
specifier: ^7.3.2 specifier: ^7.3.2
version: 7.3.6(@types/node@26.6.4)(yaml@2.9.1) version: 7.3.6(@types/node@26.6.4)(yaml@2.9.1)
diigo-week:
specifier: link:../diigo-week
version: link:../diigo-week
emdash: emdash:
specifier: ^1.0.1 specifier: ^1.0.1
version: 1.2.0(@astrojs/react@6.0.6(@types/node@26.6.4)(@types/react-dom@19.3.0(@types/react@19.3.0))(@types/react@19.3.0)(esbuild@0.28.2)(react-dom@19.2.4(react@19.2.4))(react@19.2.4)(yaml@2.9.1))(@atcute/cbor@2.3.8(@atcute/cid@2.5.0))(@atcute/cid@2.5.0)(@atcute/identity@2.0.2(@atcute/lexicons@2.1.1)(typescript@6.0.3))(@date-fns/tz@1.5.0)(@floating-ui/dom@1.8.0)(@tiptap/extensions@3.31.4(@tiptap/core@3.31.4(@tiptap/pm@3.31.4))(@tiptap/pm@3.31.4))(@types/react-dom@19.3.0(@types/react@19.3.0))(@types/react@19.3.0)(astro@7.3.6(@types/node@26.6.4)(yaml@2.9.1))(date-fns@4.4.0)(echarts@6.1.0)(prosemirror-model@1.25.12)(prosemirror-state@1.4.4)(prosemirror-view@1.42.6)(react-dom@19.2.4(react@19.2.4))(react@19.2.4)(typescript@6.0.3) version: 1.2.0(@astrojs/react@6.0.6(@types/node@26.6.4)(@types/react-dom@19.3.0(@types/react@19.3.0))(@types/react@19.3.0)(esbuild@0.28.2)(react-dom@19.2.4(react@19.2.4))(react@19.2.4)(yaml@2.9.1))(@atcute/cbor@2.3.8(@atcute/cid@2.5.0))(@atcute/cid@2.5.0)(@atcute/identity@2.0.2(@atcute/lexicons@2.1.1)(typescript@6.0.3))(@date-fns/tz@1.5.0)(@floating-ui/dom@1.8.0)(@tiptap/extensions@3.31.4(@tiptap/core@3.31.4(@tiptap/pm@3.31.4))(@tiptap/pm@3.31.4))(@types/react-dom@19.3.0(@types/react@19.3.0))(@types/react@19.3.0)(astro@7.3.6(@types/node@26.6.4)(yaml@2.9.1))(date-fns@4.4.0)(echarts@6.1.0)(prosemirror-model@1.25.12)(prosemirror-state@1.4.4)(prosemirror-view@1.42.6)(react-dom@19.2.4(react@19.2.4))(react@19.2.4)(typescript@6.0.3)
+3
View File
@@ -16,6 +16,9 @@
"bucket_name": "my-emdash-media", "bucket_name": "my-emdash-media",
}, },
], ],
// diigo-cache's public pages, for the diigo-week plugin (reads its
// microformats; in dev, diigo-cache's own `pnpm dev` serves it)
"services": [{ "binding": "DIIGO_CACHE", "service": "diigo-cache" }],
// Sandboxed plugins require Worker Loader, available on Workers paid plans. Uncomment to enable: // Sandboxed plugins require Worker Loader, available on Workers paid plans. Uncomment to enable:
// "worker_loaders": [{ "binding": "LOADER" }], // "worker_loaders": [{ "binding": "LOADER" }],
// General maintenance cron trigger // General maintenance cron trigger
+1 -1
View File
@@ -3,7 +3,7 @@
DO-backed fluent web client factory: site-agnostic. Deploy once; instances DO-backed fluent web client factory: site-agnostic. Deploy once; instances
are attenuated handles whose **ids are bearer capabilities** — anyone are attenuated handles whose **ids are bearer capabilities** — anyone
holding the id string can drive the instance; nobody without it can. holding the id string can drive the instance; nobody without it can.
TODO: this proxy-page flow (see diigo-cache README) vs running webapp as TODO: this proxy-page flow (see [diigo-cache CONTRIBUTING](../diigo-cache/CONTRIBUTING.md)) vs running webapp as
the primary worker (`wrangler dev -c ../webapp/wrangler.jsonc -c ../diigo-cache/wrangler.jsonc`) the primary worker (`wrangler dev -c ../webapp/wrangler.jsonc -c ../diigo-cache/wrangler.jsonc`)
so its own port serves the mint form directly; reconsider when the UI so its own port serves the mint form directly; reconsider when the UI
grows past a form or two. grows past a form or two.