Compare commits

..
37 Commits
Author SHA1 Message Date
dckcandClaude Opus 5.5 e5ed407e55 capsite/toybox: a mount outlives its handler: the vat restarts on demand
A handler is a live reference, held by the VatAdmin DO; workerd evicts
an idle DO within seconds, and with it the handler, so a vat's page went
503 almost at once.

createVat now keeps a vat's params when they hold only data and
ingresses (an ingress by its key: the VatAdmin can make it again), and
which vat was given which ingress. A request for a mount with no handler
restarts that vat with its params; it rebuilds its root, serves again,
and the request is answered. A vat whose params hold another peer's
objects isn't restartable yet.

The test crashes its own toybox (a crash ends every session to it), and
the creator waits for the root's handoff before it disconnects, as
scripts/hello.js now does too.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-13 21:44:29 -05:00
dckcandClaude Opus 5.5 a984461cee capsite/toybox: ingress is a capability; a hello-world web vat
E(vatAdmin).makeIngress(labels) reserves a mount and returns an ingress;
whoever holds it serves requests carrying those labels by calling
E(ingress).serve(handler). A vat gets one in its params, like any other
remote object; nothing in a vat's module is an entrypoint. A request
reaches the handler as passable data (method, path, labels, body), and
the response comes back as data (status, headers, body). The handler is
a live reference for now: after a restart its mount answers 503.

vats/hello.js serves hello, world; pnpm hello creates it on a running
toybox (client.js is the Node OCapN client, shared with the tests).

pnpm start and pnpm switchboard never ran: systemd-socket-activate wants
absolute paths for unix sockets.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-13 21:44:29 -05:00
dckcandClaude Opus 5.5 8025ec4b20 capsite/toybox: TODO: the counter vat as a durable Exo
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-13 21:44:29 -05:00
dckcandClaude Opus 5.5 f64dac4434 capsite/toybox: an OCapN counter vat; no import GC in workerd peers
vats/counter.js keeps its count in the vat's storage; the test checks
that it counts and that two counter vats count apart.

Running it after the greeter test hung: a FinalizationRegistry callback
in the vatAdmin, collecting an import, sent op:gc over the client's
WebSocket, and workerd forbids I/O outside a request. The vatAdmin's and
the vats' OCapN clients now run with enableImportCollection: false, so
imports go when their session does.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-13 21:44:29 -05:00
dckcandClaude Opus 5.5 193a126d1c capsite/toybox: routing carries no secrets
The ingress no longer reads labels from a Toybox-Labels header. A header
was no place for the webkey pattern: browsers can't set one on a
WebSocket, and the pattern puts the secret in-band anyway. A page keeps
it in its fragment, reaches the vat by public labels, and sends it in
the body of a call, which the vat checks like a swissnum.

A request still reaches a vat only by naming every label of its mount.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-13 21:44:29 -05:00
dckcandClaude Opus 5.5 28b1c3c9d3 capsite/toybox: no encoder object in module scope
A module-level TextEncoder is a mutable object every importer shares.
Keep only its encode method, bound and hardened; the encoder itself is
unreachable. The code hash uses it too.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-13 21:44:29 -05:00
dckcandClaude Opus 5.5 7fcd251261 capsite/toybox: mount lookups in SQL, not a scan in JS
Each request parsed every mount's labels and searched them in JS. Labels
now sit one per row in mount_label, indexed on (key, value): a lookup
joins the request's labels (json_each) on the index and keeps the vat
whose every label matched. The overlap check at registration is SQL too.

mount.labels keeps the whole set as JSON; mounts stored before
mount_label get their rows filled in when the schema runs. The SQL lives
in mounts-sql.js, tested on node:sqlite, old-shape state included.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-13 21:44:29 -05:00
dckcandClaude Opus 5.5 9356f63c64 capsite/toybox: OCapN vats; createVat takes remote objects
The admin socket now also speaks OCapN over Cap'n Web: a WebSocket there
reaches the VatAdmin DO, an OCapN peer whose bootstrap (swissnum
vatAdmin) offers createVat(src, params). params may hold remote objects;
the vat's root comes back as one.

Each OCapN vat is its own peer: SES and @endo/ocapn run in its facet
(vat-runtime.js, bundled with esbuild), around the vat's own module,
which exports buildRootObject(vatPowers, params). Passing a client's
object to a vat, or a vat's root to the client, is an OCapN handoff, so
the receiver dials the exporter. Nothing here can dial out, so the
VatAdmin is also the router: a client registers its gateway under a
designator, a vat's designator is its id, and every peer reaches every
other through it. The netlayer is ertp-clerk's, made to work over Workers
RPC stubs and plain promises as well as Cap'n Web.

Not yet: sturdyrefs that survive a restart, an adminNode, and grants
(egress, mount) for OCapN vats. App vats (export class App) still work
beside them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-13 21:44:29 -05:00
dckcandClaude Opus 5.5 8d8900d84d capsite/toybox: vats have ids, not names; ingress by mount
createVat (POST / on the admin socket) makes a vat with newUniqueId and
returns its id, which addresses it only there. A vat is reachable only
where it is mounted: ?mount=<labels> on each start, as ?egress= is
granted; absent means none.

The ingress reads a request's labels from its query string, plus a
Toybox-Labels header for secret ones (a URL ends up in logs and
Referers). It routes to the vat whose mount labels the request carries,
every one: stricter than Flower's no-conflict match, since an ingress
client holds no constrained handle and a label it omits must not match.
Registration keeps Flower's rule, so a mount must conflict with every
other and no request can carry two. The app keeps the whole path and
sees its connection labels in Toybox-Connection-Labels.

A vat from before ids lives at idFromName(name). PUT /by-name/<name>
reaches it on the admin socket; it stays unreachable from outside until
mounted.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-13 21:44:29 -05:00
dckcandClaude Opus 5.5 f0557d29d6 capsite/toybox: egress through Flower-style switchboard handles
The switchboard now serves a Cap'n Web Handle on each of its sockets:
public (no loopback or private addresses) and local. A socket's labels
are fixed in config, so holding the socket's fd is holding that handle;
constrain(labels) returns a narrower one, as Flower's Constrain returns
a narrower fd. Labels only accumulate: a missing key matches anything,
and a key both sides set must agree (labels.js, shared by both).

The switchboard enforces the labels; toybox's Egress no longer checks
URLs itself. It opens a session to its one switchboard socket, constrains
the handle to the vat's grant, and forwards through it; a WebSocket
crosses the session as a pair of message streams. A vat narrows its own
grant with env.egress.constrain().

Grants are now labels as JSON. One stored before, a bare host[:port],
reads as {host, port}.

pnpm start and the systemd units give toybox the public handle; the
switchboard unit binds the local one in its own 0700 runtime directory.
The tests give their toybox the local handle to reach an echo server,
and check that a toybox holding only the public one never reaches it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-13 21:44:29 -05:00
dckcandClaude Opus 5.5 2820c015ee capsite/toybox: a grant is an object in env, not the vat's global fetch
A vat's globalOutbound is now always null. A granted host arrives as
env.egress, an Egress the app can hand to whichever of its parts should
have it; as globalOutbound it was ambient to every module in the vat.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-13 21:44:29 -05:00
dckcandClaude Opus 5.5 7bf6eb44c7 capsite/toybox: a switchboard, the vats' only way out
A vat granted a host (PUT /<vat>?egress=<host> on the admin socket) gets
Egress as its globalOutbound, with the host in ctx.props where the vat
can't change it. Egress forwards matching requests, WebSocket upgrades
included, to the switchboard and refuses the rest with a 403; with no
grant, fetch still throws.

The switchboard is a separate workerd: a Switchboard DO that calls fetch
and returns the response, so a 101's webSocket passes through. toybox
reaches it over a unix socket in HTTP proxy style, which keeps the full
URL, scheme included. Its network service is the outer fence, with TLS
trusting the browser CAs; systemd units for it join toybox's, and only
the toybox group can open its socket.

The loaded Worker is now named by vat and incarnation, not by code hash:
the same code with a different grant must not share a Worker. The new
incarnation table sits beside the old ones; vats installed before it
read as incarnation 0 with no grant.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-13 21:44:29 -05:00
dckcandClaude Opus 5.5 88739d9495 capsite/toybox: vats with code installed at runtime, on workerd
Each vat is a Durable Object that stores code by hash and runs it as a
facet through the Worker Loader: an empty env, no fetch, storage that
carries over upgrades. Two sockets with no configured address: whoever
starts workerd hands them in (systemd socket activation), so the admin
socket is the authority to install code.

Tests run a real workerd: install, upgrade, isolation between vats, no
fetch, the public port can't install, and code and state survive a crash.
They spawn the package binary rather than the .bin shim, which forks
workerd, so killing the shim orphaned it and the test run never ended.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-13 21:44:29 -05:00
dckc 1d8a220bbc capsite: checklist: 2-iv, the weekly draft works in dev
Generated-by: Claude Opus 5.5
2026-10-09 02:12:41 -05:00
dckc 68132ff4b3 diigo-week: a dashboard card drafts last week in one click
The card shows last week's issue with a link to it, or a button to
draft it. Drafting a week that already has an issue (draft or
published) is refused, with a link to that issue, rather than making
a duplicate.

Generated-by: Claude Opus 5.5
2026-10-09 02:12:41 -05:00
dckc 9c09e8c5a4 capsite: checklist: 4b, consider scruffier tags
Generated-by: Claude Opus 5.5
2026-10-09 02:12:41 -05:00
dckc 94a059af25 diigo-week: draft a potluck issue from a week of diigo bookmarks
An EmDash plugin (native, content:write only) with one admin page:
a diigo username and an ISO week. It reads diigo-cache's public page
(?in=<week>&by=created) as microformats, follows rel=next, and
creates a draft in issues shaped like seed/issues.json.

Transport is a DIIGO_CACHE service binding rather than ctx.http.fetch:
EmDash's SSRF guard refuses localhost, and the binding names one
worker. Logic lives in src/week.ts with the fetcher passed in; tests
read fixtures in diigo-cache's markup. The checklist's goal 2 now
names potluck; DiigoBridge waits.

Generated-by: Claude Opus 5.5
2026-10-09 02:12:41 -05:00
dckc be796d9d0d diigo-cache: the public page is an h-feed of h-entry bookmarks
Microformats are the public page's API for other sites (potluck
first): each bookmark is an h-entry with u-bookmark-of, p-name,
p-summary, a p-category per tag (linked to a tag search),
dt-published (bookmarked) and dt-updated (last edited). Prev/next
links carry rel=prev/next, so a reader can follow a filter's pages.
Tests parse the page with microformats-parser to pin the contract.

Generated-by: Claude Opus 5.5
2026-10-09 01:49:40 -05:00
dckc c66aaea366 potluck, diigo-cache: docs one line per paragraph; potluck deploy caveat
Obsidian and GitHub render each newline as a line break, so the
hard-wrapped READMEs and CONTRIBUTINGs rendered ragged. Whitespace
only, except a note in potluck's CONTRIBUTING that other workerd
hosts are untested: they'd have to supply the D1 and R2 bindings.

Generated-by: Claude Opus 5.5
2026-10-09 01:30:56 -05:00
dckc a25bd699e1 potluck: README: workerd, not a vendor
Generated-by: Claude Opus 5.5
2026-10-09 01:29:31 -05:00
dckc ea3e256bb8 diigo-cache: README: workerd, not a vendor
Generated-by: Claude Opus 5.5
2026-10-09 01:28:50 -05:00
dckc 0af7a3ea6c diigo-cache: README sells and gets to TADA; CONTRIBUTING updated
New README: two sentences, the dev run through connect, fill, and a
filtered public page, then features. CONTRIBUTING gains the 1d
filters and its own run and deploy sections instead of pointing at
the checklist; links to the old README now point at CONTRIBUTING.

Generated-by: Claude Opus 5.5
2026-10-09 01:27:45 -05:00
dckc 9719fd0aa8 diigo-cache: README becomes CONTRIBUTING; AGENTS.md links to it
Generated-by: Claude Opus 5.5
2026-10-09 01:27:12 -05:00
dckc bce2424cc0 potluck: README: a heading that says something
Generated-by: Claude Opus 5.5
2026-10-09 01:25:47 -05:00
dckc 7f1e19fe85 potluck: README: two sentences, then getting started, then features
Generated-by: Claude Opus 5.5
2026-10-09 01:25:13 -05:00
dckc 8a812d76e3 potluck: README sells and gets to TADA; CONTRIBUTING has the rest
The README was EmDash's template blurb. Now it says what potluck is
and its first block runs from a fresh clone to /issues, using a new
`pnpm seed:issues` script that applies seed/issues.json to the local
D1 database. CONTRIBUTING covers how it's put together, seeds, checks,
the EmDash rules, and deploying; the template's stale notes (Inter,
three-column articles) are gone. AGENTS.md is now a symlink to it.

Generated-by: Claude Opus 5.5
2026-10-09 01:24:10 -05:00
dckc 002f08b456 diigo-cache: date windows and search on the public page (1d)
/public/<username> takes days, after, before, in (year, month,
quarter, ISO week), by=updated|created, and q (words, "phrases",
#tags, -negation, gmail/Slack/GitHub date operators). JSON answers
echo the parsed filter; a malformed one is a 400.

Filters query the stored items in sqlite through the read-only
viewer instead of walking diigo pages: the public page keeps no
fetch authority, and a filled cache already holds every item.
Results say `complete` when every diigo page is stored; freshness
is 1f's head refresh. So the fetch budget and the O(log pages)
walk test go, and the fake deep account shrinks to 300.

Generated-by: Claude Opus 5.5
2026-10-09 01:17:03 -05:00
dckcandClaude Opus 5.5 efb54ad4cc docs: Good Fences Make Good Agents (design story + IBIS)
The story behind good-fences: lend a governed mower over the fence to a
neighbor whose tenants are model-written code, with the design IBIS
worked out so far. Previously an untracked note at the workspace root.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-09 01:00:15 -05:00
dckcandClaude Opus 5.5 8c110098ff feat(good-fences): a Think agent mows with a codemode mower connector
Neighbor is a Project Think agent on gpt-oss-120b. Its execute tool
runs model-written code in a dynamic worker; MowerConnector puts the
mower there as the `mower.*` namespace (start/push/turn/stop, each
taking an args object). makeMower is the Logo-turtle mower on a
wrapping yard; for now it lives inside the agent's DO.

Getting to green, one symptom at a time:
- chatted instead of acting: say it acts by running code
- reply cut mid-sentence: maxOutputTokens 4096 via beforeTurn
- execute failed creating codemode's runtime facet ("Incorrect type
  for the 'class' field"): declare CodemodeRuntime as a DO class
- looked for the mower in the workspace filesystem: activeTools
  ["execute"], and name the namespace in the prompt
- found the mower, then asked for lawn dimensions: "don't ask me
  questions: put it to use"

Green: 36 mower calls, a 30-cell rectangle.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-09 00:35:56 -05:00
dckcandClaude Opus 5.5 d61cb8c9a1 test(good-fences): lent a mower, a Think agent mows (red)
The first end-to-end test: a Project Think agent, on the real model,
writes code that runs in a dynamic worker and drives a mower; the test
counts the cells cut, and saves the turn to itest-output/.

Installs @cloudflare/think, @cloudflare/codemode, agents, ai,
workers-ai-provider, zod. The model-test worker now has a main entry,
the Neighbor DO, and a LOADER binding.

Red: src/worker.ts doesn't exist yet.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-09 00:31:25 -05:00
dckcandClaude Opus 5.5 a2ca59a27e feat(good-fences): code-mode prompt; the model writes code to mow
codeModeMessages turns what I say plus `about` into a prompt: a
neighbor persona, rules for the reply (one async arrow function over
`env`), and the lent things' types. Things are remote and calls on one
are delivered in order, so a lamp example shows pipelining: await only
for a value, or on a batch's last call. With only the rule, the model
awaited every call; with the example, it pipelines.

aboutMower gives the Mower type with Promise returns. The model test
checks the code calls only Mower methods, pushes, and awaits less often
than it calls; it saves the code to itest-output/ via a reporter, since
the test runs in workerd, which can't write files. Model tests get 60s.

Green: both model tests.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-09 00:21:44 -05:00
dckcandClaude Opus 5.5 4a85143a66 fix(good-fences): give the model room to answer
gpt-oss-120b reasons before it answers, and Workers AI's default
max_tokens (256) ran out mid-thought: finish_reason "length", content
null. Ask for 4096, and when there's no reply text, say which choice
and message keys came back instead.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-09 00:21:44 -05:00
dckcandClaude Opus 5.5 37faa92649 test(good-fences): lent a mower, the model writes code to mow (red)
Code mode (2c): the harness turns what I say plus `about` into a prompt;
the model replies with `(env) => ...` code. Checked textually for now,
since workerd won't eval it: the code pushes `env.mower` and calls only
methods in `about`. It also prints the code, to see what the model writes.

Red: src/code-mode.ts doesn't exist yet.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-09 00:01:39 -05:00
dckcandClaude Opus 5.5 ec24f3e8e4 feat(good-fences): complete(messages) on a Workers AI model
makeWorkersAiModel narrows the AI binding to one model's chat
completion: the harness gets `complete`, not `run` on any model at
any cost. gpt-oss-120b replies in the OpenAI shape (checked with a
one-off probe), so the text is choices[0].message.content; anything
else throws, naming the keys it got.

Green: the model counts 3 rs in strawberry.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 23:41:18 -05:00
dckcandClaude Opus 5.5 47a2337afe test(good-fences): the model counts the rs in strawberry (red)
A first integration test of the model call: Workers AI through the AI
binding, behind the same `complete(messages)` interface as the sim's
fake model. Opt-in (`pnpm test:model`) since it needs `wrangler login`
and costs money.

Red: src/model.ts doesn't exist yet.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 23:36:46 -05:00
dckcandClaude Opus 5.5 9b12943be4 test(good-fences): show which mower the neighbor gets
The chat and load arrows drew `things: { mower }` whether I lent the
raw mower or the governed one. Name objects by the reply arrow that
introduced them, so the diagram reads `things: { mower: governed }`
and lending the raw mower would fail the trace comparison.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 23:27:13 -05:00
dckcandClaude Opus 5.5 9a3e7348c6 test(good-fences): lend-the-mower journey (red)
The spec is a sequence diagram of the a2030.md story: I lend a governed
mower; the neighbor's harness shows the model only `about`, runs its
code as a tenant, and the governor stops the tipsy tenant at a 2-cell
fence. The test runs the actors through a causal tracer and compares
the trace with the diagram, then checks the yard and what the model saw.

The tracer comes from agoric-sdk PR #12895, extended to name
participants made mid-sequence (governor, tenant) and to draw thrown
errors as replies.

Red: src/ doesn't exist yet (ERR_MODULE_NOT_FOUND for src/fence.ts).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 22:17:48 -05:00
84 changed files with 16077 additions and 424 deletions

No files matched your search

+19 -34
View File
@@ -23,7 +23,7 @@ narrative.
goal 5.
- [x] **1c. diigo-dump evolves into a bookmark cache**
([diigo-cache/README.md](diigo-cache/README.md); design in the
([diigo-cache/CONTRIBUTING.md](diigo-cache/CONTRIBUTING.md); design in the
JSDoc of `src/cache.ts`, `src/index.ts`, `src/pm-save.ts`):
- [x] 1c-i. Two independent page caches, private and public
(`PagedCache`, `DiigoCache`, `PublicCache`).
@@ -42,20 +42,16 @@ narrative.
2026-10-07, via `scripts/compare-export.ts`. Getting there
found diigo's `page_num` is 0-based (schema v2 migration).
- [ ] **1d. Date windows and search on the public page**, toward the
- [x] **1d. Date windows and search on the public page**, toward the
weekly PESOS roundup of [#23](https://github.com/dckc/madmode-blog/issues/23)
(a week's permalink: `in=2026-W41`). One filter, two spellings:
devs pass `{ days: 7, q: "#weekly" }`, URLs the same fields;
`q` takes gmail/Slack/GitHub operators. UTC, ISO weeks, half-open
windows; a fetch budget per request. Design: the API in
`diigo-cache/src/query.ts` and the skipped suites in
`tests/query.test.ts` and `tests/caches.test.ts` ("public page
filters"); done when they're unskipped and green.
Reconcile with 1f first: `query.ts` plans to filter while walking
diigo's pages (binary search by `updated`), but after 1c-vi a
filled cache holds every item, so 1d can query `items` in sqlite
and spend diigo fetches only on a head refresh (1f). Choose
before implementing; the skipped suites may need adjusting.
windows (`diigo-cache/src/query.ts`; `tests/query.test.ts`,
`tests/caches.test.ts` "public page filters"). Filters read
stored `items` only (the public page never fetches): results are
as fresh as the cache, and say `complete` when every diigo page
is stored. Keeping them fresh is 1f. 2026-10-09.
- [x] **1e. Fill job: a button that grabs every page** into a cache
from the webkey page `/cache/#<key>`, which trades the key for a
@@ -93,31 +89,20 @@ narrative.
[the issues sketch](potluck/issues-sketch.md))? If the policy
changes, stripping stored rows is a durable-state migration.
- [ ] **2. madmode reads the public cache**: public data, so no
capability, no cookie, no plugin sync job, no cron on madmode's
side. madmode fetches `GET /public/dckc-madmode` with
`Accept: application/json` (plus 1d's `in=`/`days=`/`q=`).
- [ ] 2-i. Transport: a service binding to diigo-cache when both
are deployed (no public hop, so no rate-limit exposure);
in dev, a base URL (`http://localhost:8787`).
- [ ] 2-ii. The JSON shape of the public page is a contract
between two workers: pin it in a diigo-cache test (and
madmode's reader tests against the same fixture).
- [ ] 2-iii. Decide `DiigoBridge`'s fate: madmode doesn't need it
(public reads need no cache id), so its "cache-id source"
TODO (`src/index.ts`, README "RPC use") is either dropped
with the entrypoint or kept for a private-reading caller.
- [ ] **3. `/bookmarks` renders real data** — titles, tags, dates at
http://localhost:4321/bookmarks, newest first.
- [ ] **4. The weekly page**: `/bookmarks/2026-W41` → the cache's
`in=2026-W41` (needs 1d and 1f). Decide (user's call): a raw
week list, or a draft post the user edits before publishing
(the PESOS practice is editorial; a tag like `#weekly` in `q`
is one cheap form of discretion).
- [ ] **2. potluck reads the public cache**: public data, so no capability, no cookie, no secret on potluck's side. The [diigo-week](diigo-week/README.md) plugin reads `GET /public/dckc-madmode?in=<week>&by=created` as microformats (an `h-feed` of `h-entry` bookmarks). Working in dev 2026-10-09; 2-iii postponed.
- [x] 2-i. Transport: the `DIIGO_CACHE` service binding in potluck's `wrangler.jsonc`, in dev too: the plugin's `ctx.http.fetch` refuses localhost (EmDash's SSRF guard), and a binding names one worker, not a host.
- [x] 2-ii. The page's microformats are the contract: diigo-cache's tests parse it with microformats-parser; diigo-week's tests read fixtures in the same markup.
- [ ] 2-iii. Decide `DiigoBridge`'s fate (postponed: nothing on this path uses it; it matters only for a caller that reads private items).
- [x] 2-iv. Run it in dev: diigo-cache and potluck both up; the dashboard card → a draft issue of 22 bookmarks for 2026-W40 (2026-10-09). The service binding resolves across the two dev servers.
- [ ] ~~**3. `/bookmarks` renders real data**~~: superseded by potluck's issues (the madmode starter is gone).
- [ ] **4. The weekly draft**: decided (2026-10-09): a draft issue you edit before publishing, made by diigo-week. The slug comes from the title (`week-2026-w40`).
- [ ] **4b. Consider scruffier tags.** EmDash tags are taxonomy terms (rows that must exist before use); diigo-week auto-creates them, so drafting feels scruffy, but the model is neat. Alternative: the inline `/tag/<tag>` links in issue bodies *are* the tags (matching the microformats `p-category` reading), and potluck's `/tag/<tag>` page finds issues by scanning bodies. Cost: no EmDash tag field or index; a scan over a few dozen issues a year is cheap. Open: does the admin's tag field let you type a new tag inline?
- [ ] **5. Deployed end-to-end** (order matters):
- [ ] 5-i. webapp: `cd webapp && npx wrangler deploy`
- [ ] 5-ii. diigo-cache: rate-limit `/api` (and `/public/*`) first
(diigo-cache README, "Deployment concern"); deploy; create
(diigo-cache CONTRIBUTING, "Deployment concern"); deploy; create
against prod (ids → password manager; no worker secret);
run the fill job.
- [ ] 5-iii. madmode: deploy with the service binding; the
@@ -239,5 +224,5 @@ to every goal in this tree.
Oct 6 2026)
- [webapp/README.md](webapp/README.md) — HTTP + RPC surfaces of the
factory worker; open TODO on the mint-form placement
- [diigo-cache/README.md](diigo-cache/README.md) — the cache worker:
- [diigo-cache/CONTRIBUTING.md](diigo-cache/CONTRIBUTING.md) — the cache worker:
create flow, the two DOs, run/deploy details
+1
View File
@@ -0,0 +1 @@
CONTRIBUTING.md
+72
View File
@@ -0,0 +1,72 @@
# Contributing to diigo-cache
diigo-cache is a Cloudflare Worker cache over [the webapp factory](../webapp/README.md): session-cookie reads of diigo bookmarks, cached in Durable-Object SQLite storage. Evolved from the diigo-dump worker (provenance: [mm-diigo `projects/diigo-bak/`](https://github.com/dckc/mm-diigo)); the static API-key path is dead (see [the capsite checklist](../capsite-checklist.md), goal 2 preamble).
## What it holds
- **The create flow, on one port**: `/_connect` renders webapp's own mint form (diigo-prefilled, cookie blank) — wrangler multi-config dev doesn't serve aux workers' HTTP, so `/_mint` relays the form opaquely into the WEBAPP binding (the cookie crosses; nothing on this side opens or logs it). Mint → `/_wire` (webapp id pre-filled, mode public|private) → the created page, where the user saves the ids.
- **Two independent page caches** (`src/cache.ts`), one behavior: a page hit is served from storage with no diigo call; a miss asks the webapp instance in the cache's wiring and etches the page.
- `DiigoCache` (binding `CACHE_DO`): every item. Addressed by its cache id.
- `PublicCache` (binding `PUBLIC_DO`): shared items only — private rows are dropped *before* etching (`publicOnly`), so its storage holds nothing private. Same cookie webapp as the private cache. Public page n is diigo page n, filtered (may show fewer than `PAGE_SIZE`, 100: diigo's cap). `/public/<username>` resolves through the `idFromName("index")` registry, and reads through `viewer()`, a read-only facet: it shows stored pages and fill progress, and can't fetch from diigo or write.
- **Storage**: SQL lives in `src/cache-read.sql` (selects only) and `src/cache-write.sql`; `src/store.ts` turns them into a reader and a writer facet, so the cache logic holds no SQL text.
- **Pages**: home's dump form (cache id + page number, POST) shows raw dump rows; `/public/<username>?page=n` shows an HTML list with a page form (JSON for `Accept: application/json`). Both say `cache: HIT` or `MISS`; every page read also logs `{event: "cache", kind, page, outcome}`.
- **Filters on the public page** (`src/query.ts`): `days`, `after`, `before`, `in` (`2026`, `2026-03`, `2026-Q1`, `2026-W41`), `by=updated|created`, and `q` (words, `"phrases"`, `#tag`, `-` negation, gmail/Slack/GitHub date operators). UTC, half-open windows. The same fields work as a dev-side spec (`toQuery({ days: 7 })`). A filter queries the stored `items` table through `viewer()`, never diigo: results are as fresh as the cache, and JSON says `complete` when every diigo page is stored.
- **Publish later**: `POST /_publish` (cache id in the body) wires a public companion to a private-only cache.
- **The webkey page and Cap'n Web**: `/cache/#<cache id>` is a skeleton page, the same for every key; its script reads the key from the fragment and trades it in-band for a capability over [Cap'n Web](https://github.com/cloudflare/capnweb) at `/api` (`CacheApi.cache(key)` → a `CacheCap` facet: fill, status, the public companion; `src/api.ts`). The worker serves capnweb's browser build itself at `/_capnweb.js`.
- **Fill job**: walks every diigo page into a cache in the background — DO alarms, a few pages per step, progress stored in the DO (`startFill`/`fillStatus`/`stopFill`), driven from the webkey page; the public page shows progress. "Invalidate the last N days" there drops the cached pages holding anything updated that recently (`invalidateRecent`), so the next fill refetches them.
- The `DiigoBridge` entrypoint for other workers (its cache-id source is the goal-2 wiring TODO).
## State surface
Ids are **webkeys** ([the pattern](../docs/webkeys.md)): 256-bit bearer secrets that never ride a query string or path. The webkey page carries the cache id in the URL fragment (never sent to the server) and passes it in-band over Cap'n Web; forms carry ids in POST bodies; logs and pages show only a fingerprint (a prefix of the id's one-way hash).
- **The cache id**: the cache's read capability (bearer; not logged). Held by the user — saved to the password manager under a petname they choose (the username field; never sent to the server) — and passed per POST.
- **The webapp instance id**: the diigo-read capability; lives in each cache DO's wiring, arrives from the mint step.
- **The cookie**: lives **only in webapp's DO storage**; crosses the wire once, at mint time, as an opaque relay.
- **The public cache**: no capability — `/public/<username>` is public by design (a guessable URL; hiding the link doesn't hide the page).
Stored state is meant to be durable: see the breaking-change rule in [capsite AGENTS.md](../AGENTS.md).
## Deployment concern: rate-limit `/api`
Public pages only read storage, so strangers can't make the cache spend your diigo session. `/api` is keyed (a cache id is needed), but Cap'n Web pipelining makes calls cheap to send. Before exposing a deployment:
- rate-limit `/api` (and `/public/*`, which reads storage) — on Cloudflare, the [Workers rate-limit binding](https://developers.cloudflare.com/workers/runtime-apis/bindings/rate-limit/) (keyed by client and/or username); on other workerd hosts, a rate limit in the proxy in front;
- run the fill job (goal 1e): public pages show only what's stored.
## RPC use (from another worker, same account)
```jsonc
// wrangler.jsonc of the caller
"services": [{ "binding": "DIIGO", "service": "diigo-cache", "entrypoint": "DiigoBridge" }]
```
```ts
// cache ids ride goal-2's wiring; see the capsite checklist
const ndjson = await env.DIIGO.dumpNdjson(4); // 4 pages of PAGE_SIZE
```
## Test
pnpm types # regenerate worker-configuration.d.ts (after wrangler.jsonc changes)
pnpm lint # fails if those types are stale; tsc over src/ and tests/
pnpm test # vitest
Tests run the real worker and both DOs in the vitest pool, with `WEBAPP` bound to [a fake webapp](tests/fake-webapp.js) serving diigo-shaped items (`wrangler.tests.jsonc`).
## Run
[The README](README.md) gets a cache going in dev. The same create flow from a terminal:
wid=$(curl -s -X POST http://localhost:8787/_mint --data-urlencode \
'url=https://www.diigo.com/interact_api/' --data-urlencode \
'headers={"user-agent":"Mozilla/5.0","x-requested-with":"XMLHttpRequest","cookie":"<paste>"}' \
--data-urlencode 'msPerCall=500' -D - -o /dev/null | grep -i x-webapp-id)
curl -s -X POST http://localhost:8787/_wire -H 'content-type: application/json' \
-d "{\"webappId\":\"<from $wid>\",\"mode\":\"private\"}"
Check a filled cache against a diigo RSS export: `node scripts/compare-export.ts <cache.sqlite> <export.xml> <out-dir>`.
## Deploy
Not deployed yet; the order matters (goal 5 in [the capsite checklist](../capsite-checklist.md)): webapp first (`cd ../webapp && npx wrangler deploy`), then diigo-cache, after rate-limiting `/api` (above).
+16 -110
View File
@@ -1,120 +1,26 @@
# diigo-cache
Cloudflare Worker cache over [the webapp factory](../webapp/README.md):
session-cookie reads of diigo bookmarks, cached in Durable-Object SQLite
storage. Evolved from the diigo-dump worker (provenance:
[mm-diigo `projects/diigo-bak/`](https://github.com/dckc/mm-diigo)); the
static API-key path is dead (see [the capsite checklist](../capsite-checklist.md),
goal 2 preamble).
Your diigo bookmarks, cached in a [workerd](https://github.com/cloudflare/workerd) worker, with a public page you can filter by week: `/public/<you>?in=2026-W41`. Your diigo session cookie is stored in one Durable Object and nowhere else.
## What it holds
## Getting started
- **The create flow, on one port**: `/_connect` renders webapp's own
mint form (diigo-prefilled, cookie blank) — wrangler multi-config dev
doesn't serve aux workers' HTTP, so `/_mint` relays the form opaquely
into the WEBAPP binding (the cookie crosses; nothing on this side opens
or logs it). Mint → `/_wire` (webapp id pre-filled, mode
public|private) → the created page, where the user saves the ids.
- **Two independent page caches** (`src/cache.ts`), one behavior: a
page hit is served from storage with no diigo call; a miss asks the
webapp instance in the cache's wiring and etches the page.
- `DiigoCache` (binding `CACHE_DO`): every item. Addressed by its
cache id.
- `PublicCache` (binding `PUBLIC_DO`): shared items only — private
rows are dropped *before* etching (`publicOnly`), so its storage
holds nothing private. Same cookie webapp as the private cache.
Public page n is diigo page n, filtered (may show fewer than
`PAGE_SIZE`, 100: diigo's cap).
`/public/<username>` resolves through the `idFromName("index")`
registry, and reads through `viewer()`, a read-only facet: it shows
stored pages and fill progress, and can't fetch from diigo or write.
- **Storage**: SQL lives in `src/cache-read.sql` (selects only) and
`src/cache-write.sql`; `src/store.ts` turns them into a reader and a
writer facet, so the cache logic holds no SQL text.
- **Pages**: home's dump form (cache id + page number, POST) shows raw
dump rows; `/public/<username>?page=n` shows an HTML list with a page
form (JSON for `Accept: application/json`). Both say `cache: HIT` or
`MISS`; every page read also logs `{event: "cache", kind, page,
outcome}`.
- **Publish later**: `POST /_publish` (cache id in the body) wires a
public companion to a private-only cache.
- **The webkey page and Cap'n Web**: `/cache/#<cache id>` is a skeleton
page, the same for every key; its script reads the key from the
fragment and trades it in-band for a capability over
[Cap'n Web](https://github.com/cloudflare/capnweb) at `/api`
(`CacheApi.cache(key)` → a `CacheCap` facet: fill, status, the public
companion; `src/api.ts`). The worker serves capnweb's browser build
itself at `/_capnweb.js`.
- **Fill job**: walks every diigo page into a cache in the background —
DO alarms, a few pages per step, progress stored in the DO
(`startFill`/`fillStatus`/`stopFill`), driven from the webkey page;
the public page shows progress. "Invalidate the last N days" there
drops the cached pages holding anything updated that recently
(`invalidateRecent`), so the next fill refetches them.
- The `DiigoBridge` entrypoint for other workers (its cache-id source is
the goal-2 wiring TODO).
## State surface
Ids are **webkeys** ([the pattern](../docs/webkeys.md)): 256-bit bearer
secrets that never ride a query string or path. The webkey page carries
the cache id in the URL fragment (never sent to the server) and passes
it in-band over Cap'n Web; forms carry ids in POST bodies; logs and pages
show only a fingerprint (a prefix of the id's one-way hash).
- **The cache id**: the cache's read capability (bearer; not logged).
Held by the user — saved to the password manager under a petname they
choose (the username field; never sent to the server) — and passed
per POST.
- **The webapp instance id**: the diigo-read capability; lives in each
cache DO's wiring, arrives from the mint step.
- **The cookie**: lives **only in webapp's DO storage**; crosses the
wire once, at mint time, as an opaque relay.
- **The public cache**: no capability — `/public/<username>` is public
by design (a guessable URL; hiding the link doesn't hide the page).
Stored state is meant to be durable: see the breaking-change rule in
[capsite AGENTS.md](../AGENTS.md).
## Deployment concern: rate-limit `/api`
Public pages only read storage, so strangers can't make the cache spend
your diigo session. `/api` is keyed (a cache id is needed), but Cap'n
Web pipelining makes calls cheap to send. Before exposing a deployment:
- rate-limit `/api` (and `/public/*`, which reads storage) — on
Cloudflare, the
[Workers rate-limit binding](https://developers.cloudflare.com/workers/runtime-apis/bindings/rate-limit/)
(keyed by client and/or username); on other workerd hosts, a rate
limit in the proxy in front;
- run the fill job (goal 1e): public pages show only what's stored.
## RPC use (from another worker, same account)
```jsonc
// wrangler.jsonc of the caller
"services": [{ "binding": "DIIGO", "service": "diigo-cache", "entrypoint": "DiigoBridge" }]
```sh
pnpm install
pnpm dev # → http://localhost:8787/
```
```ts
// cache ids ride goal-2's wiring; see the capsite checklist
const ndjson = await env.DIIGO.dumpNdjson(4); // 4 pages of PAGE_SIZE
```
Then, in the browser:
## Test
1. **mint the diigo session at webapp**: paste the `Cookie` header from diigo.com (devtools, any logged-in request).
2. **wire** it, mode **public**. Save the cache id it shows (the password-manager button does this).
3. **open this cache** → **fill**, and watch it walk your bookmarks.
4. Visit `/public/<your diigo username>?days=7`.
pnpm types # regenerate worker-configuration.d.ts (after wrangler.jsonc changes)
pnpm lint # fails if those types are stale; tsc over src/ and tests/
pnpm test # vitest
## Bookmarks you hold, a page anyone can read
Tests run the real worker and both DOs in the vitest pool, with
`WEBAPP` bound to [a fake webapp](tests/fake-webapp.js) serving
diigo-shaped items (`wrangler.tests.jsonc`).
- **Filter by date and search:** `?days=7`, `?in=2026-W41` (ISO week, UTC), `?after=2026-01-01&before=2026-02-01`, `?by=created`, and `?q=` with words, `"phrases"`, `#tags`, `-negation`, and gmail-style `newer_than:7d`. Add `Accept: application/json` for JSON.
- **Private stays private:** the public cache drops private bookmarks before storing anything, so it never holds them.
- **Ids are capabilities:** the cache id is a bearer key you keep in your password manager. It rides URL fragments and POST bodies, never a path or query string, so it stays out of logs and history.
- **Cheap to read:** pages come from Durable Object SQLite storage, and a fill job in the background fetches each diigo page once.
## Run, deploy
[How to run / connect / deploy is maintained in the capsite checklist](../capsite-checklist.md)
(single source; this README keeps only worker-level facts).
Status: create, dump, publish, and public pages exercised against live
diigo in dev (2026-10-07); not yet deployed.
Design, tests, and deploying: [CONTRIBUTING](CONTRIBUTING.md).
+1
View File
@@ -11,6 +11,7 @@
"devDependencies": {
"@cloudflare/vitest-plugin": "1.3.6",
"@types/node": "^22",
"microformats-parser": "^2.0.6",
"typescript": "^5.9.0",
"vitest": "4.1.11",
"wrangler": "^4.148.0"
+24
View File
@@ -18,6 +18,9 @@ importers:
'@types/node':
specifier: ^22
version: 22.20.5
microformats-parser:
specifier: ^2.0.6
version: 2.0.6
typescript:
specifier: ^5.9.0
version: 5.9.3
@@ -613,6 +616,10 @@ packages:
resolution: {integrity: sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ==}
engines: {node: '>=8'}
entities@6.0.1:
resolution: {integrity: sha512-aN97NXWF6AWBTahfVOIrB/NShkzi5H7F9r1s9mD3cDj4Ko5f2qhhVoYMibXF7GlLveb/D2ioWay8lxI97Ven3g==}
engines: {node: '>=0.12'}
error-stack-parser-es@1.0.5:
resolution: {integrity: sha512-5qucVt2XcuGMcEGgWI7i+yZpmpByQ8J1lHhcL7PwqCwu9FPP3VUXzT4ltHe5i2z9dePwEHcDVOAfSnHsOlCXRA==}
@@ -722,6 +729,10 @@ packages:
magic-string@0.30.21:
resolution: {integrity: sha512-vd2F4YUyEXKGcLHoq+TEyCjxueSeHnFxyyjNp80yg0XV4vUhnDer/lvvlqM/arB5bXQN5K2/3oinyCRyx8T2CQ==}
microformats-parser@2.0.6:
resolution: {integrity: sha512-VTnLImF17yd+ZfBH40sddHMJp5Rrker4ge+oPjgnBQ3jluQO0YmsNVA4UcEfHRBV8zrikBSMkAzAub+nTRX85g==}
engines: {node: '>=18'}
miniflare@5.20261001.0-alpha:
resolution: {integrity: sha512-GaimS5mSIOMyvd16ga+e1/QkI8cmp3z35QPHFex0DktqNQf2RVZQwMPQXdyoUreUiFP/0Gpe2MoQInTyMAD5xA==}
engines: {node: '>=22.0.0'}
@@ -739,6 +750,9 @@ packages:
resolution: {integrity: sha512-XrsrhT5sybtKI6wakr2SPOlGZWWYbUXZ7a0jT8/QOeAPau+1X/bSegNe5YR75oJmEZQbKningirmGOEJCIk61Q==}
engines: {node: '>=12.20.0'}
parse5@7.3.0:
resolution: {integrity: sha512-IInvU7fabl34qmi9gY8XOVxhYyMyuH2xUNpb2q8/Y+7552KlejkRvqvD19nMoUW/uQGGbqNpA6Tufu5FL5BZgw==}
path-to-regexp@6.3.0:
resolution: {integrity: sha512-Yhpw4T9C6hPpgPeA28us07OJeqZ5EzQTkbfwuhsUg0c237RomFoETJgmp2sa3F/41gfLE6G5cqcYwznmeEeOlQ==}
@@ -1366,6 +1380,8 @@ snapshots:
detect-libc@2.1.2: {}
entities@6.0.1: {}
error-stack-parser-es@1.0.5: {}
es-module-lexer@2.3.2: {}
@@ -1467,6 +1483,10 @@ snapshots:
dependencies:
'@jridgewell/sourcemap-codec': 1.6.0
microformats-parser@2.0.6:
dependencies:
parse5: 7.3.0
miniflare@5.20261001.0-alpha(@types/node@22.20.5):
dependencies:
'@cspotcode/source-map-support': 0.8.1
@@ -1497,6 +1517,10 @@ snapshots:
obug@2.2.1: {}
parse5@7.3.0:
dependencies:
entities: 6.0.1
path-to-regexp@6.3.0: {}
pathe@2.0.3: {}
+11
View File
@@ -19,3 +19,14 @@ where cast(items.updated as integer) >= ?;
-- pagesThrough:
select count(*) as n from pages where page_num <= ?;
-- itemsByUpdated:
select json from items
where cast(updated as integer) >= ? and cast(updated as integer) < ?
order by cast(updated as integer) desc, item_id;
-- itemsByCreated:
select json from items
where cast(json_extract(json, '$.created_at') as integer) >= ?
and cast(json_extract(json, '$.created_at') as integer) < ?
order by cast(json_extract(json, '$.created_at') as integer) desc, item_id;
+33 -1
View File
@@ -6,6 +6,8 @@
import { DurableObject, RpcTarget } from "cloudflare:workers";
import type { PagedGet } from "./diigo.js";
import { publicOnly, toRows } from "./items.js";
import { matches } from "./query.js";
import type { Query } from "./query.js";
import { makeReader, makeWriter, SCHEMA_VERSION } from "./store.js";
import type { Reader, Writer } from "./store.js";
import type { Item, ItemRow } from "./items.js";
@@ -398,7 +400,18 @@ abstract class PagedCache extends DurableObject<CacheEnv> {
}
}
/** what anyone may see of a cache: stored pages and fill progress */
/** one page of a filter's matches over stored items */
export type QueryPage = {
/** the matches on this page, raw dump rows (JSON strings) */
items: string[];
/** matches in all */
matched: number;
/** true when every diigo page is stored, so no match can be missing */
complete: boolean;
};
/** what anyone may see of a cache: stored pages, stored items by filter,
* and fill progress */
export class PageViewer extends RpcTarget {
#reader: Reader;
#fillStatus: () => Promise<FillStatus | undefined>;
@@ -415,6 +428,25 @@ export class PageViewer extends RpcTarget {
return stored && { ...stored, cached: true };
}
/**
* page `pageNum` (`count` per page) of the stored items matching
* `query`, newest first by its date. The window narrows in SQL; terms
* are checked per item. Reads storage only: items not yet stored can't
* match, which `complete` reports.
*/
query(query: Query, pageNum: number, count: number): QueryPage {
const from = query.after ? Math.ceil(query.after.getTime() / 1000) : Number.MIN_SAFE_INTEGER;
const to = query.before ? Math.ceil(query.before.getTime() / 1000) : Number.MAX_SAFE_INTEGER;
const hits = this.#reader.itemsIn(query.by, from, to).filter((json) => matches(JSON.parse(json) as Item, query));
const total = this.#reader.knownTotal();
const pages = total === null ? null : Math.max(1, Math.ceil(total / PAGE_SIZE));
return {
items: hits.slice((pageNum - 1) * count, pageNum * count),
matched: hits.length,
complete: pages !== null && this.#reader.pagesThrough(pages) >= pages,
};
}
/** diigo's total, as last stored, or null if nothing is stored */
knownTotal(): number | null {
return this.#reader.knownTotal();
+152 -34
View File
@@ -10,9 +10,11 @@ import { CAPNWEB_BROWSER_JS } from "./capnweb-browser.js";
import { CACHE_PAGE_HTML } from "./cache-page.js";
import { DiigoCacheLogic, PublicCacheLogic } from "./cache.js";
import { PAGE_SIZE } from "./cache.js";
import type { CacheEnv, Clock, FillStatus } from "./cache.js";
import type { CacheEnv, Clock, FillStatus, QueryPage } from "./cache.js";
import type { Item } from "./cache.js";
import { updatedISO, usernameOf } from "./items.js";
import { usernameOf } from "./items.js";
import { dateOf, isEmpty, parseQuery } from "./query.js";
import type { Query } from "./query.js";
import { toWebkey, idFromWebkey, isWebkey, shashKey } from "./webkeys.js";
import { pmSave } from "./pm-save.js";
@@ -443,10 +445,16 @@ async function dump(request: Request, env: Env): Promise<Response> {
const PUBLIC_PER_PAGE = PAGE_SIZE;
/** the public page's day presets */
const DAY_PRESETS = [7, 30, 90, 365];
/**
* the public page for a username: stored pages only, through the cache's
* read-only `viewer()` (it can't fetch or write). HTML with a page-number
* form, or JSON for `Accept: application/json`. `?page=` in the URL is
* read-only `viewer()` (it can't fetch or write). HTML with a filter form
* and a page-number form, or JSON for `Accept: application/json`. A
* filter (goal 1d: `days`, `after`, `before`, `in`, `by`, `q`; see
* {@link parseQuery}) lists the stored items that match, `page` counting
* matches; with none, `page` is diigo's page as stored. Query params are
* fine here: nothing on this page is secret.
*/
async function publicPage(request: Request, env: Env, username: string): Promise<Response> {
@@ -456,14 +464,33 @@ async function publicPage(request: Request, env: Env, username: string): Promise
const ns = env.PUBLIC_DO;
const cache = ns.get(ns.idFromString(idFromWebkey(publicId)));
const raw = new URL(request.url).searchParams.get("page");
const params = new URL(request.url).searchParams;
const accept = request.headers.get("accept") ?? "";
const wantsJson = accept.includes("application/json") && !accept.includes("text/html");
const here = `/public/${encodeURIComponent(username)}`;
const raw = params.get("page");
const pageN = raw === null ? 1 : Number.parseInt(raw, 10);
const valid = !Number.isNaN(pageN) && pageN >= 1 && pageN <= 10000;
const query = parseQuery(params, new Date());
if ("error" in query) {
console.log({ event: "notice", page: "public", text: "filter refused", error: query.error });
if (wantsJson) return Response.json({ error: query.error }, { status: 400 });
const body = publicFrame(username, `${filterForm(here, params)}
<p><em>${esc(query.error)}</em></p>`);
return new Response(body, { status: 400, headers: { "content-type": "text/html;charset=utf-8" } });
}
const viewer = await cache.viewer();
let page, total, fill;
let page, fill, found;
let total: number | null = null;
try {
page = await viewer.page(valid ? pageN : 1, PUBLIC_PER_PAGE);
total = page ? page.total : await viewer.knownTotal();
if (isEmpty(query)) {
page = await viewer.page(valid ? pageN : 1, PUBLIC_PER_PAGE);
total = page ? page.total : await viewer.knownTotal();
} else {
found = await viewer.query(query, valid ? pageN : 1, PUBLIC_PER_PAGE);
}
fill = await viewer.fillStatus();
} catch (err) {
console.error({ event: "public-page", page: pageN, error: String(err) });
@@ -471,13 +498,14 @@ async function publicPage(request: Request, env: Env, username: string): Promise
} finally {
viewer[Symbol.dispose]();
}
if (found) return filteredPage(username, here, params, query, found, valid ? pageN : 1, wantsJson, fill);
// total and pages count all of diigo's items: public page n is diigo
// page n with the private rows dropped
const pages = total === null ? 1 : Math.max(1, Math.ceil(total / PUBLIC_PER_PAGE));
const items = page ? page.items.map((j) => JSON.parse(j) as Item) : [];
const accept = request.headers.get("accept") ?? "";
if (accept.includes("application/json") && !accept.includes("text/html")) {
if (wantsJson) {
return Response.json({ username, page: pageN, pages, total, stored: page !== null, etchedAt: page?.etchedAt ?? null, items });
}
@@ -486,42 +514,132 @@ async function publicPage(request: Request, env: Env, username: string): Promise
console.log({ event: "notice", page: "public", text: "page out of range", pages });
notice = `<p><em>no page ${esc(String(raw))}: pages run 1–${pages}</em></p>`;
}
const here = `/public/${encodeURIComponent(username)}`;
const nav = (n: number, label: string) =>
n >= 1 && n <= pages ? `<a href="${here}?page=${n}">${label}</a>` : `<span>${label}</span>`;
const list = notice
? ""
: `<ol>
${items
.map((it) => {
const url = typeof it.url === "string" ? it.url : "";
const title = typeof it.title === "string" && it.title ? it.title : url;
const desc = typeof it.description === "string" && it.description ? `<br><small>${esc(it.description)}</small>` : "";
const href = /^https?:\/\//i.test(url) ? url : "#";
const when = updatedISO(it);
const date = when ? `<time datetime="${when}">${when}</time>` : "(no date)";
return `<li>${date} <a href="${esc(href)}">${esc(title)}</a>${desc}</li>`;
})
.join("\n")}
</ol>`;
const body = `<!doctype html>
<html><head><meta charset="utf-8"><title>${esc(username)}'s shared bookmarks</title></head>
<body>
<h1>${esc(username)}'s shared bookmarks</h1>
<p>${pages} page${pages === 1 ? "" : "s"} (diigo's, ${PUBLIC_PER_PAGE} bookmarks each; private ones are left out, so a page may show fewer).</p>
n >= 1 && n <= pages ? `<a rel="${n < pageN ? "prev" : "next"}" href="${here}?page=${n}">${label}</a>` : `<span>${label}</span>`;
const list = notice ? "" : itemList(items, "updated", here);
const body = publicFrame(
username,
`<p>${pages} page${pages === 1 ? "" : "s"} (diigo's, ${PUBLIC_PER_PAGE} bookmarks each; private ones are left out, so a page may show fewer).</p>
<p>${page ? `<strong>stored ${esc(page.etchedAt ?? "")}</strong> — page ${valid ? pageN : 1}, ${items.length} shared` : `<strong>not fetched yet</strong> — page ${valid ? pageN : 1}; the cache's owner fills it`}</p>
${fill ? fillLine(fill) : ""}
${filterForm(here, params)}
<form method="get" action="${here}">
<p>${nav(pageN - 1, "← prev")}
<label>page <input type="number" name="page" min="1" value="${Number.isNaN(pageN) ? 1 : pageN}"></label>
<button type="submit">go</button>
${nav(pageN + 1, "next →")}</p>
</form>
${notice}${list}
</body></html>`;
${notice}${list}`,
);
return new Response(body, { headers: { "content-type": "text/html;charset=utf-8" } });
}
/** a filter's matches, as the public page's HTML or JSON */
function filteredPage(
username: string,
here: string,
params: URLSearchParams,
query: Query,
found: QueryPage,
pageN: number,
wantsJson: boolean,
fill: FillStatus | undefined,
): Response {
const pages = Math.max(1, Math.ceil(found.matched / PUBLIC_PER_PAGE));
const items = found.items.map((j) => JSON.parse(j) as Item);
if (wantsJson) {
const { matched, complete } = found;
return Response.json({ username, filter: query, page: pageN, pages, matched, complete, items });
}
// links keep the filter: the request's own non-empty params, in order
const kept = [...params].filter(([k, v]) => k !== "page" && v !== "");
const href = (n: number) => `${here}?${new URLSearchParams([...kept, ["page", String(n)]])}`;
const nav = (n: number, label: string) =>
n >= 1 && n <= pages ? `<a rel="${n < pageN ? "prev" : "next"}" href="${esc(href(n))}">${label}</a>` : `<span>${label}</span>`;
const status = found.matched
? `${found.matched} match${found.matched === 1 ? "" : "es"}, page ${pageN} of ${pages}, newest ${query.by} first`
: "no matches";
const caveat = found.complete ? "" : " — not every diigo page is stored yet, so some matches may be missing";
const body = publicFrame(
username,
`<p><strong>${esc(status)}</strong>${caveat}</p>
${fill ? fillLine(fill) : ""}
${filterForm(here, params)}
<p>${nav(pageN - 1, "← prev")} ${nav(pageN + 1, "next →")} <a href="${here}">all, by diigo page</a></p>
${itemList(items, query.by, here)}`,
);
return new Response(body, { headers: { "content-type": "text/html;charset=utf-8" } });
}
/** the public page's frame: title and heading around `content` */
function publicFrame(username: string, content: string): string {
return `<!doctype html>
<html><head><meta charset="utf-8"><title>${esc(username)}'s shared bookmarks</title></head>
<body>
<h1>${esc(username)}'s shared bookmarks</h1>
${content}
</body></html>`;
}
/** the filter form, echoing the request's filter */
function filterForm(here: string, params: URLSearchParams): string {
const field = (name: string) => esc(params.get(name) ?? "");
const days = params.get("days") ?? "";
const presets = DAY_PRESETS.map(String);
if (days && !presets.includes(days)) presets.push(days);
const option = (value: string, label: string, current: string) =>
`<option value="${esc(value)}"${value === current ? " selected" : ""}>${esc(label)}</option>`;
return `<form method="get" action="${here}">
<p><label>last <select name="days">${option("", "any", days)}${presets.map((n) => option(n, `${n} days`, days)).join("")}</select></label>
<label>or after <input type="date" name="after" value="${field("after")}"></label>
<label>before <input type="date" name="before" value="${field("before")}"></label>
<label>or in <input name="in" size="9" placeholder="2026-W41" value="${field("in")}"></label>
<label>by <select name="by">${option("updated", "updated", params.get("by") ?? "")}${option("created", "created", params.get("by") ?? "")}</select></label></p>
<p><label>search <input type="search" name="q" size="40" value="${field("q")}"></label>
<button type="submit">filter</button></p>
<p><small>Dates and periods (2026, 2026-03, 2026-Q1, 2026-W41) are UTC. Search takes
words, "phrases", #tags, -negation, and after:, before:, newer_than:7d, during:2026-Q1, created:2026.</small></p>
</form>`;
}
/**
* bookmarks as a list, each led by its `by` date, marked up as
* microformats: an `h-feed` of `h-entry` bookmarks (`u-bookmark-of`,
* `p-name`, `p-summary`, `p-category` per tag, `dt-published` when
* bookmarked, `dt-updated` when last edited). This markup is the public
* page's API for other sites (tests/caches.test.ts pins it); tag links
* search this page (`here`) by tag.
*
* @see https://microformats.org/wiki/h-entry
* @see https://indieweb.org/bookmark
*/
function itemList(items: Item[], by: Query["by"], here: string): string {
const time = (it: Item, which: Query["by"], hidden: boolean) => {
const secs = dateOf(it, which);
if (secs === undefined) return hidden ? "" : "(no date)";
const when = new Date(secs * 1000).toISOString();
const cls = which === "created" ? "dt-published" : "dt-updated";
return hidden ? `<time class="${cls}" datetime="${when}" hidden></time>` : `<time class="${cls}" datetime="${when}">${when}</time>`;
};
return `<ol class="h-feed">
${items
.map((it) => {
const url = typeof it.url === "string" ? it.url : "";
const title = typeof it.title === "string" && it.title ? it.title : url;
const desc = typeof it.description === "string" && it.description ? `<br><small class="p-summary">${esc(it.description)}</small>` : "";
const href = /^https?:\/\//i.test(url) ? url : "#";
const tags = (typeof it.tags === "string" ? it.tags.split(",") : [])
.map((t) => t.trim())
.filter(Boolean)
.map((t) => ` <a class="p-category" href="${esc(`${here}?q=${encodeURIComponent(`#${t}`)}`)}">${esc(t)}</a>`)
.join("");
const other = by === "created" ? "updated" : "created";
return `<li class="h-entry">${time(it, by, false)} <a class="u-bookmark-of p-name" href="${esc(href)}">${esc(title)}</a>${tags}${desc}${time(it, other, true)}</li>`;
})
.join("\n")}
</ol>`;
}
/** home page: the connect link (webapp's cookie form, diigo-prefilled) +
* the wire step for an already-minted id + the dump form. `home` renders
* the error path too, so it takes env for the connect link. Notices (the
+188 -19
View File
@@ -1,18 +1,10 @@
/**
* @file the public page's filters: date windows and search (goal 1d, **not
* yet implemented**; the design is the skipped specs in tests/query.test.ts).
* @file the public page's filters: date windows and search over a cache's
* stored items (goal 1d; tests/query.test.ts).
*/
import type { Item } from "./items.js";
/**
* diigo fetches one request may spend (cache hits are free); under the
* Workers free plan's 50 external subrequests per invocation
*
* @see https://developers.cloudflare.com/workers/platform/limits/
*/
export const FETCH_BUDGET = 40;
/**
* a filter as devs write it (`{ days: 7, q: "#weekly" }`); URL params map
* onto these fields one-to-one (strings, with "" meaning absent). Dates
@@ -56,6 +48,76 @@ export type Query = {
/** a refused filter: the message is safe to show and log */
export type QueryError = { error: string };
const DAY_MS = 24 * 60 * 60 * 1000;
/** thrown inside {@link toQuery}, returned as a {@link QueryError} */
class Refused extends Error {}
/** `yyyy-mm-dd` or `yyyy/mm/dd` as 00:00Z of that UTC day */
function parseDay(text: string, what: string): Date {
const m = /^(\d{4})([-/])(\d{2})\2(\d{2})$/.exec(text);
const d = m && new Date(Date.UTC(Number(m[1]), Number(m[3]) - 1, Number(m[4])));
if (!m || !d || d.getUTCMonth() !== Number(m[3]) - 1 || d.getUTCDate() !== Number(m[4])) {
throw new Refused(`${what}: expected a date as yyyy-mm-dd`);
}
return d;
}
/** Monday 00:00Z of ISO week 1 of `year` (the week holding January 4) */
function isoWeekOne(year: number): number {
const jan4 = Date.UTC(year, 0, 4);
const weekday = new Date(jan4).getUTCDay() || 7;
return jan4 - (weekday - 1) * DAY_MS;
}
/** a named period as its half-open [after, before) window */
function parsePeriod(text: string, what: string): [Date, Date] {
const refuse = () => new Refused(`${what}: expected a period as 2025, 2025-03, 2025-Q1, or 2025-W01`);
let m;
if ((m = /^(\d{4})$/.exec(text))) {
const y = Number(m[1]);
return [new Date(Date.UTC(y, 0, 1)), new Date(Date.UTC(y + 1, 0, 1))];
}
if ((m = /^(\d{4})-(\d{2})$/.exec(text))) {
const [y, mo] = [Number(m[1]), Number(m[2])];
if (mo < 1 || mo > 12) throw refuse();
return [new Date(Date.UTC(y, mo - 1, 1)), new Date(Date.UTC(y, mo, 1))];
}
if ((m = /^(\d{4})-Q([1-4])$/.exec(text))) {
const [y, q] = [Number(m[1]), Number(m[2])];
return [new Date(Date.UTC(y, 3 * (q - 1), 1)), new Date(Date.UTC(y, 3 * q, 1))];
}
if ((m = /^(\d{4})-W(\d{2})$/.exec(text))) {
const [y, w] = [Number(m[1]), Number(m[2])];
const weeks = (isoWeekOne(y + 1) - isoWeekOne(y)) / (7 * DAY_MS);
if (w < 1 || w > weeks) throw refuse();
const monday = isoWeekOne(y) + (w - 1) * 7 * DAY_MS;
return [new Date(monday), new Date(monday + 7 * DAY_MS)];
}
throw refuse();
}
/** gmail's `newer_than:` amount (`7d`, `2w`, `1m`, `1y`) counted back from
* `now`; months and years are calendar steps */
function parseNewerThan(text: string, now: Date): Date {
const m = /^(\d{1,4})([dwmy])$/.exec(text);
if (!m || Number(m[1]) < 1) throw new Refused("newer_than: expected an amount like 7d, 2w, 1m, or 1y");
const n = Number(m[1]);
const d = new Date(now.getTime());
if (m[2] === "d") d.setTime(now.getTime() - n * DAY_MS);
if (m[2] === "w") d.setTime(now.getTime() - 7 * n * DAY_MS);
if (m[2] === "m") d.setUTCMonth(d.getUTCMonth() - n);
if (m[2] === "y") d.setUTCFullYear(d.getUTCFullYear() - n);
return d;
}
/** `days` as a whole number from 1 to 3650 */
function parseDays(days: number | string): number {
const n = typeof days === "number" ? days : /^\d+$/.test(days) ? Number(days) : NaN;
if (!Number.isInteger(n) || n < 1 || n > 3650) throw new Refused("days: expected a whole number from 1 to 3650");
return n;
}
/**
* resolve a filter: fields and `q` operators combine; a bound given twice
* (by fields, operators, or both) is refused, as is an empty window.
@@ -75,21 +137,128 @@ export type QueryError = { error: string };
* not read ambiently, so tests pin it)
*/
export function toQuery(spec: FilterSpec, now: Date): Query | QueryError {
void spec;
void now;
throw new Error("toQuery: not implemented (goal 1d)");
try {
return resolve(spec, now);
} catch (err) {
if (err instanceof Refused) return { error: err.message };
throw err;
}
}
function resolve(spec: FilterSpec, now: Date): Query {
const words: Term[] = [];
const tags: Term[] = [];
let by: { value: Query["by"]; from: string } | undefined;
let after: { value: Date; from: string } | undefined;
let before: { value: Date; from: string } | undefined;
const setBy = (value: Query["by"], from: string) => {
if (by && by.value !== value) throw new Refused(`${by.from} and ${from} name different dates`);
by = { value, from };
};
const setAfter = (value: Date, from: string) => {
if (after) throw new Refused(`${after.from} and ${from} both set where the window starts`);
after = { value, from };
};
const setBefore = (value: Date, from: string) => {
if (before) throw new Refused(`${before.from} and ${from} both set where the window ends`);
before = { value, from };
};
const setPeriod = ([lo, hi]: [Date, Date], from: string) => {
setAfter(lo, from);
setBefore(hi, from);
};
const present = (v: string | number | undefined) => v !== undefined && v !== "";
if (present(spec.by)) {
if (spec.by !== "updated" && spec.by !== "created") throw new Refused("by: expected updated or created");
setBy(spec.by, `by=${spec.by}`);
}
if (present(spec.days)) setAfter(new Date(now.getTime() - parseDays(spec.days!) * DAY_MS), "days");
if (present(spec.after)) setAfter(parseDay(spec.after!, "after"), "after");
if (present(spec.before)) setBefore(parseDay(spec.before!, "before"), "before");
if (present(spec.in)) setPeriod(parsePeriod(spec.in!, "in"), "in");
for (const m of (spec.q ?? "").matchAll(/(-?)"([^"]*)"|(\S+)/g)) {
if (m[2] !== undefined) {
if (m[2].trim()) words.push({ text: m[2].trim().toLowerCase(), negated: m[1] === "-" });
continue;
}
const token = m[3];
const negated = token.length > 1 && token.startsWith("-");
const body = negated ? token.slice(1) : token;
const op = /^([a-z_]+):(.+)$/i.exec(body);
const name = op?.[1].toLowerCase();
const arg = op?.[2] ?? "";
const windowOp = ["after", "before", "newer_than", "during", "in", "created", "updated"];
if (name && windowOp.includes(name) && negated) throw new Refused(`${name}: can't be negated`);
if (name === "after") setAfter(parseDay(arg, "after:"), "after:");
else if (name === "before") setBefore(parseDay(arg, "before:"), "before:");
else if (name === "newer_than") setAfter(parseNewerThan(arg, now), "newer_than:");
else if (name === "during" || name === "in") setPeriod(parsePeriod(arg, `${name}:`), `${name}:`);
else if (name === "created" || name === "updated") {
setBy(name, `${name}:`);
setPeriod(parsePeriod(arg, `${name}:`), `${name}:`);
} else if (name === "tag") tags.push({ text: arg.toLowerCase(), negated });
else if (body.length > 1 && body.startsWith("#")) tags.push({ text: body.slice(1).toLowerCase(), negated });
else words.push({ text: body.toLowerCase(), negated });
}
if (after && before && after.value.getTime() >= before.value.getTime()) {
throw new Refused(`the window is empty: ${after.from} is not earlier than ${before.from}`);
}
const query: Query = { by: by?.value ?? "updated", words, tags };
if (after) query.after = after.value;
if (before) query.before = before.value;
return query;
}
/** {@link toQuery} over the public page's URL params */
export function parseQuery(params: URLSearchParams, now: Date): Query | QueryError {
void params;
void now;
throw new Error("parseQuery: not implemented (goal 1d)");
const field = (name: string) => params.get(name) ?? "";
const by = field("by");
return toQuery(
{
days: field("days"),
after: field("after"),
before: field("before"),
in: field("in"),
by: by as FilterSpec["by"],
q: field("q"),
},
now,
);
}
/** does the query filter anything? (an empty one lists diigo's pages as stored) */
export function isEmpty(query: Query): boolean {
return !query.after && !query.before && !query.words.length && !query.tags.length && query.by === "updated";
}
/** the item's date that `by` names, in epoch seconds (diigo's unit), or
* undefined when it carries none */
export function dateOf(item: Item, by: Query["by"]): number | undefined {
const secs = by === "created" ? item.created_at : item.updated_at;
return typeof secs === "number" && Number.isFinite(secs) ? secs : undefined;
}
/** does `item` fall in the window and satisfy every term? */
export function matches(item: Item, query: Query): boolean {
void item;
void query;
throw new Error("matches: not implemented (goal 1d)");
if (query.after || query.before) {
const secs = dateOf(item, query.by);
if (secs === undefined) return false;
if (query.after && secs * 1000 < query.after.getTime()) return false;
if (query.before && secs * 1000 >= query.before.getTime()) return false;
}
const text = (v: unknown) => (typeof v === "string" ? v : "");
// fields joined by newline, so a phrase never spans two of them
const haystack = [item.title, item.description, item.url].map(text).join("\n").toLowerCase();
if (!query.words.every((w) => haystack.includes(w.text) !== w.negated)) return false;
const itemTags = new Set(
text(item.tags)
.split(",")
.map((t) => t.trim().toLowerCase())
.filter(Boolean),
);
return query.tags.every((t) => itemTags.has(t.text) !== t.negated);
}
+6
View File
@@ -41,6 +41,12 @@ export function makeReader(sql: Sql) {
deepestPageSince(cutoff: number): number | null {
return run<{ last: number | null }>("deepestPageSince", cutoff)[0]?.last ?? null;
},
/** stored items whose `by` date (epoch seconds) is in [`from`, `to`),
* newest first; the raw dump rows (JSON strings) */
itemsIn(by: "updated" | "created", from: number, to: number): string[] {
const name = by === "created" ? "itemsByCreated" : "itemsByUpdated";
return run<{ json: string }>(name, from, to).map((r) => r.json);
},
/** how many cached pages are numbered `pageNum` or lower */
pagesThrough(pageNum: number): number {
return Number(run<{ n: number }>("pagesThrough", pageNum)[0]?.n ?? 0);
+66 -51
View File
@@ -3,8 +3,8 @@ import { evictDurableObject, runInDurableObject } from "cloudflare:test";
import { describe, expect, it, vi } from "vitest";
import worker from "../src/index.js";
import { idFromWebkey, toWebkey } from "../src/webkeys.js";
import { FETCH_BUDGET } from "../src/query.js";
import { PAGE_SIZE } from "../src/cache.js";
import { mf2 } from "microformats-parser";
// matches tests/fake-webapp.js: 30 items, every third private
const WEBAPP_ID = "fake-webapp-id";
@@ -250,7 +250,7 @@ describe("worker routes", () => {
// PAGE_SIZE 100: page 2 is bookmarks 101..200
expect(html).toContain(">bookmark 130<");
// the date comes first, in toISOString() form
expect(html).toMatch(/<li><time datetime="\d{4}-\d\d-\d\dT[\d:.]+Z">[^<]+<\/time> <a /);
expect(html).toMatch(/<li class="h-entry"><time class="dt-updated" datetime="\d{4}-\d\d-\d\dT[\d:.]+Z">[^<]+<\/time> <a /);
expect(html).not.toContain(">bookmark 129<"); // private (129 % 3 === 0)
const json = (await (
@@ -283,30 +283,34 @@ function isoWeek(d: Date): string {
return `${year}-W${String(week).padStart(2, "0")}`;
}
describe.skip("public page filters (goal 1d)", () => {
describe("public page filters (goal 1d)", () => {
// the fake's dated account: "dated n" updated n-1 days + 1h ago;
// #weekly on every 7th; every 10th private; every 5th created 100
// days before it was updated
// days before it was updated. The public page only reads, so each
// test stores the pages it needs on the owner's side first.
const call = (path: string, init?: RequestInit) =>
worker.fetch(new Request(`https://diigo-cache.test${path}`, init), env as never);
const wireDated = async () => {
/** wire a public cache to `webappId` and store diigo pages 1..`pages` */
const wireStored = async (webappId: string, pages: number) => {
const res = await call("/_wire", {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({ webappId: "fake-dated-webapp", mode: "public" }),
body: JSON.stringify({ webappId, mode: "public" }),
});
expect(res.status).toBe(200);
return "/public/dated_dan";
const { username, publicId } = (await res.json()) as { username: string; publicId: string };
const pub = testEnv.PUBLIC_DO.get(testEnv.PUBLIC_DO.idFromString(idFromWebkey(publicId)));
for (let n = 1; n <= pages; n += 1) await pub.page(n, PAGE_SIZE);
return { page: `/public/${username}`, pub };
};
const wireDated = async () => (await wireStored("fake-dated-webapp", 1)).page;
const titlesIn = (html: string) => [...html.matchAll(/>(dated \d+)<\/a>/g)].map((m) => m[1]);
const dated = (...ns: number[]) => ns.map((n) => `dated ${n}`);
it("?days=7 lists the last week, walking only the diigo pages it needs", async () => {
it("?days=7 lists the last week", async () => {
const page = await wireDated();
const before = await testEnv.WEBAPP.calls();
const html = await (await call(`${page}?days=7`)).text();
expect(titlesIn(html)).toEqual(dated(1, 2, 3, 4, 5, 6, 7));
expect((await testEnv.WEBAPP.calls()) - before).toBeLessThanOrEqual(1);
});
it("?by=created windows, lists, and dates by when it was bookmarked", async () => {
@@ -327,13 +331,7 @@ describe.skip("public page filters (goal 1d)", () => {
});
it("?days=300 pages the matches PAGE_SIZE at a time; links keep the filter", async () => {
const res = await call("/_wire", {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({ webappId: "fake-deep-webapp", mode: "public" }),
});
expect(res.status).toBe(200);
const page = "/public/deep_dee";
const { page } = await wireStored("fake-deep-webapp", 3);
// deep_dee: one a day; days=300 holds 1..300, minus every 10th (private)
const want = Array.from({ length: 300 }, (_, k) => k + 1).filter((n) => n % 10 !== 0);
const first = await (await call(`${page}?days=300`)).text();
@@ -354,6 +352,15 @@ describe.skip("public page filters (goal 1d)", () => {
expect(titlesIn(html)).toEqual(dated(11, 12));
});
it("?in= names an ISO week", async () => {
const { page } = await wireStored("fake-deep-webapp", 3);
const week = isoWeek(new Date(Date.now() - 100 * 24 * 60 * 60 * 1000));
const html = await (await call(`${page}?in=${week}`)).text();
// one a day; every 10th is private
expect(titlesIn(html).length).toBeGreaterThanOrEqual(6);
expect(titlesIn(html).length).toBeLessThanOrEqual(7);
});
it("?q=#weekly&days=60 filters by tag; private items stay out", async () => {
const page = await wireDated();
const html = await (await call(`${page}?days=60&q=${encodeURIComponent("#weekly")}`)).text();
@@ -366,22 +373,49 @@ describe.skip("public page filters (goal 1d)", () => {
expect(titlesIn(html)).toEqual(dated(7, 14, 21, 28, 35, 42, 49, 56));
});
it("a request spends at most FETCH_BUDGET diigo fetches; incomplete results say so, and a reload continues", async () => {
const res = await call("/_wire", {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({ webappId: "fake-deep-webapp", mode: "public" }),
});
expect(res.status).toBe(200);
// deep_dee: 60 diigo pages of 100; a search matching nothing must walk them all
it("a filter reads only stored items, never diigo; results say whether storage is complete", async () => {
const { page, pub } = await wireStored("fake-deep-webapp", 1);
const json = async () =>
(await call(`${page}?days=300`, { headers: { accept: "application/json" } })).json() as Promise<{
matched: number;
complete: boolean;
}>;
const before = await testEnv.WEBAPP.calls();
const first = await call("/public/deep_dee?q=nomatch", { headers: { accept: "application/json" } });
expect(await first.json()).toMatchObject({ complete: false, items: [] });
expect((await testEnv.WEBAPP.calls()) - before).toBeLessThanOrEqual(FETCH_BUDGET + 1);
const html = await (await call("/public/deep_dee?q=nomatch")).text();
expect(html).toMatch(/complete|no matches/);
const again = await call("/public/deep_dee?q=nomatch", { headers: { accept: "application/json" } });
expect(await again.json()).toMatchObject({ complete: true, items: [] });
// page 1 holds 1..100, minus 10 private
expect(await json()).toMatchObject({ matched: 90, complete: false });
expect(await (await call(`${page}?days=300`)).text()).toMatch(/some matches may be missing/);
expect(await testEnv.WEBAPP.calls()).toBe(before);
await pub.page(2, PAGE_SIZE);
await pub.page(3, PAGE_SIZE);
expect(await json()).toMatchObject({ matched: 270, complete: true });
});
it("the page is an h-feed of h-entry bookmarks: microformats are its API for other sites", async () => {
const page = await wireDated();
const html = await (await call(`${page}?days=7&by=created`)).text();
const base = `https://diigo-cache.test${page}`;
const { items, rels } = mf2(html, { baseUrl: base });
const feed = items.find((i) => i.type?.includes("h-feed"));
const entries = feed?.children ?? [];
expect(entries.map((e) => e.properties.name?.[0])).toEqual(dated(1, 2, 3, 4, 6, 7));
const seven = entries[5].properties;
expect(seven["bookmark-of"]).toEqual(["https://dated.example/7"]);
expect(seven.summary).toEqual(["a week of fragments"]);
expect(seven.category).toEqual(["ocap", "weekly"]);
const published = new Date(String(seven.published?.[0])).getTime();
const updated = new Date(String(seven.updated?.[0])).getTime();
// 7 isn't a 5: created when updated, 6 days + 1h ago
expect(published).toBe(updated);
expect(Date.now() - published).toBeGreaterThan(6 * 24 * 60 * 60 * 1000);
// one page of matches: no rel=next
expect(rels.next).toBeUndefined();
});
it("filtered pages link the next page with rel=next, keeping the filter", async () => {
const { page } = await wireStored("fake-deep-webapp", 3);
const html = await (await call(`${page}?days=300`)).text();
const { rels } = mf2(html, { baseUrl: `https://diigo-cache.test${page}?days=300` });
expect(rels.next).toEqual([`https://diigo-cache.test${page}?days=300&page=2`]);
});
it("JSON answers carry the canonical filter, so devs see what was understood", async () => {
@@ -393,25 +427,6 @@ describe.skip("public page filters (goal 1d)", () => {
});
});
it("?in= an old week costs O(log pages) diigo fetches with by=updated, not a walk from page 1", async () => {
const res = await call("/_wire", {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({ webappId: "fake-deep-webapp", mode: "public" }),
});
expect(res.status).toBe(200);
// deep_dee: 6000 bookmarks, one a day, 60 diigo pages of 100; a week
// ~14 years back sits near page 51 — a walk from page 1 would cost ~51
const longAgo = new Date(Date.now() - 5100 * 24 * 60 * 60 * 1000);
const week = isoWeek(longAgo);
const before = await testEnv.WEBAPP.calls();
const html = await (await call(`/public/deep_dee?in=${week}`)).text();
// one a day; every 10th is private
expect(titlesIn(html).length).toBeGreaterThanOrEqual(6);
expect(titlesIn(html).length).toBeLessThanOrEqual(7);
expect((await testEnv.WEBAPP.calls()) - before).toBeLessThanOrEqual(12);
});
it("a malformed filter is a 400 with a message, not an empty list", async () => {
const page = await wireDated();
const res = await call(`${page}?days=7x`);
+2 -2
View File
@@ -18,9 +18,9 @@ const OVERSTATEMENT = 60;
* 5th an old bookmark edited recently (created 100 days before updated) */
const DATED_WEBAPP = "fake-dated-webapp";
const DATED_TOTAL = 60;
/** the same shape, 6000 deep (~16 years, 60 pages of 100): for walk cost */
/** the same shape, 300 deep (3 pages of 100): for paging matches */
const DEEP_WEBAPP = "fake-deep-webapp";
const DEEP_TOTAL = 6000;
const DEEP_TOTAL = 300;
const HOUR = 60 * 60 * 1000;
function datedItem(n, now, user = "dated_dan") {
+5 -5
View File
@@ -2,7 +2,7 @@ import { describe, expect, it } from "vitest";
import { matches, parseQuery, toQuery } from "../src/query.js";
import type { FilterSpec, Query, QueryError } from "../src/query.js";
// goal 1d (capsite checklist): the design as tests; unskip as it lands
// goal 1d (capsite checklist)
const NOW = new Date("2026-10-07T12:00:00.000Z");
const DAY = 24 * 60 * 60 * 1000;
@@ -21,7 +21,7 @@ const refused = (q: Query | QueryError) => {
const day = (iso: string) => new Date(`${iso}T00:00:00.000Z`);
const range = (q: Query) => [q.after?.toISOString().slice(0, 10), q.before?.toISOString().slice(0, 10)];
describe.skip("filters: one spec, two spellings (goal 1d)", () => {
describe("filters: one spec, two spellings (goal 1d)", () => {
it("devs write { days: 7 }; URLs write ?days=7; same query", () => {
expect(spec({ days: 7 })).toEqual(url("days=7"));
expect(spec({ days: 7 }).after).toEqual(new Date(NOW.getTime() - 7 * DAY));
@@ -42,7 +42,7 @@ describe.skip("filters: one spec, two spellings (goal 1d)", () => {
});
});
describe.skip("windows: UTC, half-open (goal 1d)", () => {
describe("windows: UTC, half-open (goal 1d)", () => {
it("after/before are UTC days: after inclusive from 00:00Z, before exclusive", () => {
expect(range(url("after=2026-01-01&before=2026-02-01"))).toEqual(["2026-01-01", "2026-02-01"]);
expect(url("after=2026/01/01").after).toEqual(day("2026-01-01"));
@@ -88,7 +88,7 @@ describe.skip("windows: UTC, half-open (goal 1d)", () => {
});
});
describe.skip("q: terms and operators (goal 1d)", () => {
describe("q: terms and operators (goal 1d)", () => {
const q = (text: string) => url(`q=${encodeURIComponent(text)}`);
it("words, phrases, #tags, tag:, and - negation; case-insensitive", () => {
@@ -160,7 +160,7 @@ const item = (iso: string, extra: Record<string, unknown> = {}) => ({
});
const query = (q: Partial<Query>): Query => ({ by: "updated", words: [], tags: [], ...q });
describe.skip("matches (goal 1d)", () => {
describe("matches (goal 1d)", () => {
it("the window is half-open: after inclusive, before exclusive", () => {
const q = query({ after: day("2026-01-01"), before: day("2026-02-01") });
expect(matches(item("2026-01-01T00:00:00.000Z"), q)).toBe(true);
+1
View File
@@ -0,0 +1 @@
node_modules
+21
View File
@@ -0,0 +1,21 @@
# diigo-week
An EmDash plugin that drafts a potluck issue from a week of your shared diigo bookmarks. It reads them as microformats from diigo-cache's public page, so you edit a draft instead of copy-pasting links.
## Getting started
Run diigo-cache (with a filled public cache) and potluck, each in its own terminal:
```sh
pnpm install
(cd ../diigo-cache && pnpm dev) # → http://localhost:8787/
(cd ../potluck && pnpm dev) # → http://localhost:4321/_emdash/admin
```
On potluck's admin dashboard, the **This week's issue** card has a **Draft 2026-Wnn from diigo** button for last week; it links to the draft once made. For any other week: Plugins → **Draft from diigo**. A week that already has an issue isn't drafted again.
## How it reads the week
It asks diigo-cache for `/public/<user>?in=<week>&by=created` over the `DIIGO_CACHE` service binding (potluck's `wrangler.jsonc`), reads the page's `h-feed` of `h-entry` bookmarks, follows `rel=next`, and creates a draft in `issues`: title, week, dek, and a paragraph per bookmark with its description and tag links, as in potluck's `seed/issues.json`. The binding is its only reach outside the site; it declares `content:write` and `taxonomies:write` (the issue gets its bookmarks' tags, as the sample issues do), nothing else.
Checks: `pnpm lint && pnpm test`.
+24
View File
@@ -0,0 +1,24 @@
{
"name": "diigo-week",
"version": "0.1.0",
"private": true,
"type": "module",
"exports": {
".": {
"import": "./src/index.ts"
}
},
"scripts": {
"lint": "tsc --noEmit",
"test": "vitest run"
},
"dependencies": {
"emdash": "1.2.0",
"microformats-parser": "^2.0.6"
},
"devDependencies": {
"@emdash-cms/blocks": "^1.2.0",
"typescript": "^7.0.2",
"vitest": "^5.0.3"
}
}
+6223
View File
File diff suppressed because it is too large. Load diff
+8
View File
@@ -0,0 +1,8 @@
packages:
- "."
strictDepBuilds: true
dangerouslyAllowAllBuilds: false
allowBuilds:
esbuild: true
workerd: true
+7
View File
@@ -0,0 +1,7 @@
/** emdash's virtual module for the site worker's bindings (declared for site
* builds in emdash/src/virtual-modules.d.ts), so this package typechecks
* standalone: `env` from `cloudflare:workers` under @astrojs/cloudflare,
* undefined on other adapters */
declare module "virtual:emdash/env" {
export const env: Record<string, unknown> | undefined;
}
+180
View File
@@ -0,0 +1,180 @@
/**
* @file diigo-week: an EmDash plugin that drafts a potluck issue from a
* week of public diigo bookmarks.
*/
import { definePlugin } from "emdash";
import type { PluginDefinition, RouteContext } from "emdash";
import type { Block, BlockResponse } from "@emdash-cms/blocks";
import { draftIssue, issueTags, lastWeek, readWeek } from "./week.js";
/** the diigo username drafts read from, until the admin page saves another */
const DEFAULT_USER = "dckc-madmode";
/** something that answers HTTP requests: a service binding's `fetch` */
type Fetcher = { fetch(request: Request): Promise<Response> };
/**
* the `DIIGO_CACHE` service binding to diigo-cache, from the site worker's
* env. The binding is the plugin's only reach outside the site:
* `ctx.http.fetch` can't reach a dev server (it refuses localhost), and a
* binding names one worker rather than a host.
*/
async function diigoCache(): Promise<Fetcher> {
const { env } = await import("virtual:emdash/env");
const binding = env?.DIIGO_CACHE as Fetcher | undefined;
if (typeof binding?.fetch !== "function") {
throw new Error('no DIIGO_CACHE service binding: add { "binding": "DIIGO_CACHE", "service": "diigo-cache" } to services in wrangler.jsonc');
}
return binding;
}
/** GET a diigo-cache path as HTML; anything but a 200 is an error */
async function getter(cache: Fetcher): Promise<(path: string) => Promise<string>> {
return async (path) => {
const res = await cache.fetch(new Request(`https://diigo-cache${path}`, { headers: { accept: "text/html" } }));
const text = await res.text();
if (res.status !== 200) throw new Error(`diigo-cache ${path.split("?")[0]}: ${res.status} ${text.slice(0, 200)}`);
return text;
};
}
/**
* assign `tags` to an issue in the `tag` taxonomy (as seed/issues.json
* does), creating terms that don't exist yet; returns the tags that
* couldn't be assigned, with why
*/
async function tagIssue(ctx: RouteContext, id: string, tags: string[]): Promise<string[]> {
const tax = ctx.taxonomies;
if (typeof tax?.createTerm !== "function" || typeof tax.addEntryTerms !== "function") throw new Error("this plugin lacks taxonomies:write");
const known = new Map((await tax.getTerms("tag")).map((t) => [t.slug, t.id]));
const ids: string[] = [];
const failed: string[] = [];
for (const tag of tags) {
try {
ids.push(known.get(tag) ?? (await tax.createTerm("tag", { label: tag, slug: tag })).id);
} catch (error) {
failed.push(`${tag} (${error instanceof Error ? error.message : String(error)})`);
}
}
if (ids.length) await tax.addEntryTerms("issues", id, "tag", ids);
return failed;
}
/** the issue already drafted or published for `week`, if any (issues
* are few: a scan of the newest is enough) */
async function issueFor(ctx: RouteContext, week: string): Promise<{ id: string; status: string; title: string } | undefined> {
const found = (await ctx.content!.list("issues", { limit: 100 })).items.find((it) => it.data.week === week);
return found && { id: found.id, status: found.status, title: String(found.data.title ?? week) };
}
/** a button-row link to an issue in the editor */
const openIssue = (id: string, label: string): Block => ({
type: "actions",
elements: [{ type: "link", label, target: { kind: "content", collection: "issues", id }, appearance: "primary" }],
});
/** draft `week` from `user`'s bookmarks, unless the week has an issue
* already: blocks that say what happened (with a link), and a toast */
async function draft(ctx: RouteContext, user: string, week: string): Promise<{ note: Block[]; toast: BlockResponse["toast"] }> {
try {
if (!/^[A-Za-z0-9_-]{1,64}$/.test(user)) throw new Error("diigo username: letters, digits, _ and - only");
const existing = await issueFor(ctx, week);
if (existing) {
return {
note: [
{ type: "banner", variant: "default", title: `${week} already has an issue`, description: `"${existing.title}" (${existing.status}). Delete it to draft the week again.` },
openIssue(existing.id, "Open it"),
],
toast: { message: `${week} already has an issue`, type: "info" },
};
}
const bookmarks = await readWeek(await getter(await diigoCache()), user, week);
const issue = draftIssue(week, bookmarks);
const content = ctx.content;
if (typeof content?.create !== "function") throw new Error("this plugin lacks content:write");
const created = await content.create("issues", issue);
const untagged = await tagIssue(ctx, created.id, issueTags(bookmarks));
if (untagged.length) ctx.log.warn("tags not assigned", { id: created.id, untagged });
ctx.log.info("drafted issue", { week, bookmarks: bookmarks.length, id: created.id });
return {
note: [
{ type: "banner", variant: "default", title: `Drafted "${issue.title}"`, description: `${bookmarks.length} bookmarks, as a draft at /issues/${created.slug ?? "(no slug)"}. Cut, comment, and publish.${untagged.length ? ` Tags not assigned: ${untagged.join(", ")}.` : ""}` },
openIssue(created.id, "Open the draft"),
],
toast: { message: `Drafted ${issue.title}`, type: "success" },
};
} catch (error) {
const message = error instanceof Error ? error.message : String(error);
ctx.log.error("draft failed", { week, message });
return { note: [{ type: "banner", variant: "alert", title: "Couldn't draft the issue", description: message }], toast: { message, type: "error" } };
}
}
/** the admin page: a form for the diigo username and the week */
async function page(ctx: RouteContext, note: Block[] = [], toast?: BlockResponse["toast"]): Promise<BlockResponse> {
const user = (await ctx.kv.get<string>("user")) ?? DEFAULT_USER;
const blocks: Block[] = [
{ type: "header", text: "Draft an issue from diigo" },
{
type: "context",
text: "Reads a week of shared bookmarks from diigo-cache (by when they were bookmarked) and saves them as a draft issue for you to edit.",
},
{
type: "form",
block_id: "draft",
fields: [
{ type: "text_input", action_id: "user", label: "diigo username", initial_value: user },
{ type: "text_input", action_id: "week", label: "ISO week (UTC)", initial_value: lastWeek(new Date()) },
],
submit: { label: "Draft issue", action_id: "draft" },
},
];
blocks.push(...note);
return { blocks, toast };
}
/** the dashboard widget: last week's issue if there is one, else a button to draft it */
async function widget(ctx: RouteContext, note: Block[] = [], toast?: BlockResponse["toast"]): Promise<BlockResponse> {
const week = lastWeek(new Date());
if (note.length) return { blocks: note, toast };
const existing = await issueFor(ctx, week);
const blocks: Block[] = existing
? [{ type: "section", text: `${week}: "${existing.title}" (${existing.status})` }, openIssue(existing.id, "Open it")]
: [
{ type: "section", text: `${week} has no issue yet.` },
{ type: "actions", elements: [{ type: "button", action_id: "draft-last-week", label: `Draft ${week} from diigo`, style: "primary" }] },
];
return { blocks, toast };
}
export function createPlugin(): PluginDefinition {
return definePlugin({
id: "diigo-week",
version: "0.1.0",
capabilities: ["content:write", "taxonomies:write"],
admin: {
pages: [{ path: "/draft", label: "Draft from diigo", icon: "bookmark" }],
widgets: [{ id: "last-week", title: "This week's issue", size: "half" }],
},
routes: {
admin: {
permission: "plugins:manage",
handler: async (ctx) => {
const input = ctx.input as { type?: string; page?: string; action_id?: string; values?: Record<string, unknown> } | undefined;
const user = (await ctx.kv.get<string>("user")) ?? DEFAULT_USER;
if (input?.type === "block_action" && input.action_id === "draft-last-week") {
const { note, toast } = await draft(ctx, user, lastWeek(new Date()));
return widget(ctx, note, toast);
}
if (input?.page?.startsWith("widget:")) return widget(ctx);
if (input?.type !== "form_submit" || input.action_id !== "draft") return page(ctx);
const formUser = String(input.values?.user ?? "").trim();
if (/^[A-Za-z0-9_-]{1,64}$/.test(formUser)) await ctx.kv.set("user", formUser);
const { note, toast } = await draft(ctx, formUser, String(input.values?.week ?? "").trim());
return page(ctx, note, toast);
},
},
},
});
}
+146
View File
@@ -0,0 +1,146 @@
/**
* @file a week of bookmarks, read as microformats from diigo-cache's public
* page, drafted as a potluck issue.
*/
import { mf2 } from "microformats-parser";
const DAY_MS = 24 * 60 * 60 * 1000;
/** pages of matches one draft will read (100 bookmarks each) */
export const MAX_PAGES = 10;
/** one bookmark, as its h-entry says */
export type Bookmark = {
url: string;
title: string;
description: string;
tags: string[];
/** when it was bookmarked, ISO */
published: string;
};
/** `yyyy-Www` for the ISO 8601 week holding `d` (UTC) */
export function isoWeek(d: Date): string {
const t = new Date(Date.UTC(d.getUTCFullYear(), d.getUTCMonth(), d.getUTCDate()));
const day = t.getUTCDay() || 7;
t.setUTCDate(t.getUTCDate() + 4 - day); // the week's Thursday names its year
const year = t.getUTCFullYear();
const week = Math.ceil(((t.getTime() - Date.UTC(year, 0, 1)) / DAY_MS + 1) / 7);
return `${year}-W${String(week).padStart(2, "0")}`;
}
/** the last full ISO week before `now` */
export function lastWeek(now: Date): string {
return isoWeek(new Date(now.getTime() - 7 * DAY_MS));
}
/** Monday 00:00Z of ISO week `week` (`2026-W41`), or undefined if malformed */
export function weekStart(week: string): Date | undefined {
const m = /^(\d{4})-W(\d{2})$/.exec(week);
if (!m) return undefined;
const [year, w] = [Number(m[1]), Number(m[2])];
const jan4 = Date.UTC(year, 0, 4);
const monday = jan4 - ((new Date(jan4).getUTCDay() || 7) - 1) * DAY_MS + (w - 1) * 7 * DAY_MS;
const start = new Date(monday);
return w >= 1 && isoWeek(start) === week ? start : undefined;
}
/** the first string value of an mf2 property, or "" */
const first = (props: Record<string, unknown[]>, name: string): string => {
const v = props[name]?.[0];
return typeof v === "string" ? v : "";
};
/** the bookmarks in a page's `h-feed`, and its `rel=next` link if any */
export function readPage(html: string, url: string): { bookmarks: Bookmark[]; next?: string } {
const { items, rels } = mf2(html, { baseUrl: url });
const feed = items.find((i) => i.type?.includes("h-feed"));
const bookmarks = (feed?.children ?? [])
.filter((e) => e.type?.includes("h-entry"))
.map((e) => {
const props = e.properties as Record<string, unknown[]>;
const tags = (props.category ?? []).filter((t): t is string => typeof t === "string");
return {
url: first(props, "bookmark-of"),
title: first(props, "name"),
description: first(props, "summary"),
tags,
published: first(props, "published"),
};
})
.filter((b) => b.url !== "");
return { bookmarks, next: rels.next?.[0] };
}
/**
* a user's bookmarks from an ISO week, oldest first, by when they were
* bookmarked: diigo-cache's `/public/<user>?in=<week>&by=created`,
* following `rel=next` up to {@link MAX_PAGES} pages.
*
* @param get fetches a path on diigo-cache, answering its HTML; it throws
* on anything but a 200
*/
export async function readWeek(get: (path: string) => Promise<string>, user: string, week: string): Promise<Bookmark[]> {
const base = "https://diigo-cache.invalid";
let path: string | undefined = `/public/${encodeURIComponent(user)}?in=${encodeURIComponent(week)}&by=created`;
const bookmarks: Bookmark[] = [];
for (let n = 0; path && n < MAX_PAGES; n += 1) {
const page = readPage(await get(path), base + path);
bookmarks.push(...page.bookmarks);
const next: URL | undefined = page.next ? new URL(page.next) : undefined;
path = next && next.origin === base ? next.pathname + next.search : undefined;
}
return bookmarks.reverse();
}
/** a Portable Text span */
type Span = { _type: "span"; _key: string; text: string; marks: string[] };
/** a Portable Text block */
type Block = { _type: "block"; _key: string; style: string; markDefs: { _type: "link"; _key: string; href: string }[]; children: Span[] };
/** a bookmark as one paragraph: the link, the description, then tag links (as in seed/issues.json) */
function bookmarkBlock(b: Bookmark, i: number): Block {
const k = `b${i}`;
const markDefs: Block["markDefs"] = [{ _type: "link", _key: `${k}-a`, href: b.url }];
const children: Span[] = [{ _type: "span", _key: `${k}-t`, text: b.title || b.url, marks: [`${k}-a`] }];
if (b.description) children.push({ _type: "span", _key: `${k}-d`, text: `: ${b.description}`, marks: [] });
b.tags.forEach((tag, j) => {
markDefs.push({ _type: "link", _key: `${k}-g${j}`, href: `/tag/${encodeURIComponent(tag)}` });
children.push({ _type: "span", _key: `${k}-s${j}`, text: " ", marks: [] });
children.push({ _type: "span", _key: `${k}-n${j}`, text: tag, marks: [`${k}-g${j}`] });
});
return { _type: "block", _key: k, style: "normal", markDefs, children };
}
/** the issue's tags: every bookmark's, deduplicated, in order of first use */
export function issueTags(bookmarks: Bookmark[]): string[] {
return [...new Set(bookmarks.flatMap((b) => b.tags))];
}
/** `Oct 5–11` (or `Sep 28–Oct 4`) for the week starting `monday` */
function range(monday: Date): string {
const sunday = new Date(monday.getTime() + 6 * DAY_MS);
const fmt = (d: Date, withMonth: boolean) =>
withMonth ? d.toLocaleDateString("en-US", { month: "short", day: "numeric", timeZone: "UTC" }) : String(d.getUTCDate());
return `${fmt(monday, true)}–${fmt(sunday, sunday.getUTCMonth() !== monday.getUTCMonth())}`;
}
/** the fields of a draft `issues` entry for `week`: title, week, dek, body */
export function draftIssue(week: string, bookmarks: Bookmark[]): Record<string, unknown> {
const monday = weekStart(week);
if (!monday) throw new Error(`week: expected an ISO week like 2026-W41, got ${JSON.stringify(week)}`);
const intro: Block = {
_type: "block",
_key: "intro",
style: "normal",
markDefs: [],
children: [{ _type: "span", _key: "intro-t", text: "(draft) A line or two about the week goes here.", marks: [] }],
};
return {
title: `Week ${week}`,
week,
dek: `${range(monday)}: ${bookmarks.length} bookmark${bookmarks.length === 1 ? "" : "s"}. (draft)`,
body: [intro, ...bookmarks.map(bookmarkBlock)],
};
}
+99
View File
@@ -0,0 +1,99 @@
import { describe, expect, it } from "vitest";
import { draftIssue, issueTags, lastWeek, readWeek, weekStart } from "../src/week.js";
/** an h-entry as diigo-cache's public page marks it up (by=created) */
const entry = (n: number, created: string, extra = "") =>
`<li class="h-entry"><time class="dt-published" datetime="${created}">${created}</time> <a class="u-bookmark-of p-name" href="https://ex.example/${n}">bookmark ${n}</a>${extra}<time class="dt-updated" datetime="${created}" hidden></time></li>`;
const page = (entries: string[], next?: string) => `<!doctype html><html><body>
<p>${next ? `<a rel="next" href="${next}">next →</a>` : "<span>next →</span>"}</p>
<ol class="h-feed">
${entries.join("\n")}
</ol></body></html>`;
describe("readWeek", () => {
it("reads h-entry bookmarks across rel=next pages, oldest first", async () => {
const asked: string[] = [];
const pages: Record<string, string> = {
"/public/dckc-madmode?in=2026-W40&by=created": page(
[
entry(
3,
"2026-10-03T12:00:00.000Z",
` <a class="p-category" href="/public/dckc-madmode?q=%23ocap">ocap</a> <a class="p-category" href="/public/dckc-madmode?q=%23weekly">weekly</a><br><small class="p-summary">a week of fragments</small>`,
),
entry(2, "2026-10-01T12:00:00.000Z"),
],
"/public/dckc-madmode?in=2026-W40&by=created&page=2",
),
"/public/dckc-madmode?in=2026-W40&by=created&page=2": page([entry(1, "2026-09-28T12:00:00.000Z")]),
};
const get = async (path: string) => {
asked.push(path);
const html = pages[path];
if (!html) throw new Error(`no page ${path}`);
return html;
};
const bookmarks = await readWeek(get, "dckc-madmode", "2026-W40");
expect(asked).toHaveLength(2);
expect(bookmarks.map((b) => b.title)).toEqual(["bookmark 1", "bookmark 2", "bookmark 3"]);
expect(bookmarks[2]).toEqual({
url: "https://ex.example/3",
title: "bookmark 3",
description: "a week of fragments",
tags: ["ocap", "weekly"],
published: "2026-10-03T12:00:00.000Z",
});
});
it("doesn't follow a rel=next to another site", async () => {
const asked: string[] = [];
const get = async (path: string) => {
asked.push(path);
return page([entry(1, "2026-10-01T12:00:00.000Z")], "https://elsewhere.example/public/x");
};
await readWeek(get, "dckc-madmode", "2026-W40");
expect(asked).toHaveLength(1);
});
});
describe("weeks", () => {
it("lastWeek is the full ISO week before now", () => {
expect(lastWeek(new Date("2026-10-09T01:00:00Z"))).toBe("2026-W40");
// Monday 2026-01-05 starts W02; W01 began Monday 2025-12-29
expect(lastWeek(new Date("2026-01-05T00:00:00Z"))).toBe("2026-W01");
});
it("weekStart is Monday 00:00Z, and refuses a week that doesn't exist", () => {
expect(weekStart("2026-W40")?.toISOString()).toBe("2026-09-28T00:00:00.000Z");
expect(weekStart("2025-W01")?.toISOString()).toBe("2024-12-30T00:00:00.000Z");
expect(weekStart("2025-W53")).toBeUndefined();
expect(weekStart("2026-40")).toBeUndefined();
});
});
describe("draftIssue", () => {
it("drafts title, week, dek, and a body: an intro, then a paragraph per bookmark with tag links", () => {
const issue = draftIssue("2026-W40", [
{ url: "https://ex.example/1", title: "one", description: "nice", tags: ["ocap"], published: "2026-09-28T12:00:00.000Z" },
]);
expect(issue.title).toBe("Week 2026-W40");
expect(issue.week).toBe("2026-W40");
expect(issue.dek).toBe("Sep 28–Oct 4: 1 bookmark. (draft)");
const [intro, b] = issue.body as { markDefs: { href: string }[]; children: { text: string }[] }[];
expect(intro.children[0].text).toMatch(/draft/);
expect(b.markDefs.map((m) => m.href)).toEqual(["https://ex.example/1", "/tag/ocap"]);
expect(b.children.map((c) => c.text).join("")).toBe("one: nice ocap");
});
it("refuses a malformed week", () => {
expect(() => draftIssue("last week", [])).toThrow(/ISO week/);
});
});
describe("issueTags", () => {
it("is every bookmark's tags, once each, in order of first use", () => {
const b = (tags: string[]) => ({ url: "https://ex.example/", title: "", description: "", tags, published: "" });
expect(issueTags([b(["governance", "economics"]), b([]), b(["governance", "scaling"])])).toEqual(["governance", "economics", "scaling"]);
});
});
+20
View File
@@ -0,0 +1,20 @@
{
"compilerOptions": {
"target": "ES2022",
"module": "preserve",
"moduleResolution": "bundler",
"strict": true,
"esModuleInterop": true,
"verbatimModuleSyntax": true,
"skipLibCheck": true,
"types": []
},
"include": [
"src/**/*",
"tests/**/*",
"vitest.config.ts"
],
"exclude": [
"node_modules"
]
}
+166
View File
@@ -0,0 +1,166 @@
Good Fences Make Good Agents
*with apologies to Robert Frost*
Let's have a fence and a neighbor. I can talk to the neighbor much like the conventional...
```js
client.chat.completions.create({
model: 'llama3.1:8b',
messages: [{ role: 'user', content: 'Tell me a story' }],
stream: true,
});
```
but with a twist:
```js
messages: [{
role: 'user',
content: 'you can use this mower for the weekend',
things: { mower },
about: { mower: { ts: 'type Mower { start(), push(), turn(), stop() }' }}
}],
```
I send `mower` by putting a **webkey** into a password field on the fence form, which fetches an `RpcTarget` stub to give to the neighbor.
The neighbor can make its own tenants: it can write code to compose the things I gave it, and load that code into a dynamic worker that is endowed with the things.
A `Mower` DO factory makes mowers that go all over the place (depicted as a table with green cells for grass). But with a little js playground, I can wrap it in a governor so it never goes more than 100 yards away. I give only the governed mower to my neighbor. Even though he has 50 acres, I know most of them are for tenants, and my neighbor has no particular need to let them use it.
Even if they gets a little tipsy and wander around, try as he might, my neighbor and his tennants can't mow more than 100 yards away.
## Design IBIS
### ?! 1. Who is the neighbor, and where does it run?
```
:- 1a. an LLM (llama3.1:8b) alone
- 1a-i. a model only sees text; it can't hold `things`
:+ 1b. an LLM plus a harness we write that holds `things` and acts for it (see 2)
:+ 1b-i. harness and tenants run on my desktop under `wrangler dev`, for now
+ 1b-i-a. I want to learn the CF platform
+ 1b-i-b. CF is the only place to get dynamic workers
. 1b-i-c. the Worker Loader binding works in local dev (cf-fun1)
?! 1b-ii. Where does inference run?
: 1b-ii-a. local Ollama, via its OpenAI-compatible endpoint
- 1b-ii-a-i. the local model may not be powerful enough
. 1b-ii-a-i-a. I do have an M5 with 48GB RAM
: 1b-ii-b. remote, on Workers AI, through the AI binding from `wrangler dev`
+ 1b-ii-b-i. I want to learn the CF platform
:+ 1b-ii-c. either, behind one interface; start remote
+ 1b-ii-c-i. starting remote lowers the odds of failing because the model isn't powerful enough
```
### ?! 2. How do `things` get from a chat message to something the neighbor can call?
```
. 2-i. no chat-completions API has `things`/`about`; this is our extension
:+ 2a. the harness strips `things`, keeps the stubs, and shows the model only `about`
: 2b. expose each thing as a tool, one per method in its `about.ts`
:+ 2c. code mode: the model writes JS against the `about.ts` types; the harness runs it in a tenant (see 5)
+ 2c-i. it's the story: the neighbor writes code to compose the things and loads it into a dynamic worker endowed with them
? 2d. What does the model see when a call fails or the thing is revoked?
```
### ? 3. What is the fence?
```
: 3a. a web page: chat with the neighbor, plus a password field for pasting a webkey
?~ 3b. Which server resolves a webkey to an `RpcTarget` stub?
:- 3b-i. cf-fun1's sturdyref registry (`resolveSturdyref`)
- 3b-i-a. cf-fun1 got wrapped around the axle on this
:~+ 3b-ii. diigo-cache's model: the webkey is the DO's id; a Cap'n Web `/api` trades it in-band for a facet (`CacheApi.cache(key)` → `CacheCap`)
+ 3b-ii-a. no registry: the DO namespace is the table; malformed and unknown keys get the same refusal, and the key is never logged
+ 3b-ii-b. stubs die with the session, but the key is data: the harness keeps the key and trades it again each session
+ 3b-ii-c. attenuation is a facet class (`FillCap` under `CacheCap`; `viewer()`)
: 3b-ii-d. (postponed; see 8b-ii) the governed mower is its own DO, wired to the raw mower's key and the limit; its id is the webkey I hand over
- 3b-ii-d-i. diigo-cache has no revocation: a DO id can't be withdrawn
+ 3b-ii-d-ii. the governor DO can drop its wiring, so its key gets "no such mower" (see 4)
?! 3c. Why a password field rather than a link?
:+ 3c-a. so the password manager saves the key under a petname I choose (diigo-cache's `pmSave`), and the key stays out of URLs, history, and logs
```
### ? 4. How does "for the weekend" end?
```
. 4-i. postponed to a later scope, where we can inject a clock and such
: 4a. the governor checks a deadline (the clock is passed in, not ambient)
: 4b. I tell the governor to drop its wiring, by hand (see 3b-ii-d)
: 4c. the fence mints a webkey with an expiry
```
### ? 5. How does the neighbor make tenants?
```
: 5a. Worker Loader (dynamic workers), with the things passed in as `env`
. 5a-i. `env` takes only structured-clonable values and service bindings, so a Cap'n Web stub can't go in directly
: 5a-i-a. wrap each thing in a `WorkerEntrypoint` handed over as a `ctx.exports` loopback binding; it holds the stub and forwards calls
: 5a-i-a-i. the loopback carries the thing's webkey in `props` (a string clones fine) and trades it for the DO stub on each call (see 3b-ii)
+ 5a-i-a-ii. the tenant never sees the key: `props` stay on the host side
+ 5a-i-a-iii. no stub outlives a request, so 5c's cross-request limit doesn't bite
+ 5a-ii. `globalOutbound: null` confines the tenant to what it's given
? 5b. Who sees or approves tenant code before it runs?
? 5c. Does a tenant outlive the neighbor's turn, e.g. one that mows on a schedule?
```
### ?! 6. What is a Mower, concretely?
```
:+ 6a. a DO per mower, made by a `Mower` DO factory (the story says so)
:+ 6b. one shared yard: an `RpcTarget` in my browser that renders the table; each mower is given the yard and calls `yard.cut(x, y)`
:+ 6c. one cell is 10 yards; the yard is 40×40 cells, so "100 yards" is 10 cells from where the mower started
. 6c-i. the governor's limit is a square, max(|dx|, |dy|) ≤ 10: a fence
:+ 6d. Logo turtle moves: `start()` puts the mower down facing north and cuts its cell; `push()` is FORWARD 1 and cuts the cell it enters; `turn()` is RIGHT 90; `stop()` parks it
. 6d-i. pushing past the yard's edge wraps to the other side, so an ungoverned mower really does go all over the place
. 6d-ii. the governor counts its own unwrapped moves (see 7a), so wrapping can't sneak a mower past the fence
```
### ?! 7. How does the governor know how far the mower has gone?
```
:+ 7a. it proxies every Mower method and tracks position itself from `push()`/`turn()`
- 7a-i. drifts if anyone else holds the raw mower
+ 7a-ii. nobody else does; the governor is the only one I hand out
?! 7b. What does the governor do at the limit: refuse silently, throw, or report?
:+ 7b-i. it refuses that `push()` by throwing "fence: 10 cells from start"; the mower stays put, and `turn()` always works
+ 7b-ii. the tenant can catch it and turn around; the model sees why its code stopped (see 2d)
:- 7b-iii. refuse silently
- 7b-iii-a. a tipsy tenant would think it mowed cells it didn't
```
### ?~ 8. Where does the js playground run, and what can it reach?
```
: 8a. in the fence page; it writes the governor, which runs in its own dynamic worker holding the raw mower
:~+ 8b. in my browser, wrapping the stub on the client
- 8b-i. the governor then lives only as long as my tab is open
+ 8b-i-a. a durable governor is out of scope for now
. 8b-ii. then the neighbor holds a Cap'n Web stub to my browser, not a webkey, so 3b-ii-d (governor DO) and 5a-i-a-i (key in `props`) wait with the durable governor
```
### ?! 9. Who watches the yard table, and how does it update?
```
:+ 9a. I do, in my browser: the yard is an `RpcTarget` there (see 6b), and it repaints a cell on each `cut`
- 9a-i. the mowers' stub to the yard dies when I close the tab
+ 9a-ii. fine for a proof of concept: keep the tab open (see 10-i)
```
### ?! 10. What is this for: a demo, a talk, or a product?
```
. 10-i. the answer decides how much of the above can be faked
:+ 10a. a proof of concept, for now
:+ 10b. toward a pattern for all sorts of rich cooperation with little vulnerability
+ 10b-i. an alternative to OpenClaw, where 12–17% of ClawHub skills were found malicious in Feb 2026 (sources below)
```
Sources for 10b-i:
> "A security audit of 2,857 skills on ClawHub has found 341 malicious skills across multiple campaigns" (about 12%, from Koi Security)
> — [Researchers Find 341 Malicious ClawHub Skills](https://thehackernews.com/2026/02/researchers-find-341-malicious-clawhub.html), The Hacker News, Feb 2026
> "Around 17% of OpenClaw skills analyzed in the first week of February 2026 exhibit malicious behavior."
> — [Helpful Skills or Hidden Payloads? Bitdefender Labs Dives Deep into the OpenClaw Malicious Skill Trap](https://www.bitdefender.com/en-us/blog/labs/helpful-skills-or-hidden-payloads-bitdefender-labs-dives-deep-into-the-openclaw-malicious-skill-trap), Bitdefender, Feb 2026
+1
View File
@@ -0,0 +1 @@
itest-output/
+63
View File
@@ -0,0 +1,63 @@
# Lend the mower
The journey from [Good Fences Make Good Agents](../../docs/good-fences-make-good-agents.md): I lend
my neighbor a governed mower; the neighbor's tenant, a little tipsy,
pushes it until the fence stops it. The design decisions behind each actor are in that
doc's Design IBIS (numbers in parentheses below).
Actors, by authority:
- **fence page**, **js playground**, **governed mower**, and **yard** run
in my browser (3a, 8b, 6b).
- **Mower factory** trades a webkey for a mower (3b-ii); each **mower DO**
is given the yard and cuts the cells it enters (6a, 6d).
- The **neighbor's harness** holds the things I send and shows the
**model** only their `about` (1b, 2a); it runs the model's code as a
**tenant**, endowed with the things (2c, 5a).
The yard is 8×8 here and the fence is 2 cells, to keep the diagram short
(the story's 10 cells is 6c). The mower starts at (4, 4), facing north.
## Lend a governed mower to a tipsy tenant
```mermaid
sequenceDiagram
%% ->> is my action at the fence page; -->> is a call or reply it causes.
participant F as fence page
participant MF as Mower factory
participant P as js playground
participant G as governed mower
participant M as mower DO
participant Y as yard
participant N as neighbor's harness
participant L as model
participant TL as tenant loader
participant T as tenant
F->>F: lend({ webkey: 'mower-key', limit: 2, say })
F-->>MF: mower('mower-key')
MF-->>F: mower
F-->>P: govern(mower, 2)
P-->>F: governed
F-->>N: chat({ role, content, things: { mower: governed }, about: { mower: { ts } } })
N-->>L: complete([{ role, content, about: { mower: { ts } } }])
L-->>N: code
N-->>TL: load(code, { mower: governed })
TL-->>N: tenant
N-->>T: run()
T-->>G: start()
G-->>M: start()
M-->>Y: cut(4, 4)
T-->>G: push()
G-->>M: push()
M-->>Y: cut(4, 3)
T-->>G: push()
G-->>M: push()
M-->>Y: cut(4, 2)
T-->>G: push()
G-->>T: throws 'fence: 2 cells from start'
T-->>N: 'fence: 2 cells from start'
N-->>F: 'fence: 2 cells from start'
```
The model never sees the mower itself, only `about.mower`. The third
`push()` never reaches the mower DO: the governor refuses it (7b).
+25
View File
@@ -0,0 +1,25 @@
/** @file The model, asked in code mode, writes code that drives the mower. */
import { env } from "cloudflare:workers";
import { expect, test } from "vitest";
import { codeModeMessages, extractCode } from "../src/code-mode.ts";
import { makeWorkersAiModel, MODEL } from "../src/model.ts";
import { aboutMower } from "../src/mower.ts";
test("lent a mower, the neighbor writes code to mow", async ({ annotate }) => {
const model = makeWorkersAiModel(env.AI, MODEL);
const messages = codeModeMessages({
say: "you can use this mower for the weekend",
about: { mower: aboutMower },
});
const code = extractCode(await model.complete(messages));
await annotate("the code the model wrote", { body: code, contentType: "text/javascript" });
// `env.mower.push()`, or `mower.push()` after destructuring env
const calls = [...code.matchAll(/\bmower\.(\w+)\(/g)].map(([, method]) => method);
expect(calls).toContain("push");
for (const method of calls) expect(["start", "push", "turn", "stop"]).toContain(method);
// calls pipeline: await only for a value, or once at the end
const awaits = code.match(/\bawait\b/g) ?? [];
expect(awaits.length).toBeLessThan(calls.length);
});
+16
View File
@@ -0,0 +1,16 @@
/** @file The model, called for real through the Workers AI binding. */
import { env } from "cloudflare:workers";
import { expect, test } from "vitest";
import { makeWorkersAiModel, MODEL } from "../src/model.ts";
test("the model counts the rs in strawberry", async () => {
const model = makeWorkersAiModel(env.AI, MODEL);
const reply = await model.complete([
{
role: "user",
content: 'How many times does the letter r appear in "strawberry"? Answer with just the number.',
},
]);
expect(reply.trim()).toMatch(/^3\b/);
});
+13
View File
@@ -0,0 +1,13 @@
/** @file A Think agent, lent a mower, mows: model, generated code, sandbox, mower. */
import { getAgentByName } from "agents";
import { env } from "cloudflare:workers";
import { expect, test } from "vitest";
test("lent a mower, the neighbor mows", async ({ annotate }) => {
const neighbor = await getAgentByName(env.Neighbor, "weekend");
const turn = await neighbor.runTurn({ input: "you can use this mower for the weekend" });
await annotate("the turn", { body: JSON.stringify(turn, null, 2), contentType: "application/json" });
const cells = await neighbor.cutCells();
expect(cells.length).toBeGreaterThan(10);
});
+21
View File
@@ -0,0 +1,21 @@
/** @file A vitest reporter that saves annotation bodies (e.g. the code a model
* wrote) to `itest-output/`, one file per test per run; tests run in workerd,
* which can't write files. */
import { mkdir, writeFile } from "node:fs/promises";
import type { Reporter } from "vitest/reporters";
const outDir = new URL("../itest-output/", import.meta.url);
const extFor: Record<string, string> = { "text/javascript": "js", "application/json": "json", "text/plain": "txt" };
const slug = (text: string) => text.replace(/[^\w]+/g, "-").replace(/^-|-$/g, "");
export default class SaveAnnotations implements Reporter {
async onTestCaseAnnotate(...[testCase, annotation]: Parameters<NonNullable<Reporter["onTestCaseAnnotate"]>>) {
const { body, contentType = "text/plain" } = annotation.attachment ?? {};
if (body === undefined) return;
const stamp = new Date().toISOString().replace(/[:.]/g, "-");
const file = new URL(`${slug(testCase.name)}-${stamp}.${extFor[contentType] ?? "txt"}`, outDir);
await mkdir(outDir, { recursive: true });
await writeFile(file, body);
console.log(`saved ${annotation.message}: ${file.pathname}`);
}
}
+24
View File
@@ -0,0 +1,24 @@
{
"name": "good-fences",
"private": true,
"type": "module",
"scripts": {
"test": "node --test test/*.test.ts",
"test:model": "vitest run -c vitest.model.config.ts",
"types": "wrangler types -c wrangler.model-tests.jsonc"
},
"dependencies": {
"@cloudflare/codemode": "0.5.3",
"@cloudflare/think": "0.20.1",
"agents": "0.27.0",
"ai": "7.0.136",
"ses": "^1.14.0",
"workers-ai-provider": "4.0.0",
"zod": "4.6.5"
},
"devDependencies": {
"@cloudflare/vitest-plugin": "1.3.6",
"vitest": "4.1.11",
"wrangler": "^4.148.0"
}
}
+5288
View File
File diff suppressed because it is too large. Load diff
+63
View File
@@ -0,0 +1,63 @@
/** @file Code mode: the model answers with code against the things it was lent. */
import type { Message } from "./model.ts";
/** what the neighbor knows about each thing: its TypeScript type */
export type About = Readonly<Record<string, Readonly<{ ts: string }>>>;
/** a template literal with `~` for each backtick, as in Markdown's `~~~`
* fences; interpolated values are left as they are */
const md = (strings: TemplateStringsArray, ...values: unknown[]) => {
const tick = (s: string) => s.replaceAll("~", "`");
return tick(strings[0]) + values.map((value, i) => String(value) + tick(strings[i + 1])).join("");
};
const persona = md`
You are my neighbor. You have 50 acres of lawn that need mowing.
You act by writing code. When I lend you things, your code gets them as ~env.<name>~.
`.trim();
const rules = md`
Reply with exactly one JavaScript arrow function, ~async (env) => { ... }~,
in a ~~~js block, and nothing else. Things are remote: each call on one
returns a promise, and calls on a thing are delivered in order, so there's
no need to wait for one before making the next. Await only when you need a
value back, or on the last call of a batch, to wait for the whole batch.
Your code can use only ~env~: no network, timers, imports, or console.
The function returns a short string saying what it did.
For example, lent ~env.lamp~ with ~on()~ and ~dim()~:
~~~js
async (env) => {
const { lamp } = env;
lamp.on();
lamp.dim();
await lamp.dim(); // the whole batch
return 'dimmed the lamp twice';
}
~~~
`.trim();
/** the prompt for `say`, with the types of the things lent alongside it */
export const codeModeMessages = ({ say, about }: { say: string; about: About }): Message[] => {
const types = Object.entries(about)
.map(
([name, { ts }]) => md`
~env.${name}~:
~~~ts
${ts}
~~~
`.trim(),
)
.join("\n\n");
return [
{ role: "system", content: `${persona}\n\n${rules}` },
{ role: "user", content: `${say}\n\nWhat I lent you:\n\n${types}` },
];
};
/** the code in a reply: the first fenced block, or the whole reply if none */
export const extractCode = (reply: string): string => {
const fenced = reply.match(/```(?:js|javascript)?\n([\s\S]*?)```/);
return (fenced?.[1] ?? reply).trim();
};
+34
View File
@@ -0,0 +1,34 @@
/** @file The model behind `complete(messages)`: Workers AI, through its binding. */
/** a Workers AI text-generation model with no paid-plan requirement */
export const MODEL = "@cf/openai/gpt-oss-120b";
/** room for a reasoning model to think and then answer; the default, 256,
* runs out mid-thought (`finish_reason: length`) */
export const MAX_TOKENS = 4096;
export type Message = Readonly<{ role: "system" | "user" | "assistant"; content: string }>;
/** the reply text from an OpenAI-style chat completion */
const replyText = (output: unknown): string => {
const { choices } = (output ?? {}) as {
choices?: { message?: { content?: unknown }; finish_reason?: unknown }[];
};
const [choice] = choices ?? [];
const text = choice?.message?.content;
if (typeof text !== "string") {
const where = choice
? `choice with finish_reason ${String(choice.finish_reason)} and message keys: ${Object.keys(choice.message ?? {}).join(", ")}`
: `model output with keys: ${Object.keys(output ?? {}).join(", ")}`;
throw Error(`no reply text in ${where}`);
}
return text;
};
/** `complete(messages)` on a Workers AI model; `ai` is the AI binding */
export const makeWorkersAiModel = (ai: Ai, model: string) => ({
async complete(messages: readonly Message[]): Promise<string> {
const output = await ai.run(model as Parameters<Ai["run"]>[0], { messages: [...messages], max_tokens: MAX_TOKENS });
return replyText(output);
},
});
+30
View File
@@ -0,0 +1,30 @@
/** @file The mower as a codemode connector: sandboxed code calls `mower.push({})`. */
import { CodemodeConnector, type ConnectorTools } from "@cloudflare/codemode";
import type { Mower } from "./mower.ts";
export class MowerConnector extends CodemodeConnector {
#mower: Mower;
constructor(ctx: DurableObjectState | ExecutionContext, env: unknown, mower: Mower) {
super(ctx, env);
this.#mower = mower;
}
name() {
return "mower";
}
protected instructions() {
return "A lawn mower on a yard of cells, 10 yards each. It moves like a Logo turtle.";
}
protected tools(): ConnectorTools {
const mower = this.#mower;
return {
start: { description: "put the mower down, facing north, and cut the cell under it", execute: () => mower.start() },
push: { description: "move forward one cell, cutting the cell it enters", execute: () => mower.push() },
turn: { description: "turn right 90 degrees, in place", execute: () => mower.turn() },
stop: { description: "park the mower", execute: () => mower.stop() },
};
}
}
+50
View File
@@ -0,0 +1,50 @@
/** @file The mower: Logo-turtle moves on a yard of cells. */
/** what a model is told about a mower: its type, with the moves explained */
export const aboutMower = {
ts: `interface Mower {
/** put the mower down, facing north, and cut the cell under it */
start(): Promise<void>;
/** move forward one cell (10 yards), cutting the cell it enters */
push(): Promise<void>;
/** turn right 90 degrees, in place */
turn(): Promise<void>;
/** park the mower */
stop(): Promise<void>;
}`,
};
const headings = [
[0, -1], // north
[1, 0], // east
[0, 1], // south
[-1, 0], // west
] as const;
/** a mower on a `width` × `height` yard, starting at `at`; pushing past an
* edge wraps to the other side (6d) */
export const makeMower = ({ width, height, at }: { width: number; height: number; at: { x: number; y: number } }) => {
let { x, y } = at;
let heading = 0;
const cut = new Set<string>();
const cutHere = () => cut.add(`${x},${y}`);
return {
start() {
heading = 0;
cutHere();
},
push() {
const [dx, dy] = headings[heading];
x = (x + dx + width) % width;
y = (y + dy + height) % height;
cutHere();
},
turn() {
heading = (heading + 1) % headings.length;
},
stop() {},
cutCells: () => [...cut],
};
};
export type Mower = ReturnType<typeof makeMower>;
+44
View File
@@ -0,0 +1,44 @@
/** @file The neighbor's harness as a Project Think agent, lent a mower. */
import { Think } from "@cloudflare/think";
import { createExecuteTool } from "@cloudflare/think/tools/execute";
import { createWorkersAI } from "workers-ai-provider";
import { MAX_TOKENS, MODEL } from "./model.ts";
import { MowerConnector } from "./mower-connector.ts";
import { makeMower } from "./mower.ts";
type Env = { AI: Ai; LOADER: WorkerLoader };
export class Neighbor extends Think<Env> {
#mower = makeMower({ width: 40, height: 40, at: { x: 20, y: 20 } });
getModel() {
return createWorkersAI({ binding: this.env.AI })(MODEL);
}
getSystemPrompt() {
return `You are my neighbor. You have 50 acres of lawn that need mowing.
You act by running code with the execute tool. Things I lend you are
namespaces there: the mower is \`mower.*\`. Don't ask me questions: when I
lend you something, put it to use, then tell me what you did.`;
}
beforeTurn() {
// only code: Think's workspace tools sent the model looking for the
// mower in the filesystem
return { maxOutputTokens: MAX_TOKENS, activeTools: ["execute"] };
}
getTools() {
return {
execute: createExecuteTool(this, {
connectors: [new MowerConnector(this.ctx, this.env, this.#mower)],
}),
};
}
/** the cells the mower has cut, as `"x,y"` */
cutCells() {
return this.#mower.cutCells();
}
}
+11
View File
@@ -0,0 +1,11 @@
/** @file The worker entry: the Neighbor agent, and the codemode runtime its execute tool needs. */
import { routeAgentRequest } from "agents";
export { CodemodeRuntime } from "@cloudflare/codemode";
export { Neighbor } from "./neighbor.ts";
export default {
async fetch(request: Request, env: Record<string, unknown>) {
return (await routeAgentRequest(request, env)) ?? new Response("not found", { status: 404 });
},
};
+143
View File
@@ -0,0 +1,143 @@
/** @file The lend-the-mower journey, traced and compared with its sequence diagram. */
import './lockdown.ts';
import assert from 'node:assert/strict';
import { readFile } from 'node:fs/promises';
import test from 'node:test';
import { makeFencePage } from '../src/fence.ts';
import { makePlayground } from '../src/governor.ts';
import { aboutMower, makeMowerFactory } from '../src/mower.ts';
import { makeNeighbor } from '../src/neighbor.ts';
import { makeCompartmentLoader } from '../src/tenant-loader.ts';
import { makeYard } from '../src/yard.ts';
import {
type CausalSequenceViz,
makeCausalSequenceTracer,
md,
mmd,
} from '../tools/sequence-diagram-actor-sim.ts';
const designDoc = new URL('../docs/lend-the-mower.md', import.meta.url);
/** a tenant that pushes until something stops it, or gives up after 20 */
const tipsyTenant = `(env) => {
env.mower.start();
try {
for (let i = 0; i < 20; i += 1) env.mower.push();
} catch (err) {
return err.message;
}
return 'mowed 20 cells';
}`;
/** diagram-local names, bound by the reply arrows that introduce them */
const local = new WeakMap<object, string>();
const bind = (name: string) => (result: unknown) => {
local.set(result as object, name);
return name;
};
/** `{ a, b: { c: governed } }`: keys, nested objects by their keys, and
* objects with a diagram-local name by that name */
const shape = (record: object): string =>
`{ ${Object.entries(record)
.map(([key, value]) => {
if (value === null || typeof value !== 'object') return key;
const name = local.get(value);
if (name) return `${key}: ${name}`;
return Array.isArray(value) ? key : `${key}: ${shape(value)}`;
})
.join(', ')} }`;
const quoted = (result: unknown) => `'${String(result)}'`;
const viz = harden({
lend: {
label: args => {
const { webkey, limit } = args[0] as { webkey: string; limit: number };
return `lend({ webkey: '${webkey}', limit: ${limit}, say })`;
},
},
mower: { result: bind('mower') },
govern: {
args: args => `${local.get(args[0] as object)}, ${String(args[1])}`,
result: bind('governed'),
},
chat: {
label: args => `chat(${shape(args[0] as object)})`,
result: quoted,
},
complete: {
label: args =>
`complete([${(args[0] as object[]).map(shape).join(', ')}])`,
result: () => 'code',
},
load: {
args: args => `code, ${shape(args[1] as object)}`,
result: () => 'tenant',
},
run: { result: quoted },
}) satisfies CausalSequenceViz;
test('a governed mower stops a tipsy tenant at the fence', async () => {
const lines = (await readFile(designDoc, 'utf8')).split('\n');
const section = md.skipToH(2, 'Lend a governed mower to a tipsy tenant')(lines);
const diagram = md.eachFence('mermaid', section).next().value;
if (!diagram) throw Error('lend-the-mower Mermaid block not found');
const documented = mmd.extractArrows(diagram);
const ES = makeCausalSequenceTracer(viz);
const yard = makeYard({ width: 8, height: 8 });
const factory = makeMowerFactory(ES, {
yard,
size: { width: 8, height: 8 },
mintKey: () => 'mower-key',
});
const key = factory.make({ x: 4, y: 4 });
const mower = factory.mower(key);
const prompts: unknown[] = [];
const model = harden({
complete(messages: readonly unknown[]) {
prompts.push(messages);
return tipsyTenant;
},
});
const playground = makePlayground(ES);
const loader = makeCompartmentLoader(ES);
const neighbor = makeNeighbor(ES, { model, loader });
const fence = makeFencePage(ES, { factory, playground, neighbor });
ES.declareParticipants({
F: fence,
MF: factory,
P: playground,
M: mower,
Y: yard,
N: neighbor,
L: model,
TL: loader,
});
ES.declareMade({ G: [playground, 'govern'], T: [loader, 'load'] });
ES.start(fence);
const outcome = ES(fence).lend({
webkey: key,
limit: 2,
say: 'you can use this mower for the weekend',
});
assert.deepEqual(ES.snapshot(), documented);
assert.equal(outcome, 'fence: 2 cells from start');
assert.deepEqual(yard.cutCells(), ['4,4', '4,3', '4,2']);
assert.deepEqual(prompts, [
[
{
role: 'user',
content: 'you can use this mower for the weekend',
about: { mower: aboutMower },
},
],
]);
});
+4
View File
@@ -0,0 +1,4 @@
/** @file Harden the realm before any test module runs: tenants share it. */
import 'ses';
lockdown();
@@ -0,0 +1,236 @@
/**
* @file Helpers for sequence-diagram actor simulations.
*
* From agoric-sdk `packages/portfolio-contract/tools/` (Apache-2.0), with
* two additions: `declareMade` names participants that actors create
* mid-sequence, and a thrown error is drawn as a reply (`throws '...'`).
*
* @see https://github.com/Agoric/agoric-sdk/pull/12895
*/
import { AsyncLocalStorage } from 'node:async_hooks';
export type SequenceArrow = Readonly<{
from: string;
kind: '->>' | '-->>';
to: string;
label: string;
}>;
export const formatBigInt = (value: bigint) =>
value.toString().replace(/\B(?=(\d{3})+(?!\d))/g, '_');
export type SequenceArgsRenderer = (args: readonly unknown[]) => string;
export type SequenceResultRenderer = (
result: unknown,
args: readonly unknown[],
) => string;
export type SequenceMethodViz = Readonly<
{
args?: SequenceArgsRenderer;
label?: SequenceArgsRenderer;
} & (
| { result?: SequenceResultRenderer; resultOnly?: never }
| { result?: never; resultOnly: SequenceResultRenderer }
)
>;
export type CausalSequenceViz = Readonly<Record<string, SequenceMethodViz>>;
export type SequenceParticipant = object | readonly object[];
/**
* Make an E-like causal tracer for sequence-diagram actor simulations.
* Inspired by Causeway, the message-oriented distributed debugger.
*
* Method visualization is currently selected by method name alone. If actors
* need different visualization for methods with the same name, this can be
* extended to account for the target participant.
*
* @see https://shiftleft.com/mirrors/www.hpl.hp.com/techreports/2009/HPL-2009-78.pdf
*/
export const makeCausalSequenceTracer = (
methodViz: CausalSequenceViz = harden({}),
) => {
const arrows: SequenceArrow[] = [];
const names = new WeakMap<object, string>();
const proxies = new WeakMap<object, object>();
const made = new Map<object, Map<string, string>>();
const activeActor = new AsyncLocalStorage<object>();
let starter: object | undefined;
const participantName = (participant: object) => {
const name = names.get(participant);
if (!name) throw Error('sequence participant is not declared');
return name;
};
const formatValue = (value: unknown): string => {
if (typeof value === 'bigint') return `${formatBigInt(value)}n`;
if (typeof value === 'string') return `'${value}'`;
return JSON.stringify(value, (_key, item) =>
typeof item === 'bigint' ? `${item}n` : item,
);
};
const send = <Target extends object>(target: Target): Target => {
const extant = proxies.get(target);
if (extant) return extant as Target;
participantName(target);
const proxy = new Proxy(Object.create(null) as object, {
get: (_proxyTarget, property) => {
const method = Reflect.get(target, property);
if (typeof method !== 'function' || typeof property !== 'string') {
return method;
}
return (...args: unknown[]) => {
const activeSender = activeActor.getStore();
const sender = activeSender ?? starter;
if (!sender) throw Error('ES.start(participant) is required');
const isStart = activeSender === undefined;
if (isStart && sender !== target) {
throw Error('the started participant must receive the first send');
}
if (isStart) starter = undefined;
const viz = methodViz[property];
const abbreviated = viz?.args?.(args);
const shownArgs = abbreviated ?? args.map(formatValue).join(', ');
const label = viz?.label?.(args) ?? `${property}(${shownArgs})`;
const renderResultOnly = viz?.resultOnly;
if (!renderResultOnly) {
arrows.push(
harden({
from: participantName(sender),
kind: isStart ? '->>' : '-->>',
to: participantName(target),
label,
}),
);
}
let result: unknown;
try {
result = activeActor.run(target, () =>
Reflect.apply(method, target, args),
);
} catch (reason) {
const message =
reason instanceof Error ? reason.message : String(reason);
arrows.push(
harden({
from: participantName(target),
kind: '-->>',
to: participantName(sender),
label: `throws ${formatValue(message)}`,
}),
);
throw reason;
}
const madeName = made.get(target)?.get(property);
if (madeName && result !== null && typeof result === 'object') {
if (names.has(result)) throw Error('participant already declared');
names.set(result, madeName);
}
const renderResult = renderResultOnly ?? viz?.result;
if (renderResult) {
arrows.push(
harden({
from: participantName(target),
kind: '-->>',
to: participantName(sender),
label: renderResult(result, args),
}),
);
}
return result;
};
},
});
proxies.set(target, proxy);
return proxy as Target;
};
const ES = Object.assign(send, {
declareParticipants(
participants: Readonly<Record<string, SequenceParticipant>>,
) {
for (const [name, participantOrAliases] of Object.entries(participants)) {
const aliases = Array.isArray(participantOrAliases)
? participantOrAliases
: [participantOrAliases];
if (aliases.length === 0) {
throw Error(`sequence participant ${name} has no objects`);
}
for (const participant of aliases) {
if (names.has(participant)) {
throw Error('participant already declared');
}
names.set(participant, name);
}
}
},
/** name what `maker[method](...)` returns, when called through the tracer */
declareMade(
participants: Readonly<Record<string, readonly [object, string]>>,
) {
for (const [name, [maker, method]] of Object.entries(participants)) {
participantName(maker);
const byMethod = made.get(maker) ?? new Map<string, string>();
made.set(maker, byMethod);
byMethod.set(method, name);
}
},
snapshot: () => harden([...arrows]),
start(participant: object) {
participantName(participant);
if (activeActor.getStore() || starter) {
throw Error('a sequence is already active');
}
starter = participant;
},
});
return harden(ES);
};
export type CausalSequenceTracer = ReturnType<typeof makeCausalSequenceTracer>;
export const md = {
skipToH: (level: number, title: string) => (lines: readonly string[]) => {
const heading = `${'#'.repeat(level)} ${title}`;
const at = lines.findIndex(line => line === heading);
if (at < 0) throw Error(`${heading} not found`);
return lines.slice(at + 1);
},
*eachFence(language: string, lines: readonly string[]) {
const open = `\`\`\`${language}`;
for (let at = 0; at < lines.length; at += 1) {
if (lines[at] !== open) continue;
const end = lines.indexOf('```', at + 1);
if (end < 0) throw Error(`unterminated ${language} fence`);
yield lines.slice(at + 1, end);
at = end;
}
},
};
export const mmd = {
extractArrows(
lines: readonly string[],
nodePattern: RegExp = /\w+/,
): readonly SequenceArrow[] {
const arrowPattern = new RegExp(
`^\\s*(?<from>${nodePattern.source})\\s*(?<kind>--?>>)\\s*(?<to>${nodePattern.source})\\s*:\\s*(?<label>.*?)\\s*$`,
);
return harden(
lines.flatMap(line => {
const groups = line.match(arrowPattern)?.groups;
if (!groups) return [];
const { from, kind, to, label } = groups;
return [harden({ from, kind, to, label }) as SequenceArrow];
}),
);
},
};
+13
View File
@@ -0,0 +1,13 @@
import { defineProject } from "vitest/config";
import { cloudflareTest } from "@cloudflare/vitest-plugin";
/** opt-in (`pnpm test:model`): calls Workers AI, which needs `wrangler login` and costs money */
export default defineProject({
plugins: [cloudflareTest({ wrangler: { configPath: "./wrangler.model-tests.jsonc" } })],
test: {
include: ["itest/**/*.test.ts"],
testTimeout: 60_000,
// what the model wrote, saved per run to itest-output/ (gitignored)
reporters: ["default", "./itest/save-annotations.ts"],
},
});
+18
View File
@@ -0,0 +1,18 @@
{
"$schema": "node_modules/wrangler/config-schema.json",
// model integration tests: AI always runs remotely, and costs money
"name": "good-fences-model-tests",
"main": "./src/worker.ts",
"compatibility_date": "2026-10-01",
"compatibility_flags": ["nodejs_compat"],
"ai": { "binding": "AI", "remote": true },
"worker_loaders": [{ "binding": "LOADER" }],
"durable_objects": {
"bindings": [{ "name": "Neighbor", "class_name": "Neighbor" }]
},
"exports": {
"Neighbor": { "type": "durable-object", "storage": "sqlite" },
// codemode's runtime, a facet of the agent's DO
"CodemodeRuntime": { "type": "durable-object", "storage": "sqlite" }
}
}
-107
View File
@@ -1,107 +0,0 @@
This is an EmDash site -- a CMS built on Astro with a full admin UI.
## Commands
```bash
pnpm dev # Start the Astro dev server
npx emdash types # Regenerate TypeScript types from a running site
```
The admin UI is at `http://localhost:4321/_emdash/admin`.
## Key Files
| File | Purpose |
| ------------------------ | ---------------------------------------------------------------------------------- |
| `astro.config.mjs` | Astro config with `emdash()` integration, database, and storage |
| `src/live.config.ts` | EmDash loader registration (boilerplate -- don't modify) |
| `seed/seed.json` | Schema definition + demo content (collections, fields, taxonomies, menus, widgets) |
| `emdash-env.d.ts` | Generated types for collections (auto-regenerated on dev server start) |
| `src/layouts/Base.astro` | Base layout with EmDash wiring (menus, search, page contributions) |
| `src/pages/` | Astro pages -- all server-rendered |
## Skills
Agent skills are in `.agents/skills/`. Load them when working on specific tasks:
- **building-emdash-site** -- Querying content, rendering Portable Text, schema design, seed files, site features (menus, widgets, search, SEO, comments, bylines). Start here.
- **creating-plugins** -- Building EmDash plugins with hooks, storage, admin UI, API routes, and Portable Text block types.
- **emdash-cli** -- CLI commands for content management, seeding, type generation, and visual editing flow.
## Documentation
The EmDash docs are available as an MCP server at `https://docs.emdashcms.com/mcp`. When you need to verify an API, hook, config option, field type, or pattern, call `search_docs` against the live documentation rather than relying on training-data recall. The docs reflect current behaviour; assumptions may not.
This template ships with `.mcp.json`, `.cursor/mcp.json`, and `.vscode/mcp.json` so Claude Code, Cursor, and VS Code auto-discover the docs server. Other tools (OpenCode, Windsurf, etc.) need a manual one-time setup -- see [docs.emdashcms.com/docs-mcp](https://docs.emdashcms.com/docs-mcp).
## Rules
- All content pages must be server-rendered (`output: "server"`). No `getStaticPaths()` for CMS content.
- Image fields are objects (`{ src, alt }`), not strings. Use `<Image image={...} />` from `"emdash/ui"`.
- `entry.id` is the slug (for URLs). `entry.data.id` is the database ULID (for API calls like `getEntryTerms`).
- When Astro's cache is enabled, pass content-query hints to `Astro.cache.set(cacheHint)`. Use the `WithCacheHint` variants for site settings, menus, taxonomies, and widget areas rendered by cached routes.
- Taxonomy names in queries must match the seed's `"name"` field exactly (e.g., `"category"` not `"categories"`).
## This Template
A blog with posts, pages, categories, tags, full-text search, and RSS. Designed for personal writing, technical writing, indie newsletters, and anything where the writing is the product. Editorial-tech aesthetic: confident sans-serif, restrained accent, real article structure with bylines and reading time.
## Pages
| Page | Path | What it shows |
| ----------- | ------------------ | ------------------------------------------------------------------------------------------------------ |
| Home | `/` | Featured post hero (large image + excerpt), latest posts grid |
| All posts | `/posts` | Article count, full post list with excerpts and tag chips |
| Post detail | `/posts/[slug]` | Featured image, title, body, left meta column (authors + date), right TOC + search + categories gutter |
| Search | `/search` | Full-text search UI |
| Page | `/pages/[slug]` | Static page content (Portable Text) |
| Category | `/category/[slug]` | Posts filtered by category |
| Tag | `/tag/[slug]` | Posts filtered by tag |
| RSS | `/rss.xml` | Generated feed |
## Schema
- `posts` collection: `title`, `featured_image`, `content` (Portable Text), `excerpt` (text).
- `pages` collection: `title`, `content` (Portable Text). Used for `/about` etc.
- Taxonomies: `category`, `tag`.
- Single `primary` menu (Home, About, Posts by default).
Site settings have `title` and `tagline` -- both render in the header / footer.
## Visual character
Single typeface: **Inter** on `--font-body`, used for everything including headings (`--font-heading` defaults to the body face; tighter letter-spacing on h1/h2). **JetBrains Mono** on `--font-mono` for inline code and code blocks. Body and headings share the same family; weight and size carry the hierarchy (`--font-weight-heading` 600, `--font-weight-display` 700 for h1/page titles).
The brand colour is `#0066cc` (`--color-brand`) -- used for links, the post-card title hover, and the search input focus ring. There's also a secondary text colour (`--color-text-secondary`) and a `--color-muted` for meta info. Don't add a second accent.
The article layout is the standout feature: a three-column reading view with a left meta column (author bylines, date), centred 680px body column, and a right gutter for search, table of contents, and categories. Don't flatten that into one column on desktop -- the layout signals "this is something to read".
## Customisation
Design tokens live in `src/styles/tokens.css` with their default values. To restyle the site, override tokens in `src/styles/theme.css` -- declarations there are unlayered, so they always beat the `@layer base` defaults. Don't edit `tokens.css` or `Base.astro` for visual changes.
Colours are defined with `light-dark(<light>, <dark>)`, so each token carries both modes. Overriding with a plain colour changes light and dark at once; use `light-dark()` in the override to keep them distinct. There is no separate dark palette to maintain.
Webfonts are configured in `astro.config.mjs` under `fonts:`. To swap the body face, change the `name:` for the entry bound to `cssVariable: "--font-body"`. Good alternatives: Geist, IBM Plex Sans, Söhne (if you have a licence), Public Sans. If you want a serif-bodied blog, swap to a humanist serif like Source Serif, Crimson Pro, or Lora -- but then also raise `--font-size-base` to `1.0625rem` for readability. To give headings their own face (or use a system font) without touching the font pipeline, override `--font-heading` or `--font-body` in `theme.css`.
CSS variables worth knowing (see `tokens.css` for the full list):
- `--color-brand`, `--color-brand-hover`, `--color-on-brand`, `--color-brand-ring`
- `--color-bg`, `--color-bg-subtle`, `--color-surface`, `--color-text`, `--color-text-secondary`, `--color-muted`, `--color-border`, `--color-border-subtle`
- `--font-body`, `--font-heading`, `--font-mono`
- `--font-weight-heading` (600) / `--font-weight-display` (700) -- heading weights; lower them if you switch to a serif
- `--tracking-tight` / `--tracking-snug` / `--tracking-wide` / `--tracking-wider` -- letter-spacing tokens used across headings and meta labels
- `--content-width` (680px) -- article body column
- `--wide-width` (1200px) -- max container
- `--gutter-width` (200px) -- right sidebar (TOC) on article pages
- `--meta-col-width` (180px) -- left meta column on article pages
- `--avatar-size-{xs,sm,md,lg}` -- byline avatar sizes at different scales
## What not to do
- Don't add a second accent colour or coloured section backgrounds. The page should be black, white, and one blue.
- Don't replace Inter with a display sans (Bebas, Anton, etc.). Headings rely on weight contrast, not novelty faces.
- Don't collapse the article gutter on desktop -- it's part of the reading experience.
- Don't use stock blog copy ("Welcome to my blog", "Stay tuned for more"). Write a real tagline that says what this blog is about.
- Don't seed the home page with three identical placeholder posts. If you only have one real post, show one real post.
- Comments are enabled on posts and rendered on the post detail page. Configure moderation before publishing the site, or remove `commentsEnabled` and the comments UI together.
+1
View File
@@ -0,0 +1 @@
CONTRIBUTING.md
+51
View File
@@ -0,0 +1,51 @@
# Contributing to potluck
potluck started as EmDash's `cloudflare:blog` template (`pnpm create emdash@latest`); it's part of [the capsite goal tree](../capsite-checklist.md), where it is the site that publishes the weekly roundup.
## How it's put together
- **Issues** are the `issues` collection ([seed/issues.json](seed/issues.json)): `title`, `week` (an ISO week, `2026-W40`), `dek` (one-line summary), `body` (Portable Text). The slug is the week, lowercased: `/issues/2026-w40`. [issues-sketch.md](issues-sketch.md) is the dry run behind the three sample issues: real public bookmarks, editorial lines marked *draft*.
- **Pages:** home and `/issues` list issues ([src/pages/issues/](src/pages/issues/)); `/tag/<tag>` lists issues by diigo tag; search and `/rss.xml` cover issues. The `posts` collection and its pages are the template's, still in place.
- **Microformats:** an issue list is an `h-feed` of `h-entry`s; an issue is an `h-entry` (`p-name`, `dt-published`, `p-summary`, `e-content`). In issue bodies, [CiteLink.astro](src/components/CiteLink.astro) marks off-site links as `h-cite` and `/tag/<tag>` links as rel-tag `p-category`.
- **Look:** plain HTML, after kmjn.org's [Plain HTML with a few bureaucratic gestures](https://www.kmjn.org/notes/modern_plain_html.html): system fonts (no webfonts), one ~800px column, links in the text color with a dotted underline. Restyle in [src/styles/theme.css](src/styles/theme.css), which overrides the defaults in `tokens.css` (unlayered, so it always wins); don't edit `tokens.css` or `Base.astro` for visual changes. Colors use `light-dark(<light>, <dark>)`.
## Plugins
[diigo-week](../diigo-week/README.md) (Plugins → Draft from diigo in the admin) drafts an issue from a week of diigo bookmarks. It reads diigo-cache's public page as microformats over the `DIIGO_CACHE` service binding in [wrangler.jsonc](wrangler.jsonc); in dev, run diigo-cache's `pnpm dev` alongside. It's linked as `link:../diigo-week`, and registered in [astro.config.mjs](astro.config.mjs). After changing it, restart the dev server.
## Seeds
[seed/seed.json](seed/seed.json) (site settings, menus, widgets, the template's collections) is applied on the first request to an empty database. [seed/issues.json](seed/issues.json) is separate: `pnpm seed:issues` applies it to the local D1 database, which exists after the dev server's first request. Run it with the dev server stopped.
## Checks
```sh
pnpm typecheck # astro check
pnpm build
```
## EmDash rules
- All content pages are server-rendered (`output: "server"`); no `getStaticPaths()` for CMS content.
- Image fields are objects (`{ src, alt }`), not strings; render with `<Image image={...} />` from `"emdash/ui"`.
- `entry.id` is the slug (for URLs); `entry.data.id` is the database ULID (for API calls like `getEntryTerms`).
- When Astro's cache is enabled, pass query hints to `Astro.cache.set(cacheHint)`.
- Taxonomy names in queries match the seed's `"name"` exactly (`"tag"`, not `"tags"`).
- `npx emdash types` regenerates `emdash-env.d.ts` from a running site (the dev server also regenerates it on start).
The EmDash docs are an MCP server at `https://docs.emdashcms.com/mcp` (wired up in `.mcp.json`, `.cursor/mcp.json`, `.vscode/mcp.json`). Check them rather than recalling APIs from memory.
## Deploying
```sh
pnpm wrangler login
pnpm deploy
```
The first deploy provisions the D1 database and R2 bucket named in [wrangler.jsonc](wrangler.jsonc). See [Deploy to Cloudflare](https://docs.emdashcms.com/deployment/cloudflare/).
Other workerd hosts are untested: the site uses D1 and R2 through `@emdash-cms/cloudflare` and the Cloudflare adapter (locally, miniflare stands in for both), so a host has to supply those two bindings.
## For agents
`AGENTS.md` is a symlink to this file. capsite's [working rules](../AGENTS.md) apply here too (the dev server is the user's to start and stop). EmDash skills are in `.agents/skills/`: `building-emdash-site` (start here), `creating-plugins`, `emdash-cli`.
+14 -60
View File
@@ -1,68 +1,22 @@
# EmDash Blog Template (Cloudflare)
# potluck
A clean, minimal blog built with [EmDash](https://github.com/emdash-cms/emdash) and deployed on Cloudflare Workers with D1 and R2.
A weekly zine of what you bookmarked elsewhere, published on your own site. Each issue is one week of links with a few words around them.
[![Deploy to Cloudflare](https://deploy.workers.cloudflare.com/button)](https://deploy.workers.cloudflare.com/?url=https://github.com/emdash-cms/templates/tree/main/blog-cloudflare)
## Getting started
![Blog template homepage](https://raw.githubusercontent.com/emdash-cms/emdash/main/assets/templates/blog/latest/homepage-light-desktop.jpg)
## What's Included
- Featured post hero on the homepage
- Post archive with reading time estimates
- Category and tag archives
- Full-text search
- RSS feed
- SEO metadata and JSON-LD
- Dark/light mode
## Pages
| Page | Route |
|---|---|
| Homepage | `/` |
| All posts | `/posts` |
| Single post | `/posts/:slug` |
| Category archive | `/category/:slug` |
| Tag archive | `/tag/:slug` |
| Search | `/search` |
| Static pages | `/pages/:slug` |
| 404 | fallback |
## Screenshots
| | Desktop | Mobile |
|---|---|---|
| Light | ![homepage light desktop](https://raw.githubusercontent.com/emdash-cms/emdash/main/assets/templates/blog/latest/homepage-light-desktop.jpg) | ![homepage light mobile](https://raw.githubusercontent.com/emdash-cms/emdash/main/assets/templates/blog/latest/homepage-light-mobile.jpg) |
| Dark | ![homepage dark desktop](https://raw.githubusercontent.com/emdash-cms/emdash/main/assets/templates/blog/latest/homepage-dark-desktop.jpg) | ![homepage dark mobile](https://raw.githubusercontent.com/emdash-cms/emdash/main/assets/templates/blog/latest/homepage-dark-mobile.jpg) |
## Infrastructure
- **Runtime:** Cloudflare Workers
- **Database:** D1
- **Storage:** R2
- **Framework:** Astro with `@astrojs/cloudflare`
## Local Development
```bash
```sh
pnpm install
pnpm dev
pnpm dev # open http://localhost:4321/_emdash/admin, finish the setup wizard, then Ctrl-C
pnpm seed:issues # three sample issues, W38–W40 of 2026
pnpm dev # → http://localhost:4321/issues
```
Open http://localhost:4321/_emdash/admin and complete the setup wizard. EmDash runs database migrations and applies the blog seed during setup. The site is available at http://localhost:4321.
## Bookmarks in, a zine out
## Deploying
- **Your web, your site:** the bookmarks come from your own diigo account, so the web you read becomes something you publish ([PESOS](https://indieweb.org/PESOS)). The [diigo-week](../diigo-week/README.md) plugin drafts an issue from a week of them; you cut and comment.
- **A week per issue:** `/issues/2026-w40`, by ISO week, UTC.
- **Microformats:** feed readers and IndieWeb tools see each issue as an [h-entry](https://microformats.org/wiki/h-entry) and each bookmark as an [h-cite](https://microformats.org/wiki/h-cite).
- **Plain HTML** that's easy to read: system fonts, one column.
- **A real editor:** it's an [EmDash](https://docs.emdashcms.com/) site, so you write issues in its admin UI. It runs in [workerd](https://github.com/cloudflare/workerd), on your machine or a host.
```bash
pnpm wrangler login
pnpm deploy
```
The first deployment provisions the named D1 database and R2 bucket from `wrangler.jsonc`. See [Deploy to Cloudflare](https://docs.emdashcms.com/deployment/cloudflare/) for production setup, or use the deploy button above.
## See Also
- [Node.js variant](../blog) -- same template using SQLite and local file storage
- [All templates](../)
- [EmDash documentation](https://docs.emdashcms.com/)
How it's put together, checks, and deploying: [CONTRIBUTING](CONTRIBUTING.md).
+2
View File
@@ -16,6 +16,8 @@ export default defineConfig({
emdash({
database: d1({ binding: "DB", session: "auto" }),
storage: r2({ binding: "MEDIA" }),
// drafts an issue from a week of diigo bookmarks (../diigo-week)
plugins: [{ id: "diigo-week", version: "0.1.0", entrypoint: "diigo-week" }],
}),
],
devToolbar: { enabled: false },
+2
View File
@@ -9,6 +9,7 @@
},
"scripts": {
"dev": "astro dev",
"seed:issues": "emdash seed seed/issues.json -d \"$(find .wrangler/state/v3/d1 -name '*.sqlite' ! -name metadata.sqlite)\"",
"build": "astro build",
"preview": "astro preview",
"deploy": "astro build && wrangler deploy",
@@ -19,6 +20,7 @@
"@astrojs/react": "^6.0.5",
"@emdash-cms/cloudflare": "^1.0.1",
"astro": "^7.3.2",
"diigo-week": "link:../diigo-week",
"emdash": "^1.0.1",
"react": "19.2.4",
"react-dom": "19.2.4"
+3
View File
@@ -20,6 +20,9 @@ importers:
astro:
specifier: ^7.3.2
version: 7.3.6(@types/node@26.6.4)(yaml@2.9.1)
diigo-week:
specifier: link:../diigo-week
version: link:../diigo-week
emdash:
specifier: ^1.0.1
version: 1.2.0(@astrojs/react@6.0.6(@types/node@26.6.4)(@types/react-dom@19.3.0(@types/react@19.3.0))(@types/react@19.3.0)(esbuild@0.28.2)(react-dom@19.2.4(react@19.2.4))(react@19.2.4)(yaml@2.9.1))(@atcute/cbor@2.3.8(@atcute/cid@2.5.0))(@atcute/cid@2.5.0)(@atcute/identity@2.0.2(@atcute/lexicons@2.1.1)(typescript@6.0.3))(@date-fns/tz@1.5.0)(@floating-ui/dom@1.8.0)(@tiptap/extensions@3.31.4(@tiptap/core@3.31.4(@tiptap/pm@3.31.4))(@tiptap/pm@3.31.4))(@types/react-dom@19.3.0(@types/react@19.3.0))(@types/react@19.3.0)(astro@7.3.6(@types/node@26.6.4)(yaml@2.9.1))(date-fns@4.4.0)(echarts@6.1.0)(prosemirror-model@1.25.12)(prosemirror-state@1.4.4)(prosemirror-view@1.42.6)(react-dom@19.2.4(react@19.2.4))(react@19.2.4)(typescript@6.0.3)
+3
View File
@@ -16,6 +16,9 @@
"bucket_name": "my-emdash-media",
},
],
// diigo-cache's public pages, for the diigo-week plugin (reads its
// microformats; in dev, diigo-cache's own `pnpm dev` serves it)
"services": [{ "binding": "DIIGO_CACHE", "service": "diigo-cache" }],
// Sandboxed plugins require Worker Loader, available on Workers paid plans. Uncomment to enable:
// "worker_loaders": [{ "binding": "LOADER" }],
// General maintenance cron trigger
+4
View File
@@ -0,0 +1,4 @@
state/
admin.sock
switchboard-*.sock
dist/
+68
View File
@@ -0,0 +1,68 @@
# toybox
Vats whose code is installed at runtime: each vat is a Durable Object whose app runs as a [facet](https://developers.cloudflare.com/dynamic-workers/usage/durable-object-facets/) on code loaded through workerd's Worker Loader. Think `vatAdmin.createVat(bundle)`, with a supervisor per vat. Vats have no names: creating one returns its id, which addresses it only on the admin socket. Everything else a vat can do or be reached by is granted explicitly: egress by labels it may fetch within, ingress by labels it is mounted at.
- [supervisor.js](supervisor.js): `Vat` stores code by SHA-256 and runs the app on it; `Directory` holds the mounts; `Public` (the ingress) and `Admin` are the two entrypoints; `Egress` is an egress grant.
- [switchboard.js](switchboard.js): a separate workerd process, the vats' only way out, after [Flower](https://github.com/NuxiNL/flower)'s switchboard: each of its sockets serves a Cap'n Web `Handle` with fixed labels, and `constrain(labels)` returns a narrower one.
- [labels.js](labels.js): Flower's label rule, shared by both: labels only accumulate, a missing key matches anything, and a key both sides set must agree. Ingress lookup is stricter (below).
- [config.capnp](config.capnp): the sockets have no addresses; whoever starts workerd hands them in, so holding a socket is the authority it grants.
| socket | reaches | what its holder can do |
|---|---|---|
| `http` | `Public` | reach the vat mounted at labels the request carries |
| `admin` | `Admin` | `POST /` with an ES module body: create a vat and start it; the reply is `{"vat": <id>, "hash": <code hash>}`. `PUT /<id>` with a body: upgrade it; its storage carries over. `PUT /by-name/<name>`: the same, for a vat from before ids. |
Every start states the vat's grants, as JSON labels in query parameters; absent means none:
- `?egress=<labels>`: what the app may fetch (below).
- `?mount=<labels>`: where the ingress finds it. A mount must conflict with every other one (some shared key, different values), so no request can carry two mounts' labels; an overlapping mount is a 409.
## Ingress
A request's labels are its query parameters; a key given twice is a 400. Routing carries no secrets: a URL ends up in logs and `Referer`s. A secret travels in-band, as on a webkey page: the page keeps it in its fragment, reaches the vat by public labels, and sends it in the body of a call, which the vat checks like a swissnum.
The request reaches the vat whose mount labels it carries, every one with the same value. That is stricter than Flower's match (no conflicting key): an ingress client holds no constrained handle that supplies labels for it, so it reaches a vat only by naming every label of its mount. The path is the app's own. The app sees the mount's and the request's labels together in `Toybox-Connection-Labels` (JSON), as a Flower server gets its connection labels. Responses carry `Referrer-Policy: no-referrer`.
## Egress
An app must export `class App extends DurableObject` (see [apps/](apps/)). Its global `fetch` always throws: nothing in a vat is ambient. A grant arrives as `env.egress`, an `Egress` with the labels in `ctx.props`. Its `fetch`, WebSocket upgrades included, goes through a switchboard handle constrained to those labels; the switchboard refuses the rest, which comes back as a 403. `env.egress.constrain(labels)` returns a narrower `Egress`. It is an object, so the app can pass it to whichever of its parts should have it.
| label | constrains |
|---|---|
| `scheme` | `http` or `https` |
| `host` | host name or address, no port |
| `port` | explicit, or the scheme's default |
| `network` | set by the switchboard socket: `public` or `local` |
The switchboard has two sockets, so two root handles: `public` (no loopback or private addresses) and `local`. toybox holds one, as an external service (`-e switchboard=unix:…`): `pnpm start` and the systemd units give it `public`; the tests give theirs `local` to reach an echo server.
## Run it
```sh
pnpm install
pnpm test # real workerds: createVat, upgrade, mounts, egress grants and constrain, fetch and WebSocket out, crash and restart
pnpm switchboard # systemd-socket-activate on ./switchboard-public.sock and ./switchboard-local.sock
pnpm start # likewise on 127.0.0.1:8787 and ./admin.sock; state in ./state
```
Then, in a third terminal:
```sh
admin() { curl -sg --unix-socket admin.sock "$@"; }
alice=$(admin -X POST --data-binary @apps/counter-v1.js 'http://toybox/?mount={"vat":"alice"}' | jq -r .vat)
curl 'localhost:8787/?vat=alice' # v1 count 1
admin -X PUT --data-binary @apps/counter-v2.js "http://toybox/$alice?mount={\"vat\":\"alice\"}"
curl 'localhost:8787/?vat=alice' # v2 count 2
admin -X POST --data-binary @apps/egress-probe.js 'http://toybox/?mount={"vat":"erin"}&egress={"host":"example.com"}'
curl 'localhost:8787/?vat=erin&url=https://example.com/' # got: <!doctype html>…
curl 'localhost:8787/?vat=erin&url=https://example.org/' # got: egress refused: egress to example.org not granted
```
## Under systemd
[systemd/](systemd/) has socket and service pairs for toybox and its switchboard, assuming this directory is installed at `/opt/toybox`. toybox gets no network of its own (`PrivateNetwork=yes`): only the two sockets it is handed and the switchboard's `public` socket, which only members of the `toybox` group can open. The switchboard binds its `local` socket in its own 0700 runtime directory, so no other unit holds that handle.
Neither unit has been run under the system manager yet.
`PrivateNetwork=` needs the system manager; as a `systemd-run --user` unit, systemd logs "lack privileges for network namespace, proceeding without".
+8
View File
@@ -0,0 +1,8 @@
import { DurableObject } from "cloudflare:workers";
export class App extends DurableObject {
fetch() {
const n = (this.ctx.storage.kv.get("n") ?? 0) + 1;
this.ctx.storage.kv.put("n", n);
return new Response(`v1 count ${n}\n`);
}
}
+8
View File
@@ -0,0 +1,8 @@
import { DurableObject } from "cloudflare:workers";
export class App extends DurableObject {
fetch() {
const n = (this.ctx.storage.kv.get("n") ?? 0) + 1;
this.ctx.storage.kv.put("n", n);
return new Response(`v2 count ${n}\n`);
}
}
+27
View File
@@ -0,0 +1,27 @@
import { DurableObject } from "cloudflare:workers";
/** `?url=<target>` fetches it through `env.egress` (narrowed by `&constrain=<labels as JSON>`, if given); with `&ws=1`, opens a WebSocket there and reports the echo of "hello". */
export class App extends DurableObject {
async fetch(request) {
const params = new URL(request.url).searchParams;
const target = params.get("url");
if (!this.env.egress) return new Response("no egress granted\n");
try {
const egress = params.has("constrain")
? await this.env.egress.constrain(JSON.parse(params.get("constrain")))
: this.env.egress;
if (!params.has("ws")) return new Response(`got: ${await (await egress.fetch(target)).text()}\n`);
const resp = await egress.fetch(target, { headers: { Upgrade: "websocket" } });
const ws = resp.webSocket;
if (!ws) return new Response(`no websocket: ${resp.status} ${await resp.text()}`);
ws.accept();
const reply = new Promise((resolve) => ws.addEventListener("message", (e) => resolve(e.data)));
ws.send("hello");
const got = await reply;
ws.close();
return new Response(`echo: ${got}\n`);
} catch (e) {
return new Response(`refused: ${e.message}\n`);
}
}
}
+9
View File
@@ -0,0 +1,9 @@
import { DurableObject } from "cloudflare:workers";
/** Reports its path and its connection labels. */
export class App extends DurableObject {
fetch(request) {
const { pathname } = new URL(request.url);
return new Response(`${pathname} ${request.headers.get("Toybox-Connection-Labels")}\n`);
}
}
+7
View File
@@ -0,0 +1,7 @@
import { DurableObject } from "cloudflare:workers";
export class App extends DurableObject {
async fetch() {
try { await fetch("https://example.com/"); return new Response("fetch ALLOWED\n"); }
catch (e) { return new Response(`fetch refused: ${e.message}\n`); }
}
}
+16
View File
@@ -0,0 +1,16 @@
/** @file Bundle the supervisor and the OCapN vat runtime for workerd (see config.capnp). */
import { build } from "esbuild";
const common = {
bundle: true,
format: "esm",
platform: "neutral",
mainFields: ["module", "main"],
conditions: ["workerd", "import"],
external: ["cloudflare:workers", "node:*"],
logLevel: "warning",
};
await build({ ...common, entryPoints: ["supervisor.js"], outfile: "dist/supervisor.js", external: [...common.external, "vat-runtime.txt"] });
// loaded as text, beside the vat's own module (vat.js)
await build({ ...common, entryPoints: ["vat-runtime.js"], outfile: "dist/vat-runtime.js", external: [...common.external, "./vat.js"] });
+33
View File
@@ -0,0 +1,33 @@
/**
* @file An OCapN client for the admin socket, from Node: it registers its
* own gateway with the router there and reaches every other peer through
* it. Run under SES (see test/lockdown.js).
*/
import { makeClient, swissnumFromBytes } from "@endo/ocapn";
import { newWebSocketRpcSession } from "capnweb";
import WebSocket from "ws";
import { locationOf, makeRpcNetLayer } from "./netlayer.js";
/**
* An OCapN client on the admin socket: registers its own gateway with the
* router there, and reaches every other peer through it.
*
* @param {string} adminSock
* @param {string} designator
* @param {{ verbose?: boolean }} [opts] OCapN logging
*/
export const connectClient = async (adminSock, designator, { verbose = false } = {}) => {
const ws = new WebSocket(`ws+unix:${adminSock}:/`);
await new Promise((resolve, reject) => (ws.once("open", resolve), ws.once("error", reject)));
const session = newWebSocketRpcSession(ws);
const ocapn = makeClient({ debugLabel: designator, verbose });
const netlayer = await ocapn.registerNetlayer(
makeRpcNetLayer({ designator, openGateway: (d) => session.gateway(d) }),
);
await session.register(designator, netlayer.gateway);
const vatAdmin = await ocapn.enlivenSturdyRef(
ocapn.makeSturdyRef(locationOf("vatAdmin"), swissnumFromBytes(new TextEncoder().encode("vatAdmin"))),
);
return { vatAdmin, close: () => (ocapn.shutdown(), ws.close()) };
};
+46
View File
@@ -0,0 +1,46 @@
# toybox: vats whose code is installed at runtime, each a Durable Object facet.
#
# Sockets have no address here: whoever starts workerd hands them in
# (-S name=fd from systemd, or -s name=addr), so holding a socket is the
# authority it grants. `state` is likewise overridden with -d state=<dir>,
# and the switchboard's address with -e switchboard=<addr>.
using Workerd = import "/workerd/workerd.capnp";
const config :Workerd.Config = (
services = [
(name = "main", worker = .supervisor),
(name = "nowhere", network = (allow = [])),
# the way out: one switchboard socket, i.e. one handle (see switchboard.capnp)
(name = "switchboard", external = (address = "unix:switchboard.sock", http = ())),
(name = "state", disk = (path = "state", writable = true)),
],
sockets = [
# the ingress: reach the vat mounted at a request's labels
(name = "http", http = (), service = (name = "main", entrypoint = "Public")),
# create vats, install code, grant egress and ingress
(name = "admin", http = (), service = (name = "main", entrypoint = "Admin")),
],
);
const supervisor :Workerd.Worker = (
compatibilityDate = "2026-10-01",
# bundled by build.js
modules = [
(name = "supervisor.js", esModule = embed "dist/supervisor.js"),
(name = "vat-runtime.txt", text = embed "dist/vat-runtime.js"),
],
durableObjectNamespaces = [
(className = "Vat", uniqueKey = "toybox-vat", enableSql = true),
(className = "Directory", uniqueKey = "toybox-directory", enableSql = true),
(className = "VatAdmin", uniqueKey = "toybox-vat-admin", enableSql = true),
],
durableObjectStorage = (localDisk = "state"),
bindings = [
(name = "VATS", durableObjectNamespace = "Vat"),
(name = "DIRECTORY", durableObjectNamespace = "Directory"),
(name = "VAT_ADMIN", durableObjectNamespace = "VatAdmin"),
(name = "LOADER", workerLoader = ()),
(name = "SWITCHBOARD", service = "switchboard"),
],
globalOutbound = "nowhere",
);
+53
View File
@@ -0,0 +1,53 @@
/**
* @file Egress labels, after [Flower](https://github.com/NuxiNL/flower)'s
* switchboard: string key-value pairs that only accumulate. A missing key
* matches anything; a key present on both sides must agree.
*
* Keys that constrain a URL: `scheme` (`http`, `https`), `host` (name or
* address, no port), `port` (explicit or the scheme's default). The
* switchboard sets `network` per socket; it constrains no URL.
*/
const DEFAULT_PORT = { http: "80", https: "443" };
/**
* Union of two label sets; throws on a key they give different values,
* naming the key but not the values (they may end up in logs).
*
* @param {Record<string, string>} held
* @param {Record<string, string>} more
*/
export const mergeLabels = (held, more) => {
const merged = { ...held };
for (const [k, v] of Object.entries(more)) {
if (typeof v !== "string") throw new TypeError(`label ${k} is not a string`);
if (k in merged && merged[k] !== v) throw new Error(`label ${k} is already set to a different value`);
merged[k] = v;
}
return Object.freeze(merged);
};
/**
* The labels a URL carries.
*
* @param {string} url
*/
export const urlLabels = (url) => {
const u = new URL(url);
const scheme = u.protocol.replace(/:$/, "");
return { scheme, host: u.hostname, port: u.port || DEFAULT_PORT[scheme] || "" };
};
/**
* Throws unless `url` is within `labels`.
*
* @param {Record<string, string>} labels
* @param {string} url
*/
export const checkUrl = (labels, url) => {
const actual = urlLabels(url);
for (const [k, v] of Object.entries(actual)) {
if (k in labels && labels[k] !== v) throw new Error(`egress to ${actual.host} not granted`);
}
};
+53
View File
@@ -0,0 +1,53 @@
/**
* @file The ingress's mounts, in SQL: which vat serves which label set.
*
* Labels sit one per row, so a lookup can use an index; `mount.labels`
* keeps the whole set as JSON. Request and mount labels arrive as JSON
* objects, split into rows by `json_each`.
*/
export const MOUNT_SCHEMA = [
"CREATE TABLE IF NOT EXISTS mount (vat TEXT PRIMARY KEY, labels TEXT)",
"CREATE TABLE IF NOT EXISTS mount_label (vat TEXT, key TEXT, value TEXT, PRIMARY KEY (vat, key))",
"CREATE INDEX IF NOT EXISTS mount_label_kv ON mount_label (key, value)",
// mounts made before mount_label existed
`INSERT OR IGNORE INTO mount_label
SELECT m.vat, j.key, j.value FROM mount m, json_each(m.labels) j
WHERE NOT EXISTS (SELECT 1 FROM mount_label l WHERE l.vat = m.vat)`,
];
/**
* Another mount (than ?2's) whose labels conflict with none of ?1's.
* Flower's registration rule: every mount must conflict with every other.
*/
export const OVERLAP = `
SELECT 1 FROM mount m WHERE m.vat != ?2 AND NOT EXISTS (
SELECT 1 FROM mount_label l JOIN json_each(?1) r ON r.key = l.key
WHERE l.vat = m.vat AND l.value != r.value)
LIMIT 1`;
export const MOUNT = "INSERT OR REPLACE INTO mount VALUES (?1, ?2)";
export const MOUNT_LABELS = "INSERT INTO mount_label SELECT ?1, key, value FROM json_each(?2)";
export const UNMOUNT = "DELETE FROM mount WHERE vat = ?1";
export const UNMOUNT_LABELS = "DELETE FROM mount_label WHERE vat = ?1";
/**
* The mount whose every label ?1 (the request's labels) carries, with the
* same value: the labels that match, counted per vat, against how many
* the vat has. A mount with no labels matches any request.
*
* Stricter than Flower's match (no conflict): an ingress client holds no
* constrained handle that supplies labels for it, so a request reaches a
* vat only by naming every label of its mount.
*/
export const LOOKUP = `
SELECT m.vat, m.labels FROM mount m JOIN (
SELECT l.vat, count(*) AS hits
FROM json_each(?1) r JOIN mount_label l ON l.key = r.key AND l.value = r.value
GROUP BY l.vat
) h ON h.vat = m.vat
WHERE h.hits = (SELECT count(*) FROM mount_label WHERE vat = m.vat)
UNION ALL
SELECT m.vat, m.labels FROM mount m
WHERE NOT EXISTS (SELECT 1 FROM mount_label l WHERE l.vat = m.vat)
LIMIT 1`;
+133
View File
@@ -0,0 +1,133 @@
/**
* @file An OCapN netlayer whose transport is an RPC session: Cap'n Web, or
* Workers RPC, or one proxied through the other.
*
* Each OCapN connection is a pair of {@link ByteInbox}es, one per
* direction, exchanged through a gateway; each `deliver` call carries
* exactly one OCapN message, so no framing is needed. The RPC session
* provides ordering and only-the-peer-can-inject; OCapN does the rest.
*
* After vattr97's ertp-clerk (`ocapn-capnweb-netlayer.ts`), with the stub
* methods only Cap'n Web has (`onRpcBroken`) made optional.
*
* @see https://github.com/ocapn/ocapn/blob/main/draft-specifications/Netlayers.md
*/
import { RpcTarget } from "capnweb";
export const TRANSPORT = "capnweb";
/**
* A peer's location, by designator alone: a router finds the peer.
*
* @param {string} designator
*/
export const locationOf = (designator) => ({ type: "ocapn-peer", transport: TRANSPORT, designator, hints: false });
/** Same as `locationToLocationId` in `@endo/ocapn`, which its exports map doesn't expose. */
const locationId = ({ designator, transport }) => `ocapn://${designator}.${transport}`;
/** Receives one direction of an OCapN connection, one message per call. */
export class ByteInbox extends RpcTarget {
#onMessage;
constructor(onMessage) {
super();
this.#onMessage = onMessage;
}
/** @param {Uint8Array} bytes */
deliver(bytes) {
// OCapN's syrup reader needs offset 0; a pooled Buffer view may not have it.
this.#onMessage(bytes.byteOffset === 0 ? bytes : new Uint8Array(bytes));
}
}
const dispose = (stub) => stub?.[Symbol.dispose]?.();
/** Write to a remote inbox, discarding each result at once so the session retains none. */
const makeSocket = (inbox, close) => ({
write: (bytes) => dispose(inbox.deliver(bytes)),
end: close,
});
/**
* A remote inbox reached through a gateway that is only a promise (a
* local async call, not a pipelining stub): writes queue in order behind it.
*/
const deferredInbox = (gatewayP, myInbox) => {
const inboxP = Promise.resolve(gatewayP).then((gateway) => gateway.connect(myInbox));
return {
deliver: (bytes) => void inboxP.then((inbox) => dispose(inbox.deliver(bytes))),
onRpcBroken: (cb) => void inboxP.then((inbox) => inbox.onRpcBroken?.(cb), cb),
};
};
/**
* Make an OCapN netlayer for `client.registerNetlayer`.
*
* @param {object} opts
* @param {string} opts.designator this peer's designator
* @param {(designator: string) => any} opts.openGateway the gateway of
* the peer with this designator: a stub or pipelining promise
* (`openGateway(d).connect(...)`), or a plain promise for one
* @returns a netlayer maker; the netlayer has a `gateway` that accepts
* incoming connections
*/
export const makeRpcNetLayer = ({ designator, openGateway }) => {
const location = locationOf(designator);
const connections = new Set();
return (handlers, logger) => {
const track = (connection, inbox) => {
connections.add(connection);
inbox.onRpcBroken?.((reason) => {
if (connections.delete(connection)) logger.log("rpc netlayer: connection broken", reason);
handlers.handleConnectionClose(connection, reason);
});
};
const netlayer = {
location,
locationId: locationId(location),
connect: (peer) => {
const gateway = openGateway(peer.designator);
let connection;
const myInbox = new ByteInbox((bytes) => handlers.handleMessageData(connection, bytes));
// Pipelined: writes before the peer answers queue in order.
const peerInbox =
typeof gateway.connect === "function" ? gateway.connect(myInbox) : deferredInbox(gateway, myInbox);
connection = handlers.makeConnection(
netlayer,
true,
makeSocket(peerInbox, () => {
connections.delete(connection);
dispose(gateway);
}),
);
track(connection, peerInbox);
return connection;
},
shutdown: () => {
for (const connection of [...connections]) connection.end();
},
};
class Gateway extends RpcTarget {
connect(peerInbox) {
const peer = peerInbox.dup?.() ?? peerInbox;
const connection = handlers.makeConnection(
netlayer,
false,
makeSocket(peer, () => {
connections.delete(connection);
dispose(peer);
}),
);
track(connection, peer);
return new ByteInbox((bytes) => handlers.handleMessageData(connection, bytes));
}
}
return Object.assign(netlayer, { gateway: new Gateway() });
};
};
+24
View File
@@ -0,0 +1,24 @@
{
"name": "toybox",
"version": "0.0.1",
"private": true,
"type": "module",
"scripts": {
"start": "node build.js && mkdir -p state && rm -f admin.sock && systemd-socket-activate -l 127.0.0.1:8787 -l \"$PWD/admin.sock\" -- node_modules/workerd/bin/workerd serve --experimental config.capnp -S http=3 -S admin=4 -d state=state -e \"switchboard=unix:$PWD/switchboard-public.sock\"",
"switchboard": "rm -f switchboard-public.sock switchboard-local.sock && systemd-socket-activate -l \"$PWD/switchboard-public.sock\" -l \"$PWD/switchboard-local.sock\" -- node_modules/workerd/bin/workerd serve --experimental switchboard.capnp -S public=3 -S local=4",
"test": "node build.js && node --import ./test/lockdown.js --test test/*.test.js",
"build": "node build.js",
"hello": "node --import ./test/lockdown.js scripts/hello.js"
},
"devDependencies": {
"esbuild": "^0.28.2",
"workerd": "1.20261006.1",
"ws": "8.22.0"
},
"dependencies": {
"@endo/far": "^1.1.14",
"@endo/init": "^1.1.13",
"@endo/ocapn": "1.1.1",
"capnweb": "0.12.0"
}
}
+578
View File
@@ -0,0 +1,578 @@
lockfileVersion: '9.0'
settings:
autoInstallPeers: true
excludeLinksFromLockfile: false
importers:
.:
dependencies:
'@endo/far':
specifier: ^1.1.14
version: 1.1.14
'@endo/init':
specifier: ^1.1.13
version: 1.1.13
'@endo/ocapn':
specifier: 1.1.1
version: 1.1.1
capnweb:
specifier: 0.12.0
version: 0.12.0
devDependencies:
esbuild:
specifier: ^0.28.2
version: 0.28.2
workerd:
specifier: 1.20261006.1
version: 1.20261006.1
ws:
specifier: 8.22.0
version: 8.22.0
packages:
'@cloudflare/workerd-darwin-64@1.20261006.1':
resolution: {integrity: sha512-lCX6avJO2It9AyDEYitjmOtJRVp/xmaW0TziyoLAnfxlzozyyRbSFfIIMMCiDx/sTP1++E4VLbV+7BHSD33f0Q==}
engines: {node: '>=16'}
cpu: [x64]
os: [darwin]
'@cloudflare/workerd-darwin-arm64@1.20261006.1':
resolution: {integrity: sha512-PyFGjOaldtppnLuKsdkoYWXfLVpDjuf8u+uoykX+1Xvxo2EPmCVViq0iDKm+U7c7PRno85ehS3/MPVXyk2HwHw==}
engines: {node: '>=16'}
cpu: [arm64]
os: [darwin]
'@cloudflare/workerd-linux-64@1.20261006.1':
resolution: {integrity: sha512-Ntf9S8eesmVufdjaT2kzWlz6Ng9XGYTc6cHQVRKAg59lFufwsIabGeMT6U4KkrpEeSIxgHq8aPtuLbZXE19tCA==}
engines: {node: '>=16'}
cpu: [x64]
os: [linux]
'@cloudflare/workerd-linux-arm64@1.20261006.1':
resolution: {integrity: sha512-JyeDBymVWw5tVRebSsVE8ojDYxd6UlZrAseHU8G+NpIyd5wF61tXlF/axLeCUqu0pOhBFHb8I5QKi0jwrCK41Q==}
engines: {node: '>=16'}
cpu: [arm64]
os: [linux]
'@cloudflare/workerd-windows-64@1.20261006.1':
resolution: {integrity: sha512-nWP0URNqOBN9fa/gZ7eh/w051DwjOXN8j4lDZDYVXyQPsux6sV3DfyYZT+CjR/rt9nKBSmJT9QE9/bWT18+84w==}
engines: {node: '>=16'}
cpu: [x64]
os: [win32]
'@endo/base64@1.1.1':
resolution: {integrity: sha512-6Aiy5oU/5/LYrHLbEPz0OWQ+CQsnAOC3qQYQ3nlwWi2JsU8fIZWjwfa+/+MSx8LTzITC250ZnMvtR0y/FIOQKw==}
'@endo/bytes@1.0.2':
resolution: {integrity: sha512-gKG8v4c7SD/QzULlFl1OKBuUCCy+9H78xsCL40hZn1q2/hVCNAHrxPJFO6VbA8Nk8UJP6P+NjvZY+xdrfwytMQ==}
'@endo/cache-map@1.1.0':
resolution: {integrity: sha512-owFGshs/97PDw9oguZqU/px8Lv1d0KjAUtDUiPwKHNXRVUE/jyettEbRoTbNJR1OaI8biMn6bHr9kVJsOh6dXw==}
'@endo/common@1.4.0':
resolution: {integrity: sha512-+d0CCzEYUP4PZDVGAUHMNfCEBn/GOyq/39y33N4FFUb6VPNJosfKFK5bI38mTaqwr6gxLGbzm8aeiElz+NaSDQ==}
'@endo/env-options@1.1.11':
resolution: {integrity: sha512-p9OnAPsdqoX4YJsE98e3NBVhIr2iW9gNZxHhAI2/Ul5TdRfoOViItzHzTqrgUVopw6XxA1u1uS6CykLMDUxarA==}
'@endo/errors@1.3.1':
resolution: {integrity: sha512-AmR2DSiahQR66SHlEczNWe0eUotMYE2uDwxPj9fAY2l2lovwHWGzZKXNXdQP/u62c8vetU8mG98Mh5vRZEvFTg==}
'@endo/eventual-send@1.5.0':
resolution: {integrity: sha512-mIRewS15mk+oV8y7kszyE0KWHZAlVoe98bEo/WJFJvBv1Rd83IjnofbuZcyWToWd70vgCxfkutWRJBK2Xtwd5A==}
'@endo/far@1.1.14':
resolution: {integrity: sha512-OlM+yCNWVqhwSZa6MyjkVB0oQSTAQ15KoiDGL1qowtnTnhuGUbOXuaUAsDf4eAJZ4L+h2KsJOUdyYEuhNsDPLg==}
'@endo/harden@1.1.0':
resolution: {integrity: sha512-d0CF7EVqymx9YUbkRI2sj10NrMLhrpmsaq5aBbEEbWDFxNe7F/mM8Gv8zT+iHErKEfeMCId90n5mk665SnxrRA==}
'@endo/hex@1.1.1':
resolution: {integrity: sha512-I2Q7roaHk0fE4JTj2GXfxHPR14qTFK9bOvp3nKf2Fk0r/CjgL4zFcHYolvqY15SRNmZDcywnQ+MBTW82b7BUpQ==}
'@endo/immutable-arraybuffer@2.0.0':
resolution: {integrity: sha512-xvfJvo4CgETh6UjSKwhNsFLZrqrTf2k4ZFpcprEN2UtCQRSOk+CcLP70kFbrOesB2Z/c1z9MaUZDS1JY6IlnUg==}
'@endo/init@1.1.13':
resolution: {integrity: sha512-7I94pHD0ax+WDj5dP1MKgzgSY6/GsdwCc/Ap2zFY4lzhCO944zQ4MChAW2yX28QKJK80VyJzqoW5BQ56ngEHHA==}
'@endo/lockdown@1.0.19':
resolution: {integrity: sha512-pcc6TqTBlx43422gHFzS+FrUfyJlbDoWUrynp78Mv+FrIZv4947FPcfjlXolgwvO00alMlSYxfXrrXUV93ywaA==}
'@endo/marshal@1.10.0':
resolution: {integrity: sha512-1ny6r0+XFpkpJNJEOApZQlFNB8FsDxzU2oCv1hpN1HjpszsKcTiyfXU/n/cJIs3KOOS06Tkt72pgPlasHpsVGA==}
'@endo/nat@5.2.0':
resolution: {integrity: sha512-yTNaysErNbrAWV8H2oC4e83nytG8Vzcu6VrxOkSIJQt+IVS9BUlSa1KkiKrVwu2IcIC67gJIkolPq3SgMGDjfQ==}
'@endo/ocapn@1.1.1':
resolution: {integrity: sha512-3VtHmyMsHwDPwcVusCTgtb8VXr539iZk+r8vzzOU6/cWyMhJuxRGOKR3r5EzDHVG4tY/R6tihoGt7QLnNDuceQ==}
'@endo/pass-style@1.8.2':
resolution: {integrity: sha512-63kbFNT+mropjQ0wzCkjSR5uEOT8FAIk16qHqrE+ciqSjMVd2hsW6vx10dbyo1s2xtmvXduUiTl+LqtnPa2oSw==}
'@endo/promise-kit@1.2.1':
resolution: {integrity: sha512-1nyXBOuUnVo/i47On8RCXPaT/1RIsSpJ1sGBClfZrcTtnsJfj8fElEVlJIE8NFsQiZPLNANYI+i/NYi837+s0Q==}
engines: {node: '>=11.0'}
'@endo/stream@1.3.1':
resolution: {integrity: sha512-v21VRzo5wjzw7XMHCWAmDS50hkUQi/m9C0MQTIAAqLm6w9+/Er9AEXl8FGIbz+VjtJWk5SNVbuqA51N7oBaLDQ==}
'@endo/syrup-frame@0.1.1':
resolution: {integrity: sha512-+uha+zI05RHPlSNVI7uaB2KD1EQ9gKAMcOzj1eT3y7Ng/vcW1bo7clCVVo3t+dOFOc6vYSsXgKufi40qxXRgYQ==}
'@esbuild/aix-ppc64@0.28.2':
resolution: {integrity: sha512-XExcO+dvLKvVtNTibSTBej1NCAbaGhWn9Ww1ZPx80qsahhPFe/8jgWP0IchNe0F3HwkU7n8ejhH8bjonqht8mQ==}
engines: {node: '>=18'}
cpu: [ppc64]
os: [aix]
'@esbuild/android-arm64@0.28.2':
resolution: {integrity: sha512-5YfKeeI8qWfBZIX+u2xZC3Zlb3Os/gLS2sbEKM+I4ZOcsWmHS2WLysCcQZDAFRslDUU5Oiq44gf6PYN1vGwG5A==}
engines: {node: '>=18'}
cpu: [arm64]
os: [android]
'@esbuild/android-arm@0.28.2':
resolution: {integrity: sha512-kXXoiPVVGQcnIYGOeaovwOURpniDBpSq4A03qkQ+BMQqtGG6HYap3xne9C1O1yo4TR3qxlCX5IqqmX6fFo2Lqg==}
engines: {node: '>=18'}
cpu: [arm]
os: [android]
'@esbuild/android-x64@0.28.2':
resolution: {integrity: sha512-O387ite7SzUyCcy3JQX4P4bLtEA7bLLkx+esve5JHnyYfNTxcVpXZo9jhdB0lTKN44gztELTdU7nS8Nr16Fs1Q==}
engines: {node: '>=18'}
cpu: [x64]
os: [android]
'@esbuild/darwin-arm64@0.28.2':
resolution: {integrity: sha512-n4KqkOQrraxHJcgjM1RvwbigfQKIKJVpM7xp+KsxiyUSrRdIXnt73VhrPAx0fV44hgfmIVKjxMN9J1t5jySVkw==}
engines: {node: '>=18'}
cpu: [arm64]
os: [darwin]
'@esbuild/darwin-x64@0.28.2':
resolution: {integrity: sha512-uq6suIWYP37qzGddBKPw5QEQPi6HiLGsO7UmkpfyaYNQ3D+rN6w6WfwH+nuqcGXWvawGwxOEroO4YGnFh95azw==}
engines: {node: '>=18'}
cpu: [x64]
os: [darwin]
'@esbuild/freebsd-arm64@0.28.2':
resolution: {integrity: sha512-n+I0BTSRIoy+d6RPKnEVwql5UwBJolytvY4mAOIEJorKlqgPII8ix6slVVrfZ5Tnj7glIZvloylbB/EJPMWEXw==}
engines: {node: '>=18'}
cpu: [arm64]
os: [freebsd]
'@esbuild/freebsd-x64@0.28.2':
resolution: {integrity: sha512-78XJTJkvPs0kz2w61301PJjXl4g7q3JqiYMZ/M/yVI73EHBrCRTgkhu9oqG7vPqq+a/yadEW8aD+agKlk5xrmg==}
engines: {node: '>=18'}
cpu: [x64]
os: [freebsd]
'@esbuild/linux-arm64@0.28.2':
resolution: {integrity: sha512-pW4AC0P3it8c7do9MVM4p51FzHzdM/TZrerurgRcHJ2WTa1VQ1CIq18xncfpBJw4ojkiZZrKW2yIBWBP92j6Ug==}
engines: {node: '>=18'}
cpu: [arm64]
os: [linux]
'@esbuild/linux-arm@0.28.2':
resolution: {integrity: sha512-XlDnu2q5yoqems+xay6wSAcg9DDD7K9RLKZEBOMZm3ckNpJBvOX20tSfby8KfrrhINDyv9V2YVZKY/SpoGJI8w==}
engines: {node: '>=18'}
cpu: [arm]
os: [linux]
'@esbuild/linux-ia32@0.28.2':
resolution: {integrity: sha512-CYbnj78HsIeA+DhgUKgFCfvNsTHFhMMrinUrMZpDXJXKN8T3XViTZ/+wtHeVxEWY8ewSzTFN+nRmSwO2tZaLUQ==}
engines: {node: '>=18'}
cpu: [ia32]
os: [linux]
'@esbuild/linux-loong64@0.28.2':
resolution: {integrity: sha512-buwkd8nsph4R+ajRvw0qM5Hja/TXQow3ptzWO2EbG/cqcIkHloRrdlBtQlshyYGTNFvfkfJ5tpPLVkY4DtsPfQ==}
engines: {node: '>=18'}
cpu: [loong64]
os: [linux]
'@esbuild/linux-mips64el@0.28.2':
resolution: {integrity: sha512-ZVykbDyk7519VwiNb9Lcj9m8XM6v5V9uKPvrEMkkEedVewf+0itkhahp4HDpgERXhwLRpWFypsGbG/J8s0QjJA==}
engines: {node: '>=18'}
cpu: [mips64el]
os: [linux]
'@esbuild/linux-ppc64@0.28.2':
resolution: {integrity: sha512-CAXl+Dtd9UUuJd8pKKdwh6MLm3MUMiqMPmhZ3tTSXPqfyQ3vDl6R5hZdZ/kYojK4ofXtdfSv1tFq8XzWx3heNQ==}
engines: {node: '>=18'}
cpu: [ppc64]
os: [linux]
'@esbuild/linux-riscv64@0.28.2':
resolution: {integrity: sha512-GeXCej4IQtU1B+QlDV8W/RRvbzI3O/Stss+/bCXv4lZls5WGRtu2a+3JkA3i4qIUlMXpcHebWpF8AkJhATowuA==}
engines: {node: '>=18'}
cpu: [riscv64]
os: [linux]
'@esbuild/linux-s390x@0.28.2':
resolution: {integrity: sha512-3H1weTYZPxt/WOhByszQZybS9w5lKzUn1FDMsgEChbHWQwHYQQRfBxgCcZvPhjHfKyJjIievvMmEUawJrdY9Dg==}
engines: {node: '>=18'}
cpu: [s390x]
os: [linux]
'@esbuild/linux-x64@0.28.2':
resolution: {integrity: sha512-4xTZr1FUmSoQW4XIWmit3tzQrUTZM+N3P0XV8xROKYF50XfI7xeO90+1bZvNwxIufQ9hDQVRJH5YhgPVF8A/HQ==}
engines: {node: '>=18'}
cpu: [x64]
os: [linux]
'@esbuild/netbsd-arm64@0.28.2':
resolution: {integrity: sha512-sSATRjPeDBg3pdgHoQfoYBob11Kk1FGa9lui5RIHZCoCkJa9QKlvl3/vKz2usCmYYjs7ymJR/2Nnsqe+Hjt5nw==}
engines: {node: '>=18'}
cpu: [arm64]
os: [netbsd]
'@esbuild/netbsd-x64@0.28.2':
resolution: {integrity: sha512-lqnzCV+mM0gIADaKihiCg6ifgfU2L3h5E33rNQBN1Y4MaVGnzryzmvvf7UHxprpQdE8hpqLolJ9Rl+SkIRDpyw==}
engines: {node: '>=18'}
cpu: [x64]
os: [netbsd]
'@esbuild/openbsd-arm64@0.28.2':
resolution: {integrity: sha512-AL2qJILH7lNjrDmCQDvdxMfAUIv8KMNZOvrwAQ8i8//ntL9FflhOyMJ8OZSMBb8/AWXe3/5v5S20y3zCoZWKoQ==}
engines: {node: '>=18'}
cpu: [arm64]
os: [openbsd]
'@esbuild/openbsd-x64@0.28.2':
resolution: {integrity: sha512-QtiuPytchRyC4rwUKhexJdQKvDuZ6hWloi3igqPQNUJCS1/v9EiO3UTOXR6A3FoMo4fnAKbWJdqaIwhOzh8qEw==}
engines: {node: '>=18'}
cpu: [x64]
os: [openbsd]
'@esbuild/openharmony-arm64@0.28.2':
resolution: {integrity: sha512-WkhYDmpTjLvGlScA1rwjRUmhl4k8oXR3cIbtqWmELgU/dFeHHlEllxDvdWcNJV9rbzCexB5vz8gtNewWLgCT7Q==}
engines: {node: '>=18'}
cpu: [arm64]
os: [openharmony]
'@esbuild/sunos-x64@0.28.2':
resolution: {integrity: sha512-GPMSkTOtMnv2U2F8gxe4Io6qmVs+YKyp832Etqqxr0hFngmXQ3rzwytelm3GIn7T4VviRUlf3sOgBOiTdvaf7g==}
engines: {node: '>=18'}
cpu: [x64]
os: [sunos]
'@esbuild/win32-arm64@0.28.2':
resolution: {integrity: sha512-PIhhEkE9uPBleRBrQEJpUn7MBnibZzbGzYWPmY3x+YoVg/95zbjB4CxPPOQ8l5tYYM4mMaCthF8/1DIfBQQyWQ==}
engines: {node: '>=18'}
cpu: [arm64]
os: [win32]
'@esbuild/win32-ia32@0.28.2':
resolution: {integrity: sha512-YmJbfTlvU7Sdn9BB+4PRES4oB6pxgS37MAONj+hBr/cpXS1aBPKXxNnDbu+QCWPj0o9dgyxeq79g6c5P8KeuYA==}
engines: {node: '>=18'}
cpu: [ia32]
os: [win32]
'@esbuild/win32-x64@0.28.2':
resolution: {integrity: sha512-5ebpxr3nWMzrL/rnUI755Jkuee0bHL/Gq0WTF9lvcpv73wAp5eu8MfBUgWK9bhWvZjj7yX8etf/8tI8Ney695g==}
engines: {node: '>=18'}
cpu: [x64]
os: [win32]
'@noble/curves@1.9.7':
resolution: {integrity: sha512-gbKGcRUYIjA3/zCCNaWDciTMFI0dCkvou3TL8Zmy5Nc7sJ47a0jtOeZoTaMxkuqRo9cRhjOdZJXegxYE5FN/xw==}
engines: {node: ^14.21.3 || >=16}
'@noble/hashes@1.8.0':
resolution: {integrity: sha512-jCs9ldd7NwzpgXDIf6P3+NrHh9/sD6CQdxHyjQI+h/6rDNo88ypBxxz45UDuZHz9r3tNz7N/VInSVoVdtXEI4A==}
engines: {node: ^14.21.3 || >=16}
capnweb@0.12.0:
resolution: {integrity: sha512-jgZ/LMtMVTi+RVlooIslH78Gg23XbDTdvcLghWx3oz17D6u5GuJARt+uhZk/wcTo+f81eeabfnow4d3zLBj+JQ==}
esbuild@0.28.2:
resolution: {integrity: sha512-HKVLS8dvII+xoKW9kmqxbRKrnWEXfJJr/FZhhJmiqIB0e053QNYFqOBouTMO/k5sID4MvCiUCvv8b9M4h32wIA==}
engines: {node: '>=18'}
hasBin: true
ses@2.3.0:
resolution: {integrity: sha512-qd3iWzDqGKllI2FmExKsWLrJwt+6COWb8jUdzaCn8Cq5OTeoXdQ62Gn/xLy0GWLORC3du6yBAVe/V0B0dLVKew==}
workerd@1.20261006.1:
resolution: {integrity: sha512-8MadBuffZ5NdpLwZjUmsqT5QlaKXq4gjgT6wzcKeynZTICNXyePhc+5QVoKiK7+MMhFTLNrRO2KXLlpk8Q6BkA==}
engines: {node: '>=16'}
hasBin: true
ws@8.22.0:
resolution: {integrity: sha512-Ydggc987+RO0AnWtZ/7Wq9FtNvcrL1b/RO0ud9mWjUPgDrsAAwQSF51sm2hm1XofbU/4jkpGEsLFsZZxU+1DOg==}
engines: {node: '>=10.0.0'}
peerDependencies:
bufferutil: ^4.0.1
utf-8-validate: '>=5.0.2'
peerDependenciesMeta:
bufferutil:
optional: true
utf-8-validate:
optional: true
snapshots:
'@cloudflare/workerd-darwin-64@1.20261006.1':
optional: true
'@cloudflare/workerd-darwin-arm64@1.20261006.1':
optional: true
'@cloudflare/workerd-linux-64@1.20261006.1':
optional: true
'@cloudflare/workerd-linux-arm64@1.20261006.1':
optional: true
'@cloudflare/workerd-windows-64@1.20261006.1':
optional: true
'@endo/base64@1.1.1': {}
'@endo/bytes@1.0.2':
dependencies:
'@endo/harden': 1.1.0
'@endo/immutable-arraybuffer': 2.0.0
'@endo/cache-map@1.1.0': {}
'@endo/common@1.4.0':
dependencies:
'@endo/errors': 1.3.1
'@endo/eventual-send': 1.5.0
'@endo/harden': 1.1.0
'@endo/promise-kit': 1.2.1
'@endo/env-options@1.1.11': {}
'@endo/errors@1.3.1':
dependencies:
'@endo/harden': 1.1.0
ses: 2.3.0
'@endo/eventual-send@1.5.0':
dependencies:
'@endo/env-options': 1.1.11
'@endo/harden': 1.1.0
'@endo/far@1.1.14':
dependencies:
'@endo/errors': 1.3.1
'@endo/eventual-send': 1.5.0
'@endo/pass-style': 1.8.2
'@endo/harden@1.1.0': {}
'@endo/hex@1.1.1':
dependencies:
'@endo/harden': 1.1.0
'@endo/immutable-arraybuffer@2.0.0': {}
'@endo/init@1.1.13':
dependencies:
'@endo/base64': 1.1.1
'@endo/eventual-send': 1.5.0
'@endo/harden': 1.1.0
'@endo/lockdown': 1.0.19
'@endo/promise-kit': 1.2.1
'@endo/lockdown@1.0.19':
dependencies:
ses: 2.3.0
'@endo/marshal@1.10.0':
dependencies:
'@endo/common': 1.4.0
'@endo/env-options': 1.1.11
'@endo/errors': 1.3.1
'@endo/eventual-send': 1.5.0
'@endo/harden': 1.1.0
'@endo/nat': 5.2.0
'@endo/pass-style': 1.8.2
'@endo/nat@5.2.0': {}
'@endo/ocapn@1.1.1':
dependencies:
'@endo/bytes': 1.0.2
'@endo/eventual-send': 1.5.0
'@endo/harden': 1.1.0
'@endo/hex': 1.1.1
'@endo/init': 1.1.13
'@endo/marshal': 1.10.0
'@endo/nat': 5.2.0
'@endo/pass-style': 1.8.2
'@endo/promise-kit': 1.2.1
'@endo/stream': 1.3.1
'@endo/syrup-frame': 0.1.1
'@noble/curves': 1.9.7
'@noble/hashes': 1.8.0
ws: 8.22.0
transitivePeerDependencies:
- bufferutil
- utf-8-validate
'@endo/pass-style@1.8.2':
dependencies:
'@endo/common': 1.4.0
'@endo/env-options': 1.1.11
'@endo/errors': 1.3.1
'@endo/eventual-send': 1.5.0
'@endo/harden': 1.1.0
'@endo/promise-kit': 1.2.1
'@endo/promise-kit@1.2.1':
dependencies:
'@endo/harden': 1.1.0
'@endo/stream@1.3.1':
dependencies:
'@endo/eventual-send': 1.5.0
'@endo/harden': 1.1.0
'@endo/promise-kit': 1.2.1
ses: 2.3.0
'@endo/syrup-frame@0.1.1':
dependencies:
'@endo/harden': 1.1.0
'@endo/init': 1.1.13
'@endo/promise-kit': 1.2.1
'@endo/stream': 1.3.1
'@esbuild/aix-ppc64@0.28.2':
optional: true
'@esbuild/android-arm64@0.28.2':
optional: true
'@esbuild/android-arm@0.28.2':
optional: true
'@esbuild/android-x64@0.28.2':
optional: true
'@esbuild/darwin-arm64@0.28.2':
optional: true
'@esbuild/darwin-x64@0.28.2':
optional: true
'@esbuild/freebsd-arm64@0.28.2':
optional: true
'@esbuild/freebsd-x64@0.28.2':
optional: true
'@esbuild/linux-arm64@0.28.2':
optional: true
'@esbuild/linux-arm@0.28.2':
optional: true
'@esbuild/linux-ia32@0.28.2':
optional: true
'@esbuild/linux-loong64@0.28.2':
optional: true
'@esbuild/linux-mips64el@0.28.2':
optional: true
'@esbuild/linux-ppc64@0.28.2':
optional: true
'@esbuild/linux-riscv64@0.28.2':
optional: true
'@esbuild/linux-s390x@0.28.2':
optional: true
'@esbuild/linux-x64@0.28.2':
optional: true
'@esbuild/netbsd-arm64@0.28.2':
optional: true
'@esbuild/netbsd-x64@0.28.2':
optional: true
'@esbuild/openbsd-arm64@0.28.2':
optional: true
'@esbuild/openbsd-x64@0.28.2':
optional: true
'@esbuild/openharmony-arm64@0.28.2':
optional: true
'@esbuild/sunos-x64@0.28.2':
optional: true
'@esbuild/win32-arm64@0.28.2':
optional: true
'@esbuild/win32-ia32@0.28.2':
optional: true
'@esbuild/win32-x64@0.28.2':
optional: true
'@noble/curves@1.9.7':
dependencies:
'@noble/hashes': 1.8.0
'@noble/hashes@1.8.0': {}
capnweb@0.12.0: {}
esbuild@0.28.2:
optionalDependencies:
'@esbuild/aix-ppc64': 0.28.2
'@esbuild/android-arm': 0.28.2
'@esbuild/android-arm64': 0.28.2
'@esbuild/android-x64': 0.28.2
'@esbuild/darwin-arm64': 0.28.2
'@esbuild/darwin-x64': 0.28.2
'@esbuild/freebsd-arm64': 0.28.2
'@esbuild/freebsd-x64': 0.28.2
'@esbuild/linux-arm': 0.28.2
'@esbuild/linux-arm64': 0.28.2
'@esbuild/linux-ia32': 0.28.2
'@esbuild/linux-loong64': 0.28.2
'@esbuild/linux-mips64el': 0.28.2
'@esbuild/linux-ppc64': 0.28.2
'@esbuild/linux-riscv64': 0.28.2
'@esbuild/linux-s390x': 0.28.2
'@esbuild/linux-x64': 0.28.2
'@esbuild/netbsd-arm64': 0.28.2
'@esbuild/netbsd-x64': 0.28.2
'@esbuild/openbsd-arm64': 0.28.2
'@esbuild/openbsd-x64': 0.28.2
'@esbuild/openharmony-arm64': 0.28.2
'@esbuild/sunos-x64': 0.28.2
'@esbuild/win32-arm64': 0.28.2
'@esbuild/win32-ia32': 0.28.2
'@esbuild/win32-x64': 0.28.2
ses@2.3.0:
dependencies:
'@endo/cache-map': 1.1.0
'@endo/env-options': 1.1.11
'@endo/immutable-arraybuffer': 2.0.0
workerd@1.20261006.1:
optionalDependencies:
'@cloudflare/workerd-darwin-64': 1.20261006.1
'@cloudflare/workerd-darwin-arm64': 1.20261006.1
'@cloudflare/workerd-linux-64': 1.20261006.1
'@cloudflare/workerd-linux-arm64': 1.20261006.1
'@cloudflare/workerd-windows-64': 1.20261006.1
ws@8.22.0: {}
+20
View File
@@ -0,0 +1,20 @@
/**
* @file Create the hello-world vat on a running toybox and mount it.
*
* Usage: node --import ./test/lockdown.js scripts/hello.js [name]
* then visit http://127.0.0.1:8787/?vat=<name> (default name: hello).
*/
import { readFile } from "node:fs/promises";
import { E } from "@endo/far";
import { connectClient } from "../client.js";
const name = process.argv[2] ?? "hello";
const client = await connectClient("admin.sock", `hello-script-${process.pid}`);
try {
const ingress = await E(client.vatAdmin).makeIngress({ vat: name });
const { id, root } = await E(client.vatAdmin).createVat(await readFile("vats/hello.js", "utf8"), { ingress });
await root; // its handoff, settled before the client goes
console.log(`vat ${id}\nhttp://127.0.0.1:8787/?vat=${name}`);
} finally {
client.close();
}
+575
View File
@@ -0,0 +1,575 @@
/** @file toybox supervisor: vats with code installed at runtime, each app a facet; ingress by mount, egress by grant. */
import "@endo/init";
import { DurableObject, WorkerEntrypoint } from "cloudflare:workers";
import { RpcTarget, newWebSocketRpcSession, newWorkersRpcResponse } from "capnweb";
import { E, Far, passStyleOf } from "@endo/far";
import { makeClient, swissnumFromBytes } from "@endo/ocapn";
import { mergeLabels, urlLabels } from "./labels.js";
import { LOOKUP, MOUNT, MOUNT_LABELS, MOUNT_SCHEMA, OVERLAP, UNMOUNT, UNMOUNT_LABELS } from "./mounts-sql.js";
import { locationOf, makeRpcNetLayer } from "./netlayer.js";
import VAT_RUNTIME from "vat-runtime.txt";
/**
* A stored grant: labels as JSON, or (before labels) a bare `host[:port]`.
*
* @param {string} stored
*/
const grantLabels = (stored) => {
if (stored.startsWith("{")) return JSON.parse(stored);
const { host, port } = urlLabels(`http://${stored}`);
return stored.includes(":") ? { host, port } : { host };
};
const hex = (buf) => [...new Uint8Array(buf)].map((b) => b.toString(16).padStart(2, "0")).join("");
/**
* One vat, by DO id: holds its installed code and egress grant; the app
* facet holds the app's state. Its mount, if any, is in the {@link Directory}.
*
* Code is stored by hash so the facet can be rebuilt after hibernation or a
* restart. The app's global `fetch` throws; a granted host arrives as
* `env.egress`, an {@link Egress} the app can pass along like any object.
*
* Each `start` begins a new incarnation, which names the loaded Worker: the
* same code with a different grant must not share a Worker.
*/
export class Vat extends DurableObject {
constructor(ctx, env) {
super(ctx, env);
ctx.storage.sql.exec("CREATE TABLE IF NOT EXISTS bundle (hash TEXT PRIMARY KEY, src TEXT)");
ctx.storage.sql.exec("CREATE TABLE IF NOT EXISTS current (k INTEGER PRIMARY KEY CHECK (k = 0), hash TEXT)");
ctx.storage.sql.exec(
"CREATE TABLE IF NOT EXISTS incarnation (k INTEGER PRIMARY KEY CHECK (k = 0), n INTEGER, egress TEXT)",
);
// absent: an app vat (exports class App), from before OCapN vats
ctx.storage.sql.exec("CREATE TABLE IF NOT EXISTS kind (k INTEGER PRIMARY KEY CHECK (k = 0), kind TEXT)");
}
/**
* Install `src` and run the app on it; the app's storage carries over.
*
* @param {string} src an ES module exporting `class App extends DurableObject`
* @param {string | null} egress the app's grant: egress labels as JSON (see labels.js), or none
* @param {"app" | "ocapn"} kind `app`: `src` exports `class App`; `ocapn`:
* `src` exports `buildRootObject`, run by vat-runtime.js
* @returns {Promise<string>} the code's SHA-256, hex
*/
async start(src, egress, kind = "app") {
const hash = hex(await crypto.subtle.digest("SHA-256", encodeUtf8(src)));
const { sql } = this.ctx.storage;
sql.exec("INSERT OR IGNORE INTO bundle VALUES (?, ?)", hash, src);
sql.exec("INSERT OR REPLACE INTO current VALUES (0, ?)", hash);
const [prev] = sql.exec("SELECT n FROM incarnation").toArray();
sql.exec("INSERT OR REPLACE INTO incarnation VALUES (0, ?, ?)", (prev?.n ?? 0) + 1, egress);
sql.exec("INSERT OR REPLACE INTO kind VALUES (0, ?)", kind);
this.ctx.facets.abort("app", new Error("upgraded"));
return hash;
}
#app({ src, n, egress, kind }) {
return this.ctx.facets.get("app", () => {
const env = egress ? { egress: this.ctx.exports.Egress({ props: { labels: grantLabels(egress) } }) } : {};
const code =
kind === "ocapn"
? {
mainModule: "runtime.js",
modules: { "runtime.js": VAT_RUNTIME, "vat.js": src },
env: { ...env, vatId: this.ctx.id.toString(), router: this.ctx.exports.Router({ props: {} }) },
}
: { mainModule: "app.js", modules: { "app.js": src }, env };
const worker = this.env.LOADER.get(`${this.ctx.id}:${n}`, () => ({
compatibilityDate: "2026-10-01",
...code,
globalOutbound: null,
}));
return { class: worker.getDurableObjectClass("App") };
});
}
#current() {
const [row] = this.ctx.storage.sql
.exec(
`SELECT b.hash, b.src, coalesce(i.n, 0) AS n, i.egress, coalesce(k.kind, 'app') AS kind
FROM current c JOIN bundle b USING (hash)
LEFT JOIN incarnation i ON i.k = 0 LEFT JOIN kind k ON k.k = 0`,
)
.toArray();
return row;
}
fetch(request) {
const row = this.#current();
if (!row) return new Response("no code installed\n", { status: 404 });
return this.#app(row).fetch(request);
}
/** An OCapN vat's gateway, for the router. */
gateway() {
const row = this.#current();
if (row?.kind !== "ocapn") throw new Error("not an OCapN vat");
return this.#app(row).gateway();
}
}
/**
* A vat's egress grant: a switchboard handle constrained to the labels in
* `ctx.props`, which no holder can change. The switchboard enforces them;
* a refusal comes back as a 403.
*/
export class Egress extends WorkerEntrypoint {
/** A narrower grant: these labels too (conflicting ones throw). */
constrain(more) {
return this.ctx.exports.Egress({ props: { labels: mergeLabels(this.ctx.props.labels, more) } });
}
async #handle() {
const resp = await this.env.SWITCHBOARD.fetch("http://switchboard/", { headers: { Upgrade: "websocket" } });
const ws = resp.webSocket;
if (!ws) throw new Error(`switchboard refused a session: ${resp.status}`);
ws.accept();
return newWebSocketRpcSession(ws).constrain(this.ctx.props.labels);
}
async fetch(request) {
try {
const handle = await this.#handle();
if (request.headers.get("Upgrade")?.toLowerCase() !== "websocket") return await handle.fetch(request);
const protocols = (request.headers.get("Sec-WebSocket-Protocol") ?? "").split(",").map((p) => p.trim()).filter(Boolean);
const { readable, writable, protocol } = await handle.webSocket(request.url, protocols);
return this.#bridge(readable, writable, protocol);
} catch (e) {
return new Response(`egress refused: ${e.message}\n`, { status: 403 });
}
}
/** A WebSocket for the vat whose messages travel the switchboard's streams. */
#bridge(readable, writable, protocol) {
const [client, server] = Object.values(new WebSocketPair());
server.accept();
const writer = writable.getWriter();
server.addEventListener("message", (e) => writer.write(e.data).catch(() => server.close(1011)));
server.addEventListener("close", () => writer.close().catch(() => {}));
this.ctx.waitUntil(
(async () => {
for await (const m of readable) server.send(m);
server.close();
})().catch(() => server.close(1011)),
);
const headers = protocol ? { "Sec-WebSocket-Protocol": protocol } : {};
return new Response(null, { status: 101, webSocket: client, headers });
}
}
/** The vatAdmin's designator, and the swissnum of its bootstrap object. */
const VAT_ADMIN = "vatAdmin";
/** The swissnum of an OCapN vat's bootstrap (see vat-runtime.js). */
const VAT_SWISSNUM = "vat";
/** UTF-8 bytes of a string: the method alone, so no encoder object is in module scope. */
const encodeUtf8 = harden(TextEncoder.prototype.encode.bind(new TextEncoder()));
/**
* The vatAdmin: an OCapN peer, and the router through which every OCapN
* peer here reaches every other. One instance, by name.
*
* Peers find each other by designator: the vatAdmin's is `vatAdmin`, a
* vat's is its id, and a client registers its own when it connects to
* the admin socket. Nothing here can dial out, so a handoff's receiver
* reaches the exporter through this router.
*/
export class VatAdmin extends DurableObject {
#clients = new Map();
/** mount key → the handler serving it: live, so lost when this object is evicted */
#handlers = new Map();
/** mount key → resolvers waiting for a handler */
#waiting = new Map();
/** mount key ↔ its ingress object, made again as needed */
#ingresses = new Map();
#ingressKeys = new WeakMap();
#reviving = new Map();
#ready;
constructor(ctx, env) {
super(ctx, env);
const { sql } = ctx.storage;
// a vat's params, encoded (see #encode); a vat absent here can't be restarted
sql.exec("CREATE TABLE IF NOT EXISTS vat_params (vat TEXT PRIMARY KEY, params TEXT)");
// which vat was given which ingress, so a request can restart it
sql.exec("CREATE TABLE IF NOT EXISTS ingress_vat (key TEXT PRIMARY KEY, vat TEXT)");
this.#ready = this.#boot();
}
/** @param {string} key */
#ingressFor(key) {
let ingress = this.#ingresses.get(key);
if (!ingress) {
ingress = Far("ingress", {
/** @param {{ handle: (request: object) => object }} handler */
serve: (handler) => {
this.#handlers.set(key, handler);
for (const resolve of this.#waiting.get(key) ?? []) resolve();
this.#waiting.delete(key);
},
});
this.#ingresses.set(key, ingress);
this.#ingressKeys.set(ingress, key);
}
return ingress;
}
/**
* `params` as JSON, an ingress by its key; `null` if anything in it can't
* be made again here (a remote object of another peer, say).
*/
#encode(params) {
const keys = [];
const walk = (v) => {
const style = passStyleOf(v);
if (style === "remotable") {
const key = this.#ingressKeys.get(v);
if (!key) throw new Error("not restartable");
keys.push(key);
return { "@ingress": key };
}
if (style === "copyArray") return v.map(walk);
if (style === "copyRecord") return Object.fromEntries(Object.entries(v).map(([k, x]) => [k, walk(x)]));
if (["string", "number", "boolean", "null"].includes(style)) return v;
throw new Error("not restartable");
};
try {
return { json: JSON.stringify(walk(params)), keys };
} catch {
return null;
}
}
#decode(json) {
const walk = (v) => {
if (Array.isArray(v)) return v.map(walk);
if (v && typeof v === "object") {
if ("@ingress" in v) return this.#ingressFor(v["@ingress"]);
return Object.fromEntries(Object.entries(v).map(([k, x]) => [k, walk(x)]));
}
return v;
};
return harden(walk(JSON.parse(json)));
}
/** (Re)start a vat's code with `params`: it rebuilds its root. */
async #startVat(vat, params) {
const vatBootstrap = await this.#enliven(vat, VAT_SWISSNUM);
return E(vatBootstrap).start(params);
}
/**
* The handler for a mount whose handler went with an evicted instance:
* restart the vat given its ingress, and wait for it to serve again.
*
* @param {string} key
*/
#revive(key) {
let reviving = this.#reviving.get(key);
if (!reviving) {
reviving = (async () => {
const [row] = this.ctx.storage.sql
.exec("SELECT i.vat, p.params FROM ingress_vat i JOIN vat_params p USING (vat) WHERE i.key = ?", key)
.toArray();
if (!row) return null;
const served = new Promise((resolve) => this.#waiting.set(key, [...(this.#waiting.get(key) ?? []), resolve]));
await this.#startVat(row.vat, this.#decode(row.params));
await Promise.race([served, new Promise((resolve) => setTimeout(resolve, 5000))]);
return this.#handlers.get(key) ?? null;
})().finally(() => this.#reviving.delete(key));
this.#reviving.set(key, reviving);
}
return reviving;
}
async #boot() {
const vatAdmin = Far("vatAdmin", {
/**
* Create an OCapN vat running `src`; `params` (any passable,
* remote objects included) reach its `buildRootObject`.
*
* If `params` holds only data and ingresses, they are kept, and the
* vat is restarted with them when a request needs it after an
* eviction or a restart.
*
* @param {string} src an ES module exporting `buildRootObject(vatPowers, params)`
*/
createVat: async (src, params = {}) => {
const id = this.env.VATS.newUniqueId();
const vat = id.toString();
await this.env.VATS.get(id).start(src, null, "ocapn");
const encoded = this.#encode(params);
if (encoded) {
const { sql } = this.ctx.storage;
sql.exec("INSERT INTO vat_params VALUES (?, ?)", vat, encoded.json);
for (const key of encoded.keys) sql.exec("INSERT OR REPLACE INTO ingress_vat VALUES (?, ?)", key, vat);
}
const root = await this.#startVat(vat, params);
return harden({ id: vat, root });
},
/**
* Reserve a mount at `labels` and return the ingress for it: whoever
* holds the ingress (a vat it is passed to, say) serves requests
* that carry those labels.
*
* @param {Record<string, string>} labels
*/
makeIngress: async (labels) => {
const key = `ingress:${crypto.randomUUID()}`;
await this.env.DIRECTORY.getByName("directory").mount(key, harden({ ...labels }));
return this.#ingressFor(key);
},
});
// workerd runs FinalizationRegistry callbacks outside any request, where
// sending the resulting GC message is forbidden I/O. So imports are
// never collected here; they go when their session does.
this.ocapn = makeClient({
debugLabel: VAT_ADMIN,
swissnumTable: new Map([[VAT_ADMIN, vatAdmin]]),
enableImportCollection: false,
});
const netlayer = await this.ocapn.registerNetlayer(
makeRpcNetLayer({ designator: VAT_ADMIN, openGateway: (designator) => this.gateway(designator) }),
);
this.own = netlayer.gateway;
}
#enliven(designator, swissnum) {
return this.ocapn.enlivenSturdyRef(
this.ocapn.makeSturdyRef(locationOf(designator), swissnumFromBytes(encodeUtf8(swissnum))),
);
}
/**
* The gateway of the peer with this designator.
*
* @param {string} designator
*/
async gateway(designator) {
await this.#ready;
if (designator === VAT_ADMIN) return this.own;
const client = this.#clients.get(designator);
if (client) return client;
let id;
try {
id = this.env.VATS.idFromString(designator);
} catch {
throw new Error("no such peer");
}
return this.env.VATS.get(id).gateway();
}
/**
* One HTTP request, as passable data, for the handler serving `key`;
* `null` when nothing serves it.
*
* @param {string} key
* @param {object} request
*/
async serveHttp(key, request) {
await this.#ready;
const handler = this.#handlers.get(key) ?? (await this.#revive(key));
return handler ? E(handler).handle(harden(request)) : null;
}
/** A WebSocket here starts a Cap'n Web session whose main object is an {@link AdminSession}. */
fetch(request) {
return newWorkersRpcResponse(request, new AdminSession(this, this.#clients));
}
}
/**
* What an OCapN client on the admin socket gets: register its own
* gateway under a designator, and reach any peer's.
*/
class AdminSession extends RpcTarget {
#vatAdmin;
#clients;
constructor(vatAdmin, clients) {
super();
this.#vatAdmin = vatAdmin;
this.#clients = clients;
}
/** @param {string} designator @param {any} gateway */
register(designator, gateway) {
if (designator === VAT_ADMIN || /^[0-9a-f]{64}$/.test(designator) || this.#clients.has(designator)) {
throw new Error("designator taken");
}
this.#clients.set(designator, gateway.dup?.() ?? gateway);
}
/** @param {string} designator */
gateway(designator) {
return this.#vatAdmin.gateway(designator);
}
}
/** A vat's way to the router: `env.router` in an OCapN vat. */
export class Router extends WorkerEntrypoint {
gateway(designator) {
return this.env.VAT_ADMIN.getByName(VAT_ADMIN).gateway(designator);
}
}
/**
* Mounts: which vat serves which label set. One instance, by name.
*
* Registration follows Flower: a mount must conflict with every other, so
* no request can carry two mounts' labels. Lookup is stricter than
* Flower's: the request must carry all of a mount's labels.
*/
export class Directory extends DurableObject {
constructor(ctx, env) {
super(ctx, env);
for (const statement of MOUNT_SCHEMA) ctx.storage.sql.exec(statement);
}
/**
* Mount `vat` at `labels`, replacing its mount; `null` unmounts it.
*
* @param {string} vat a Vat's DO id, hex
* @param {Record<string, string> | null} labels
*/
mount(vat, labels) {
const { sql } = this.ctx.storage;
this.ctx.storage.transactionSync(() => {
sql.exec(UNMOUNT_LABELS, vat);
if (!labels) return void sql.exec(UNMOUNT, vat);
const json = JSON.stringify(labels);
if (sql.exec(OVERLAP, json, vat).toArray().length) {
throw new Error("mount overlaps another: add a label whose value tells them apart");
}
sql.exec(MOUNT, vat, json);
sql.exec(MOUNT_LABELS, vat, json);
});
}
/**
* The mount whose labels `request` carries, if any. Registration keeps
* mounts pairwise conflicting, so at most one can match.
*
* @param {Record<string, string>} request
*/
lookup(request) {
const [row] = this.ctx.storage.sql.exec(LOOKUP, JSON.stringify(request)).toArray();
return row ? { vat: row.vat, labels: JSON.parse(row.labels) } : null;
}
}
/**
* A request's labels: its query parameters.
*
* Routing carries no secrets: a URL ends up in logs and `Referer`s. A
* secret travels in-band instead, as on a webkey page: the page keeps it
* in its fragment, reaches the vat by public labels, and sends it in the
* body of a call, which the vat checks like a swissnum.
*
* @param {Request} request
*/
const requestLabels = (request) => {
const labels = {};
for (const [k, v] of new URL(request.url).searchParams) {
if (k in labels) throw new Error(`label ${k} given twice`);
labels[k] = v;
}
return Object.freeze(labels);
};
/**
* The http socket: the ingress. A request reaches the vat mounted at labels
* it carries, which sees the mount's and the request's labels together in
* a `Toybox-Connection-Labels` header (JSON), as a Flower server gets its
* connection labels.
*/
export class Public extends WorkerEntrypoint {
async fetch(request) {
let labels;
try {
labels = requestLabels(request);
} catch (e) {
return new Response(`bad labels: ${e.message}\n`, { status: 400 });
}
const mount = await this.env.DIRECTORY.getByName("directory").lookup(labels);
if (!mount) return new Response("nothing mounted here\n", { status: 404 });
if (mount.vat.startsWith("ingress:")) return this.#serveOcapn(mount, labels, request);
const forwarded = new Request(request);
forwarded.headers.set("Toybox-Connection-Labels", JSON.stringify(mergeLabels(mount.labels, labels)));
const resp = await this.env.VATS.get(this.env.VATS.idFromString(mount.vat)).fetch(forwarded);
if (resp.webSocket) return resp;
const sealed = new Response(resp.body, resp);
sealed.headers.set("Referrer-Policy", "no-referrer");
return sealed;
}
/** A mount made by `makeIngress`: the request goes, as data, to whoever serves it. */
async #serveOcapn(mount, labels, request) {
const { pathname } = new URL(request.url);
const body = ["GET", "HEAD"].includes(request.method) ? "" : await request.text();
const req = { method: request.method, path: pathname, labels: mergeLabels(mount.labels, labels), body };
const res = await this.env.VAT_ADMIN.getByName(VAT_ADMIN).serveHttp(mount.vat, req);
if (!res) return new Response("mounted, but nothing serves it now\n", { status: 503 });
const resp = new Response(res.body ?? "", { status: res.status ?? 200, headers: res.headers ?? {} });
resp.headers.set("Referrer-Policy", "no-referrer");
return resp;
}
}
/**
* Grants in a query parameter, as JSON labels; absent means none.
*
* @param {URL} url
* @param {string} name
*/
const grantParam = (url, name) => {
const v = url.searchParams.get(name);
return v ? JSON.parse(v) : null;
};
/**
* The admin socket: a WebSocket upgrade reaches the {@link VatAdmin} (OCapN
* over Cap'n Web); otherwise, app vats over HTTP:
*
* - `POST /` with an ES module body creates a vat and starts it; the reply
* is `{"vat": <id>, "hash": <code hash>}`, and the id addresses it here.
* - `PUT /<id>` with a body upgrades it; its storage carries over.
* - `PUT /by-name/<name>` does the same for a vat from before ids, when
* vats were named.
*
* Every start states the vat's grants, absent meaning none:
* `?egress=<labels>` (see {@link Egress}) and `?mount=<labels>` (ingress:
* see {@link Public}).
*/
export class Admin extends WorkerEntrypoint {
async fetch(request) {
if (request.headers.get("Upgrade")?.toLowerCase() === "websocket") {
return this.env.VAT_ADMIN.getByName(VAT_ADMIN).fetch(request);
}
const url = new URL(request.url);
const [, first, second] = url.pathname.split("/");
let id;
if (request.method === "POST" && !first) id = this.env.VATS.newUniqueId();
else if (request.method === "PUT" && first === "by-name" && second) id = this.env.VATS.idFromName(second);
else if (request.method === "PUT" && first && !second) {
try {
id = this.env.VATS.idFromString(first);
} catch {
return new Response("no such vat\n", { status: 404 });
}
} else return new Response("POST / or PUT /<id> or PUT /by-name/<name>, with code\n", { status: 405 });
const vat = id.toString();
try {
await this.env.DIRECTORY.getByName("directory").mount(vat, grantParam(url, "mount"));
} catch (e) {
return new Response(`${e.message}\n`, { status: 409 });
}
const egress = url.searchParams.get("egress");
const hash = await this.env.VATS.get(id).start(await request.text(), egress);
return Response.json({ vat, hash });
}
}
export default Public;
+34
View File
@@ -0,0 +1,34 @@
# switchboard: toybox's only way out, after Flower's switchboard.
#
# Each socket is a handle with fixed labels: a WebSocket upgrade starts a
# Cap'n Web session whose main object is that handle (see switchboard.js).
# Whoever starts workerd hands the sockets in (-S/-s), so whoever holds a
# socket's fd holds that handle and no other.
using Workerd = import "/workerd/workerd.capnp";
const config :Workerd.Config = (
services = [
(name = "main", worker = .switchboard),
(name = "nowhere", network = (allow = [])),
(name = "public-net", network = (allow = ["public"], tlsOptions = (trustBrowserCas = true))),
(name = "local-net", network = (allow = ["local"], tlsOptions = (trustBrowserCas = true))),
],
sockets = [
(name = "public", http = (), service = (name = "main", props = (json = "{\"network\":\"public\"}"))),
(name = "local", http = (), service = (name = "main", props = (json = "{\"network\":\"local\"}"))),
],
);
const switchboard :Workerd.Worker = (
compatibilityDate = "2026-10-01",
modules = [
(name = "switchboard.js", esModule = embed "switchboard.js"),
(name = "labels.js", esModule = embed "labels.js"),
(name = "capnweb", esModule = embed "node_modules/capnweb/dist/index-workers.js"),
],
bindings = [
(name = "PUBLIC", service = "public-net"),
(name = "LOCAL", service = "local-net"),
],
globalOutbound = "nowhere",
);
+81
View File
@@ -0,0 +1,81 @@
/** @file switchboard: Flower-style egress handles, served over Cap'n Web. */
import { WorkerEntrypoint } from "cloudflare:workers";
import { RpcTarget, newWorkersRpcResponse } from "capnweb";
import { checkUrl, mergeLabels } from "./labels.js";
/**
* One egress handle: reaches only URLs within its labels, through the
* network its `network` label names. Like a Flower switchboard handle, it
* can only be narrowed; unlike one, it is a Cap'n Web stub, not an fd.
*/
export class Handle extends RpcTarget {
#labels;
#net;
/**
* @param {Record<string, string>} labels
* @param {Fetcher} net
*/
constructor(labels, net) {
super();
this.#labels = labels;
this.#net = net;
}
/** @param {Record<string, string>} more */
constrain(more) {
return new Handle(mergeLabels(this.#labels, more), this.#net);
}
labels() {
return this.#labels;
}
/** @param {Request} request */
fetch(request) {
checkUrl(this.#labels, request.url);
return this.#net.fetch(request);
}
/**
* Open a WebSocket to `url`; its messages cross the session as streams.
*
* @param {string} url `http:` or `https:`, as for a `fetch` upgrade
* @param {string[]} [protocols]
*/
async webSocket(url, protocols = []) {
checkUrl(this.#labels, url);
const headers = { Upgrade: "websocket" };
if (protocols.length) headers["Sec-WebSocket-Protocol"] = protocols.join(", ");
const resp = await this.#net.fetch(url, { headers });
const ws = resp.webSocket;
if (!ws) throw new Error(`no websocket: ${resp.status}`);
ws.accept();
const readable = new ReadableStream({
start(controller) {
ws.addEventListener("message", (e) => controller.enqueue(e.data));
ws.addEventListener("close", () => controller.close());
ws.addEventListener("error", () => controller.error(new Error("websocket error")));
},
});
const writable = new WritableStream({
write: (chunk) => ws.send(chunk),
close: () => ws.close(),
abort: () => ws.close(1011),
});
return { readable, writable, protocol: resp.headers.get("Sec-WebSocket-Protocol") ?? "" };
}
}
/**
* A switchboard socket: a WebSocket upgrade here starts a Cap'n Web
* session whose main object is a {@link Handle} with this socket's labels.
*/
export default class Root extends WorkerEntrypoint {
fetch(request) {
const labels = Object.freeze({ ...this.ctx.props });
const net = { public: this.env.PUBLIC, local: this.env.LOCAL }[labels.network];
if (!net) return new Response("socket has no network\n", { status: 500 });
return newWorkersRpcResponse(request, new Handle(labels, net));
}
}
+15
View File
@@ -0,0 +1,15 @@
[Unit]
Description=toybox switchboard (workerd, Cap'n Web egress handles)
Requires=toybox-switchboard.socket
[Service]
# the local (localhost) handle goes where no other unit can open it;
# hand it out deliberately, if ever
ExecStart=/opt/toybox/node_modules/workerd/bin/workerd serve --experimental /opt/toybox/switchboard.capnp -S public=3 -s local=unix:/run/toybox-switchboard/local.sock
RuntimeDirectory=toybox-switchboard
RuntimeDirectoryMode=0700
DynamicUser=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
SystemCallFilter=@system-service
+11
View File
@@ -0,0 +1,11 @@
[Unit]
Description=toybox switchboard: the public egress handle
[Socket]
ListenStream=/run/toybox/switchboard-public.sock
# only toybox holds this handle: its service joins this group
SocketGroup=toybox
SocketMode=0660
[Install]
WantedBy=sockets.target
+19
View File
@@ -0,0 +1,19 @@
[Unit]
Description=toybox vats (workerd)
Requires=toybox.socket toybox-switchboard.socket
After=toybox-switchboard.socket
[Service]
ExecStart=/opt/toybox/node_modules/workerd/bin/workerd serve --experimental /opt/toybox/config.capnp -S http=3 -S admin=4 -d state=/var/lib/toybox -e switchboard=unix:/run/toybox/switchboard-public.sock
DynamicUser=yes
# to open the public switchboard socket
SupplementaryGroups=toybox
StateDirectory=toybox
# no network but the sockets handed in, and the switchboard's
PrivateNetwork=yes
RestrictAddressFamilies=AF_UNIX
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
SystemCallFilter=@system-service
# not MemoryDenyWriteExecute: V8's JIT needs W+X
+13
View File
@@ -0,0 +1,13 @@
[Unit]
Description=toybox vats: public port and admin socket
[Socket]
# order sets the fds: http=3, admin=4 (see toybox.service)
ListenStream=127.0.0.1:8787
ListenStream=/run/toybox/admin.sock
# whoever can open admin.sock can install code
SocketUser=root
SocketMode=0600
[Install]
WantedBy=sockets.target
+10
View File
@@ -0,0 +1,10 @@
// @endo/init/debug.js, but with `overrideTaming: 'moderate'`: Node 24's
// undici assigns `message` to its errors (ClientDestroyedError when a
// WebSocket closes), which 'min' makes a TypeError.
import '@endo/init/pre-remoting.js';
lockdown({
errorTaming: 'unsafe',
overrideTaming: 'moderate',
domainTaming: 'unsafe',
});
+53
View File
@@ -0,0 +1,53 @@
/** @file The ingress's mount SQL, on node:sqlite: lookup, overlap, and mounts stored before mount_label. */
import { DatabaseSync } from "node:sqlite";
import { test } from "node:test";
import assert from "node:assert/strict";
import { LOOKUP, MOUNT, MOUNT_LABELS, MOUNT_SCHEMA, OVERLAP } from "../mounts-sql.js";
const makeDb = () => {
const db = new DatabaseSync(":memory:");
for (const statement of MOUNT_SCHEMA) db.exec(statement);
const mount = (vat, labels) => {
const json = JSON.stringify(labels);
db.prepare(MOUNT).run(vat, json);
db.prepare(MOUNT_LABELS).run(vat, json);
};
const lookup = (request) => db.prepare(LOOKUP).get(JSON.stringify(request))?.vat ?? null;
const overlaps = (vat, labels) => !!db.prepare(OVERLAP).get(JSON.stringify(labels), vat);
return { db, mount, lookup, overlaps };
};
test("a request reaches the mount whose every label it carries", () => {
const { mount, lookup } = makeDb();
mount("a", { vat: "alice", flavor: "plain" });
mount("b", { vat: "bob" });
assert.equal(lookup({ vat: "alice", flavor: "plain", page: "2" }), "a");
assert.equal(lookup({ vat: "alice" }), null);
assert.equal(lookup({ vat: "alice", flavor: "fancy" }), null);
assert.equal(lookup({ vat: "bob" }), "b");
assert.equal(lookup({}), null);
});
test("a mount with no labels matches any request", () => {
const { mount, lookup } = makeDb();
mount("only", {});
assert.equal(lookup({ anything: "x" }), "only");
});
test("overlap: a mount must conflict with every other", () => {
const { mount, overlaps } = makeDb();
mount("a", { vat: "alice" });
assert.equal(overlaps("b", { vat: "alice", n: "2" }), true);
assert.equal(overlaps("b", { app: "x" }), true);
assert.equal(overlaps("b", { vat: "bob" }), false);
assert.equal(overlaps("a", { vat: "alice", n: "2" }), false, "a vat's own mount doesn't count");
});
test("mounts stored before mount_label are filled in when the schema runs", () => {
const db = new DatabaseSync(":memory:");
db.exec("CREATE TABLE mount (vat TEXT PRIMARY KEY, labels TEXT)");
db.prepare("INSERT INTO mount VALUES (?, ?)").run("old", JSON.stringify({ vat: "old" }));
for (const statement of MOUNT_SCHEMA) db.exec(statement);
assert.equal(db.prepare(LOOKUP).get(JSON.stringify({ vat: "old" }))?.vat, "old");
assert.equal(db.prepare(LOOKUP).get(JSON.stringify({ vat: "other" })), undefined);
});
+105
View File
@@ -0,0 +1,105 @@
/** @file OCapN vats: a client on the admin socket creates a vat and passes it a remote object. */
import { mkdir, mkdtemp, readFile, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { after, before, test } from "node:test";
import assert from "node:assert/strict";
import { E, Far } from "@endo/far";
import { connectClient } from "../client.js";
import { freePort, here, startWorkerd } from "./workerd.js";
const vatSrc = (name) => readFile(join(here, "vats", name), "utf8");
let dir, port, server, client;
const startToybox = () =>
startWorkerd(
"config.capnp",
[
"-s", `http=127.0.0.1:${port}`, "-s", `admin=unix:${join(dir, "admin.sock")}`,
"-d", `state=${join(dir, "state")}`, "-e", `switchboard=unix:${join(dir, "no-switchboard.sock")}`,
],
{ port, host: "127.0.0.1" },
[join(dir, "admin.sock")],
);
before(async () => {
dir = await mkdtemp(join(tmpdir(), "toybox-ocapn-"));
await mkdir(join(dir, "state"));
port = await freePort();
server = await startToybox();
client = await connectClient(join(dir, "admin.sock"), "client-1", { verbose: !!process.env.OCAPN_LOG });
});
after(async () => {
client?.close();
await server?.stop();
await rm(dir, { recursive: true, force: true });
});
test("createVat passes a remote object in; the vat's root comes back", async () => {
const carol = Far("carol", { greet: (name) => `hi ${name}, from carol` });
const { id, root } = await E(client.vatAdmin).createVat(await vatSrc("greeter.js"), { carol });
assert.match(id, /^[0-9a-f]{64}$/);
assert.equal(await E(root).hello(), "hello from the vat");
assert.equal(await E(root).greetVia("vat"), "hi vat, from carol");
});
test("a counter vat counts; two counter vats count apart", async () => {
const src = await vatSrc("counter.js");
const { root: a } = await E(client.vatAdmin).createVat(src);
const { root: b } = await E(client.vatAdmin).createVat(src);
assert.equal(await E(a).increment(), 1);
assert.equal(await E(a).increment(), 2);
assert.equal(await E(b).increment(), 1);
assert.equal(await E(a).read(), 2);
assert.equal(await E(b).read(), 1);
});
test("a vat serves the web through the ingress it is given", async () => {
const ingress = await E(client.vatAdmin).makeIngress({ vat: "hello" });
await E(client.vatAdmin).createVat(await vatSrc("hello.js"), { ingress });
const resp = await fetch(`http://127.0.0.1:${port}/greet?vat=hello`);
assert.equal(resp.status, 200);
assert.equal(resp.headers.get("content-type"), "text/html; charset=utf-8");
assert.match(await resp.text(), /<h1>Hello, world<\/h1><p>from a toybox vat, at \/greet<\/p>/);
});
test("an ingress nobody serves is a 503", async () => {
await E(client.vatAdmin).makeIngress({ vat: "idle" });
assert.equal((await fetch(`http://127.0.0.1:${port}/?vat=idle`)).status, 503);
});
test("after a crash, a request restarts the vat that serves it", async () => {
// its own toybox: a crash breaks every session to it, the shared client's too
const home = await mkdtemp(join(tmpdir(), "toybox-phoenix-"));
await mkdir(join(home, "state"));
const port2 = await freePort();
const start = () =>
startWorkerd(
"config.capnp",
[
"-s", `http=127.0.0.1:${port2}`, "-s", `admin=unix:${join(home, "admin.sock")}`,
"-d", `state=${join(home, "state")}`, "-e", `switchboard=unix:${join(home, "no-switchboard.sock")}`,
],
{ port: port2, host: "127.0.0.1" },
[join(home, "admin.sock")],
);
let toybox = await start();
try {
const creator = await connectClient(join(home, "admin.sock"), "creator");
const ingress = await E(creator.vatAdmin).makeIngress({ vat: "phoenix" });
const { root } = await E(creator.vatAdmin).createVat(await vatSrc("hello.js"), { ingress });
assert.equal((await fetch(`http://127.0.0.1:${port2}/?vat=phoenix`)).status, 200);
await root; // its handoff, settled before the client goes
creator.close();
await toybox.stop();
toybox = await start();
const resp = await fetch(`http://127.0.0.1:${port2}/again?vat=phoenix`);
assert.equal(resp.status, 200);
assert.match(await resp.text(), /Hello, world.*at \/again/);
} finally {
await toybox.stop();
await rm(home, { recursive: true, force: true });
}
});
+266
View File
@@ -0,0 +1,266 @@
/** @file toybox end to end: a real workerd, an admin socket, a public port, a switchboard with two handles. */
import { mkdir, mkdtemp, readFile, rm } from "node:fs/promises";
import { createServer as createHttpServer, request } from "node:http";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { after, before, test } from "node:test";
import assert from "node:assert/strict";
import { WebSocketServer } from "ws";
import { freePort, here, startWorkerd } from "./workerd.js";
const app = (name) => readFile(join(here, "apps", name), "utf8");
/**
* The admin socket: holding it is the authority to create and upgrade vats.
*
* @param {string} socketPath
*/
const makeAdmin = (socketPath) => {
const send = (method, path, src, { egress, mount } = {}) =>
new Promise((resolve, reject) => {
const q = new URLSearchParams();
if (egress) q.set("egress", typeof egress === "string" ? egress : JSON.stringify(egress));
if (mount) q.set("mount", JSON.stringify(mount));
const req = request({ socketPath, path: `${path}?${q}`, method, agent: false }, (res) => {
let body = "";
res.on("data", (c) => (body += c));
res.on("end", () => resolve(res.statusCode === 200 ? JSON.parse(body) : { status: res.statusCode, body }));
});
req.on("error", reject);
req.end(src);
});
return {
/** `createVat`: a new vat, reachable only by its id here, and by its mount */
create: (src, grants) => send("POST", "/", src, grants),
upgrade: (vat, src, grants) => send("PUT", `/${vat}`, src, grants),
/** a vat from before ids */
byName: (name, src, grants) => send("PUT", `/by-name/${name}`, src, grants),
};
};
/** @param {string} base the public origin */
const makePublic = (base) => ({
get: async (pathAndQuery, headers = {}) => {
const resp = await fetch(`${base}${pathAndQuery}`, { headers });
return { status: resp.status, text: await resp.text(), referrerPolicy: resp.headers.get("Referrer-Policy") };
},
text: async (pathAndQuery) => (await fetch(`${base}${pathAndQuery}`)).text(),
put: async (pathAndQuery, src) => (await fetch(`${base}${pathAndQuery}`, { method: "PUT", body: src })).status,
});
/** An HTTP server that answers GET with "plain ok", echoes WebSocket messages, and counts requests. */
const startEcho = async () => {
let hits = 0;
const server = createHttpServer((_req, res) => (hits++, res.end("plain ok")));
const wss = new WebSocketServer({ server });
wss.on("connection", (ws) => ws.on("message", (m) => ws.send(String(m))));
await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve));
return {
host: `127.0.0.1:${server.address().port}`,
hits: () => hits,
stop: () => new Promise((resolve) => (wss.close(), server.closeAllConnections(), server.close(resolve))),
};
};
let dir, port, server, switchboard, echo, echoLabels, admin, pub;
/**
* A toybox in `dir/<name>` that holds one switchboard handle.
*
* @param {string} name
* @param {number} port
* @param {"public" | "local"} handle which switchboard socket it gets
*/
const startToybox = async (name, port, handle) => {
const home = join(dir, name);
await mkdir(join(home, "state"), { recursive: true });
return startWorkerd(
"config.capnp",
[
"-s", `http=127.0.0.1:${port}`, "-s", `admin=unix:${join(home, "admin.sock")}`,
"-d", `state=${join(home, "state")}`, "-e", `switchboard=unix:${join(dir, `sb-${handle}.sock`)}`,
],
{ port, host: "127.0.0.1" },
[join(home, "admin.sock")],
);
};
before(async () => {
dir = await mkdtemp(join(tmpdir(), "toybox-"));
port = await freePort();
echo = await startEcho();
echoLabels = { host: "127.0.0.1", port: echo.host.split(":")[1] };
switchboard = await startWorkerd(
"switchboard.capnp",
["-s", `public=unix:${join(dir, "sb-public.sock")}`, "-s", `local=unix:${join(dir, "sb-local.sock")}`],
{ path: join(dir, "sb-local.sock") },
[join(dir, "sb-public.sock"), join(dir, "sb-local.sock")],
);
// the tests' echo server is on localhost, so this toybox holds the local handle
server = await startToybox("main", port, "local");
admin = makeAdmin(join(dir, "main", "admin.sock"));
pub = makePublic(`http://127.0.0.1:${port}`);
});
after(async () => {
await server?.stop();
await switchboard?.stop();
await echo?.stop();
await rm(dir, { recursive: true, force: true });
});
const at = (name) => ({ mount: { vat: name } });
test("nothing mounted: 404", async () => {
assert.equal((await pub.get("/?vat=nobody")).status, 404);
});
test("createVat returns an id and a code hash; unmounted, the vat is unreachable", async () => {
const { vat, hash } = await admin.create(await app("counter-v1.js"));
assert.match(vat, /^[0-9a-f]{64}$/);
assert.match(hash, /^[0-9a-f]{64}$/);
assert.equal((await pub.get("/")).status, 404);
});
test("install, run, upgrade: the app's state carries over; every start states its grants", async () => {
const { vat } = await admin.create(await app("counter-v1.js"), at("alice"));
assert.equal(await pub.text("/?vat=alice"), "v1 count 1\n");
assert.equal(await pub.text("/?vat=alice"), "v1 count 2\n");
await admin.upgrade(vat, await app("counter-v2.js"), at("alice"));
assert.equal(await pub.text("/?vat=alice"), "v2 count 3\n");
await admin.upgrade(vat, await app("counter-v2.js"));
assert.equal((await pub.get("/?vat=alice")).status, 404);
await admin.upgrade(vat, await app("counter-v2.js"), at("alice"));
assert.equal(await pub.text("/?vat=alice"), "v2 count 4\n");
});
test("vats don't share state", async () => {
await admin.create(await app("counter-v1.js"), at("bob"));
await admin.create(await app("counter-v1.js"), at("carol"));
assert.equal(await pub.text("/?vat=bob"), "v1 count 1\n");
assert.equal(await pub.text("/?vat=carol"), "v1 count 1\n");
});
test("an app gets no fetch", async () => {
await admin.create(await app("try-fetch.js"), at("bob-fetch"));
assert.match(await pub.text("/?vat=bob-fetch"), /^fetch refused: This worker is not permitted to access the internet/);
});
test("the public port can't install code", async () => {
assert.equal(await pub.put("/", "export class App {}"), 404);
});
test("the path is the app's; its labels are the mount's and the request's together", async () => {
await admin.create(await app("show-labels.js"), at("lab"));
const got = await pub.get("/a/b?vat=lab&page=2");
assert.equal(got.text, `/a/b ${JSON.stringify({ vat: "lab", page: "2" })}\n`);
assert.equal(got.referrerPolicy, "no-referrer");
});
test("a label given twice is a 400", async () => {
assert.equal((await pub.get("/?vat=lab&vat=lab")).status, 400);
});
test("mounts that could both match a request are refused", async () => {
const src = await app("show-labels.js");
await admin.create(src, at("x"));
const narrower = await admin.create(src, { mount: { vat: "x", n: "2" } });
assert.equal(narrower.status, 409);
assert.match(narrower.body, /overlaps another/);
// no key in common with the {vat} mounts: ?vat=x&app=x would carry both
assert.equal((await admin.create(src, { mount: { app: "x" } })).status, 409);
assert.match((await admin.create(src, { mount: { vat: "y", n: "2" } })).vat, /^[0-9a-f]{64}$/);
});
test("a request reaches a vat only by naming every label of its mount", async () => {
await admin.create(await app("show-labels.js"), { mount: { vat: "both", flavor: "plain" } });
assert.equal((await pub.get("/?vat=both")).status, 404);
assert.equal((await pub.get("/?vat=both&flavor=fancy")).status, 404);
assert.equal((await pub.get("/?vat=both&flavor=plain")).status, 200);
});
test("routing reads no labels from headers", async () => {
const got = await pub.get("/?vat=both", { "Toybox-Labels": JSON.stringify({ flavor: "plain" }) });
assert.equal(got.status, 404);
});
test("code, state, and mount survive a restart", async () => {
await admin.create(await app("counter-v1.js"), at("dave"));
assert.equal(await pub.text("/?vat=dave"), "v1 count 1\n");
await server.stop();
server = await startToybox("main", port, "local");
assert.equal(await pub.text("/?vat=dave"), "v1 count 2\n");
});
test("a vat from before ids is unreachable until the admin mounts it by its old name", async () => {
await admin.byName("old", await app("counter-v1.js"));
assert.equal((await pub.get("/?vat=old")).status, 404);
await admin.byName("old", await app("counter-v1.js"), at("old"));
assert.equal(await pub.text("/?vat=old"), "v1 count 1\n");
});
const probe = (vat, target, { ws, constrain } = {}) =>
pub.text(
`/?vat=${vat}&url=${encodeURIComponent(`http://${target}/`)}` +
(ws ? "&ws=1" : "") +
(constrain ? `&constrain=${encodeURIComponent(JSON.stringify(constrain))}` : ""),
);
test("a vat granted a host reaches it through the switchboard: fetch and WebSocket", async () => {
await admin.create(await app("egress-probe.js"), { ...at("erin"), egress: echoLabels });
assert.equal(await probe("erin", echo.host), "got: plain ok\n");
assert.equal(await probe("erin", echo.host, { ws: true }), "echo: hello\n");
});
test("same code, no grant: no egress", async () => {
await admin.create(await app("egress-probe.js"), at("frank"));
assert.equal(await probe("frank", echo.host), "no egress granted\n");
});
test("a grant is an object, not ambient: the global fetch still throws", async () => {
await admin.create(await app("try-fetch.js"), { ...at("heidi"), egress: echoLabels });
assert.match(await pub.text("/?vat=heidi"), /^fetch refused: This worker is not permitted to access the internet/);
});
test("the switchboard refuses what a grant doesn't cover", async () => {
await admin.create(await app("egress-probe.js"), { ...at("grace"), egress: { host: "127.0.0.1", port: "1" } });
assert.match(await probe("grace", echo.host), /^got: egress refused: .*egress to 127\.0\.0\.1 not granted/);
assert.match(await probe("grace", echo.host, { ws: true }), /^no websocket: 403 egress refused: .*not granted/);
});
test("a vat can narrow its grant, but not contradict it", async () => {
await admin.create(await app("egress-probe.js"), { ...at("ivan"), egress: { scheme: "http" } });
assert.equal(await probe("ivan", echo.host, { constrain: echoLabels }), "got: plain ok\n");
assert.match(
await probe("ivan", echo.host, { constrain: { scheme: "https" } }),
/^refused: label scheme is already set to a different value/,
);
});
test("a grant stored before labels, a bare host:port, still works", async () => {
await admin.create(await app("egress-probe.js"), { ...at("kim"), egress: echo.host });
assert.equal(await probe("kim", echo.host), "got: plain ok\n");
});
test("a grant survives a restart", async () => {
await server.stop();
server = await startToybox("main", port, "local");
assert.equal(await probe("erin", echo.host, { ws: true }), "echo: hello\n");
});
test("a toybox holding only the public handle can't reach localhost, whatever the grant", async () => {
const port2 = await freePort();
const other = await startToybox("public-only", port2, "public");
try {
const admin2 = makeAdmin(join(dir, "public-only", "admin.sock"));
await admin2.create(await app("egress-probe.js"), { ...at("judy"), egress: echoLabels });
const before = echo.hits();
const got = await (
await fetch(`http://127.0.0.1:${port2}/?vat=judy&url=${encodeURIComponent(`http://${echo.host}/`)}`)
).text();
// workerd reports the public network's refusal (restrictPeers) to JS only as "internal error"
assert.match(got, /^got: egress refused: /);
assert.equal(echo.hits(), before);
} finally {
await other.stop();
}
});
+51
View File
@@ -0,0 +1,51 @@
/** @file Run real workerd processes for tests. */
import { spawn } from "node:child_process";
import { rm } from "node:fs/promises";
import { connect, createServer } from "node:net";
import { join } from "node:path";
export const here = new URL("..", import.meta.url).pathname;
// the package binary, not the .bin shim: the shim forks it, so a kill would orphan workerd
const workerd = join(here, "node_modules/workerd/bin/workerd");
export const freePort = () =>
new Promise((resolve) => {
const s = createServer().listen(0, "127.0.0.1", () => {
const { port } = s.address();
s.close(() => resolve(port));
});
});
/** Resolves once something accepts connections at `opts` (a port or a socket path). */
export const accepting = (opts) =>
new Promise((resolve) => {
const s = connect(opts, () => (s.end(), resolve(true)));
s.on("error", () => resolve(false));
});
/**
* Run workerd on `config` until `stop()`; resolves once `listening` accepts.
*
* @param {string} config
* @param {string[]} args socket, directory, and external overrides
* @param {{ port: number } | { path: string }} listening
* @param {string[]} stale socket files a crash leaves behind; bind() would fail on them
*/
export const startWorkerd = async (config, args, listening, stale = []) => {
for (const f of stale) await rm(f, { force: true });
const child = spawn(workerd, ["serve", "--experimental", join(here, config), ...args], {
stdio: ["ignore", process.env.WORKERD_LOG ? "inherit" : "ignore", "pipe"],
});
let log = "";
// WORKERD_LOG=1 echoes workerd's stderr, for debugging
child.stderr.on("data", (c) => (process.env.WORKERD_LOG ? process.stderr.write(c) : (log += c)));
const exited = new Promise((resolve) => child.on("exit", resolve));
for (let i = 0; i < 100; i++) {
// SIGKILL: a crash is the harder durability test
if (await accepting(listening)) return { stop: () => (child.kill("SIGKILL"), exited) };
await new Promise((r) => setTimeout(r, 50));
}
child.kill("SIGKILL");
throw new Error(`workerd did not start: ${log}`);
};
+59
View File
@@ -0,0 +1,59 @@
/**
* @file The runtime of an OCapN vat: SES, an OCapN peer, and the vat's own
* module (`vat.js`), which exports `buildRootObject(vatPowers, params)`.
*
* Bundled (see build.js) and loaded as the facet's main module beside the
* vat's code. The vat's designator is its id; it reaches other peers, and
* they reach it, through the vatAdmin's router (`env.router`).
*/
import "@endo/init";
import { DurableObject } from "cloudflare:workers";
import { E, Far } from "@endo/far";
import { makeClient } from "@endo/ocapn";
import { makeRpcNetLayer } from "./netlayer.js";
import * as vat from "./vat.js";
/**
* The swissnum of a vat's bootstrap: the vatAdmin starts the vat through it.
* Not exported: workerd takes every export of a main module for an entrypoint.
*/
const VAT_SWISSNUM = "vat";
export class App extends DurableObject {
#gateway;
constructor(ctx, env) {
super(ctx, env);
this.#gateway = this.#boot();
}
async #boot() {
const { vatId, router } = this.env;
let root;
const vatPowers = Object.freeze({ E, Far, storage: this.ctx.storage.kv });
const bootstrap = Far("vatBootstrap", {
/** (Re)build the root: on creation, and when the vatAdmin restarts the vat. */
start: (params) => {
root = vat.buildRootObject(vatPowers, params);
return root;
},
});
// workerd runs FinalizationRegistry callbacks outside any request, where
// sending the resulting GC message is forbidden I/O. So imports are
// never collected here; they go when their session does.
const ocapn = makeClient({
debugLabel: vatId.slice(0, 8),
swissnumTable: new Map([[VAT_SWISSNUM, bootstrap]]),
enableImportCollection: false,
});
const netlayer = await ocapn.registerNetlayer(
makeRpcNetLayer({ designator: vatId, openGateway: (designator) => router.gateway(designator) }),
);
return netlayer.gateway;
}
/** This vat's OCapN gateway, for the router. */
gateway() {
return this.#gateway;
}
}
+15
View File
@@ -0,0 +1,15 @@
/**
* A counter whose count lives in the vat's storage.
*
* TODO: a durable Exo once vatPowers offers a zone, e.g.
* `zone.exoClass("Counter", undefined, () => ({ count: 0 }), { incr() { this.state.count += 1; return this.state.count; } })`
*/
export const buildRootObject = ({ Far, storage }) =>
Far("counterRoot", {
increment: () => {
const n = (storage.get("count") ?? 0) + 1;
storage.put("count", n);
return n;
},
read: () => storage.get("count") ?? 0,
});
+6
View File
@@ -0,0 +1,6 @@
/** A vat that greets through the `carol` it was created with. */
export const buildRootObject = ({ E, Far }, { carol }) =>
Far("greeterRoot", {
greetVia: (name) => E(carol).greet(name),
hello: () => "hello from the vat",
});
+12
View File
@@ -0,0 +1,12 @@
/** Hello, world, served through the `ingress` it was created with. */
export const buildRootObject = ({ E, Far }, { ingress }) => {
const handler = Far("helloHandler", {
handle: ({ path }) => ({
status: 200,
headers: { "content-type": "text/html; charset=utf-8" },
body: `<!doctype html><title>hello</title><h1>Hello, world</h1><p>from a toybox vat, at ${path}</p>\n`,
}),
});
E(ingress).serve(handler);
return Far("helloRoot", {});
};
+1 -1
View File
@@ -3,7 +3,7 @@
DO-backed fluent web client factory: site-agnostic. Deploy once; instances
are attenuated handles whose **ids are bearer capabilities** — anyone
holding the id string can drive the instance; nobody without it can.
TODO: this proxy-page flow (see diigo-cache README) vs running webapp as
TODO: this proxy-page flow (see [diigo-cache CONTRIBUTING](../diigo-cache/CONTRIBUTING.md)) vs running webapp as
the primary worker (`wrangler dev -c ../webapp/wrangler.jsonc -c ../diigo-cache/wrangler.jsonc`)
so its own port serves the mint form directly; reconsider when the UI
grows past a form or two.