capsite/toybox: a switchboard, the vats' only way out
A vat granted a host (PUT /<vat>?egress=<host> on the admin socket) gets Egress as its globalOutbound, with the host in ctx.props where the vat can't change it. Egress forwards matching requests, WebSocket upgrades included, to the switchboard and refuses the rest with a 403; with no grant, fetch still throws. The switchboard is a separate workerd: a Switchboard DO that calls fetch and returns the response, so a 101's webSocket passes through. toybox reaches it over a unix socket in HTTP proxy style, which keeps the full URL, scheme included. Its network service is the outer fence, with TLS trusting the browser CAs; systemd units for it join toybox's, and only the toybox group can open its socket. The loaded Worker is now named by vat and incarnation, not by code hash: the same code with a different grant must not share a Worker. The new incarnation table sits beside the old ones; vats installed before it read as incarnation 0 with no grant. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
88739d9495
commit
7bf6eb44c7
13 files changed
+277
-49
No files matched your search
@@ -1,2 +1,3 @@
|
|||||||
state/
|
state/
|
||||||
admin.sock
|
admin.sock
|
||||||
|
switchboard.sock
|
||||||
+17
-7
@@ -2,35 +2,45 @@
|
|||||||
|
|
||||||
Vats whose code is installed at runtime: each vat is a Durable Object whose app runs as a [facet](https://developers.cloudflare.com/dynamic-workers/usage/durable-object-facets/) on code loaded through workerd's Worker Loader. Think `vatAdmin.createVat(bundle)`, with a supervisor per vat.
|
Vats whose code is installed at runtime: each vat is a Durable Object whose app runs as a [facet](https://developers.cloudflare.com/dynamic-workers/usage/durable-object-facets/) on code loaded through workerd's Worker Loader. Think `vatAdmin.createVat(bundle)`, with a supervisor per vat.
|
||||||
|
|
||||||
- [supervisor.js](supervisor.js): `Vat` stores code by SHA-256 and runs the app on it; `Public` and `Admin` are the two entrypoints.
|
- [supervisor.js](supervisor.js): `Vat` stores code by SHA-256 and runs the app on it; `Public` and `Admin` are the two entrypoints; `Egress` is a vat's `fetch`.
|
||||||
|
- [switchboard.js](switchboard.js): a separate workerd process, the vats' only way out: a forward proxy for HTTP and WebSockets.
|
||||||
- [config.capnp](config.capnp): the sockets have no addresses; whoever starts workerd hands them in, so holding a socket is the authority it grants.
|
- [config.capnp](config.capnp): the sockets have no addresses; whoever starts workerd hands them in, so holding a socket is the authority it grants.
|
||||||
|
|
||||||
| socket | reaches | what its holder can do |
|
| socket | reaches | what its holder can do |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `http` | `Public` | `GET /<vat>/…`: talk to a vat's app |
|
| `http` | `Public` | `GET /<vat>/…`: talk to a vat's app |
|
||||||
| `admin` | `Admin` | `PUT /<vat>` with an ES module body: install it and (re)start the app; its storage carries over |
|
| `admin` | `Admin` | `PUT /<vat>` with an ES module body: install it and (re)start the app; its storage carries over. `?egress=<host>` grants the app that one host. |
|
||||||
|
|
||||||
An app must export `class App extends DurableObject` (see [apps/](apps/)). It gets an empty `env` and no `fetch`.
|
An app must export `class App extends DurableObject` (see [apps/](apps/)). It gets an empty `env`. With no grant, `fetch` throws; with one, its `globalOutbound` is `Egress` with the granted host in `ctx.props`, which forwards matching requests, WebSocket upgrades included, to the switchboard and refuses the rest with a 403.
|
||||||
|
|
||||||
|
toybox reaches the switchboard over a unix socket (`-e switchboard=unix:…`), in HTTP proxy style so the full URL, scheme included, survives the hop.
|
||||||
|
|
||||||
## Run it
|
## Run it
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
pnpm install
|
pnpm install
|
||||||
pnpm test # a real workerd: install, upgrade, isolation, no fetch, crash and restart
|
pnpm test # real workerds: install, upgrade, isolation, egress grants, fetch and WebSocket out, crash and restart
|
||||||
pnpm start # systemd-socket-activate on 127.0.0.1:8787 and ./admin.sock; state in ./state
|
pnpm switchboard # systemd-socket-activate on ./switchboard.sock
|
||||||
|
pnpm start # likewise on 127.0.0.1:8787 and ./admin.sock; state in ./state
|
||||||
```
|
```
|
||||||
|
|
||||||
Then, in another terminal:
|
Then, in a third terminal:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
curl -X PUT --unix-socket admin.sock --data-binary @apps/counter-v1.js http://toybox/alice
|
curl -X PUT --unix-socket admin.sock --data-binary @apps/counter-v1.js http://toybox/alice
|
||||||
curl localhost:8787/alice # v1 count 1
|
curl localhost:8787/alice # v1 count 1
|
||||||
curl -X PUT --unix-socket admin.sock --data-binary @apps/counter-v2.js http://toybox/alice
|
curl -X PUT --unix-socket admin.sock --data-binary @apps/counter-v2.js http://toybox/alice
|
||||||
curl localhost:8787/alice # v2 count 2
|
curl localhost:8787/alice # v2 count 2
|
||||||
|
|
||||||
|
curl -X PUT --unix-socket admin.sock --data-binary @apps/egress-probe.js 'http://toybox/erin?egress=example.com'
|
||||||
|
curl 'localhost:8787/erin?url=https://example.com/' # got: <!doctype html>…
|
||||||
|
curl 'localhost:8787/erin?url=https://example.org/' # got: egress to example.org not granted
|
||||||
```
|
```
|
||||||
|
|
||||||
## Under systemd
|
## Under systemd
|
||||||
|
|
||||||
[systemd/](systemd/) has a socket and service pair, assuming this directory is installed at `/opt/toybox`. The service gets no network of its own (`PrivateNetwork=yes`): only the two sockets it is handed.
|
[systemd/](systemd/) has socket and service pairs for toybox and its switchboard, assuming this directory is installed at `/opt/toybox`. toybox gets no network of its own (`PrivateNetwork=yes`): only the two sockets it is handed and the switchboard's socket, which only members of the `toybox` group can open. The switchboard has the network, fenced by `IPAddressDeny=`; `switchboard.capnp` allows `local` addresses only so the tests can reach a local echo server.
|
||||||
|
|
||||||
|
Neither unit has been run under the system manager yet.
|
||||||
|
|
||||||
`PrivateNetwork=` needs the system manager; as a `systemd-run --user` unit, systemd logs "lack privileges for network namespace, proceeding without".
|
`PrivateNetwork=` needs the system manager; as a `systemd-run --user` unit, systemd logs "lack privileges for network namespace, proceeding without".
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
import { DurableObject } from "cloudflare:workers";
|
||||||
|
|
||||||
|
/** `?url=<target>` fetches it; with `&ws=1`, opens a WebSocket there and reports the echo of "hello". */
|
||||||
|
export class App extends DurableObject {
|
||||||
|
async fetch(request) {
|
||||||
|
const params = new URL(request.url).searchParams;
|
||||||
|
const target = params.get("url");
|
||||||
|
try {
|
||||||
|
if (!params.has("ws")) return new Response(`got: ${await (await fetch(target)).text()}\n`);
|
||||||
|
const resp = await fetch(target, { headers: { Upgrade: "websocket" } });
|
||||||
|
const ws = resp.webSocket;
|
||||||
|
if (!ws) return new Response(`no websocket: ${resp.status} ${await resp.text()}`);
|
||||||
|
ws.accept();
|
||||||
|
const reply = new Promise((resolve) => ws.addEventListener("message", (e) => resolve(e.data)));
|
||||||
|
ws.send("hello");
|
||||||
|
const got = await reply;
|
||||||
|
ws.close();
|
||||||
|
return new Response(`echo: ${got}\n`);
|
||||||
|
} catch (e) {
|
||||||
|
return new Response(`refused: ${e.message}\n`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
+5
-1
@@ -2,13 +2,16 @@
|
|||||||
#
|
#
|
||||||
# Sockets have no address here: whoever starts workerd hands them in
|
# Sockets have no address here: whoever starts workerd hands them in
|
||||||
# (-S name=fd from systemd, or -s name=addr), so holding a socket is the
|
# (-S name=fd from systemd, or -s name=addr), so holding a socket is the
|
||||||
# authority it grants. `state` is likewise overridden with -d state=<dir>.
|
# authority it grants. `state` is likewise overridden with -d state=<dir>,
|
||||||
|
# and the switchboard's address with -e switchboard=<addr>.
|
||||||
using Workerd = import "/workerd/workerd.capnp";
|
using Workerd = import "/workerd/workerd.capnp";
|
||||||
|
|
||||||
const config :Workerd.Config = (
|
const config :Workerd.Config = (
|
||||||
services = [
|
services = [
|
||||||
(name = "main", worker = .supervisor),
|
(name = "main", worker = .supervisor),
|
||||||
(name = "nowhere", network = (allow = [])),
|
(name = "nowhere", network = (allow = [])),
|
||||||
|
# the way out: see switchboard.capnp
|
||||||
|
(name = "switchboard", external = (address = "unix:switchboard.sock", http = (style = proxy))),
|
||||||
(name = "state", disk = (path = "state", writable = true)),
|
(name = "state", disk = (path = "state", writable = true)),
|
||||||
],
|
],
|
||||||
sockets = [
|
sockets = [
|
||||||
@@ -27,6 +30,7 @@ const supervisor :Workerd.Worker = (
|
|||||||
bindings = [
|
bindings = [
|
||||||
(name = "VATS", durableObjectNamespace = "Vat"),
|
(name = "VATS", durableObjectNamespace = "Vat"),
|
||||||
(name = "LOADER", workerLoader = ()),
|
(name = "LOADER", workerLoader = ()),
|
||||||
|
(name = "SWITCHBOARD", service = "switchboard"),
|
||||||
],
|
],
|
||||||
globalOutbound = "nowhere",
|
globalOutbound = "nowhere",
|
||||||
);
|
);
|
||||||
+4
-2
@@ -4,10 +4,12 @@
|
|||||||
"private": true,
|
"private": true,
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"scripts": {
|
"scripts": {
|
||||||
"start": "mkdir -p state && rm -f admin.sock && systemd-socket-activate -l 127.0.0.1:8787 -l ./admin.sock -- node_modules/workerd/bin/workerd serve --experimental config.capnp -S http=3 -S admin=4 -d state=state",
|
"start": "mkdir -p state && rm -f admin.sock && systemd-socket-activate -l 127.0.0.1:8787 -l ./admin.sock -- node_modules/workerd/bin/workerd serve --experimental config.capnp -S http=3 -S admin=4 -d state=state -e switchboard=unix:./switchboard.sock",
|
||||||
|
"switchboard": "rm -f switchboard.sock && systemd-socket-activate -l ./switchboard.sock -- node_modules/workerd/bin/workerd serve --experimental switchboard.capnp -S http=3",
|
||||||
"test": "node --test"
|
"test": "node --test"
|
||||||
},
|
},
|
||||||
"devDependencies": {
|
"devDependencies": {
|
||||||
"workerd": "1.20261006.1"
|
"workerd": "1.20261006.1",
|
||||||
|
"ws": "8.22.0"
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
Generated
+17
@@ -11,6 +11,9 @@ importers:
|
|||||||
workerd:
|
workerd:
|
||||||
specifier: 1.20261006.1
|
specifier: 1.20261006.1
|
||||||
version: 1.20261006.1
|
version: 1.20261006.1
|
||||||
|
ws:
|
||||||
|
specifier: 8.22.0
|
||||||
|
version: 8.22.0
|
||||||
|
|
||||||
packages:
|
packages:
|
||||||
|
|
||||||
@@ -49,6 +52,18 @@ packages:
|
|||||||
engines: {node: '>=16'}
|
engines: {node: '>=16'}
|
||||||
hasBin: true
|
hasBin: true
|
||||||
|
|
||||||
|
ws@8.22.0:
|
||||||
|
resolution: {integrity: sha512-Ydggc987+RO0AnWtZ/7Wq9FtNvcrL1b/RO0ud9mWjUPgDrsAAwQSF51sm2hm1XofbU/4jkpGEsLFsZZxU+1DOg==}
|
||||||
|
engines: {node: '>=10.0.0'}
|
||||||
|
peerDependencies:
|
||||||
|
bufferutil: ^4.0.1
|
||||||
|
utf-8-validate: '>=5.0.2'
|
||||||
|
peerDependenciesMeta:
|
||||||
|
bufferutil:
|
||||||
|
optional: true
|
||||||
|
utf-8-validate:
|
||||||
|
optional: true
|
||||||
|
|
||||||
snapshots:
|
snapshots:
|
||||||
|
|
||||||
'@cloudflare/workerd-darwin-64@1.20261006.1':
|
'@cloudflare/workerd-darwin-64@1.20261006.1':
|
||||||
@@ -73,3 +88,5 @@ snapshots:
|
|||||||
'@cloudflare/workerd-linux-64': 1.20261006.1
|
'@cloudflare/workerd-linux-64': 1.20261006.1
|
||||||
'@cloudflare/workerd-linux-arm64': 1.20261006.1
|
'@cloudflare/workerd-linux-arm64': 1.20261006.1
|
||||||
'@cloudflare/workerd-windows-64': 1.20261006.1
|
'@cloudflare/workerd-windows-64': 1.20261006.1
|
||||||
|
|
||||||
|
ws@8.22.0: {}
|
||||||
+42
-12
@@ -4,40 +4,51 @@ import { DurableObject, WorkerEntrypoint } from "cloudflare:workers";
|
|||||||
const hex = (buf) => [...new Uint8Array(buf)].map((b) => b.toString(16).padStart(2, "0")).join("");
|
const hex = (buf) => [...new Uint8Array(buf)].map((b) => b.toString(16).padStart(2, "0")).join("");
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Holds the vat's installed code; the app facet holds the app's state.
|
* Holds the vat's installed code and egress grant; the app facet holds the app's state.
|
||||||
*
|
*
|
||||||
* Code is stored by hash so the facet can be rebuilt after hibernation or a
|
* Code is stored by hash so the facet can be rebuilt after hibernation or a
|
||||||
* restart. The app gets an empty `env` and no `fetch`.
|
* restart. The app gets an empty `env`; its `fetch` reaches only the host it
|
||||||
|
* was granted, if any, through {@link Egress}.
|
||||||
|
*
|
||||||
|
* Each `start` begins a new incarnation, which names the loaded Worker: the
|
||||||
|
* same code with a different grant must not share a Worker.
|
||||||
*/
|
*/
|
||||||
export class Vat extends DurableObject {
|
export class Vat extends DurableObject {
|
||||||
constructor(ctx, env) {
|
constructor(ctx, env) {
|
||||||
super(ctx, env);
|
super(ctx, env);
|
||||||
ctx.storage.sql.exec("CREATE TABLE IF NOT EXISTS bundle (hash TEXT PRIMARY KEY, src TEXT)");
|
ctx.storage.sql.exec("CREATE TABLE IF NOT EXISTS bundle (hash TEXT PRIMARY KEY, src TEXT)");
|
||||||
ctx.storage.sql.exec("CREATE TABLE IF NOT EXISTS current (k INTEGER PRIMARY KEY CHECK (k = 0), hash TEXT)");
|
ctx.storage.sql.exec("CREATE TABLE IF NOT EXISTS current (k INTEGER PRIMARY KEY CHECK (k = 0), hash TEXT)");
|
||||||
|
ctx.storage.sql.exec(
|
||||||
|
"CREATE TABLE IF NOT EXISTS incarnation (k INTEGER PRIMARY KEY CHECK (k = 0), n INTEGER, egress TEXT)",
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Install `src` and run the app on it; the app's storage carries over.
|
* Install `src` and run the app on it; the app's storage carries over.
|
||||||
*
|
*
|
||||||
* @param {string} src an ES module exporting `class App extends DurableObject`
|
* @param {string} src an ES module exporting `class App extends DurableObject`
|
||||||
|
* @param {string | null} egress the one host (`name[:port]`) the app may fetch, or none
|
||||||
* @returns {Promise<string>} the code's SHA-256, hex
|
* @returns {Promise<string>} the code's SHA-256, hex
|
||||||
*/
|
*/
|
||||||
async start(src) {
|
async start(src, egress) {
|
||||||
const hash = hex(await crypto.subtle.digest("SHA-256", new TextEncoder().encode(src)));
|
const hash = hex(await crypto.subtle.digest("SHA-256", new TextEncoder().encode(src)));
|
||||||
this.ctx.storage.sql.exec("INSERT OR IGNORE INTO bundle VALUES (?, ?)", hash, src);
|
const { sql } = this.ctx.storage;
|
||||||
this.ctx.storage.sql.exec("INSERT OR REPLACE INTO current VALUES (0, ?)", hash);
|
sql.exec("INSERT OR IGNORE INTO bundle VALUES (?, ?)", hash, src);
|
||||||
|
sql.exec("INSERT OR REPLACE INTO current VALUES (0, ?)", hash);
|
||||||
|
const [prev] = sql.exec("SELECT n FROM incarnation").toArray();
|
||||||
|
sql.exec("INSERT OR REPLACE INTO incarnation VALUES (0, ?, ?)", (prev?.n ?? 0) + 1, egress);
|
||||||
this.ctx.facets.abort("app", new Error("upgraded"));
|
this.ctx.facets.abort("app", new Error("upgraded"));
|
||||||
return hash;
|
return hash;
|
||||||
}
|
}
|
||||||
|
|
||||||
#app(hash, src) {
|
#app({ hash, src, n, egress }) {
|
||||||
return this.ctx.facets.get("app", () => {
|
return this.ctx.facets.get("app", () => {
|
||||||
const worker = this.env.LOADER.get(hash, () => ({
|
const worker = this.env.LOADER.get(`${this.ctx.id}:${n}`, () => ({
|
||||||
compatibilityDate: "2026-10-01",
|
compatibilityDate: "2026-10-01",
|
||||||
mainModule: "app.js",
|
mainModule: "app.js",
|
||||||
modules: { "app.js": src },
|
modules: { "app.js": src },
|
||||||
env: {},
|
env: {},
|
||||||
globalOutbound: null,
|
globalOutbound: egress ? this.ctx.exports.Egress({ props: { host: egress } }) : null,
|
||||||
}));
|
}));
|
||||||
return { class: worker.getDurableObjectClass("App") };
|
return { class: worker.getDurableObjectClass("App") };
|
||||||
});
|
});
|
||||||
@@ -45,10 +56,25 @@ export class Vat extends DurableObject {
|
|||||||
|
|
||||||
fetch(request) {
|
fetch(request) {
|
||||||
const [row] = this.ctx.storage.sql
|
const [row] = this.ctx.storage.sql
|
||||||
.exec("SELECT b.hash, b.src FROM current c JOIN bundle b USING (hash)")
|
.exec(
|
||||||
|
`SELECT b.hash, b.src, coalesce(i.n, 0) AS n, i.egress
|
||||||
|
FROM current c JOIN bundle b USING (hash) LEFT JOIN incarnation i ON i.k = 0`,
|
||||||
|
)
|
||||||
.toArray();
|
.toArray();
|
||||||
if (!row) return new Response("no code installed\n", { status: 404 });
|
if (!row) return new Response("no code installed\n", { status: 404 });
|
||||||
return this.#app(row.hash, row.src).fetch(request);
|
return this.#app(row).fetch(request);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A vat's `globalOutbound`: its `fetch`, WebSocket upgrades included, reaches
|
||||||
|
* the switchboard only for the host in `ctx.props`, which the vat can't set.
|
||||||
|
*/
|
||||||
|
export class Egress extends WorkerEntrypoint {
|
||||||
|
fetch(request) {
|
||||||
|
const { host } = new URL(request.url);
|
||||||
|
if (host !== this.ctx.props.host) return new Response(`egress to ${host} not granted\n`, { status: 403 });
|
||||||
|
return this.env.SWITCHBOARD.fetch(request);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -61,11 +87,15 @@ export class Public extends WorkerEntrypoint {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/** The admin socket: `PUT /<vat>` with an ES module body installs and starts it. */
|
/**
|
||||||
|
* The admin socket: `PUT /<vat>` with an ES module body installs and starts
|
||||||
|
* it; `?egress=<host>` grants it that one host.
|
||||||
|
*/
|
||||||
export class Admin extends WorkerEntrypoint {
|
export class Admin extends WorkerEntrypoint {
|
||||||
async fetch(request) {
|
async fetch(request) {
|
||||||
if (request.method !== "PUT") return new Response("PUT code to /<vat>\n", { status: 405 });
|
if (request.method !== "PUT") return new Response("PUT code to /<vat>\n", { status: 405 });
|
||||||
return new Response((await vatOf(this.env, request.url).start(await request.text())) + "\n");
|
const egress = new URL(request.url).searchParams.get("egress");
|
||||||
|
return new Response((await vatOf(this.env, request.url).start(await request.text(), egress)) + "\n");
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,26 @@
|
|||||||
|
# switchboard: toybox's only way out. A forward proxy (HTTP and WebSocket)
|
||||||
|
# that toybox's Egress gateway reaches over a socket handed in with -S/-s.
|
||||||
|
#
|
||||||
|
# The network service is the switchboard's outer fence; under systemd,
|
||||||
|
# IPAddressAllow=/IPAddressDeny= narrow it further (see systemd/).
|
||||||
|
using Workerd = import "/workerd/workerd.capnp";
|
||||||
|
|
||||||
|
const config :Workerd.Config = (
|
||||||
|
services = [
|
||||||
|
(name = "main", worker = .switchboard),
|
||||||
|
(name = "internet", network = (allow = ["public", "local"], tlsOptions = (trustBrowserCas = true))),
|
||||||
|
],
|
||||||
|
sockets = [
|
||||||
|
# proxy style: the request line carries the full URL, scheme included
|
||||||
|
(name = "http", http = (style = proxy), service = "main"),
|
||||||
|
],
|
||||||
|
);
|
||||||
|
|
||||||
|
const switchboard :Workerd.Worker = (
|
||||||
|
compatibilityDate = "2026-10-01",
|
||||||
|
modules = [(name = "switchboard.js", esModule = embed "switchboard.js")],
|
||||||
|
durableObjectNamespaces = [(className = "Switchboard", uniqueKey = "toybox-switchboard")],
|
||||||
|
durableObjectStorage = (inMemory = void),
|
||||||
|
bindings = [(name = "BOARD", durableObjectNamespace = "Switchboard")],
|
||||||
|
globalOutbound = "internet",
|
||||||
|
);
|
||||||
@@ -0,0 +1,22 @@
|
|||||||
|
/** @file switchboard: forwards each proxied request, WebSocket upgrades included. */
|
||||||
|
import { DurableObject } from "cloudflare:workers";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The one place egress happens. A 101 response from `fetch` carries its
|
||||||
|
* `webSocket`; returning it passes the socket through to the caller.
|
||||||
|
*
|
||||||
|
* Which vat may reach which host is decided before a request gets here
|
||||||
|
* (toybox's Egress gateway); this process's network service and its
|
||||||
|
* systemd unit are the outer fence.
|
||||||
|
*/
|
||||||
|
export class Switchboard extends DurableObject {
|
||||||
|
fetch(request) {
|
||||||
|
return fetch(request);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export default {
|
||||||
|
fetch(request, env) {
|
||||||
|
return env.BOARD.getByName("switchboard").fetch(request);
|
||||||
|
},
|
||||||
|
};
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=toybox switchboard (workerd forward proxy)
|
||||||
|
Requires=toybox-switchboard.socket
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
ExecStart=/opt/toybox/node_modules/workerd/bin/workerd serve --experimental /opt/toybox/switchboard.capnp -S http=3
|
||||||
|
DynamicUser=yes
|
||||||
|
# the outer fence: switchboard.capnp allows "local" for tests; deny it here
|
||||||
|
IPAddressDeny=localhost link-local multicast
|
||||||
|
ProtectSystem=strict
|
||||||
|
ProtectHome=yes
|
||||||
|
PrivateTmp=yes
|
||||||
|
SystemCallFilter=@system-service
|
||||||
@@ -0,0 +1,11 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=toybox switchboard: the vats' only way out
|
||||||
|
|
||||||
|
[Socket]
|
||||||
|
ListenStream=/run/toybox/switchboard.sock
|
||||||
|
# only toybox may dial out: its service joins this group
|
||||||
|
SocketGroup=toybox
|
||||||
|
SocketMode=0660
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=sockets.target
|
||||||
@@ -1,12 +1,15 @@
|
|||||||
[Unit]
|
[Unit]
|
||||||
Description=toybox vats (workerd)
|
Description=toybox vats (workerd)
|
||||||
Requires=toybox.socket
|
Requires=toybox.socket toybox-switchboard.socket
|
||||||
|
After=toybox-switchboard.socket
|
||||||
|
|
||||||
[Service]
|
[Service]
|
||||||
ExecStart=/opt/toybox/node_modules/workerd/bin/workerd serve --experimental /opt/toybox/config.capnp -S http=3 -S admin=4 -d state=/var/lib/toybox
|
ExecStart=/opt/toybox/node_modules/workerd/bin/workerd serve --experimental /opt/toybox/config.capnp -S http=3 -S admin=4 -d state=/var/lib/toybox -e switchboard=unix:/run/toybox/switchboard.sock
|
||||||
DynamicUser=yes
|
DynamicUser=yes
|
||||||
|
# to open the switchboard socket
|
||||||
|
SupplementaryGroups=toybox
|
||||||
StateDirectory=toybox
|
StateDirectory=toybox
|
||||||
# no network but the sockets handed in
|
# no network but the sockets handed in, and the switchboard's
|
||||||
PrivateNetwork=yes
|
PrivateNetwork=yes
|
||||||
RestrictAddressFamilies=AF_UNIX
|
RestrictAddressFamilies=AF_UNIX
|
||||||
ProtectSystem=strict
|
ProtectSystem=strict
|
||||||
|
|||||||
+90
-24
@@ -1,12 +1,13 @@
|
|||||||
/** @file toybox end to end: a real workerd, an admin socket, a public port. */
|
/** @file toybox end to end: a real workerd, an admin socket, a public port, a switchboard. */
|
||||||
import { spawn } from "node:child_process";
|
import { spawn } from "node:child_process";
|
||||||
import { mkdir, mkdtemp, readFile, rm } from "node:fs/promises";
|
import { mkdir, mkdtemp, readFile, rm } from "node:fs/promises";
|
||||||
import { request } from "node:http";
|
import { createServer as createHttpServer, request } from "node:http";
|
||||||
import { createServer } from "node:net";
|
import { connect, createServer } from "node:net";
|
||||||
import { tmpdir } from "node:os";
|
import { tmpdir } from "node:os";
|
||||||
import { join } from "node:path";
|
import { join } from "node:path";
|
||||||
import { after, before, test } from "node:test";
|
import { after, before, test } from "node:test";
|
||||||
import assert from "node:assert/strict";
|
import assert from "node:assert/strict";
|
||||||
|
import { WebSocketServer } from "ws";
|
||||||
|
|
||||||
const here = new URL("..", import.meta.url).pathname;
|
const here = new URL("..", import.meta.url).pathname;
|
||||||
// the package binary, not the .bin shim: the shim forks it, so a kill would orphan workerd
|
// the package binary, not the .bin shim: the shim forks it, so a kill would orphan workerd
|
||||||
@@ -21,11 +22,19 @@ const freePort = () =>
|
|||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
/** Resolves once something accepts connections at `opts` (a port or a socket path). */
|
||||||
|
const accepting = (opts) =>
|
||||||
|
new Promise((resolve) => {
|
||||||
|
const s = connect(opts, () => (s.end(), resolve(true)));
|
||||||
|
s.on("error", () => resolve(false));
|
||||||
|
});
|
||||||
|
|
||||||
/** @param {string} socketPath the admin socket; holding it is the authority to install */
|
/** @param {string} socketPath the admin socket; holding it is the authority to install */
|
||||||
const makeAdmin = (socketPath) => ({
|
const makeAdmin = (socketPath) => ({
|
||||||
install: (vat, src) =>
|
install: (vat, src, egress) =>
|
||||||
new Promise((resolve, reject) => {
|
new Promise((resolve, reject) => {
|
||||||
const req = request({ socketPath, path: `/${vat}`, method: "PUT", agent: false }, (res) => {
|
const path = egress ? `/${vat}?egress=${encodeURIComponent(egress)}` : `/${vat}`;
|
||||||
|
const req = request({ socketPath, path, method: "PUT", agent: false }, (res) => {
|
||||||
let body = "";
|
let body = "";
|
||||||
res.on("data", (c) => (body += c));
|
res.on("data", (c) => (body += c));
|
||||||
res.on("end", () => resolve(body.trim()));
|
res.on("end", () => resolve(body.trim()));
|
||||||
@@ -37,49 +46,80 @@ const makeAdmin = (socketPath) => ({
|
|||||||
|
|
||||||
/** @param {string} base the public origin */
|
/** @param {string} base the public origin */
|
||||||
const makePublic = (base) => ({
|
const makePublic = (base) => ({
|
||||||
get: async (vat) => (await fetch(`${base}/${vat}`)).text(),
|
get: async (path) => (await fetch(`${base}/${path}`)).text(),
|
||||||
put: async (vat, src) => (await fetch(`${base}/${vat}`, { method: "PUT", body: src })).status,
|
put: async (vat, src) => (await fetch(`${base}/${vat}`, { method: "PUT", body: src })).status,
|
||||||
});
|
});
|
||||||
|
|
||||||
/** Run workerd on `dir`'s state until `stop()`; resolves once the port answers. */
|
/**
|
||||||
const startWorkerd = async (dir, port) => {
|
* Run workerd on `config` until `stop()`; resolves once `listening` accepts.
|
||||||
await rm(join(dir, "admin.sock"), { force: true }); // left behind by a crash; bind() would fail
|
*
|
||||||
const child = spawn(
|
* @param {string} config
|
||||||
workerd,
|
* @param {string[]} args socket, directory, and external overrides
|
||||||
["serve", "--experimental", join(here, "config.capnp"),
|
* @param {{ port: number } | { path: string }} listening
|
||||||
"-s", `http=127.0.0.1:${port}`, "-s", `admin=unix:${join(dir, "admin.sock")}`,
|
* @param {string[]} stale socket files a crash leaves behind; bind() would fail on them
|
||||||
"-d", `state=${join(dir, "state")}`],
|
*/
|
||||||
{ stdio: ["ignore", "ignore", "pipe"] },
|
const startWorkerd = async (config, args, listening, stale = []) => {
|
||||||
);
|
for (const f of stale) await rm(f, { force: true });
|
||||||
|
const child = spawn(workerd, ["serve", "--experimental", join(here, config), ...args], {
|
||||||
|
stdio: ["ignore", "ignore", "pipe"],
|
||||||
|
});
|
||||||
let log = "";
|
let log = "";
|
||||||
child.stderr.on("data", (c) => (log += c));
|
child.stderr.on("data", (c) => (log += c));
|
||||||
const exited = new Promise((resolve) => child.on("exit", resolve));
|
const exited = new Promise((resolve) => child.on("exit", resolve));
|
||||||
for (let i = 0; i < 100; i++) {
|
for (let i = 0; i < 100; i++) {
|
||||||
try {
|
|
||||||
await fetch(`http://127.0.0.1:${port}/`);
|
|
||||||
// SIGKILL: a crash is the harder durability test
|
// SIGKILL: a crash is the harder durability test
|
||||||
return { stop: () => (child.kill("SIGKILL"), exited) };
|
if (await accepting(listening)) return { stop: () => (child.kill("SIGKILL"), exited) };
|
||||||
} catch {
|
|
||||||
await new Promise((r) => setTimeout(r, 50));
|
await new Promise((r) => setTimeout(r, 50));
|
||||||
}
|
}
|
||||||
}
|
|
||||||
child.kill("SIGKILL");
|
child.kill("SIGKILL");
|
||||||
throw new Error(`workerd did not start: ${log}`);
|
throw new Error(`workerd did not start: ${log}`);
|
||||||
};
|
};
|
||||||
|
|
||||||
let dir, port, server, admin, pub;
|
/** An HTTP server that answers GET with "plain ok" and echoes WebSocket messages. */
|
||||||
|
const startEcho = async () => {
|
||||||
|
const server = createHttpServer((_req, res) => res.end("plain ok"));
|
||||||
|
const wss = new WebSocketServer({ server });
|
||||||
|
wss.on("connection", (ws) => ws.on("message", (m) => ws.send(String(m))));
|
||||||
|
await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve));
|
||||||
|
return {
|
||||||
|
host: `127.0.0.1:${server.address().port}`,
|
||||||
|
stop: () => new Promise((resolve) => (wss.close(), server.closeAllConnections(), server.close(resolve))),
|
||||||
|
};
|
||||||
|
};
|
||||||
|
|
||||||
|
let dir, port, server, switchboard, echo, admin, pub;
|
||||||
|
|
||||||
|
const startToybox = () =>
|
||||||
|
startWorkerd(
|
||||||
|
"config.capnp",
|
||||||
|
[
|
||||||
|
"-s", `http=127.0.0.1:${port}`, "-s", `admin=unix:${join(dir, "admin.sock")}`,
|
||||||
|
"-d", `state=${join(dir, "state")}`, "-e", `switchboard=unix:${join(dir, "switchboard.sock")}`,
|
||||||
|
],
|
||||||
|
{ port, host: "127.0.0.1" },
|
||||||
|
[join(dir, "admin.sock")],
|
||||||
|
);
|
||||||
|
|
||||||
before(async () => {
|
before(async () => {
|
||||||
dir = await mkdtemp(join(tmpdir(), "toybox-"));
|
dir = await mkdtemp(join(tmpdir(), "toybox-"));
|
||||||
await mkdir(join(dir, "state"));
|
await mkdir(join(dir, "state"));
|
||||||
port = await freePort();
|
port = await freePort();
|
||||||
server = await startWorkerd(dir, port);
|
echo = await startEcho();
|
||||||
|
switchboard = await startWorkerd(
|
||||||
|
"switchboard.capnp",
|
||||||
|
["-s", `http=unix:${join(dir, "switchboard.sock")}`],
|
||||||
|
{ path: join(dir, "switchboard.sock") },
|
||||||
|
[join(dir, "switchboard.sock")],
|
||||||
|
);
|
||||||
|
server = await startToybox();
|
||||||
admin = makeAdmin(join(dir, "admin.sock"));
|
admin = makeAdmin(join(dir, "admin.sock"));
|
||||||
pub = makePublic(`http://127.0.0.1:${port}`);
|
pub = makePublic(`http://127.0.0.1:${port}`);
|
||||||
});
|
});
|
||||||
|
|
||||||
after(async () => {
|
after(async () => {
|
||||||
await server?.stop();
|
await server?.stop();
|
||||||
|
await switchboard?.stop();
|
||||||
|
await echo?.stop();
|
||||||
await rm(dir, { recursive: true, force: true });
|
await rm(dir, { recursive: true, force: true });
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -114,6 +154,32 @@ test("code and state survive a restart", async () => {
|
|||||||
await admin.install("dave", await app("counter-v1.js"));
|
await admin.install("dave", await app("counter-v1.js"));
|
||||||
assert.equal(await pub.get("dave"), "v1 count 1\n");
|
assert.equal(await pub.get("dave"), "v1 count 1\n");
|
||||||
await server.stop();
|
await server.stop();
|
||||||
server = await startWorkerd(dir, port);
|
server = await startToybox();
|
||||||
assert.equal(await pub.get("dave"), "v1 count 2\n");
|
assert.equal(await pub.get("dave"), "v1 count 2\n");
|
||||||
});
|
});
|
||||||
|
|
||||||
|
const probe = (vat, target, ws) =>
|
||||||
|
pub.get(`${vat}?url=${encodeURIComponent(`http://${target}/`)}${ws ? "&ws=1" : ""}`);
|
||||||
|
|
||||||
|
test("a vat granted a host reaches it through the switchboard: fetch and WebSocket", async () => {
|
||||||
|
await admin.install("erin", await app("egress-probe.js"), echo.host);
|
||||||
|
assert.equal(await probe("erin", echo.host), "got: plain ok\n");
|
||||||
|
assert.equal(await probe("erin", echo.host, true), "echo: hello\n");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("same code, no grant: no fetch", async () => {
|
||||||
|
await admin.install("frank", await app("egress-probe.js"));
|
||||||
|
assert.match(await probe("frank", echo.host), /^refused: This worker is not permitted to access the internet/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a grant covers one host only", async () => {
|
||||||
|
await admin.install("grace", await app("egress-probe.js"), "127.0.0.1:1");
|
||||||
|
assert.equal(await probe("grace", echo.host), `got: egress to ${echo.host} not granted\n\n`);
|
||||||
|
assert.match(await probe("grace", echo.host, true), /^no websocket: 403 egress to .* not granted/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a grant survives a restart", async () => {
|
||||||
|
await server.stop();
|
||||||
|
server = await startToybox();
|
||||||
|
assert.equal(await probe("erin", echo.host, true), "echo: hello\n");
|
||||||
|
});
|
||||||
Reference in new issue
Block a user