WIP: diigo-cache: /public/<username>/all, searched client-side #2

Draft
dckc wants to merge 1 commits from dckc/diigo-search AGit into main
pull from: dckc/diigo-search
12 changed files with 357 additions and 86 deletions

No files matched your search

+3 -1
View File
@@ -10,7 +10,9 @@ diigo-cache is a Cloudflare Worker cache over [the webapp factory](../webapp/REA
- `PublicCache` (binding `PUBLIC_DO`): shared items only — private rows are dropped *before* etching (`publicOnly`), so its storage holds nothing private. Same cookie webapp as the private cache. Public page n is diigo page n, filtered (may show fewer than `PAGE_SIZE`, 100: diigo's cap). `/public/<username>` resolves through the `idFromName("index")` registry, and reads through `viewer()`, a read-only facet: it shows stored pages and fill progress, and can't fetch from diigo or write.
- **Storage**: SQL lives in `src/cache-read.sql` (selects only) and `src/cache-write.sql`; `src/store.ts` turns them into a reader and a writer facet, so the cache logic holds no SQL text.
- **Pages**: home's dump form (cache id + page number, POST) shows raw dump rows; `/public/<username>?page=n` shows an HTML list with a page form (JSON for `Accept: application/json`). Both say `cache: HIT` or `MISS`; every page read also logs `{event: "cache", kind, page, outcome}`.
- **Filters on the public page** (`src/query.ts`): `days`, `after`, `before`, `in` (`2026`, `2026-03`, `2026-Q1`, `2026-W41`), `by=updated|created`, and `q` (words, `"phrases"`, `#tag`, `-` negation, gmail/Slack/GitHub date operators). UTC, half-open windows. The same fields work as a dev-side spec (`toQuery({ days: 7 })`). A filter queries the stored `items` table through `viewer()`, never diigo: results are as fresh as the cache, and JSON says `complete` when every diigo page is stored.
- **Filters on the public page** (`src/query.js`): `days`, `after`, `before`, `in` (`2026`, `2026-03`, `2026-Q1`, `2026-W41`), `by=updated|created`, and `q` (words, `"phrases"`, `#tag`, `-` negation, gmail/Slack/GitHub date operators). UTC, half-open windows. The same fields work as a dev-side spec (`toQuery({ days: 7 })`). A filter queries the stored `items` table through `viewer()`, never diigo: results are as fresh as the cache, and JSON says `complete` when every diigo page is stored.
- **All bookmarks, searched client-side**: `/public/<username>/all` lists every stored shared bookmark in the same microformats. Without JS that's the whole list, and its search form asks the server. With JS (`src/all-page.ts`), the newest 5 show and search hides and shows entries in place, by the server's own matcher: `src/query.js` is plain JS with JSDoc types, so the worker imports it as code and also serves it, as text (`?raw`, a wrangler Text rule), at `/_query.js`.
- **All bookmarks, searched client-side**: `/public/<username>/all` lists every stored shared bookmark in the same microformats. Without JS that's the whole list, and its search form asks the server. With JS (`src/all-page.ts`), the newest 5 show and search hides and shows entries in place, by the server's own matcher: `src/query.js` is plain JS with JSDoc types, so the worker imports it as code and also serves it, as text (`?raw`, a wrangler Text rule), at `/_query.js`.
- **Publish later**: `POST /_publish` (cache id in the body) wires a public companion to a private-only cache.
- **The webkey page and Cap'n Web**: `/cache/#<cache id>` is a skeleton page, the same for every key; its script reads the key from the fragment and trades it in-band for a capability over [Cap'n Web](https://github.com/cloudflare/capnweb) at `/api` (`CacheApi.cache(key)` → a `CacheCap` facet: fill, status, the public companion; `src/api.ts`). The worker serves capnweb's browser build itself at `/_capnweb.js`.
- **Fill job**: walks every diigo page into a cache in the background — DO alarms, a few pages per step, progress stored in the DO (`startFill`/`fillStatus`/`stopFill`), driven from the webkey page; the public page shows progress. "Invalidate the last N days" there drops the cached pages holding anything updated that recently (`invalidateRecent`), so the next fill refetches them.
+1
View File
@@ -19,6 +19,7 @@ Then, in the browser:
## Bookmarks you hold, a page anyone can read
- **Filter by date and search:** `?days=7`, `?in=2026-W41` (ISO week, UTC), `?after=2026-01-01&before=2026-02-01`, `?by=created`, and `?q=` with words, `"phrases"`, `#tags`, `-negation`, and gmail-style `newer_than:7d`. Add `Accept: application/json` for JSON.
- **Everything on one page:** `/public/<you>/all` lists every stored shared bookmark; with JS it shows the newest 5 and searches as you type, with the same syntax.
- **Private stays private:** the public cache drops private bookmarks before storing anything, so it never holds them.
- **Ids are capabilities:** the cache id is a bearer key you keep in your password manager. It rides URL fragments and POST bodies, never a path or query string, so it stays out of logs and history.
- **Cheap to read:** pages come from Durable Object SQLite storage, and a fill job in the background fetches each diigo page once.
+53
View File
@@ -0,0 +1,53 @@
/** @file the all-bookmarks page's script: client-side search over the page's own microformats. */
/**
* the script for /public/<username>/all. Without it, the page lists every
* stored shared bookmark, and its form searches on the server. With it,
* the page shows the newest few and searches as you type, using
* `shown()` from /_query.js: the server's own matcher, fed items read
* back from each `h-entry`.
*/
export const ALL_PAGE_SCRIPT = `
import { shown } from "/_query.js";
const entries = [...document.querySelectorAll(".h-feed > .h-entry")];
const text = (li, selector) => li.querySelector(selector)?.textContent ?? "";
const secs = (li, cls) => {
const time = li.querySelector("time." + cls);
return time ? Date.parse(time.dateTime) / 1000 : undefined;
};
const items = entries.map((li) => ({
title: text(li, ".p-name"),
url: li.querySelector(".u-bookmark-of")?.getAttribute("href") ?? "",
description: text(li, ".p-summary"),
tags: [...li.querySelectorAll(".p-category")].map((a) => a.textContent).join(","),
created_at: secs(li, "dt-published"),
updated_at: secs(li, "dt-updated"),
}));
const form = document.getElementById("search");
const input = form.elements.q;
const status = document.getElementById("all-status");
const apply = () => {
const result = shown(items, input.value, new Date());
if ("error" in result) {
status.textContent = result.error;
return;
}
entries.forEach((li, i) => {
li.hidden = !result.show[i];
});
const n = entries.length;
status.textContent =
result.matched === undefined
? \`the newest \${Math.min(5, n)} of \${n}; search to see more\`
: \`\${result.matched} of \${n} match\`;
};
form.addEventListener("submit", (event) => {
event.preventDefault();
apply();
});
input.addEventListener("input", apply);
input.value = new URLSearchParams(location.search).get("q") ?? "";
apply();
`;
+58 -5
View File
@@ -8,6 +8,7 @@ import { newWorkersRpcResponse } from "capnweb";
import { CacheApi } from "./api.js";
import { CAPNWEB_BROWSER_JS } from "./capnweb-browser.js";
import { CACHE_PAGE_HTML } from "./cache-page.js";
import { ALL_PAGE_SCRIPT } from "./all-page.js";
import { DiigoCacheLogic, PublicCacheLogic } from "./cache.js";
import { PAGE_SIZE } from "./cache.js";
import type { CacheEnv, Clock, FillStatus, QueryPage } from "./cache.js";
@@ -15,6 +16,7 @@ import type { Item } from "./cache.js";
import { usernameOf } from "./items.js";
import { dateOf, isEmpty, parseQuery } from "./query.js";
import type { Query } from "./query.js";
import QUERY_JS from "./query.js?raw";
import { toWebkey, idFromWebkey, isWebkey, shashKey } from "./webkeys.js";
import { pmSave } from "./pm-save.js";
@@ -85,6 +87,8 @@ export class DiigoBridge extends WorkerEntrypoint {
* - `GET /cache/`: the webkey page ({@link CACHE_PAGE_HTML});
* `/api`: its Cap'n Web API ({@link CacheApi}); `GET /_capnweb.js`
* - `GET /public/<username>`: public pages, read-only ({@link publicPage})
* - `GET /public/<username>/all`: every stored shared bookmark, searched
* client-side ({@link allPage}); `GET /_query.js`, its matcher
*
* Cache ids are webkeys (docs/webkeys.md): they ride POST bodies and URL
* fragments, never query strings or paths.
@@ -95,6 +99,8 @@ export default {
const pub = url.pathname.match(/^\/public\/([A-Za-z0-9_-]+)$/);
if (pub) return publicPage(request, env, pub[1]);
const all = url.pathname.match(/^\/public\/([A-Za-z0-9_-]+)\/all$/);
if (all) return allPage(env, all[1]);
if (url.pathname === "/_connect" && request.method === "GET") {
return mintPage(env);
@@ -114,6 +120,9 @@ export default {
if (url.pathname === "/_capnweb.js" && request.method === "GET") {
return new Response(CAPNWEB_BROWSER_JS, { headers: { "content-type": "text/javascript;charset=utf-8" } });
}
if (url.pathname === "/_query.js" && request.method === "GET") {
return new Response(QUERY_JS, { headers: { "content-type": "text/javascript;charset=utf-8" } });
}
if (url.pathname === "/api") {
return newWorkersRpcResponse(request, new CacheApi(env));
}
@@ -458,11 +467,8 @@ const DAY_PRESETS = [7, 30, 90, 365];
* fine here: nothing on this page is secret.
*/
async function publicPage(request: Request, env: Env, username: string): Promise<Response> {
const index = env.PUBLIC_DO.get(env.PUBLIC_DO.idFromName("index"));
const publicId = await index.lookup(username);
if (!publicId) return new Response("no such public cache", { status: 404 });
const ns = env.PUBLIC_DO;
const cache = ns.get(ns.idFromString(idFromWebkey(publicId)));
const cache = await publicCacheOf(env, username);
if (!cache) return new Response("no such public cache", { status: 404 });
const params = new URL(request.url).searchParams;
const accept = request.headers.get("accept") ?? "";
@@ -534,6 +540,53 @@ ${notice}${list}`,
return new Response(body, { headers: { "content-type": "text/html;charset=utf-8" } });
}
/** the public cache registered for `username`, if any */
async function publicCacheOf(env: Env, username: string) {
const index = env.PUBLIC_DO.get(env.PUBLIC_DO.idFromName("index"));
const publicId = await index.lookup(username);
if (!publicId) return undefined;
return env.PUBLIC_DO.get(env.PUBLIC_DO.idFromString(idFromWebkey(publicId)));
}
/**
* every stored shared bookmark on one page, newest updated first, as the
* same microformats as the public page. Without JS, that's the whole
* list, and the search form asks the public page. With JS
* ({@link ALL_PAGE_SCRIPT}), the newest few show, and search hides and
* shows entries in place with the server's matcher.
*/
async function allPage(env: Env, username: string): Promise<Response> {
const cache = await publicCacheOf(env, username);
if (!cache) return new Response("no such public cache", { status: 404 });
const viewer = await cache.viewer();
let found, fill;
try {
found = await viewer.query({ by: "updated", words: [], tags: [] }, 1, Number.MAX_SAFE_INTEGER);
fill = await viewer.fillStatus();
} catch (err) {
console.error({ event: "all-page", error: String(err) });
return new Response(`could not load bookmarks: ${String(err)}`, { status: 502 });
} finally {
viewer[Symbol.dispose]();
}
const here = `/public/${encodeURIComponent(username)}`;
const items = found.items.map((j) => JSON.parse(j) as Item);
const caveat = found.complete ? "" : " — not every diigo page is stored yet, so some may be missing";
const body = publicFrame(
username,
`<p id="all-status">${items.length} shared bookmark${items.length === 1 ? "" : "s"}, newest first${caveat}</p>
${fill ? fillLine(fill) : ""}
<form id="search" method="get" action="${here}">
<p><label>search <input type="search" name="q" size="40"></label> <button type="submit">search</button></p>
<p><small>words, "phrases", #tags, -negation, and after:, before:, newer_than:7d, during:2026-Q1, created:2026 (UTC)</small></p>
</form>
<p><a href="${here}">by diigo page</a></p>
${itemList(items, "updated", here)}
<script type="module">${ALL_PAGE_SCRIPT}</script>`,
);
return new Response(body, { headers: { "content-type": "text/html;charset=utf-8" } });
}
/** a filter's matches, as the public page's HTML or JSON */
function filteredPage(
username: string,
@@ -1,10 +1,14 @@
/**
* @file the public page's filters: date windows and search over a cache's
* stored items (goal 1d; tests/query.test.ts).
*
* Plain JS with JSDoc types, so the worker imports it as code and serves
* the same file to browsers at /_query.js: the all-bookmarks page searches
* client-side with exactly the server's matcher.
*
* @import { Item } from "./items.js"
*/
import type { Item } from "./items.js";
/**
* a filter as devs write it (`{ days: 7, q: "#weekly" }`); URL params map
* onto these fields one-to-one (strings, with "" meaning absent). Dates
@@ -12,49 +16,49 @@ import type { Item } from "./items.js";
* `2025-03`, `2025-Q1`, or ISO week `2025-W01`. Time is UTC throughout;
* ISO weeks start Monday, keeping a weekend together.
*/
export type FilterSpec = {
/** the last N days, counted back from now (1–3650) */
days?: number | string;
/** inclusive lower bound, from 00:00Z */
after?: string;
/** exclusive upper bound, at 00:00Z */
before?: string;
/** a named period: shorthand for its after/before pair */
in?: string;
/** which date the window, the listed date, and the sort use */
by?: "updated" | "created" | "";
/** words, phrases, tags, and operators */
q?: string;
};
/**
* @typedef {object} FilterSpec
* @property {number | string} [days] the last N days, counted back from now (1–3650)
* @property {string} [after] inclusive lower bound, from 00:00Z
* @property {string} [before] exclusive upper bound, at 00:00Z
* @property {string} [in] a named period: shorthand for its after/before pair
* @property {"updated" | "created" | ""} [by] which date the window, the listed date, and the sort use
* @property {string} [q] words, phrases, tags, and operators
*/
/** one search term; `negated` for a leading `-` */
export type Term = { text: string; negated: boolean };
/**
* one search term; `negated` for a leading `-`
* @typedef {{ text: string; negated: boolean }} Term
*/
/** a parsed filter; an empty one (no window, no terms) filters nothing */
export type Query = {
/** which date the window, the listed date, and the sort use (default
* `updated`, diigo's own order) */
by: "updated" | "created";
/** inclusive lower bound */
after?: Date;
/** exclusive upper bound */
before?: Date;
/** words and phrases: each must appear in title, description, or url */
words: Term[];
/** tags: exact, case-insensitive match against the item's tags */
tags: Term[];
};
/**
* a parsed filter; an empty one (no window, no terms) filters nothing
* @typedef {object} Query
* @property {"updated" | "created"} by which date the window, the listed
* date, and the sort use (default `updated`, diigo's own order)
* @property {Date} [after] inclusive lower bound
* @property {Date} [before] exclusive upper bound
* @property {Term[]} words words and phrases: each must appear in title, description, or url
* @property {Term[]} tags tags: exact, case-insensitive match against the item's tags
*/
/** a refused filter: the message is safe to show and log */
export type QueryError = { error: string };
/**
* a refused filter: the message is safe to show and log
* @typedef {{ error: string }} QueryError
*/
const DAY_MS = 24 * 60 * 60 * 1000;
/** thrown inside {@link toQuery}, returned as a {@link QueryError} */
class Refused extends Error {}
/** `yyyy-mm-dd` or `yyyy/mm/dd` as 00:00Z of that UTC day */
function parseDay(text: string, what: string): Date {
/**
* `yyyy-mm-dd` or `yyyy/mm/dd` as 00:00Z of that UTC day
* @param {string} text
* @param {string} what
* @returns {Date}
*/
function parseDay(text, what) {
const m = /^(\d{4})([-/])(\d{2})\2(\d{2})$/.exec(text);
const d = m && new Date(Date.UTC(Number(m[1]), Number(m[3]) - 1, Number(m[4])));
if (!m || !d || d.getUTCMonth() !== Number(m[3]) - 1 || d.getUTCDate() !== Number(m[4])) {
@@ -63,15 +67,24 @@ function parseDay(text: string, what: string): Date {
return d;
}
/** Monday 00:00Z of ISO week 1 of `year` (the week holding January 4) */
function isoWeekOne(year: number): number {
/**
* Monday 00:00Z of ISO week 1 of `year` (the week holding January 4)
* @param {number} year
* @returns {number}
*/
function isoWeekOne(year) {
const jan4 = Date.UTC(year, 0, 4);
const weekday = new Date(jan4).getUTCDay() || 7;
return jan4 - (weekday - 1) * DAY_MS;
}
/** a named period as its half-open [after, before) window */
function parsePeriod(text: string, what: string): [Date, Date] {
/**
* a named period as its half-open [after, before) window
* @param {string} text
* @param {string} what
* @returns {[Date, Date]}
*/
function parsePeriod(text, what) {
const refuse = () => new Refused(`${what}: expected a period as 2025, 2025-03, 2025-Q1, or 2025-W01`);
let m;
if ((m = /^(\d{4})$/.exec(text))) {
@@ -97,9 +110,14 @@ function parsePeriod(text: string, what: string): [Date, Date] {
throw refuse();
}
/** gmail's `newer_than:` amount (`7d`, `2w`, `1m`, `1y`) counted back from
* `now`; months and years are calendar steps */
function parseNewerThan(text: string, now: Date): Date {
/**
* gmail's `newer_than:` amount (`7d`, `2w`, `1m`, `1y`) counted back from
* `now`; months and years are calendar steps
* @param {string} text
* @param {Date} now
* @returns {Date}
*/
function parseNewerThan(text, now) {
const m = /^(\d{1,4})([dwmy])$/.exec(text);
if (!m || Number(m[1]) < 1) throw new Refused("newer_than: expected an amount like 7d, 2w, 1m, or 1y");
const n = Number(m[1]);
@@ -111,8 +129,12 @@ function parseNewerThan(text: string, now: Date): Date {
return d;
}
/** `days` as a whole number from 1 to 3650 */
function parseDays(days: number | string): number {
/**
* `days` as a whole number from 1 to 3650
* @param {number | string} days
* @returns {number}
*/
function parseDays(days) {
const n = typeof days === "number" ? days : /^\d+$/.test(days) ? Number(days) : NaN;
if (!Number.isInteger(n) || n < 1 || n > 3650) throw new Refused("days: expected a whole number from 1 to 3650");
return n;
@@ -133,10 +155,12 @@ function parseDays(days: number | string): number {
* @see https://slack.com/help/articles/202528808
* @see https://docs.github.com/en/search-github/searching-on-github/searching-for-repositories
*
* @param now the clock `days`/`newer_than` count back from (passed in,
* not read ambiently, so tests pin it)
* @param {FilterSpec} spec
* @param {Date} now the clock `days`/`newer_than` count back from (passed
* in, not read ambiently, so tests pin it)
* @returns {Query | QueryError}
*/
export function toQuery(spec: FilterSpec, now: Date): Query | QueryError {
export function toQuery(spec, now) {
try {
return resolve(spec, now);
} catch (err) {
@@ -145,39 +169,54 @@ export function toQuery(spec: FilterSpec, now: Date): Query | QueryError {
}
}
function resolve(spec: FilterSpec, now: Date): Query {
const words: Term[] = [];
const tags: Term[] = [];
let by: { value: Query["by"]; from: string } | undefined;
let after: { value: Date; from: string } | undefined;
let before: { value: Date; from: string } | undefined;
/**
* @param {FilterSpec} spec
* @param {Date} now
* @returns {Query}
*/
function resolve(spec, now) {
/** @type {Term[]} */
const words = [];
/** @type {Term[]} */
const tags = [];
/** @type {{ value: Query["by"]; from: string } | undefined} */
let by;
/** @type {{ value: Date; from: string } | undefined} */
let after;
/** @type {{ value: Date; from: string } | undefined} */
let before;
const setBy = (value: Query["by"], from: string) => {
/** @type {(value: Query["by"], from: string) => void} */
const setBy = (value, from) => {
if (by && by.value !== value) throw new Refused(`${by.from} and ${from} name different dates`);
by = { value, from };
};
const setAfter = (value: Date, from: string) => {
/** @type {(value: Date, from: string) => void} */
const setAfter = (value, from) => {
if (after) throw new Refused(`${after.from} and ${from} both set where the window starts`);
after = { value, from };
};
const setBefore = (value: Date, from: string) => {
/** @type {(value: Date, from: string) => void} */
const setBefore = (value, from) => {
if (before) throw new Refused(`${before.from} and ${from} both set where the window ends`);
before = { value, from };
};
const setPeriod = ([lo, hi]: [Date, Date], from: string) => {
/** @type {(window: [Date, Date], from: string) => void} */
const setPeriod = ([lo, hi], from) => {
setAfter(lo, from);
setBefore(hi, from);
};
const present = (v: string | number | undefined) => v !== undefined && v !== "";
/** @type {(v: string | number | undefined) => boolean} */
const present = (v) => v !== undefined && v !== "";
if (present(spec.by)) {
if (spec.by !== "updated" && spec.by !== "created") throw new Refused("by: expected updated or created");
setBy(spec.by, `by=${spec.by}`);
}
if (present(spec.days)) setAfter(new Date(now.getTime() - parseDays(spec.days!) * DAY_MS), "days");
if (present(spec.after)) setAfter(parseDay(spec.after!, "after"), "after");
if (present(spec.before)) setBefore(parseDay(spec.before!, "before"), "before");
if (present(spec.in)) setPeriod(parsePeriod(spec.in!, "in"), "in");
if (spec.days !== undefined && present(spec.days)) setAfter(new Date(now.getTime() - parseDays(spec.days) * DAY_MS), "days");
if (spec.after) setAfter(parseDay(spec.after, "after"), "after");
if (spec.before) setBefore(parseDay(spec.before, "before"), "before");
if (spec.in) setPeriod(parsePeriod(spec.in, "in"), "in");
for (const m of (spec.q ?? "").matchAll(/(-?)"([^"]*)"|(\S+)/g)) {
if (m[2] !== undefined) {
@@ -207,15 +246,22 @@ function resolve(spec: FilterSpec, now: Date): Query {
if (after && before && after.value.getTime() >= before.value.getTime()) {
throw new Refused(`the window is empty: ${after.from} is not earlier than ${before.from}`);
}
const query: Query = { by: by?.value ?? "updated", words, tags };
/** @type {Query} */
const query = { by: by?.value ?? "updated", words, tags };
if (after) query.after = after.value;
if (before) query.before = before.value;
return query;
}
/** {@link toQuery} over the public page's URL params */
export function parseQuery(params: URLSearchParams, now: Date): Query | QueryError {
const field = (name: string) => params.get(name) ?? "";
/**
* {@link toQuery} over the public page's URL params
* @param {URLSearchParams} params
* @param {Date} now
* @returns {Query | QueryError}
*/
export function parseQuery(params, now) {
/** @type {(name: string) => string} */
const field = (name) => params.get(name) ?? "";
const by = field("by");
return toQuery(
{
@@ -223,34 +269,49 @@ export function parseQuery(params: URLSearchParams, now: Date): Query | QueryErr
after: field("after"),
before: field("before"),
in: field("in"),
by: by as FilterSpec["by"],
by: /** @type {FilterSpec["by"]} */ (by),
q: field("q"),
},
now,
);
}
/** does the query filter anything? (an empty one lists diigo's pages as stored) */
export function isEmpty(query: Query): boolean {
/**
* does the query filter anything? (an empty one lists diigo's pages as stored)
* @param {Query} query
* @returns {boolean}
*/
export function isEmpty(query) {
return !query.after && !query.before && !query.words.length && !query.tags.length && query.by === "updated";
}
/** the item's date that `by` names, in epoch seconds (diigo's unit), or
* undefined when it carries none */
export function dateOf(item: Item, by: Query["by"]): number | undefined {
/**
* the item's date that `by` names, in epoch seconds (diigo's unit), or
* undefined when it carries none
* @param {Item} item
* @param {Query["by"]} by
* @returns {number | undefined}
*/
export function dateOf(item, by) {
const secs = by === "created" ? item.created_at : item.updated_at;
return typeof secs === "number" && Number.isFinite(secs) ? secs : undefined;
}
/** does `item` fall in the window and satisfy every term? */
export function matches(item: Item, query: Query): boolean {
/**
* does `item` fall in the window and satisfy every term?
* @param {Item} item
* @param {Query} query
* @returns {boolean}
*/
export function matches(item, query) {
if (query.after || query.before) {
const secs = dateOf(item, query.by);
if (secs === undefined) return false;
if (query.after && secs * 1000 < query.after.getTime()) return false;
if (query.before && secs * 1000 >= query.before.getTime()) return false;
}
const text = (v: unknown) => (typeof v === "string" ? v : "");
/** @type {(v: unknown) => string} */
const text = (v) => (typeof v === "string" ? v : "");
// fields joined by newline, so a phrase never spans two of them
const haystack = [item.title, item.description, item.url].map(text).join("\n").toLowerCase();
if (!query.words.every((w) => haystack.includes(w.text) !== w.negated)) return false;
@@ -262,3 +323,22 @@ export function matches(item: Item, query: Query): boolean {
);
return query.tags.every((t) => itemTags.has(t.text) !== t.negated);
}
/**
* which of a newest-first list of bookmarks the all-bookmarks page shows
* for search text `q`: the newest `latest` when `q` is blank, else every
* match. A refused search returns only its `error`, so the page can
* leave what it shows alone.
* @param {Item[]} items
* @param {string} q
* @param {Date} now
* @param {number} [latest]
* @returns {{ show: boolean[]; matched?: number } | QueryError}
*/
export function shown(items, q, now, latest = 5) {
if (!q.trim()) return { show: items.map((_, i) => i < latest) };
const query = toQuery({ q }, now);
if ("error" in query) return query;
const show = items.map((item) => matches(item, query));
return { show, matched: show.filter(Boolean).length };
}
+6
View File
@@ -3,3 +3,9 @@ declare module "*.sql" {
const text: string;
export default text;
}
/** `?raw` imports are a file's text (wrangler.jsonc "rules": Text) */
declare module "*?raw" {
const text: string;
export default text;
}
+50
View File
@@ -5,6 +5,8 @@ import worker from "../src/index.js";
import { idFromWebkey, toWebkey } from "../src/webkeys.js";
import { PAGE_SIZE } from "../src/cache.js";
import { mf2 } from "microformats-parser";
import { shown } from "../src/query.js";
import QUERY_JS from "../src/query.js?raw";
// matches tests/fake-webapp.js: 30 items, every third private
const WEBAPP_ID = "fake-webapp-id";
@@ -411,6 +413,54 @@ describe("public page filters (goal 1d)", () => {
expect(rels.next).toBeUndefined();
});
it("/all lists every stored shared bookmark, none hidden, as h-entries", async () => {
const { page } = await wireStored("fake-deep-webapp", 3);
const html = await (await call(`${page}/all`)).text();
// 300, one a day, minus every 10th (private)
const want = Array.from({ length: 300 }, (_, k) => k + 1).filter((n) => n % 10 !== 0);
expect(titlesIn(html)).toEqual(dated(...want));
expect(html).not.toMatch(/<li[^>]* hidden/);
const { items } = mf2(html, { baseUrl: `https://diigo-cache.test${page}/all` });
expect(items.find((i) => i.type?.includes("h-feed"))?.children).toHaveLength(270);
// without JS, the search form asks the server
expect(html).toContain(`<form id="search" method="get" action="${page}">`);
expect(html).toContain('import { shown } from "/_query.js"');
});
it("/_query.js serves the worker's own matcher, as a browser module", async () => {
const res = await call("/_query.js");
expect(res.headers.get("content-type")).toMatch(/^text\/javascript/);
const js = await res.text();
expect(js).toBe(QUERY_JS);
expect(js).toMatch(/^export function shown\(/m);
expect(js).not.toMatch(/^import /m);
});
it("searching /all's microformats client-side finds what the server finds", async () => {
const page = await wireDated();
const all = await (await call(`${page}/all`)).text();
const entries = mf2(all, { baseUrl: `https://diigo-cache.test${page}/all` }).items.find((i) => i.type?.includes("h-feed"))?.children ?? [];
// as the page's script reads each h-entry
const secs = (v: unknown) => (v === undefined ? undefined : Date.parse(String(v)) / 1000);
const items = entries.map(({ properties: p }) => ({
title: p.name?.[0],
url: p["bookmark-of"]?.[0],
description: p.summary?.[0],
tags: (p.category ?? []).join(","),
created_at: secs(p.published?.[0]),
updated_at: secs(p.updated?.[0]),
}));
const names = items.map((i) => i.title);
for (const q of ["fragments", "#ocap -#weekly", "newer_than:7d", "created:2026 dated", '"dated 1"']) {
const server = titlesIn(await (await call(`${page}?q=${encodeURIComponent(q)}`)).text());
const result = shown(items, q, new Date());
if ("error" in result) throw new Error(result.error);
// the same matches; /all keeps its updated order, where created: sorts by created
expect(names.filter((_, i) => result.show[i]).sort(), q).toEqual(server.sort());
expect(server.length, q).toBeGreaterThan(0);
}
});
it("filtered pages link the next page with rel=next, keeping the filter", async () => {
const { page } = await wireStored("fake-deep-webapp", 3);
const html = await (await call(`${page}?days=300`)).text();
+24 -1
View File
@@ -1,5 +1,5 @@
import { describe, expect, it } from "vitest";
import { matches, parseQuery, toQuery } from "../src/query.js";
import { matches, parseQuery, shown, toQuery } from "../src/query.js";
import type { FilterSpec, Query, QueryError } from "../src/query.js";
// goal 1d (capsite checklist)
@@ -204,3 +204,26 @@ describe("matches (goal 1d)", () => {
expect(matches(it1, query({ tags: [{ text: "js", negated: true }] }))).toBe(false);
});
});
describe("shown: the all-bookmarks page's client-side search", () => {
const items = Array.from({ length: 8 }, (_, k) => ({
title: `item ${k}`,
url: `https://example.com/${k}`,
tags: k % 2 ? "odd" : "even",
updated_at: (NOW.getTime() - k * DAY) / 1000,
}));
it("with no search, shows the newest 5", () => {
expect(shown(items, " ", NOW)).toEqual({ show: [true, true, true, true, true, false, false, false] });
});
it("a search shows every match, however old", () => {
expect(shown(items, "#odd", NOW)).toEqual({ show: [false, true, false, true, false, true, false, true], matched: 4 });
// the window's start is inclusive: item 2 is exactly 2 days old
expect(shown(items, "newer_than:2d -#odd", NOW)).toMatchObject({ matched: 2 });
});
it("a refused search returns only its error", () => {
expect(shown(items, "newer_than:soon", NOW)).toEqual({ error: expect.stringMatching(/newer_than/) });
});
});
+2 -1
View File
@@ -2,7 +2,8 @@
"extends": "../tsconfig.json",
"compilerOptions": {
"types": ["@cloudflare/vitest-plugin/types"],
"allowJs": true
"allowJs": true,
"checkJs": false
},
"include": ["./**/*", "../worker-configuration.d.ts"]
}
+2
View File
@@ -9,6 +9,8 @@
"skipLibCheck": true,
"lib": ["ES2022", "ESNext.Disposable"],
"noEmit": true,
"allowJs": true,
"checkJs": true,
"types": []
},
"include": ["src/**/*", "worker-configuration.d.ts"]
+1 -1
View File
@@ -6,7 +6,7 @@
"observability": { "enabled": true },
// capnweb's browser build, served as text at /_capnweb.js (the worker
// itself imports capnweb's workerd build as code)
"rules": [{ "type": "Text", "globs": ["**/capnweb/dist/index.js", "**/*.sql"], "fallthrough": true }],
"rules": [{ "type": "Text", "globs": ["**/capnweb/dist/index.js", "**/*.sql", "**/*.js?raw"], "fallthrough": true }],
"services": [
{
"binding": "WEBAPP",
+1 -1
View File
@@ -6,7 +6,7 @@
"name": "diigo-cache-tests",
"main": "./tests/test-worker.ts",
"compatibility_date": "2026-09-29",
"rules": [{ "type": "Text", "globs": ["**/capnweb/dist/index.js", "**/*.sql"], "fallthrough": true }],
"rules": [{ "type": "Text", "globs": ["**/capnweb/dist/index.js", "**/*.sql", "**/*.js?raw"], "fallthrough": true }],
"services": [
{ "binding": "WEBAPP", "service": "fake-webapp", "entrypoint": "WebAppFactory" }
],