captoys/docs/diigo-diffable-export.md

design: a sqlite-diffable export of diigo-cache

Status: proposed, 2026-10-10. Not implemented.

goal

Keep a public diigo cache's bookmarks in a git repo, so each week's changes show up as a readable diff and the repo can be loaded back into SQLite. Use sqlite-diffable's format, so its load and objects commands work on the result.

the format

sqlite-diffable dump db out/ items writes two files per table:

• items.ndjson: one row per line, as a JSON array of column values in column order.
• items.metadata.json: {"name", "columns", "schema"}, where schema is the table's CREATE TABLE text. It is indented 4 spaces (Python's json.dumps(..., indent=4)).

sqlite-diffable load restored.db out/ runs each schema and inserts the rows.

what to export: items only

The cache stores three tables (src/cache-write.sql):

table	rows	export?
items	one per bookmark: item_id (diigo's link_id), url, title, private, json (the raw diigo row), updated	yes
pages	one per stored diigo page, with etched_at	no
page_members	each page's item order	no

pages and page_members are cache bookkeeping. They also churn: a new bookmark shifts every item down one position, so every page_members row would change, and every refetch changes etched_at. Neither tells a reader anything about the bookmarks.

making the diff small

• Order rows by link_id, numerically. sqlite-diffable dumps in rowid order, and for items that is insertion order, which depends on fill history. diigo ids grow over time (267 up to 837197225 in the dev cache), so with numeric order a new bookmark is an appended line and an edit changes one line in place.
• Export the json column verbatim. It is JSON.stringify of diigo's row, in diigo's key order, and the round trip depends on it. If diigo's key order turns out to vary between fetches, canonicalize at etch time instead. That would change stored state, so it gets flagged first (capsite AGENTS.md).
• No export timestamps in the files. When and how complete a dump was goes in HTTP headers and the commit message, so a dump with no bookmark changes leaves no diff.

where it comes from

The public cache holds only shared bookmarks, so the export can be public too:

• GET /public/<username>/export/items.ndjson, streamed from a cursor (the dev cache's 14,645 items come to about 15.5 MB of json).
• GET /public/<username>/export/items.metadata.json, whose schema is read from sqlite_master.
• Both read through viewer(), the read-only facet, the same way the public page does.
• A header x-complete: true|false says whether every diigo page is stored. A partial dump would look like mass deletions in git.

A script, scripts/export-diffable.ts <origin> <username> <dir>, fetches both files into <dir>. It refuses a partial dump unless given --partial. A cron job or GitHub Action runs the script and commits, in the style of Simon Willison's git scraping.

A private cache's export would go through Cap'n Web on CacheCap (the cache id is the authority) rather than a URL. That can come later.

round trip

sqlite-diffable load restored.db dir/ gives an items table with the cache's schema. scripts/compare-export.ts already reads that table from a cache.sqlite, so it can check a loaded dump against a diigo RSS export unchanged.

tests

• The two files match sqlite-diffable's format. Each .ndjson line is a JSON array with the same length as columns, and schema is the table's DDL.
• Rows come out in numeric link_id order, whatever order they were etched in.
• Etching the same pages twice yields byte-identical files.
• The public export holds no private rows, and x-complete tracks fill progress.
• Round trip: node:sqlite stands in for sqlite-diffable load, by running schema and inserting rows; the result matches the cache's own items. Optionally, CI also runs the real sqlite-diffable load.

open questions

• Deletions. The cache only upserts, so a bookmark deleted on diigo stays in the cache and in the export. Showing deletions needs the fill job to notice them, for example a full refill that marks unseen items.
• Volatile fields. diigo rows carry counters (c_count, pub_c_count, readed, pri_sticky_count, …). If they change without any user edit, every dump has noise. One option is a second, stable view with those fields dropped. Measure first: dump twice, a week apart.
• One repo per user, or one per cache? One per user, so that public/<username> maps to one repo, seems simplest.