dckc
cc298e4647
Export the public cache's items table in sqlite-diffable's format (items.ndjson + items.metadata.json), ordered by link_id so new bookmarks append and edits change one line, for a git-scraped repo that loads back into SQLite. pages/page_members stay out: cache bookkeeping that churns on every new bookmark. Generated-by: Claude Opus 5.5
66 lines
4.6 KiB
Markdown
66 lines
4.6 KiB
Markdown
# design: a sqlite-diffable export of diigo-cache
|
|
|
|
*Status: proposed, 2026-10-10. Not implemented.*
|
|
|
|
## goal
|
|
|
|
Keep a public diigo cache's bookmarks in a git repo, so each week's changes show up as a readable diff and the repo can be loaded back into SQLite. Use [sqlite-diffable](https://github.com/simonw/sqlite-diffable)'s format, so its `load` and `objects` commands work on the result.
|
|
|
|
## the format
|
|
|
|
`sqlite-diffable dump db out/ items` writes two files per table:
|
|
|
|
- `items.ndjson`: one row per line, as a JSON array of column values in column order.
|
|
- `items.metadata.json`: `{"name", "columns", "schema"}`, where `schema` is the table's `CREATE TABLE` text. It is indented 4 spaces (Python's `json.dumps(..., indent=4)`).
|
|
|
|
`sqlite-diffable load restored.db out/` runs each `schema` and inserts the rows.
|
|
|
|
## what to export: `items` only
|
|
|
|
The cache stores three tables (`src/cache-write.sql`):
|
|
|
|
| table | rows | export? |
|
|
|---|---|---|
|
|
| `items` | one per bookmark: `item_id` (diigo's `link_id`), `url`, `title`, `private`, `json` (the raw diigo row), `updated` | yes |
|
|
| `pages` | one per stored diigo page, with `etched_at` | no |
|
|
| `page_members` | each page's item order | no |
|
|
|
|
`pages` and `page_members` are cache bookkeeping. They also churn: a new bookmark shifts every item down one position, so every `page_members` row would change, and every refetch changes `etched_at`. Neither tells a reader anything about the bookmarks.
|
|
|
|
## making the diff small
|
|
|
|
- **Order rows by `link_id`, numerically.** sqlite-diffable dumps in rowid order, and for `items` that is insertion order, which depends on fill history. diigo ids grow over time (267 up to 837197225 in the dev cache), so with numeric order a new bookmark is an appended line and an edit changes one line in place.
|
|
- **Export the `json` column verbatim.** It is `JSON.stringify` of diigo's row, in diigo's key order, and the round trip depends on it. If diigo's key order turns out to vary between fetches, canonicalize at etch time instead. That would change stored state, so it gets flagged first ([capsite AGENTS.md](../AGENTS.md)).
|
|
- **No export timestamps in the files.** When and how complete a dump was goes in HTTP headers and the commit message, so a dump with no bookmark changes leaves no diff.
|
|
|
|
## where it comes from
|
|
|
|
The public cache holds only shared bookmarks, so the export can be public too:
|
|
|
|
- `GET /public/<username>/export/items.ndjson`, streamed from a cursor (the dev cache's 14,645 items come to about 15.5 MB of `json`).
|
|
- `GET /public/<username>/export/items.metadata.json`, whose `schema` is read from `sqlite_master`.
|
|
- Both read through `viewer()`, the read-only facet, the same way the public page does.
|
|
- A header `x-complete: true|false` says whether every diigo page is stored. A partial dump would look like mass deletions in git.
|
|
|
|
A script, `scripts/export-diffable.ts <origin> <username> <dir>`, fetches both files into `<dir>`. It refuses a partial dump unless given `--partial`. A cron job or GitHub Action runs the script and commits, in the style of Simon Willison's [git scraping](https://simonwillison.net/2020/Oct/9/git-scraping/).
|
|
|
|
A private cache's export would go through Cap'n Web on `CacheCap` (the cache id is the authority) rather than a URL. That can come later.
|
|
|
|
## round trip
|
|
|
|
`sqlite-diffable load restored.db dir/` gives an `items` table with the cache's schema. `scripts/compare-export.ts` already reads that table from a `cache.sqlite`, so it can check a loaded dump against a diigo RSS export unchanged.
|
|
|
|
## tests
|
|
|
|
- The two files match sqlite-diffable's format. Each `.ndjson` line is a JSON array with the same length as `columns`, and `schema` is the table's DDL.
|
|
- Rows come out in numeric `link_id` order, whatever order they were etched in.
|
|
- Etching the same pages twice yields byte-identical files.
|
|
- The public export holds no private rows, and `x-complete` tracks fill progress.
|
|
- Round trip: `node:sqlite` stands in for `sqlite-diffable load`, by running `schema` and inserting rows; the result matches the cache's own `items`. Optionally, CI also runs the real `sqlite-diffable load`.
|
|
|
|
## open questions
|
|
|
|
- **Deletions.** The cache only upserts, so a bookmark deleted on diigo stays in the cache and in the export. Showing deletions needs the fill job to notice them, for example a full refill that marks unseen items.
|
|
- **Volatile fields.** diigo rows carry counters (`c_count`, `pub_c_count`, `readed`, `pri_sticky_count`, …). If they change without any user edit, every dump has noise. One option is a second, stable view with those fields dropped. Measure first: dump twice, a week apart.
|
|
- **One repo per user, or one per cache?** One per user, so that `public/<username>` maps to one repo, seems simplest.
|