From 99ce5b69643a284da5cb4268bb988d052523c362 Mon Sep 17 00:00:00 2001 From: Dan Connolly Date: Wed, 7 Oct 2026 10:21:24 -0500 Subject: [PATCH] capsite: goal trees for a zotero cache and a social.coop cache Both follow the diigo-cache pattern (webapp holds the credential, read-only and minted narrow at the source; the cache serves a public page). Zotero's versioned sync answers diigo's stale-head problem; Mastodon's doesn't announce edits/deletes, so its tree carries that as a design question. Co-Authored-By: Claude Opus 5.5 (1M context) --- capsite-checklist.md | 6 ++- social-coop-checklist.md | 107 +++++++++++++++++++++++++++++++++++++++ zotero-checklist.md | 104 +++++++++++++++++++++++++++++++++++++ 3 files changed, 216 insertions(+), 1 deletion(-) create mode 100644 social-coop-checklist.md create mode 100644 zotero-checklist.md diff --git a/capsite-checklist.md b/capsite-checklist.md index faa2695..27c5a30 100644 --- a/capsite-checklist.md +++ b/capsite-checklist.md @@ -107,7 +107,9 @@ narrative. does re-mint mean a new cache? Prefer re-wire — durable ids. - [ ] **7. bib sync #195**: repeat the pattern for the next read-access need (webapp mints for any origin; the cache's - item shape is diigo-specific). + item shape is diigo-specific): [the zotero goal tree](zotero-checklist.md). + The social.coop account is next after that: + [the social.coop goal tree](social-coop-checklist.md). - [ ] **webapp graduates from prototype** (optional; only if it earns it) - [ ] TODO(webapp): one canonical mint form — proxy-page (current) vs running webapp as the primary worker so its own port serves the @@ -171,6 +173,8 @@ to every goal in this tree. ## Pointers +- [the zotero goal tree](zotero-checklist.md) and + [the social.coop goal tree](social-coop-checklist.md) — the next caches - [the capsite notes](capsite-notes.md) — the original goal and the EmDash native-plugin wiring details - [docs/webkeys.md](docs/webkeys.md) — the Waterken URL-capability diff --git a/social-coop-checklist.md b/social-coop-checklist.md new file mode 100644 index 0000000..d9575e1 --- /dev/null +++ b/social-coop-checklist.md @@ -0,0 +1,107 @@ +# social.coop cache goal tree + +The same conventions as [the diigo goal tree](capsite-checklist.md): each +`- [ ]` is a goal, its indented children must pass first, working notes +sit under each goal. + +- [ ] **madmode's weekly roundup includes my social.coop activity** + ([#23](https://github.com/dckc/madmode-blog/issues/23), PESOS): + a cache of [@dckc@social.coop](https://social.coop/@dckc) — posts, + boosts, and (with discretion) bookmarks and favourites; the access + token stays inside webapp, so madmode holds no credential + + - [ ] **0. Decide the shape** (user's call before code): + - [ ] 0-i. New worker `masto-cache/` beside `diigo-cache/`, or + generalize diigo-cache? Recommendation: a new worker; + extract shared pieces once two caches work. + - [ ] 0-ii. Which feeds? Public posts and boosts need no + credential (`GET /api/v1/accounts/:id/statuses`, or the + ActivityPub outbox). Bookmarks, favourites, and + followers-only posts need a token. Bookmarks are private + by nature: publish them only behind an explicit opt-in + (what counts as opting in is open). + - [ ] 0-iii. Mastodon API or ActivityPub? The API is simpler + and covers bookmarks/favourites; ActivityPub works + against any server (portability if the account moves). + Recommendation: the API now, keyed by the post's + ActivityPub `uri` so a later move keeps ids stable. + + - [ ] **1. Credential discipline**: a token from social.coop → + Preferences → Development → New application, scopes + **read-only and narrow** (`read:statuses read:accounts + read:bookmarks read:favourites`, per 0-ii) — attenuation at + the source. It rides as `Authorization: Bearer`, which webapp's + mint form already blanks. Token → webapp DO storage only; ids + → password manager. Revoking the app on social.coop is the + kill switch. + + - [ ] **2. The cache** (`masto-cache/`, the 1c pattern from diigo): + - [ ] 2-i. Mint and wire against `https://social.coop/api/`; + `accounts/lookup?acct=dckc` gives the account id. + - [ ] 2-ii. Storage: one table per feed (statuses, bookmarks, + favourites), keyed by `uri`, storing the status JSON and + its `edited_at`. **Durable state** from day one: a + `SCHEMA_V1` step. + - [ ] 2-iii. Fill: page backward with `max_id` (40 per page) + following the `Link` header — bookmarks and favourites + paginate *only* by `Link` (their cursors aren't status + ids). Honor `X-RateLimit-Remaining`/`-Reset` (300 per + 5 minutes is Mastodon's default). + - [ ] 2-iv. Tests: `pnpm lint && pnpm test`, the real worker + against a fake Mastodon, including `Link` pagination and + a rate-limit response. + - [ ] 2-v. Complete: a filled cache matches the account archive + (Preferences → Import and export → Request your archive: + `outbox.json`, `likes.json`, `bookmarks.json`), the + analogue of diigo's `scripts/compare-export.ts`. + + - [ ] **3. Freshness**: new posts are cheap (`min_id` = newest + stored, page forward). Edits and deletions of older posts are + not announced to an API poller; options: re-fetch the last N + days on each refresh (the window a roundup cares about), or + follow the streaming API. A test: a post added, one edited, and + one deleted in the fake; the weekly window reflects all three. + A cron alarm in the cache DO drives it. + + - [ ] **4. Read surfaces**: `/public/dckc` with the date window and + `q` filter of diigo's 1d (the same query module, if 0-i shares + it), HTML and JSON; JSON shape pinned by a test. Serve only + `visibility: public` posts (and opted-in bookmarks) there — + followers-only and direct posts never reach a public page; + a test says so. + + - [ ] **5. madmode's weekly page includes it**: diigo's goal 4 + (`/bookmarks/2026-W41`) grows a section from this cache, or + becomes a combined `/weekly/2026-W41`. Same editorial question + as there: raw list, or a draft the user edits. + + - [ ] **6. Deployed**: after diigo's goal 5; rate-limit `/public/*`; + mint against prod; fill. + + - [ ] **7. Token revocation**: when social.coop rejects the token + (401), refreshes fail; the public page keeps serving what's + stored and says so. Re-mint is manual; same re-wire question + as diigo's goal 6. + +## How to run + +TBD with goal 2; it should mirror +[the diigo how-to-run](capsite-checklist.md#how-to-run-continuation-prompts). + +## Working rules + +[capsite AGENTS.md](AGENTS.md): dev servers are the user's job; durable +state, loud about breaking changes; secrets discipline; generated +config types. + +## Pointers + +- [Mastodon API: accounts](https://docs.joinmastodon.org/methods/accounts/), + [bookmarks](https://docs.joinmastodon.org/methods/bookmarks/), + [favourites](https://docs.joinmastodon.org/methods/favourites/), + [OAuth scopes](https://docs.joinmastodon.org/api/oauth-scopes/), + [rate limits](https://docs.joinmastodon.org/api/rate-limits/), + [pagination](https://docs.joinmastodon.org/api/guidelines/#pagination) +- [#23](https://github.com/dckc/madmode-blog/issues/23) — the PESOS goal +- [webapp/README.md](webapp/README.md) — the mint form and its secret + header list diff --git a/zotero-checklist.md b/zotero-checklist.md new file mode 100644 index 0000000..5ac13eb --- /dev/null +++ b/zotero-checklist.md @@ -0,0 +1,104 @@ +# zotero-cache goal tree + +The same conventions as [the diigo goal tree](capsite-checklist.md): each +`- [ ]` is a goal, its indented children must pass first, working notes +sit under each goal. This tree is goal 7 of that one, carried out. + +- [ ] **The awesome-ocap bibliography is generated from Zotero** + ([bib sync #195](https://github.com/dckc/madmode-blog/issues/195)): + madmode (and a GitHub action, later) read a cache of + [my Zotero library](https://www.zotero.org/connolly/library); the + Zotero API key stays inside webapp, so neither reader holds a + credential + + - [ ] **0. Decide the shape** (user's call before code): + - [ ] 0-i. New worker `zotero-cache/` beside `diigo-cache/`, or + generalize diigo-cache? Recommendation: a new worker, and + extract shared pieces (fill job, webkey page, page cache) + only once both work — two examples before an abstraction. + - [ ] 0-ii. Does it need a credential at all? The library is + public on zotero.org; the public API + (`/users//items`, no key) may be enough for the + bibliography. A key adds unpublished items, notes, and + attachment metadata. If public is enough, webapp still + earns its place as the rate-limited client, minted with + no secret header. + - [ ] 0-iii. Which collections make up awesome-ocap? (A Zotero + collection per README section is the natural mapping.) + + - [ ] **1. Credential discipline**: a **read-only** key minted at + zotero.org → Settings → Security → API keys ("allow library + access" only; no write, no notes unless 0-ii says so) — + attenuation at the source. Send it as `Authorization: Bearer`, + not `Zotero-API-Key`: webapp's mint form blanks `authorization` + as a secret header and would echo an unknown one. Key → webapp + DO storage only; ids → password manager. + + - [ ] **2. The cache** (`zotero-cache/`, the 1c pattern from diigo): + - [ ] 2-i. Mint and wire: `/_connect` → `/_mint` → `/_wire`, the + base URL `https://api.zotero.org/users//` + (`GET /keys/current` with the key returns the userID). + - [ ] 2-ii. Storage: items keyed by Zotero `key`, each with its + `version`; one library version per cache. **Durable state** + from day one: a `SCHEMA_V1` step, so later shapes migrate. + - [ ] 2-iii. Fill: `GET items?format=json&include=data,csljson` + paged by `start`/`limit` (100 max) to `Total-Results`; + honor `Backoff` and `Retry-After` (webapp's `msPerCall` + is the floor, not the whole story). + - [ ] 2-iv. Tests: `pnpm lint && pnpm test`, the real worker + against a fake Zotero (fake-webapp style), including a + `Backoff` response. + - [ ] 2-v. Complete: a filled cache matches a Zotero export + (File → Export Library → CSL JSON), the analogue of + diigo's `scripts/compare-export.ts`. + + - [ ] **3. Freshness is cheap here** (diigo's 1f, solved by the API): + Zotero versions every write. Refresh = `items?since=` (only what changed, `Last-Modified-Version` header + gives the new version) plus `deleted?since=` for + removals. A test: an item edited and one deleted in the fake + both show after a refresh. A cron alarm in the cache DO can + drive it. + + - [ ] **4. Read surfaces**: + - [ ] 4-i. Public page `/public/` with `?collection=`, as + HTML and JSON; JSON shape pinned by a test (the + contract with readers). + - [ ] 4-ii. CSL JSON and BibTeX out, from cached data (Zotero's + own `format=bibtex` is a fetch; the cache should serve + what it has). + + - [ ] **5. awesome-ocap README generated**: a script (or madmode + page) renders the collections to the README's markdown; diff + against the hand-maintained README, the way 1c-vi diffed the + diigo export. Prerequisite from #195: the current README's + entries imported into Zotero (BibTeX / CSL JSON import). + + - [ ] **6. Deployed**: after diigo's goal 5 (webapp already + deployed); rate-limit `/public/*`; mint against prod; fill. + + - [ ] **7. Stretch (#195)**: a GitHub action regenerates the README + from the public cache — public data, so no secret in the action. + +## How to run + +TBD with goal 2; it should mirror +[the diigo how-to-run](capsite-checklist.md#how-to-run-continuation-prompts). + +## Working rules + +[capsite AGENTS.md](AGENTS.md): dev servers are the user's job; durable +state, loud about breaking changes; secrets discipline; generated +config types. + +## Pointers + +- [bib sync #195](https://github.com/dckc/madmode-blog/issues/195) — the + goal, and the import half (README → Zotero) +- [Zotero Web API v3 basics](https://www.zotero.org/support/dev/web_api/v3/basics) + and [syncing](https://www.zotero.org/support/dev/web_api/v3/syncing) — + versions, `since=`, deletions, `Backoff` +- [awesome-ocap](https://github.com/dckc/awesome-ocap) — the README to + generate +- [webapp/README.md](webapp/README.md) — the mint form and its secret + header list