capsite: goal trees for a zotero cache and a social.coop cache
Both follow the diigo-cache pattern (webapp holds the credential, read-only and minted narrow at the source; the cache serves a public page). Zotero's versioned sync answers diigo's stale-head problem; Mastodon's doesn't announce edits/deletes, so its tree carries that as a design question. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
dckcandClaude Opus 5.5 committed
1 parent
279ddb4caa
commit
99ce5b6964
3 files changed
+216
-1
No files matched your search
@@ -107,7 +107,9 @@ narrative.
|
|||||||
does re-mint mean a new cache? Prefer re-wire — durable ids.
|
does re-mint mean a new cache? Prefer re-wire — durable ids.
|
||||||
- [ ] **7. bib sync #195**: repeat the pattern for the next
|
- [ ] **7. bib sync #195**: repeat the pattern for the next
|
||||||
read-access need (webapp mints for any origin; the cache's
|
read-access need (webapp mints for any origin; the cache's
|
||||||
item shape is diigo-specific).
|
item shape is diigo-specific): [the zotero goal tree](zotero-checklist.md).
|
||||||
|
The social.coop account is next after that:
|
||||||
|
[the social.coop goal tree](social-coop-checklist.md).
|
||||||
- [ ] **webapp graduates from prototype** (optional; only if it earns it)
|
- [ ] **webapp graduates from prototype** (optional; only if it earns it)
|
||||||
- [ ] TODO(webapp): one canonical mint form — proxy-page (current) vs
|
- [ ] TODO(webapp): one canonical mint form — proxy-page (current) vs
|
||||||
running webapp as the primary worker so its own port serves the
|
running webapp as the primary worker so its own port serves the
|
||||||
@@ -171,6 +173,8 @@ to every goal in this tree.
|
|||||||
|
|
||||||
## Pointers
|
## Pointers
|
||||||
|
|
||||||
|
- [the zotero goal tree](zotero-checklist.md) and
|
||||||
|
[the social.coop goal tree](social-coop-checklist.md) — the next caches
|
||||||
- [the capsite notes](capsite-notes.md) — the original goal and the EmDash
|
- [the capsite notes](capsite-notes.md) — the original goal and the EmDash
|
||||||
native-plugin wiring details
|
native-plugin wiring details
|
||||||
- [docs/webkeys.md](docs/webkeys.md) — the Waterken URL-capability
|
- [docs/webkeys.md](docs/webkeys.md) — the Waterken URL-capability
|
||||||
|
|||||||
@@ -0,0 +1,107 @@
|
|||||||
|
# social.coop cache goal tree
|
||||||
|
|
||||||
|
The same conventions as [the diigo goal tree](capsite-checklist.md): each
|
||||||
|
`- [ ]` is a goal, its indented children must pass first, working notes
|
||||||
|
sit under each goal.
|
||||||
|
|
||||||
|
- [ ] **madmode's weekly roundup includes my social.coop activity**
|
||||||
|
([#23](https://github.com/dckc/madmode-blog/issues/23), PESOS):
|
||||||
|
a cache of [@dckc@social.coop](https://social.coop/@dckc) — posts,
|
||||||
|
boosts, and (with discretion) bookmarks and favourites; the access
|
||||||
|
token stays inside webapp, so madmode holds no credential
|
||||||
|
|
||||||
|
- [ ] **0. Decide the shape** (user's call before code):
|
||||||
|
- [ ] 0-i. New worker `masto-cache/` beside `diigo-cache/`, or
|
||||||
|
generalize diigo-cache? Recommendation: a new worker;
|
||||||
|
extract shared pieces once two caches work.
|
||||||
|
- [ ] 0-ii. Which feeds? Public posts and boosts need no
|
||||||
|
credential (`GET /api/v1/accounts/:id/statuses`, or the
|
||||||
|
ActivityPub outbox). Bookmarks, favourites, and
|
||||||
|
followers-only posts need a token. Bookmarks are private
|
||||||
|
by nature: publish them only behind an explicit opt-in
|
||||||
|
(what counts as opting in is open).
|
||||||
|
- [ ] 0-iii. Mastodon API or ActivityPub? The API is simpler
|
||||||
|
and covers bookmarks/favourites; ActivityPub works
|
||||||
|
against any server (portability if the account moves).
|
||||||
|
Recommendation: the API now, keyed by the post's
|
||||||
|
ActivityPub `uri` so a later move keeps ids stable.
|
||||||
|
|
||||||
|
- [ ] **1. Credential discipline**: a token from social.coop →
|
||||||
|
Preferences → Development → New application, scopes
|
||||||
|
**read-only and narrow** (`read:statuses read:accounts
|
||||||
|
read:bookmarks read:favourites`, per 0-ii) — attenuation at
|
||||||
|
the source. It rides as `Authorization: Bearer`, which webapp's
|
||||||
|
mint form already blanks. Token → webapp DO storage only; ids
|
||||||
|
→ password manager. Revoking the app on social.coop is the
|
||||||
|
kill switch.
|
||||||
|
|
||||||
|
- [ ] **2. The cache** (`masto-cache/`, the 1c pattern from diigo):
|
||||||
|
- [ ] 2-i. Mint and wire against `https://social.coop/api/`;
|
||||||
|
`accounts/lookup?acct=dckc` gives the account id.
|
||||||
|
- [ ] 2-ii. Storage: one table per feed (statuses, bookmarks,
|
||||||
|
favourites), keyed by `uri`, storing the status JSON and
|
||||||
|
its `edited_at`. **Durable state** from day one: a
|
||||||
|
`SCHEMA_V1` step.
|
||||||
|
- [ ] 2-iii. Fill: page backward with `max_id` (40 per page)
|
||||||
|
following the `Link` header — bookmarks and favourites
|
||||||
|
paginate *only* by `Link` (their cursors aren't status
|
||||||
|
ids). Honor `X-RateLimit-Remaining`/`-Reset` (300 per
|
||||||
|
5 minutes is Mastodon's default).
|
||||||
|
- [ ] 2-iv. Tests: `pnpm lint && pnpm test`, the real worker
|
||||||
|
against a fake Mastodon, including `Link` pagination and
|
||||||
|
a rate-limit response.
|
||||||
|
- [ ] 2-v. Complete: a filled cache matches the account archive
|
||||||
|
(Preferences → Import and export → Request your archive:
|
||||||
|
`outbox.json`, `likes.json`, `bookmarks.json`), the
|
||||||
|
analogue of diigo's `scripts/compare-export.ts`.
|
||||||
|
|
||||||
|
- [ ] **3. Freshness**: new posts are cheap (`min_id` = newest
|
||||||
|
stored, page forward). Edits and deletions of older posts are
|
||||||
|
not announced to an API poller; options: re-fetch the last N
|
||||||
|
days on each refresh (the window a roundup cares about), or
|
||||||
|
follow the streaming API. A test: a post added, one edited, and
|
||||||
|
one deleted in the fake; the weekly window reflects all three.
|
||||||
|
A cron alarm in the cache DO drives it.
|
||||||
|
|
||||||
|
- [ ] **4. Read surfaces**: `/public/dckc` with the date window and
|
||||||
|
`q` filter of diigo's 1d (the same query module, if 0-i shares
|
||||||
|
it), HTML and JSON; JSON shape pinned by a test. Serve only
|
||||||
|
`visibility: public` posts (and opted-in bookmarks) there —
|
||||||
|
followers-only and direct posts never reach a public page;
|
||||||
|
a test says so.
|
||||||
|
|
||||||
|
- [ ] **5. madmode's weekly page includes it**: diigo's goal 4
|
||||||
|
(`/bookmarks/2026-W41`) grows a section from this cache, or
|
||||||
|
becomes a combined `/weekly/2026-W41`. Same editorial question
|
||||||
|
as there: raw list, or a draft the user edits.
|
||||||
|
|
||||||
|
- [ ] **6. Deployed**: after diigo's goal 5; rate-limit `/public/*`;
|
||||||
|
mint against prod; fill.
|
||||||
|
|
||||||
|
- [ ] **7. Token revocation**: when social.coop rejects the token
|
||||||
|
(401), refreshes fail; the public page keeps serving what's
|
||||||
|
stored and says so. Re-mint is manual; same re-wire question
|
||||||
|
as diigo's goal 6.
|
||||||
|
|
||||||
|
## How to run
|
||||||
|
|
||||||
|
TBD with goal 2; it should mirror
|
||||||
|
[the diigo how-to-run](capsite-checklist.md#how-to-run-continuation-prompts).
|
||||||
|
|
||||||
|
## Working rules
|
||||||
|
|
||||||
|
[capsite AGENTS.md](AGENTS.md): dev servers are the user's job; durable
|
||||||
|
state, loud about breaking changes; secrets discipline; generated
|
||||||
|
config types.
|
||||||
|
|
||||||
|
## Pointers
|
||||||
|
|
||||||
|
- [Mastodon API: accounts](https://docs.joinmastodon.org/methods/accounts/),
|
||||||
|
[bookmarks](https://docs.joinmastodon.org/methods/bookmarks/),
|
||||||
|
[favourites](https://docs.joinmastodon.org/methods/favourites/),
|
||||||
|
[OAuth scopes](https://docs.joinmastodon.org/api/oauth-scopes/),
|
||||||
|
[rate limits](https://docs.joinmastodon.org/api/rate-limits/),
|
||||||
|
[pagination](https://docs.joinmastodon.org/api/guidelines/#pagination)
|
||||||
|
- [#23](https://github.com/dckc/madmode-blog/issues/23) — the PESOS goal
|
||||||
|
- [webapp/README.md](webapp/README.md) — the mint form and its secret
|
||||||
|
header list
|
||||||
@@ -0,0 +1,104 @@
|
|||||||
|
# zotero-cache goal tree
|
||||||
|
|
||||||
|
The same conventions as [the diigo goal tree](capsite-checklist.md): each
|
||||||
|
`- [ ]` is a goal, its indented children must pass first, working notes
|
||||||
|
sit under each goal. This tree is goal 7 of that one, carried out.
|
||||||
|
|
||||||
|
- [ ] **The awesome-ocap bibliography is generated from Zotero**
|
||||||
|
([bib sync #195](https://github.com/dckc/madmode-blog/issues/195)):
|
||||||
|
madmode (and a GitHub action, later) read a cache of
|
||||||
|
[my Zotero library](https://www.zotero.org/connolly/library); the
|
||||||
|
Zotero API key stays inside webapp, so neither reader holds a
|
||||||
|
credential
|
||||||
|
|
||||||
|
- [ ] **0. Decide the shape** (user's call before code):
|
||||||
|
- [ ] 0-i. New worker `zotero-cache/` beside `diigo-cache/`, or
|
||||||
|
generalize diigo-cache? Recommendation: a new worker, and
|
||||||
|
extract shared pieces (fill job, webkey page, page cache)
|
||||||
|
only once both work — two examples before an abstraction.
|
||||||
|
- [ ] 0-ii. Does it need a credential at all? The library is
|
||||||
|
public on zotero.org; the public API
|
||||||
|
(`/users/<userID>/items`, no key) may be enough for the
|
||||||
|
bibliography. A key adds unpublished items, notes, and
|
||||||
|
attachment metadata. If public is enough, webapp still
|
||||||
|
earns its place as the rate-limited client, minted with
|
||||||
|
no secret header.
|
||||||
|
- [ ] 0-iii. Which collections make up awesome-ocap? (A Zotero
|
||||||
|
collection per README section is the natural mapping.)
|
||||||
|
|
||||||
|
- [ ] **1. Credential discipline**: a **read-only** key minted at
|
||||||
|
zotero.org → Settings → Security → API keys ("allow library
|
||||||
|
access" only; no write, no notes unless 0-ii says so) —
|
||||||
|
attenuation at the source. Send it as `Authorization: Bearer`,
|
||||||
|
not `Zotero-API-Key`: webapp's mint form blanks `authorization`
|
||||||
|
as a secret header and would echo an unknown one. Key → webapp
|
||||||
|
DO storage only; ids → password manager.
|
||||||
|
|
||||||
|
- [ ] **2. The cache** (`zotero-cache/`, the 1c pattern from diigo):
|
||||||
|
- [ ] 2-i. Mint and wire: `/_connect` → `/_mint` → `/_wire`, the
|
||||||
|
base URL `https://api.zotero.org/users/<userID>/`
|
||||||
|
(`GET /keys/current` with the key returns the userID).
|
||||||
|
- [ ] 2-ii. Storage: items keyed by Zotero `key`, each with its
|
||||||
|
`version`; one library version per cache. **Durable state**
|
||||||
|
from day one: a `SCHEMA_V1` step, so later shapes migrate.
|
||||||
|
- [ ] 2-iii. Fill: `GET items?format=json&include=data,csljson`
|
||||||
|
paged by `start`/`limit` (100 max) to `Total-Results`;
|
||||||
|
honor `Backoff` and `Retry-After` (webapp's `msPerCall`
|
||||||
|
is the floor, not the whole story).
|
||||||
|
- [ ] 2-iv. Tests: `pnpm lint && pnpm test`, the real worker
|
||||||
|
against a fake Zotero (fake-webapp style), including a
|
||||||
|
`Backoff` response.
|
||||||
|
- [ ] 2-v. Complete: a filled cache matches a Zotero export
|
||||||
|
(File → Export Library → CSL JSON), the analogue of
|
||||||
|
diigo's `scripts/compare-export.ts`.
|
||||||
|
|
||||||
|
- [ ] **3. Freshness is cheap here** (diigo's 1f, solved by the API):
|
||||||
|
Zotero versions every write. Refresh = `items?since=<library
|
||||||
|
version>` (only what changed, `Last-Modified-Version` header
|
||||||
|
gives the new version) plus `deleted?since=<version>` for
|
||||||
|
removals. A test: an item edited and one deleted in the fake
|
||||||
|
both show after a refresh. A cron alarm in the cache DO can
|
||||||
|
drive it.
|
||||||
|
|
||||||
|
- [ ] **4. Read surfaces**:
|
||||||
|
- [ ] 4-i. Public page `/public/<name>` with `?collection=`, as
|
||||||
|
HTML and JSON; JSON shape pinned by a test (the
|
||||||
|
contract with readers).
|
||||||
|
- [ ] 4-ii. CSL JSON and BibTeX out, from cached data (Zotero's
|
||||||
|
own `format=bibtex` is a fetch; the cache should serve
|
||||||
|
what it has).
|
||||||
|
|
||||||
|
- [ ] **5. awesome-ocap README generated**: a script (or madmode
|
||||||
|
page) renders the collections to the README's markdown; diff
|
||||||
|
against the hand-maintained README, the way 1c-vi diffed the
|
||||||
|
diigo export. Prerequisite from #195: the current README's
|
||||||
|
entries imported into Zotero (BibTeX / CSL JSON import).
|
||||||
|
|
||||||
|
- [ ] **6. Deployed**: after diigo's goal 5 (webapp already
|
||||||
|
deployed); rate-limit `/public/*`; mint against prod; fill.
|
||||||
|
|
||||||
|
- [ ] **7. Stretch (#195)**: a GitHub action regenerates the README
|
||||||
|
from the public cache — public data, so no secret in the action.
|
||||||
|
|
||||||
|
## How to run
|
||||||
|
|
||||||
|
TBD with goal 2; it should mirror
|
||||||
|
[the diigo how-to-run](capsite-checklist.md#how-to-run-continuation-prompts).
|
||||||
|
|
||||||
|
## Working rules
|
||||||
|
|
||||||
|
[capsite AGENTS.md](AGENTS.md): dev servers are the user's job; durable
|
||||||
|
state, loud about breaking changes; secrets discipline; generated
|
||||||
|
config types.
|
||||||
|
|
||||||
|
## Pointers
|
||||||
|
|
||||||
|
- [bib sync #195](https://github.com/dckc/madmode-blog/issues/195) — the
|
||||||
|
goal, and the import half (README → Zotero)
|
||||||
|
- [Zotero Web API v3 basics](https://www.zotero.org/support/dev/web_api/v3/basics)
|
||||||
|
and [syncing](https://www.zotero.org/support/dev/web_api/v3/syncing) —
|
||||||
|
versions, `since=`, deletions, `Backoff`
|
||||||
|
- [awesome-ocap](https://github.com/dckc/awesome-ocap) — the README to
|
||||||
|
generate
|
||||||
|
- [webapp/README.md](webapp/README.md) — the mint form and its secret
|
||||||
|
header list
|
||||||
Reference in new issue
Block a user