synthesize: anonymous braindump CSV → themed review doc via the co-op gateway
Stdlib-only pipeline for the pre-meeting braindump: CryptPad Form CSV in, review.md + synthesis.json out. Two model stages (tag atomic points with verbatim quotes, cluster into themes/novel/dissent) wrapped in deterministic Python that verifies every quote against the source and computes counts from item ids, so the model never writes the final document. Defaults to tinfoil/glm-5.3 with reasoning_effort=low: without it the model spent the whole 8000-token budget on hidden reasoning and returned nothing (probed 2026-10-19; findings in README). Refuses non-TEE models without a flag. Content-hash cache so prompt tuning reruns are free. Offline unittest suite with a fake gateway; smoke-tested live on 5 synthetic responses for ~$0.03. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
solo.istandClaude Fable 5.1 committed
1 parent
a8908c84cf
commit
05f4e88867
12 files changed
+2212
-15
No files matched your search
@@ -1,3 +1,131 @@
|
||||
# coopertron
|
||||
# braindump
|
||||
|
||||
Using AI to synthesize viewpoints to build consensus while highlighting novel ideas or dissent.
|
||||
Anonymous braindump form + LLM synthesis for Inference Cooperative meetings.
|
||||
Members answer a short form before a meeting (ideas, wants, needs, goals,
|
||||
dissent). This script turns the export into a review document the group reads
|
||||
together: where energy is concentrated, what is unusual, where people disagree.
|
||||
|
||||
It runs on the co-op's **own gateway**, on a **Tinfoil TEE model** by default,
|
||||
so anonymous responses are processed inside an enclave rather than handed to an
|
||||
outside vendor. Python standard library only.
|
||||
|
||||
## How it works
|
||||
|
||||
```
|
||||
CryptPad Form ──export CSV──▶ synthesize.py ──▶ out/review.md + out/synthesis.json
|
||||
│
|
||||
├─ tag model extracts atomic points per response,
|
||||
│ each with a verbatim quote that Python verifies
|
||||
├─ cluster model groups points into themes, flags novel
|
||||
│ points and dissent
|
||||
└─ render Python writes the document from the structured
|
||||
data: counts and quotes are exact, the model
|
||||
never writes the final text
|
||||
```
|
||||
|
||||
The model only ever returns structured data. Theme sizes are counted from item
|
||||
ids in Python, quotes are checked to be real substrings of the source, and the
|
||||
appendix reprints every response verbatim so anyone can audit the synthesis.
|
||||
|
||||
## Running
|
||||
|
||||
Requires Python 3.10+ and the [1Password CLI](https://developer.1password.com/docs/cli/)
|
||||
for the key (or set `INFERENCE_COOP_API_KEY` yourself).
|
||||
|
||||
```bash
|
||||
# 1. Dry run: parse the export, estimate tokens and cost, make no calls
|
||||
python3 synthesize.py responses.csv --dry-run
|
||||
|
||||
# 2. Real run
|
||||
op run --env-file .env.op -- python3 synthesize.py responses.csv --out out/
|
||||
```
|
||||
|
||||
Useful flags:
|
||||
|
||||
| Flag | Purpose |
|
||||
|---|---|
|
||||
| `--model` | default `tinfoil/glm-5.3`. Non-Tinfoil models are refused unless `--allow-non-tee` is passed |
|
||||
| `--limit N` | first N responses only (smoke tests) |
|
||||
| `--batch-size N` | responses per tagging call (default 15; keeps each call well inside the gateway's 180 s timeout) |
|
||||
| `--drop-col H` / `--choice-col H` | ignore a column, or force a column to be tallied as a checkbox question |
|
||||
| `--use-tools` | request structured output via a forced function call instead of a fenced JSON block |
|
||||
| `--reasoning-effort` | `reasoning_effort` sent with every call; default `low` (see below). `default` sends nothing |
|
||||
| `--no-thinking` | send `chat_template_kwargs.enable_thinking=false` (use with `tinfoil/deepseek-v4-1-flash`) |
|
||||
| `--max-tokens N` | completion budget per tagging call (default 8000; the cluster call gets 1.5×) |
|
||||
| `--save-raw DIR` | keep raw request/response JSON (fixtures, debugging) |
|
||||
| `--no-cache` | bypass `.cache/` (every call is cached by content hash, so prompt tuning reruns are free) |
|
||||
|
||||
Exit codes: `0` ok, `1` nothing extracted, `2` refused (non-TEE model without the flag).
|
||||
|
||||
### Reasoning models on the gateway (probed 2026-10-19)
|
||||
|
||||
The Tinfoil chat models are reasoning models. Left alone on a real extraction
|
||||
batch, `tinfoil/glm-5.3` spent its entire 8000-token completion budget on hidden
|
||||
reasoning and returned **no content** (`finish_reason: length`,
|
||||
`reasoning_tokens: 8000`). The script now stops immediately in that case
|
||||
instead of retrying and billing the same tokens again. Small probes of the
|
||||
available knobs:
|
||||
|
||||
| Model | Setting | Result |
|
||||
|---|---|---|
|
||||
| `tinfoil/glm-5.3` | default | thinks, then answers; reasoning scales with task size and can exhaust the budget |
|
||||
| `tinfoil/glm-5.3` | `reasoning_effort: low` | **a few reasoning tokens, clean JSON** (the default here) |
|
||||
| `tinfoil/glm-5.3` | `reasoning_effort: none` or `chat_template_kwargs.enable_thinking=false` | reasoning tokens drop to 0 but the thinking leaks into the visible content; avoid |
|
||||
| `tinfoil/deepseek-v4-1-flash` | `--no-thinking` | clean JSON, no reasoning, cheaper; good fallback |
|
||||
| `tinfoil/gpt-oss-120b` | default or `reasoning_effort: low` | clean JSON; does not report reasoning tokens |
|
||||
|
||||
The gateway forwards these parameters unchanged, so top-level
|
||||
`reasoning_effort` and `chat_template_kwargs` both reach the model.
|
||||
|
||||
## Input format
|
||||
|
||||
A CSV whose header row is the questions and each later row is one response,
|
||||
which is what CryptPad Form's **Responses → EXPORT → CSV** produces. Columns
|
||||
named like `Time`, `Timestamp`, `ID` are ignored; fully empty columns are
|
||||
ignored. A column whose cells are short comma- or semicolon-separated tokens is
|
||||
treated as a checkbox question and tallied in Python. Check the `Q:` lines the
|
||||
script prints to confirm it read the header the way you expect.
|
||||
|
||||
`tests/fixtures/sample.csv` is a synthetic 12-response export in that shape.
|
||||
|
||||
## Output
|
||||
|
||||
- `out/review.md`: headline numbers, priorities tally, themes ranked by how
|
||||
many distinct people raised them (with for/against counts and verbatim
|
||||
quotes), standout single-person ideas, notable dissent, open questions, and
|
||||
an appendix with every response.
|
||||
- `out/synthesis.json`: the structured intermediate (items, clusters, token
|
||||
usage) for auditing or re-rendering.
|
||||
|
||||
`tests/fixtures/sample.review.md` shows the shape, rendered offline from a fake
|
||||
model so the themes there are placeholders.
|
||||
|
||||
## Privacy and cost
|
||||
|
||||
- Responses are sent to `gateway.inference.coop`. With the default model they
|
||||
are processed inside a Tinfoil TEE enclave; the gateway sees metering
|
||||
metadata (tokens, model, key) and, per the co-op's docs, does not store prompt
|
||||
content. Other providers on the gateway are policy-based privacy, which is
|
||||
why the script refuses them without an explicit flag.
|
||||
- The API key is read from the environment only. Use `op run` so it never
|
||||
lands in a file or shell history. The script never prints request headers.
|
||||
- Cost: a 50-response, 8-question survey is on the order of 60k input tokens
|
||||
and a few k output through GLM-5.3, well under $1 of a member allowance.
|
||||
`--dry-run` prints the estimate; the real run prints the actual usage.
|
||||
|
||||
## Tests
|
||||
|
||||
```bash
|
||||
python3 -m unittest -v
|
||||
```
|
||||
|
||||
Offline. A fake gateway returns canned JSON so the tests cover CSV parsing,
|
||||
quote verification, JSON extraction and retry, caching, and rendering.
|
||||
|
||||
## Status
|
||||
|
||||
Built for the first-week-of-November 2026 members' meeting. The form itself
|
||||
lives on CryptPad for now; a small co-op-hosted app (anonymous form + a
|
||||
synthesize button, same pipeline) is a possible next step if the output is
|
||||
useful. Discussion and ideas: open an issue here or on
|
||||
[co-op/support](https://git.inference.coop/co-op/support/issues).
|
||||
Reference in new issue
Block a user