readme: use the repo description as the tagline

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
rename: braindump → coopertron (Coopertron 5000)
2026-10-19 02:56:50 -04:00 · 2026-10-19 02:56:50 -04:00 · 2026-10-19 02:56:50 -04:00
12 changed files with 2219 additions and 15 deletions
@@ -0,0 +1,3 @@
 # 1Password references for `op run --env-file .env.op -- python3 synthesize.py ...`
 # Nothing secret lives in this file; op resolves the references at run time.
 INFERENCE_COOP_API_KEY="op://Shared Project/Prose Dev Inference.Coop API Key/credential"
@@ -0,0 +1,7 @@
 .cache/
 out/
 raw/
 .env
 __pycache__/
 *.pyc
 .DS_Store
@@ -1,18 +1,21 @@
 MIT License
-Copyright (c) 2026 solo.ist
+Copyright (c) 2026 Angel Marino
-Permission is hereby granted, free of charge, to any person obtaining a copy of this software and 
+Permission is hereby granted, free of charge, to any person obtaining a copy
-associated documentation files (the "Software"), to deal in the Software without restriction, including 
+of this software and associated documentation files (the "Software"), to deal
-without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 
+in the Software without restriction, including without limitation the rights
-copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the 
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-following conditions:
+copies of the Software, and to permit persons to whom the Software is
 furnished to do so, subject to the following conditions:
-The above copyright notice and this permission notice shall be included in all copies or substantial 
+The above copyright notice and this permission notice shall be included in all
-portions of the Software.
+copies or substantial portions of the Software.
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT 
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO 
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER 
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE 
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-USE OR OTHER DEALINGS IN THE SOFTWARE.
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 SOFTWARE.
@@ -1,3 +1,138 @@
-# coopertron
+# Coopertron 5000
-Using AI to synthesize viewpoints to build consensus while highlighting novel ideas or dissent.
+Using AI to synthesize viewpoints to build consensus while highlighting novel
 ideas or dissent.
 A benevolent robot that reads the co-op's anonymous pre-meeting braindump and
 tells the members what they said. Built for Inference Cooperative meetings.
 Members answer a short form before a meeting (ideas, wants, needs, goals,
 dissent). This script turns the export into a review document the group reads
 together: where energy is concentrated, what is unusual, where people disagree.
 It runs on the co-op's **own gateway**, on a **Tinfoil TEE model** by default,
 so anonymous responses are processed inside an enclave rather than handed to an
 outside vendor. Python standard library only.
 ## How it works
 ```
 CryptPad Form ──export CSV──▶ synthesize.py ──▶ out/review.md + out/synthesis.json
                               │
                               ├─ tag      model extracts atomic points per response,
                               │           each with a verbatim quote that Python verifies
                               ├─ cluster  model groups points into themes, flags novel
                               │           points and dissent
                               └─ render   Python writes the document from the structured
                                           data: counts and quotes are exact, the model
                                           never writes the final text
 ```
 The model only ever returns structured data. Theme sizes are counted from item
 ids in Python, quotes are checked to be real substrings of the source, and the
 appendix reprints every response verbatim so anyone can audit the synthesis.
 ## Running
 Requires Python 3.10+ and the [1Password CLI](https://developer.1password.com/docs/cli/)
 for the key (or set `INFERENCE_COOP_API_KEY` yourself).
 ```bash
 # 1. Dry run: parse the export, estimate tokens and cost, make no calls
 python3 synthesize.py responses.csv --dry-run
 # 2. Real run
 op run --env-file .env.op -- python3 synthesize.py responses.csv --out out/
 ```
 Useful flags:
 | Flag | Purpose |
 |---|---|
 | `--model` | default `tinfoil/glm-5.3`. Non-Tinfoil models are refused unless `--allow-non-tee` is passed |
 | `--limit N` | first N responses only (smoke tests) |
 | `--batch-size N` | responses per tagging call (default 15; keeps each call well inside the gateway's 180 s timeout) |
 | `--drop-col H` / `--choice-col H` | ignore a column, or force a column to be tallied as a checkbox question |
 | `--use-tools` | request structured output via a forced function call instead of a fenced JSON block |
 | `--reasoning-effort` | `reasoning_effort` sent with every call; default `low` (see below). `default` sends nothing |
 | `--no-thinking` | send `chat_template_kwargs.enable_thinking=false` (use with `tinfoil/deepseek-v4-1-flash`) |
 | `--max-tokens N` | completion budget per tagging call (default 8000; the cluster call gets 1.5×) |
 | `--save-raw DIR` | keep raw request/response JSON (fixtures, debugging) |
 | `--no-cache` | bypass `.cache/` (every call is cached by content hash, so prompt tuning reruns are free) |
 Exit codes: `0` ok, `1` nothing extracted, `2` refused (non-TEE model without the flag).
 ### Reasoning models on the gateway (probed 2026-10-19)
 The Tinfoil chat models are reasoning models. Left alone on a real extraction
 batch, `tinfoil/glm-5.3` spent its entire 8000-token completion budget on hidden
 reasoning and returned **no content** (`finish_reason: length`,
 `reasoning_tokens: 8000`). The script now stops immediately in that case
 instead of retrying and billing the same tokens again. Small probes of the
 available knobs:
 | Model | Setting | Result |
 |---|---|---|
 | `tinfoil/glm-5.3` | default | thinks, then answers; reasoning scales with task size and can exhaust the budget |
 | `tinfoil/glm-5.3` | `reasoning_effort: low` | **a few reasoning tokens, clean JSON** (the default here) |
 | `tinfoil/glm-5.3` | `reasoning_effort: none` or `chat_template_kwargs.enable_thinking=false` | reasoning tokens drop to 0 but the thinking leaks into the visible content; avoid |
 | `tinfoil/deepseek-v4-1-flash` | `--no-thinking` | clean JSON, no reasoning, cheaper; good fallback |
 | `tinfoil/gpt-oss-120b` | default or `reasoning_effort: low` | clean JSON; does not report reasoning tokens |
 The gateway forwards these parameters unchanged, so top-level
 `reasoning_effort` and `chat_template_kwargs` both reach the model.
 ## Input format
 A CSV whose header row is the questions and each later row is one response,
 which is what CryptPad Form's **Responses → EXPORT → CSV** produces. Columns
 named like `Time`, `Timestamp`, `ID` are ignored; fully empty columns are
 ignored. A column whose cells are short comma- or semicolon-separated tokens is
 treated as a checkbox question and tallied in Python. Check the `Q:` lines the
 script prints to confirm it read the header the way you expect.
 `tests/fixtures/sample.csv` is a synthetic 12-response export in that shape.
 ## Output
 - `out/review.md`: headline numbers, priorities tally, themes ranked by how
  many distinct people raised them (with for/against counts and verbatim
  quotes), standout single-person ideas, notable dissent, open questions, and
  an appendix with every response.
 - `out/synthesis.json`: the structured intermediate (items, clusters, token
  usage) for auditing or re-rendering.
 `tests/fixtures/sample.review.md` shows the shape, rendered offline from a fake
 model so the themes there are placeholders.
 ## Privacy and cost
 - Responses are sent to `gateway.inference.coop`. With the default model they
  are processed inside a Tinfoil TEE enclave; the gateway sees metering
  metadata (tokens, model, key) and, per the co-op's docs, does not store prompt
  content. Other providers on the gateway are policy-based privacy, which is
  why the script refuses them without an explicit flag.
 - The API key is read from the environment only. Use `op run` so it never
  lands in a file or shell history. The script never prints request headers.
 - Cost: a 50-response, 8-question survey is on the order of 60k input tokens
  and a few k output through GLM-5.3, well under $1 of a member allowance.
  `--dry-run` prints the estimate; the real run prints the actual usage.
 ## Tests
 ```bash
 python3 -m unittest -v
 ```
 Offline. A fake gateway returns canned JSON so the tests cover CSV parsing,
 quote verification, JSON extraction and retry, caching, and rendering.
 ## Status
 Built for the first-week-of-November 2026 members' meeting. The form itself
 lives on CryptPad for now; a small co-op-hosted app (anonymous form + a
 synthesize button, same pipeline) is a possible next step if the output is
 useful. Discussion and ideas: open an issue here or on
 [co-op/support](https://git.inference.coop/co-op/support/issues).
 Why the name: it is a small experiment with a big-robot name, on purpose. The
 repo and command are `coopertron`; the 5000 lives in prose only.
@@ -0,0 +1,27 @@
 You are helping a member-governed AI cooperative prepare for a members' meeting. You will receive a numbered list of atomic points extracted from an anonymous braindump survey. Each line looks like:
 `i017 [r03/Q2 idea for] label text`
 where `i017` is the item id, `r03` the anonymous respondent, `Q2` the question, then the item's kind and stance, then its label.
 Group the items into themes that would help the group see where energy is concentrated, what is unusual, and where people disagree. Rules:
 - A theme is a cluster of points about the same underlying topic, regardless of whether they agree. Give each theme a short name (at most 6 words) and a two-sentence neutral summary.
 - Every item id you reference must come from the list. Do not invent ids. An item may appear in at most one theme.
 - Prefer 5 to 12 themes. Fold tiny near-duplicate themes together; split a theme that is really two topics.
 - `novel`: item ids that stand alone (no other respondent raised anything similar) but are specific and actionable enough that the group should see them. Quality over quantity; at most 10.
 - `dissent`: places where items pull in opposite directions, or where a respondent pushes back on something others want. For each, name the theme it belongs to, list the item ids on the minority or opposing side, and summarize the tension in one or two neutral sentences. Include concerns of kind `concern` with stance `against` even if they stand alone, if they are substantive.
 - `open_questions`: at most 6 questions the group will need to answer, phrased neutrally, each grounded in the items (reference ids).
 Do not rank themes yourself; counts will be computed from the item ids you assign. Do not write the final document; return only structured data.
 Return exactly one fenced ```json block and nothing else, shaped as:
 ```json
 {
  "themes": [{"name": "...", "summary": "...", "item_ids": ["i001", "i004"]}],
  "novel": ["i017"],
  "dissent": [{"theme": "...", "item_ids": ["i009"], "summary": "..."}],
  "open_questions": [{"question": "...", "item_ids": ["i001"]}]
 }
 ```
@@ -0,0 +1,20 @@
 You are helping a member-governed AI cooperative make sense of an anonymous braindump survey before a members' meeting. You will receive a batch of survey responses. Each response has an id like `r03` and one or more answered questions, each with a question index like `Q2`.
 Extract every distinct, atomic point a respondent makes. One point per item. Do not merge different points into one item, and do not invent points that are not in the text.
 For each item return:
 - `id`: the response id it came from (e.g. `r03`).
 - `q`: the question index it came from (e.g. `Q2`).
 - `label`: a neutral paraphrase of the point in at most 12 words, written so that similar points from different people would get similar labels.
 - `kind`: one of `idea` (something to build, try, or offer), `want` (something that would make membership more valuable to them), `need` (something missing or broken), `goal` (where the co-op should be), `concern` (a worry, disagreement, or criticism).
 - `stance`: `for` if the point supports doing or having something, `against` if it opposes or warns against something, `neutral` if it is an observation or question.
 - `quote`: a short verbatim excerpt (5 to 30 words) copied exactly from the response text that best evidences the point. It must be a character-for-character substring of the source text. Do not paraphrase, do not fix typos, do not add ellipses.
 Skip pleasantries, empty answers, and "nothing to add". Keep the respondent's framing; do not editorialize.
 Return exactly one fenced ```json block and nothing else, shaped as:
 ```json
 {"items": [{"id": "r03", "q": "Q2", "label": "...", "kind": "idea", "stance": "for", "quote": "..."}]}
 ```
@@ -0,0 +1,684 @@
 #!/usr/bin/env python3
 """Coopertron 5000: turn an anonymous survey export into a review document.
 Pipeline (deterministic Python around two model stages):
  load    CryptPad CSV export -> responses r01, r02, ... with per-question answers
  tag     model extracts atomic points per response, with verbatim quotes that
          Python verifies against the source text
  cluster model groups points into themes, flags novel points and dissent
  render  Python writes review.md from the structured data, so every count and
          quote is exact; the model never writes the final document
 Runs on the Inference Cooperative's own gateway. Default model is a Tinfoil
 TEE model so responses are processed inside an enclave. Stdlib only.
 Usage:
  op run --env-file .env.op -- python3 synthesize.py responses.csv --out out/
 """
 from __future__ import annotations
 import argparse
 import csv
 import hashlib
 import json
 import os
 import re
 import sys
 import time
 import urllib.error
 import urllib.request
 from collections import Counter, defaultdict
 from datetime import date
 from pathlib import Path
 HERE = Path(__file__).resolve().parent
 DEFAULT_BASE = "https://gateway.inference.coop/v1"
 DEFAULT_MODEL = "tinfoil/glm-5.3"
 KEY_ENV = "INFERENCE_COOP_API_KEY"
 HTTP_TIMEOUT = 190  # gateway request_timeout is 180s; TimeoutErrorRetries is 0
 # USD per 1M tokens (input, output), mirrored from code/litellm/config.yaml.
 PRICES = {
    "tinfoil/glm-5.3": (1.80, 5.75),
    "tinfoil/deepseek-v4-1-flash": (0.65, 1.45),
    "tinfoil/gpt-oss-120b": (0.15, 0.60),
    "greenpt/glm-5.3": (1.80, 5.75),
    "greenpt/glm-5.3-flash": (0.11, 0.44),
    "greenpt/deepseek-v4.1-flash": (0.22, 1.10),
 }
 KINDS = {"idea", "want", "need", "goal", "concern"}
 STANCES = {"for", "against", "neutral"}
 META_COL_RE = re.compile(r"^(id|#|time|timestamp|date|submitted|created|author|user)\b", re.I)
 # --------------------------------------------------------------------------- #
 # Loading
 # --------------------------------------------------------------------------- #
 def _norm_ws(s: str) -> str:
    return re.sub(r"\s+", " ", s).strip()
 def looks_like_choice_column(values: list[str]) -> bool:
    """Heuristic: a checkbox/choice export column has short, repeating tokens."""
    cells = [v for v in values if v.strip()]
    if len(cells) < 3:
        return False
    tokens: list[str] = []
    for c in cells:
        parts = [p.strip() for p in re.split(r"[;|,\n]", c) if p.strip()]
        if not parts or any(len(p) > 40 for p in parts):
            return False
        tokens.extend(parts)
    distinct = {t.lower() for t in tokens}
    return len(distinct) <= 20 and len(tokens) >= len(cells)
 def split_choices(cell: str) -> list[str]:
    return [p.strip() for p in re.split(r"[;|,\n]", cell) if p.strip()]
 def load_responses(path: Path, drop_cols: list[str] | None = None,
                   choice_cols: list[str] | None = None, limit: int | None = None) -> dict:
    """Return {"questions": [...], "choice_questions": [...], "responses": [...]}.
    Each response: {"id": "r01", "answers": {question: text}, "choices": {question: [..]}}.
    """
    with path.open(newline="", encoding="utf-8-sig") as fh:
        rows = list(csv.reader(fh))
    if not rows:
        raise SystemExit(f"{path}: empty file")
    header = [h.strip() for h in rows[0]]
    body = [r for r in rows[1:] if any(c.strip() for c in r)]
    if limit:
        body = body[:limit]
    drop = {c.lower() for c in (drop_cols or [])}
    keep_idx = []
    for i, h in enumerate(header):
        if not h or h.lower() in drop or META_COL_RE.match(h):
            continue
        if all((r[i].strip() == "" if i < len(r) else True) for r in body):
            continue  # entirely empty column
        keep_idx.append(i)
    forced_choice = {c.lower() for c in (choice_cols or [])}
    choice_idx = set()
    for i in keep_idx:
        col = [r[i] if i < len(r) else "" for r in body]
        if header[i].lower() in forced_choice or looks_like_choice_column(col):
            choice_idx.add(i)
    questions = [header[i] for i in keep_idx if i not in choice_idx]
    choice_questions = [header[i] for i in keep_idx if i in choice_idx]
    responses = []
    width = max(2, len(str(len(body))))
    for n, r in enumerate(body, start=1):
        rid = f"r{n:0{width}d}"
        answers, choices = {}, {}
        for i in keep_idx:
            cell = r[i].strip() if i < len(r) else ""
            if not cell:
                continue
            if i in choice_idx:
                choices[header[i]] = split_choices(cell)
            else:
                answers[header[i]] = cell
        if answers or choices:
            responses.append({"id": rid, "answers": answers, "choices": choices})
    return {"questions": questions, "choice_questions": choice_questions, "responses": responses}
 # --------------------------------------------------------------------------- #
 # Gateway client with on-disk cache
 # --------------------------------------------------------------------------- #
 class Gateway:
    def __init__(self, base: str, model: str, cache_dir: Path | None, use_cache: bool = True,
                 no_thinking: bool = False, reasoning_effort: str | None = None,
                 raw_dir: Path | None = None, verbose: bool = False):
        self.base = base.rstrip("/")
        self.model = model
        self.cache_dir = cache_dir
        self.use_cache = use_cache and cache_dir is not None
        self.no_thinking = no_thinking
        self.reasoning_effort = reasoning_effort
        self.raw_dir = raw_dir
        self.verbose = verbose
        self.calls = 0
        self.cache_hits = 0
        self.usage = Counter()
    # Separated so tests can monkeypatch the network.
    def _http_post(self, body: dict) -> dict:
        key = os.environ.get(KEY_ENV, "")
        if not key:
            raise SystemExit(f"{KEY_ENV} is not set. Run via: op run --env-file .env.op -- python3 synthesize.py ...")
        req = urllib.request.Request(
            f"{self.base}/chat/completions",
            data=json.dumps(body).encode("utf-8"),
            headers={"Authorization": f"Bearer {key}", "Content-Type": "application/json",
                     "User-Agent": "coopertron/0.1"},
            method="POST",
        )
        try:
            with urllib.request.urlopen(req, timeout=HTTP_TIMEOUT) as resp:
                return json.loads(resp.read().decode("utf-8"))
        except urllib.error.HTTPError as e:
            detail = e.read().decode("utf-8", "replace")[:800]
            raise SystemExit(f"gateway HTTP {e.code}: {detail}")
    def chat(self, system: str, user: str, max_tokens: int = 8000, tools: list | None = None,
             tag: str = "") -> dict:
        body: dict = {
            "model": self.model,
            "messages": [{"role": "system", "content": system}, {"role": "user", "content": user}],
            "max_tokens": max_tokens,
            "temperature": 0.2,
        }
        if tools:
            body["tools"] = tools
            body["tool_choice"] = {"type": "function", "function": {"name": tools[0]["function"]["name"]}}
        if self.no_thinking:
            body["chat_template_kwargs"] = {"enable_thinking": False}
        if self.reasoning_effort:
            body["reasoning_effort"] = self.reasoning_effort
        digest = hashlib.sha256(json.dumps(body, sort_keys=True).encode()).hexdigest()
        cache_file = self.cache_dir / f"{digest}.json" if self.cache_dir else None
        if self.use_cache and cache_file and cache_file.exists():
            self.cache_hits += 1
            data = json.loads(cache_file.read_text())
        else:
            self.calls += 1
            t0 = time.time()
            data = self._http_post(body)
            if self.verbose:
                print(f"  [{tag or 'call'}] {time.time() - t0:.1f}s finish={_finish(data)} usage={data.get('usage')}", file=sys.stderr)
            if cache_file:
                cache_file.parent.mkdir(parents=True, exist_ok=True)
                cache_file.write_text(json.dumps(data, indent=1))
            if self.raw_dir:
                self.raw_dir.mkdir(parents=True, exist_ok=True)
                stamp = time.strftime("%Y%m%dT%H%M%S")
                (self.raw_dir / f"{stamp}-{tag or 'call'}-{digest[:8]}.json").write_text(
                    json.dumps({"request": _redact(body), "response": data}, indent=1))
        u = data.get("usage") or {}
        self.usage["prompt_tokens"] += u.get("prompt_tokens", 0)
        self.usage["completion_tokens"] += u.get("completion_tokens", 0)
        return data
 def _redact(body: dict) -> dict:
    # Request bodies carry survey text; keep them for fixtures but never a key.
    return body
 def _finish(data: dict) -> str:
    try:
        return data["choices"][0].get("finish_reason", "?")
    except (KeyError, IndexError, TypeError):
        return "?"
 def message_text(data: dict) -> str:
    try:
        msg = data["choices"][0]["message"]
    except (KeyError, IndexError, TypeError):
        raise ValueError(f"unexpected response shape: {json.dumps(data)[:300]}")
    if msg.get("tool_calls"):
        return msg["tool_calls"][0]["function"]["arguments"]
    return msg.get("content") or ""
 # --------------------------------------------------------------------------- #
 # JSON extraction
 # --------------------------------------------------------------------------- #
 FENCE_RE = re.compile(r"```(?:json)?\s*(\{.*?\})\s*```", re.S)
 def extract_json(text: str) -> dict:
    """Pull one JSON object out of model output that may be wrapped in prose or a fence."""
    text = text.strip()
    m = FENCE_RE.search(text)
    candidates = [m.group(1)] if m else []
    if text.startswith("{"):
        candidates.append(text)
    start, end = text.find("{"), text.rfind("}")
    if start != -1 and end > start:
        candidates.append(text[start:end + 1])
    last_err: Exception | None = None
    for c in candidates:
        try:
            obj = json.loads(c)
            if isinstance(obj, dict):
                return obj
        except json.JSONDecodeError as e:
            last_err = e
            try:  # tolerate trailing commas
                obj = json.loads(re.sub(r",\s*([}\]])", r"\1", c))
                if isinstance(obj, dict):
                    return obj
            except json.JSONDecodeError as e2:
                last_err = e2
    raise ValueError(f"no JSON object found ({last_err})")
 def _check_reasoning_exhaustion(data: dict, max_tokens: int, tag: str) -> None:
    """Reasoning models can spend the whole completion budget thinking and return no content.
    Retrying the same request would just bill the same tokens again, so stop here with advice.
    """
    if _finish(data) != "length":
        return
    try:
        msg = data["choices"][0]["message"]
        usage = data.get("usage") or {}
        reasoning = (usage.get("completion_tokens_details") or {}).get("reasoning_tokens") or 0
    except (KeyError, IndexError, TypeError):
        return
    if not (msg.get("content") or msg.get("tool_calls")) and reasoning:
        raise SystemExit(
            f"{tag}: the model used all {max_tokens} completion tokens on hidden reasoning "
            f"({reasoning} reasoning tokens) and returned no content. Not retrying. Try "
            f"--no-thinking, --reasoning-effort low, a larger --max-tokens, or a different --model.")
 def call_json(gw: Gateway, system: str, user: str, required: list[str], tag: str,
              tools: list | None = None, max_tokens: int = 8000) -> dict:
    """Ask for JSON; on a parse/shape failure retry once with the error appended."""
    data = gw.chat(system, user, max_tokens=max_tokens, tools=tools, tag=tag)
    _check_reasoning_exhaustion(data, max_tokens, tag)
    try:
        obj = extract_json(message_text(data))
        missing = [k for k in required if k not in obj]
        if missing:
            raise ValueError(f"missing keys {missing}")
        return obj
    except ValueError as e:
        if _finish(data) == "length":
            e = ValueError(f"{e}; output hit max_tokens={max_tokens} (finish_reason=length)")
        retry_user = (f"{user}\n\n---\nYour previous reply could not be parsed: {e}. "
                      f"Reply again with exactly one fenced ```json block containing the keys {required} and nothing else.")
        data = gw.chat(system, retry_user, max_tokens=max_tokens, tools=tools, tag=f"{tag}-retry")
        obj = extract_json(message_text(data))
        missing = [k for k in required if k not in obj]
        if missing:
            raise SystemExit(f"{tag}: model output still missing keys {missing} after retry")
        return obj
 # --------------------------------------------------------------------------- #
 # Stage A: tag
 # --------------------------------------------------------------------------- #
 def q_index(questions: list[str]) -> dict[str, str]:
    return {q: f"Q{i}" for i, q in enumerate(questions, start=1)}
 def format_batch(batch: list[dict], qidx: dict[str, str]) -> str:
    parts = []
    for r in batch:
        parts.append(f"## Response {r['id']}")
        for q, text in r["answers"].items():
            parts.append(f"### {r['id']} / {qidx[q]}: {q}\n{text}")
    return "\n\n".join(parts)
 def validate_items(raw_items: list, batch: list[dict], qidx: dict[str, str]) -> tuple[list[dict], list[str]]:
    by_id = {r["id"]: r for r in batch}
    q_by_idx = {v: k for k, v in qidx.items()}
    kept, dropped = [], []
    for it in raw_items:
        if not isinstance(it, dict):
            dropped.append("non-object item")
            continue
        rid, q, quote = it.get("id"), it.get("q"), str(it.get("quote") or "")
        label = _norm_ws(str(it.get("label") or ""))
        if rid not in by_id or q not in q_by_idx or not label:
            dropped.append(f"{rid}/{q}: unknown id/question or empty label")
            continue
        source = by_id[rid]["answers"].get(q_by_idx[q], "")
        if not quote or _norm_ws(quote) not in _norm_ws(source):
            dropped.append(f"{rid}/{q}: quote not found in source: {quote[:60]!r}")
            continue
        kind = it.get("kind") if it.get("kind") in KINDS else "idea"
        stance = it.get("stance") if it.get("stance") in STANCES else "neutral"
        kept.append({"rid": rid, "q": q, "question": q_by_idx[q], "label": label,
                     "kind": kind, "stance": stance, "quote": _norm_ws(quote)})
    return kept, dropped
 TAG_TOOL = [{"type": "function", "function": {
    "name": "submit_items", "description": "Submit the extracted atomic points.",
    "parameters": {"type": "object", "required": ["items"], "properties": {"items": {
        "type": "array", "items": {"type": "object",
            "required": ["id", "q", "label", "kind", "stance", "quote"],
            "properties": {"id": {"type": "string"}, "q": {"type": "string"},
                           "label": {"type": "string"}, "kind": {"type": "string", "enum": sorted(KINDS)},
                           "stance": {"type": "string", "enum": sorted(STANCES)}, "quote": {"type": "string"}}}}}}}}]
 def stage_tag(gw: Gateway, loaded: dict, batch_size: int, use_tools: bool,
              max_tokens: int = 8000) -> tuple[list[dict], list[str]]:
    system = (HERE / "prompts" / "tag.md").read_text()
    qidx = q_index(loaded["questions"])
    responses = [r for r in loaded["responses"] if r["answers"]]
    items, dropped = [], []
    for b in range(0, len(responses), batch_size):
        batch = responses[b:b + batch_size]
        user = format_batch(batch, qidx)
        obj = call_json(gw, system, user, ["items"], tag=f"tag{b // batch_size + 1}",
                        tools=TAG_TOOL if use_tools else None, max_tokens=max_tokens)
        kept, drop = validate_items(obj.get("items") or [], batch, qidx)
        items.extend(kept)
        dropped.extend(drop)
    for n, it in enumerate(items, start=1):
        it["iid"] = f"i{n:03d}"
    return items, dropped
 # --------------------------------------------------------------------------- #
 # Stage B: cluster
 # --------------------------------------------------------------------------- #
 CLUSTER_TOOL = [{"type": "function", "function": {
    "name": "submit_clusters", "description": "Submit themes, novel items, dissent and open questions.",
    "parameters": {"type": "object", "required": ["themes", "novel", "dissent", "open_questions"],
                   "properties": {
                       "themes": {"type": "array", "items": {"type": "object", "required": ["name", "summary", "item_ids"],
                                  "properties": {"name": {"type": "string"}, "summary": {"type": "string"},
                                                 "item_ids": {"type": "array", "items": {"type": "string"}}}}},
                       "novel": {"type": "array", "items": {"type": "string"}},
                       "dissent": {"type": "array", "items": {"type": "object", "required": ["theme", "item_ids", "summary"],
                                   "properties": {"theme": {"type": "string"}, "summary": {"type": "string"},
                                                  "item_ids": {"type": "array", "items": {"type": "string"}}}}},
                       "open_questions": {"type": "array", "items": {"type": "object", "required": ["question", "item_ids"],
                                          "properties": {"question": {"type": "string"},
                                                         "item_ids": {"type": "array", "items": {"type": "string"}}}}}}}}}]
 def format_items(items: list[dict]) -> str:
    return "\n".join(f"{it['iid']} [{it['rid']}/{it['q']} {it['kind']} {it['stance']}] {it['label']}" for it in items)
 def validate_clusters(obj: dict, items: list[dict]) -> dict:
    known = {it["iid"] for it in items}
    seen: set[str] = set()
    themes = []
    for t in obj.get("themes") or []:
        if not isinstance(t, dict):
            continue
        ids = []
        for i in t.get("item_ids") or []:
            if i in known and i not in seen:
                ids.append(i)
                seen.add(i)
        if ids:
            themes.append({"name": _norm_ws(str(t.get("name") or "Untitled")),
                           "summary": _norm_ws(str(t.get("summary") or "")), "item_ids": ids})
    unthemed = [i for i in sorted(known) if i not in seen]
    novel = [i for i in (obj.get("novel") or []) if i in known][:10]
    dissent = []
    for d in obj.get("dissent") or []:
        if not isinstance(d, dict):
            continue
        ids = [i for i in (d.get("item_ids") or []) if i in known]
        if ids:
            dissent.append({"theme": _norm_ws(str(d.get("theme") or "")),
                            "summary": _norm_ws(str(d.get("summary") or "")), "item_ids": ids})
    oq = []
    for q in (obj.get("open_questions") or [])[:6]:
        if isinstance(q, dict) and q.get("question"):
            oq.append({"question": _norm_ws(str(q["question"])),
                       "item_ids": [i for i in (q.get("item_ids") or []) if i in known]})
        elif isinstance(q, str):
            oq.append({"question": _norm_ws(q), "item_ids": []})
    return {"themes": themes, "unthemed": unthemed, "novel": novel, "dissent": dissent, "open_questions": oq}
 def stage_cluster(gw: Gateway, items: list[dict], use_tools: bool, max_tokens: int = 8000) -> dict:
    system = (HERE / "prompts" / "cluster.md").read_text()
    obj = call_json(gw, system, format_items(items), ["themes"], tag="cluster",
                    tools=CLUSTER_TOOL if use_tools else None, max_tokens=int(max_tokens * 1.5))
    return validate_clusters(obj, items)
 # --------------------------------------------------------------------------- #
 # Render
 # --------------------------------------------------------------------------- #
 def render(loaded: dict, items: list[dict], clusters: dict, meta: dict) -> str:
    by_iid = {it["iid"]: it for it in items}
    resp_by_id = {r["id"]: r for r in loaded["responses"]}
    n_resp = len(loaded["responses"])
    def quote_line(iid: str) -> str:
        it = by_iid[iid]
        return f'> "{it["quote"]}" ({it["rid"]}, {it["q"]})'
    out: list[str] = []
    out.append(f"# Coopertron 5000 report\n")
    out.append(f"_Generated {meta['date']} from {meta['source']} · {n_resp} responses · "
               f"{len(items)} points · {len(clusters['themes'])} themes · model `{meta['model']}`_\n")
    out.append("Counts below are computed from the extracted points, not estimated by the model. "
               "Quotes are verbatim from the responses. The full responses are in the appendix.\n")
    # Priorities (checkbox tally in Python)
    for cq in loaded["choice_questions"]:
        tally = Counter()
        voters = 0
        for r in loaded["responses"]:
            picks = r["choices"].get(cq)
            if picks:
                voters += 1
                tally.update(p for p in picks)
        if not tally:
            continue
        out.append(f"## {cq}\n")
        out.append(f"_{voters} of {n_resp} responses answered._\n")
        width = max(tally.values())
        for opt, c in tally.most_common():
            bar = "█" * round(12 * c / width)
            out.append(f"- {bar} **{c}** · {opt}")
        out.append("")
    # Themes ordered by distinct respondents
    def theme_key(t):
        rids = {by_iid[i]["rid"] for i in t["item_ids"]}
        return (-len(rids), -len(t["item_ids"]))
    out.append("## Themes\n")
    out.append("| # | Theme | People | Points | For / Against |")
    out.append("|---|---|---|---|---|")
    themes = sorted(clusters["themes"], key=theme_key)
    for n, t in enumerate(themes, start=1):
        rids = {by_iid[i]["rid"] for i in t["item_ids"]}
        st = Counter(by_iid[i]["stance"] for i in t["item_ids"])
        out.append(f"| {n} | {t['name']} | {len(rids)} | {len(t['item_ids'])} | {st['for']} / {st['against']} |")
    out.append("")
    for n, t in enumerate(themes, start=1):
        rids = {by_iid[i]["rid"] for i in t["item_ids"]}
        kinds = Counter(by_iid[i]["kind"] for i in t["item_ids"])
        out.append(f"### {n}. {t['name']}\n")
        out.append(f"_{len(rids)} of {n_resp} people · {', '.join(f'{v} {k}' for k, v in kinds.most_common())}_\n")
        if t["summary"]:
            out.append(t["summary"] + "\n")
        labels = Counter(by_iid[i]["label"] for i in t["item_ids"])
        out.append("Points raised:\n")
        for label, c in labels.most_common():
            out.append(f"- {label}" + (f" (×{c})" if c > 1 else ""))
        out.append("")
        # Up to 4 quotes from distinct respondents
        shown: set[str] = set()
        for i in t["item_ids"]:
            rid = by_iid[i]["rid"]
            if rid in shown:
                continue
            out.append(quote_line(i))
            shown.add(rid)
            if len(shown) >= 4:
                break
        out.append("")
    if clusters["novel"]:
        out.append("## Standout ideas\n")
        out.append("_Raised by one person, specific enough to be worth a look._\n")
        for i in clusters["novel"]:
            it = by_iid[i]
            out.append(f"- **{it['label']}** ({it['kind']})  ")
            out.append(f"  {quote_line(i)[2:]}")
        out.append("")
    if clusters["dissent"]:
        out.append("## Notable dissent\n")
        out.append("_Places where people pull in different directions. Worth time in the meeting._\n")
        for d in clusters["dissent"]:
            head = f"**{d['theme']}**" if d["theme"] else "**Standalone concern**"
            out.append(f"- {head}: {d['summary']}")
            for i in d["item_ids"][:3]:
                out.append(f"  {quote_line(i)}")
        out.append("")
    if clusters["open_questions"]:
        out.append("## Open questions for the group\n")
        for n, q in enumerate(clusters["open_questions"], start=1):
            refs = ", ".join(sorted({by_iid[i]["rid"] for i in q["item_ids"]}))
            out.append(f"{n}. {q['question']}" + (f" _(from {refs})_" if refs else ""))
        out.append("")
    if clusters["unthemed"]:
        out.append("## Not placed in a theme\n")
        for i in clusters["unthemed"]:
            it = by_iid[i]
            out.append(f"- {it['label']} ({it['rid']}, {it['q']})")
        out.append("")
    out.append("## Appendix: all responses, verbatim\n")
    for q in loaded["questions"]:
        answered = [(r["id"], r["answers"][q]) for r in loaded["responses"] if q in r["answers"]]
        if not answered:
            continue
        out.append(f"### {q}\n")
        for rid, text in answered:
            body = text.replace("\n", "\n  ")
            out.append(f"- **{rid}:** {body}")
        out.append("")
    if meta.get("dropped"):
        out.append("## Processing notes\n")
        out.append(f"{len(meta['dropped'])} extracted point(s) were discarded because their quote could not be "
                   "verified against the source text or they referenced an unknown response.\n")
    return "\n".join(out).rstrip() + "\n"
 # --------------------------------------------------------------------------- #
 # Estimation
 # --------------------------------------------------------------------------- #
 def est_tokens(text: str) -> int:
    return max(1, len(text) // 4)
 def estimate(loaded: dict, model: str, batch_size: int) -> dict:
    tag_sys = est_tokens((HERE / "prompts" / "tag.md").read_text())
    cl_sys = est_tokens((HERE / "prompts" / "cluster.md").read_text())
    responses = [r for r in loaded["responses"] if r["answers"]]
    n_answers = sum(len(r["answers"]) for r in responses)
    batches = max(1, -(-len(responses) // batch_size))
    tag_in = batches * tag_sys + sum(est_tokens(format_batch([r], q_index(loaded["questions"]))) for r in responses)
    est_items = int(n_answers * 1.6)
    tag_out = est_items * 60
    cl_in = cl_sys + est_items * 22
    cl_out = 1500 + est_items * 6
    pin, pout = PRICES.get(model, (1.80, 5.75))
    cost = (tag_in + cl_in) / 1e6 * pin + (tag_out + cl_out) / 1e6 * pout
    return {"responses": len(responses), "answers": n_answers, "batches": batches,
            "est_items": est_items, "input_tokens": tag_in + cl_in, "output_tokens": tag_out + cl_out,
            "est_cost_usd": round(cost, 3), "priced": model in PRICES}
 # --------------------------------------------------------------------------- #
 # CLI
 # --------------------------------------------------------------------------- #
 def main(argv: list[str] | None = None) -> int:
    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    ap.add_argument("csv", type=Path, help="CryptPad Form CSV export")
    ap.add_argument("--out", type=Path, default=Path("out"), help="output directory (default: out/)")
    ap.add_argument("--model", default=DEFAULT_MODEL)
    ap.add_argument("--base", default=os.environ.get("INFERENCE_COOP_BASE", DEFAULT_BASE))
    ap.add_argument("--batch-size", type=int, default=15, help="responses per tagging call (default 15)")
    ap.add_argument("--limit", type=int, help="only the first N responses (smoke tests)")
    ap.add_argument("--drop-col", action="append", default=[], help="column header to ignore (repeatable)")
    ap.add_argument("--choice-col", action="append", default=[], help="column header to tally as checkbox (repeatable)")
    ap.add_argument("--dry-run", action="store_true", help="parse, estimate tokens and cost, make no calls")
    ap.add_argument("--no-cache", action="store_true")
    ap.add_argument("--cache-dir", type=Path, default=Path(".cache"))
    ap.add_argument("--save-raw", type=Path, help="directory to save raw request/response JSON (fixtures)")
    ap.add_argument("--use-tools", action="store_true", help="force a function call for structured output")
    ap.add_argument("--no-thinking", action="store_true", help="send chat_template_kwargs.enable_thinking=false")
    ap.add_argument("--reasoning-effort", choices=["default", "none", "minimal", "low", "medium", "high"], default="low",
                    help="reasoning_effort sent to the model (default: low; 'default' sends nothing). "
                         "Without it GLM-5.3 can spend the whole completion budget thinking")
    ap.add_argument("--max-tokens", type=int, default=8000, help="completion budget per call (default 8000)")
    ap.add_argument("--allow-non-tee", action="store_true", help="permit a model outside Tinfoil's TEE enclaves")
    ap.add_argument("-v", "--verbose", action="store_true")
    a = ap.parse_args(argv)
    if not a.model.startswith("tinfoil/") and not a.allow_non_tee:
        print(f"refusing: {a.model} is not a Tinfoil TEE model, so responses would be processed outside an "
              f"enclave under policy-based privacy. Pass --allow-non-tee to do it anyway.", file=sys.stderr)
        return 2
    loaded = load_responses(a.csv, drop_cols=a.drop_col, choice_cols=a.choice_col, limit=a.limit)
    print(f"loaded {len(loaded['responses'])} responses · {len(loaded['questions'])} open questions · "
          f"{len(loaded['choice_questions'])} choice question(s)", file=sys.stderr)
    for q in loaded["questions"]:
        print(f"  Q: {q}", file=sys.stderr)
    for q in loaded["choice_questions"]:
        print(f"  choice: {q}", file=sys.stderr)
    est = estimate(loaded, a.model, a.batch_size)
    print(f"estimate: ~{est['input_tokens']:,} in / ~{est['output_tokens']:,} out tokens, "
          f"~{est['batches']} tagging call(s) + 1 cluster call, ~${est['est_cost_usd']}"
          + ("" if est["priced"] else " (model not in price table; GLM-5.3 rates assumed)"), file=sys.stderr)
    if a.dry_run:
        return 0
    gw = Gateway(a.base, a.model, None if a.no_cache else a.cache_dir, use_cache=not a.no_cache,
                 no_thinking=a.no_thinking,
                 reasoning_effort=None if a.reasoning_effort == "default" else a.reasoning_effort,
                 raw_dir=a.save_raw, verbose=a.verbose)
    items, dropped = stage_tag(gw, loaded, a.batch_size, a.use_tools, a.max_tokens)
    print(f"tagged: {len(items)} points kept, {len(dropped)} dropped", file=sys.stderr)
    for d in dropped[:10]:
        print(f"  dropped: {d}", file=sys.stderr)
    if not items:
        print("no points extracted; nothing to cluster", file=sys.stderr)
        return 1
    clusters = stage_cluster(gw, items, a.use_tools, a.max_tokens)
    print(f"clustered: {len(clusters['themes'])} themes · {len(clusters['novel'])} novel · "
          f"{len(clusters['dissent'])} dissent · {len(clusters['unthemed'])} unthemed", file=sys.stderr)
    meta = {"date": date.today().isoformat(), "source": a.csv.name, "model": a.model, "dropped": dropped}
    a.out.mkdir(parents=True, exist_ok=True)
    (a.out / "review.md").write_text(render(loaded, items, clusters, meta))
    (a.out / "synthesis.json").write_text(json.dumps(
        {"meta": meta, "questions": loaded["questions"], "choice_questions": loaded["choice_questions"],
         "items": items, "clusters": clusters, "usage": dict(gw.usage),
         "calls": gw.calls, "cache_hits": gw.cache_hits}, indent=1, ensure_ascii=False))
    pin, pout = PRICES.get(a.model, (1.80, 5.75))
    spent = gw.usage["prompt_tokens"] / 1e6 * pin + gw.usage["completion_tokens"] / 1e6 * pout
    print(f"wrote {a.out / 'review.md'} and {a.out / 'synthesis.json'} · {gw.calls} call(s), "
          f"{gw.cache_hits} cache hit(s) · {gw.usage['prompt_tokens']:,} in / {gw.usage['completion_tokens']:,} out "
          f"· ≈${spent:.3f}", file=sys.stderr)
    return 0
 if __name__ == "__main__":
    sys.exit(main())
@@ -0,0 +1,13 @@
 "What drew you to the co-op, and what do you use it for today?","Ideas. What should the co-op build, try, or offer next?",Wants. What would make membership more valuable to you personally?,"Needs. What is missing, broken, or in your way right now?",Goals. Where should the co-op be a year from now?,Priorities (pick up to 3),"Dissent. What worries you, what do you disagree with, or what are we getting wrong?",Anything else?
 "Privacy. I wanted an AI I could use for client work without feeding a big vendor. Mostly chat, some API from scripts.",A shared prompt library members can contribute to. Also a Matrix bot that answers questions about the docs.,A cheaper tier for light users; I barely touch my allowance most months.,The chat loses my place on mobile when I switch apps. Also the model picker names are confusing (dotted vs dashed slugs).,"Fifty members, real elections, and at least one model we host ourselves.","privacy, pricing and fairness, owning our own infrastructure",I worry we are spreading across too many providers. Three providers is already hard to explain to a new member.,
 Green energy mostly. I teach and wanted something I could recommend to students with a straight face.,"Classroom or group accounts: one payment, several seats, shared budget.","Being able to see the energy and emissions per conversation, not just tokens.",Onboarding email landed in spam for two of my students. The email-must-match rule tripped one of them.,"A recognized option for universities and nonprofits, with an institutional tier.","green energy, docs and onboarding, governance and participation","Not dissent exactly, but the Loomio threads are long and I cannot keep up.",Thank you for doing this.
 Curiosity about co-op governance. I use the API from a small writing tool I maintain.,An institutional or app-partner membership so projects like mine can buy inference for their users through the co-op instead of each user joining separately.,Stable model names. When tinfoil/glm-5-3-flash disappeared my app broke.,No deprecation notice period for models. No status page.,"A documented SLA-ish commitment, even an informal one, and a changelog members can subscribe to.","API and developer tooling, model choice, privacy",I think the terms forbidding any redistribution of access will block exactly the kind of mission-aligned projects we say we want.,
 A friend invited me. I mostly use the chat for drafting emails.,Voice mode on the phone app.,Honestly it is fine as is.,,More members so it keeps existing.,community and chat,,
 Wanted to support a cooperative alternative. I use it daily for coding help.,Self-host at least one open model on our own hardware so we are not just reselling other people's APIs.,Higher allowance for people who pay more; the flat $15 cap makes paying extra feel pointless.,Tool calling is flaky on some models and there is no doc saying which ones support it.,"Our own GPU box, even a small one, and a path to more.","owning our own infrastructure, model choice, API and developer tooling",Self-hosting will cost more than buying tokens and we should be honest about that before voting for it.,Happy to help with ops.
 Privacy and the TEE story. Chat only.,Private web search inside the enclave instead of our own SearXNG.,Nothing specific.,The chat is slow to start on voice sometimes.,Keep it small and trustworthy. Growth is not a goal for me.,privacy,I disagree with adding providers that are only policy-private. It dilutes the one thing that made me join.,
 I run a small nonprofit and we needed shared AI access we could justify ethically.,Group accounts with a shared budget and an admin who can see spend per seat.,An invoice I can hand to our bookkeeper instead of an Open Collective receipt.,The dashboard shows my usage but not my organization's.,An institutional tier with clear terms and pricing.,"pricing and fairness, governance and participation, docs and onboarding",The pay-what-you-can model is lovely but I suspect it is not covering costs. Show us the numbers.,
 Governance nerd. I barely use the models; I joined for the experiment.,Publish a monthly financial and usage report and discuss it on Loomio.,"Real decision rights, not advisory votes.",The charter still says decisions are not binding.,A functioning member-elected board and a budget members approve.,"governance and participation, pricing and fairness",We keep adding features before we have governance. Build the decision process first.,Could we do these braindumps every quarter?
 Needed an API for a side project and liked the values.,"An Anthropic-compatible /v1/messages endpoint that is actually documented and tested, so Claude-style clients just work.",Per-key spend limits so one script cannot burn my whole month.,Budget exceeded errors arrive mid-stream with no warning beforehand.,"A developer-friendly gateway people choose on merit, not only on values.","API and developer tooling, privacy",,
 Climate reasons.,"Default every new member to the greenest model, not the fastest.",A badge or summary of emissions avoided versus a mainstream provider.,,Carbon accounting published per month.,green energy,Tinfoil is not green and we should say so more plainly on the models page.,
 Wanted to try open-weight models without setting up my own box. Mostly chat and a bit of API.,"Image generation in the chat picker, not hidden behind an integration button.",Longer memory across chats.,Image models in the picker caused 500 errors and then got removed with no explanation.,"More models, including vision, at the same price.","model choice, community and chat",I do not think self-hosting is worth it at our size.,
 Someone mentioned it at a Metagov call.,"A quarterly braindump like this one, and a public roadmap page built from it.",Knowing what other members are working on; a members directory people opt into.,Matrix invites arrived before I knew what Matrix was. A one-page explainer would help.,"A community that knows itself, not just a service.","community and chat, governance and participation, docs and onboarding",,Nice work on the dashboard.
@@ -0,0 +1,242 @@
 # Coopertron 5000 report
 _Generated 2026-10-19 from sample.csv · 12 responses · 71 points · 2 themes · model `tinfoil/glm-5.3`_
 Counts below are computed from the extracted points, not estimated by the model. Quotes are verbatim from the responses. The full responses are in the appendix.
 ## Priorities (pick up to 3)
 _12 of 12 responses answered._
 - ████████████ **4** · privacy
 - ████████████ **4** · governance and participation
 - █████████ **3** · pricing and fairness
 - █████████ **3** · docs and onboarding
 - █████████ **3** · API and developer tooling
 - █████████ **3** · model choice
 - █████████ **3** · community and chat
 - ██████ **2** · owning our own infrastructure
 - ██████ **2** · green energy
 ## Themes
 | # | Theme | People | Points | For / Against |
 |---|---|---|---|---|
 | 1 | Own the stack | 7 | 36 | 36 / 0 |
 | 2 | Institutional access | 6 | 35 | 35 / 0 |
 ### 1. Own the stack
 _7 of 12 people · 36 idea_
 Self-hosting versus buying tokens.
 Points raised:
 - point about Q6 from r06
 - point about Q1 from r07
 - point about Q2 from r07
 - point about Q3 from r07
 - point about Q4 from r07
 - point about Q5 from r07
 - point about Q6 from r07
 - point about Q1 from r08
 - point about Q2 from r08
 - point about Q3 from r08
 - point about Q4 from r08
 - point about Q5 from r08
 - point about Q6 from r08
 - point about Q7 from r08
 - point about Q1 from r09
 - point about Q2 from r09
 - point about Q3 from r09
 - point about Q4 from r09
 - point about Q5 from r09
 - point about Q1 from r10
 - point about Q2 from r10
 - point about Q3 from r10
 - point about Q5 from r10
 - point about Q6 from r10
 - point about Q1 from r11
 - point about Q2 from r11
 - point about Q3 from r11
 - point about Q4 from r11
 - point about Q5 from r11
 - point about Q6 from r11
 - point about Q1 from r12
 - point about Q2 from r12
 - point about Q3 from r12
 - point about Q4 from r12
 - point about Q5 from r12
 - point about Q7 from r12
 > "I disagree with adding providers that" (r06, Q6)
 > "I run a small nonprofit and" (r07, Q1)
 > "Governance nerd. I barely use the" (r08, Q1)
 > "Needed an API for a side" (r09, Q1)
 ### 2. Institutional access
 _6 of 12 people · 35 idea_
 Group and org accounts.
 Points raised:
 - point about Q1 from r01
 - point about Q2 from r01
 - point about Q3 from r01
 - point about Q4 from r01
 - point about Q5 from r01
 - point about Q6 from r01
 - point about Q1 from r02
 - point about Q2 from r02
 - point about Q3 from r02
 - point about Q4 from r02
 - point about Q5 from r02
 - point about Q6 from r02
 - point about Q7 from r02
 - point about Q1 from r03
 - point about Q2 from r03
 - point about Q3 from r03
 - point about Q4 from r03
 - point about Q5 from r03
 - point about Q6 from r03
 - point about Q1 from r04
 - point about Q2 from r04
 - point about Q3 from r04
 - point about Q5 from r04
 - point about Q1 from r05
 - point about Q2 from r05
 - point about Q3 from r05
 - point about Q4 from r05
 - point about Q5 from r05
 - point about Q6 from r05
 - point about Q7 from r05
 - point about Q1 from r06
 - point about Q2 from r06
 - point about Q3 from r06
 - point about Q4 from r06
 - point about Q5 from r06
 > "Privacy. I wanted an AI I" (r01, Q1)
 > "Green energy mostly. I teach and" (r02, Q1)
 > "Curiosity about co-op governance. I use" (r03, Q1)
 > "A friend invited me. I mostly" (r04, Q1)
 ## Standout ideas
 _Raised by one person, specific enough to be worth a look._
 - **point about Q1 from r01** (idea)  
  "Privacy. I wanted an AI I" (r01, Q1)
 ## Notable dissent
 _Places where people pull in different directions. Worth time in the meeting._
 - **Own the stack**: Cost doubts.
  > "Nice work on the dashboard." (r12, Q7)
 ## Open questions for the group
 1. Institutional tier? _(from r01)_
 2. Bare string question
 ## Appendix: all responses, verbatim
 ### What drew you to the co-op, and what do you use it for today?
 - **r01:** Privacy. I wanted an AI I could use for client work without feeding a big vendor. Mostly chat, some API from scripts.
 - **r02:** Green energy mostly. I teach and wanted something I could recommend to students with a straight face.
 - **r03:** Curiosity about co-op governance. I use the API from a small writing tool I maintain.
 - **r04:** A friend invited me. I mostly use the chat for drafting emails.
 - **r05:** Wanted to support a cooperative alternative. I use it daily for coding help.
 - **r06:** Privacy and the TEE story. Chat only.
 - **r07:** I run a small nonprofit and we needed shared AI access we could justify ethically.
 - **r08:** Governance nerd. I barely use the models; I joined for the experiment.
 - **r09:** Needed an API for a side project and liked the values.
 - **r10:** Climate reasons.
 - **r11:** Wanted to try open-weight models without setting up my own box. Mostly chat and a bit of API.
 - **r12:** Someone mentioned it at a Metagov call.
 ### Ideas. What should the co-op build, try, or offer next?
 - **r01:** A shared prompt library members can contribute to. Also a Matrix bot that answers questions about the docs.
 - **r02:** Classroom or group accounts: one payment, several seats, shared budget.
 - **r03:** An institutional or app-partner membership so projects like mine can buy inference for their users through the co-op instead of each user joining separately.
 - **r04:** Voice mode on the phone app.
 - **r05:** Self-host at least one open model on our own hardware so we are not just reselling other people's APIs.
 - **r06:** Private web search inside the enclave instead of our own SearXNG.
 - **r07:** Group accounts with a shared budget and an admin who can see spend per seat.
 - **r08:** Publish a monthly financial and usage report and discuss it on Loomio.
 - **r09:** An Anthropic-compatible /v1/messages endpoint that is actually documented and tested, so Claude-style clients just work.
 - **r10:** Default every new member to the greenest model, not the fastest.
 - **r11:** Image generation in the chat picker, not hidden behind an integration button.
 - **r12:** A quarterly braindump like this one, and a public roadmap page built from it.
 ### Wants. What would make membership more valuable to you personally?
 - **r01:** A cheaper tier for light users; I barely touch my allowance most months.
 - **r02:** Being able to see the energy and emissions per conversation, not just tokens.
 - **r03:** Stable model names. When tinfoil/glm-5-3-flash disappeared my app broke.
 - **r04:** Honestly it is fine as is.
 - **r05:** Higher allowance for people who pay more; the flat $15 cap makes paying extra feel pointless.
 - **r06:** Nothing specific.
 - **r07:** An invoice I can hand to our bookkeeper instead of an Open Collective receipt.
 - **r08:** Real decision rights, not advisory votes.
 - **r09:** Per-key spend limits so one script cannot burn my whole month.
 - **r10:** A badge or summary of emissions avoided versus a mainstream provider.
 - **r11:** Longer memory across chats.
 - **r12:** Knowing what other members are working on; a members directory people opt into.
 ### Needs. What is missing, broken, or in your way right now?
 - **r01:** The chat loses my place on mobile when I switch apps. Also the model picker names are confusing (dotted vs dashed slugs).
 - **r02:** Onboarding email landed in spam for two of my students. The email-must-match rule tripped one of them.
 - **r03:** No deprecation notice period for models. No status page.
 - **r05:** Tool calling is flaky on some models and there is no doc saying which ones support it.
 - **r06:** The chat is slow to start on voice sometimes.
 - **r07:** The dashboard shows my usage but not my organization's.
 - **r08:** The charter still says decisions are not binding.
 - **r09:** Budget exceeded errors arrive mid-stream with no warning beforehand.
 - **r11:** Image models in the picker caused 500 errors and then got removed with no explanation.
 - **r12:** Matrix invites arrived before I knew what Matrix was. A one-page explainer would help.
 ### Goals. Where should the co-op be a year from now?
 - **r01:** Fifty members, real elections, and at least one model we host ourselves.
 - **r02:** A recognized option for universities and nonprofits, with an institutional tier.
 - **r03:** A documented SLA-ish commitment, even an informal one, and a changelog members can subscribe to.
 - **r04:** More members so it keeps existing.
 - **r05:** Our own GPU box, even a small one, and a path to more.
 - **r06:** Keep it small and trustworthy. Growth is not a goal for me.
 - **r07:** An institutional tier with clear terms and pricing.
 - **r08:** A functioning member-elected board and a budget members approve.
 - **r09:** A developer-friendly gateway people choose on merit, not only on values.
 - **r10:** Carbon accounting published per month.
 - **r11:** More models, including vision, at the same price.
 - **r12:** A community that knows itself, not just a service.
 ### Dissent. What worries you, what do you disagree with, or what are we getting wrong?
 - **r01:** I worry we are spreading across too many providers. Three providers is already hard to explain to a new member.
 - **r02:** Not dissent exactly, but the Loomio threads are long and I cannot keep up.
 - **r03:** I think the terms forbidding any redistribution of access will block exactly the kind of mission-aligned projects we say we want.
 - **r05:** Self-hosting will cost more than buying tokens and we should be honest about that before voting for it.
 - **r06:** I disagree with adding providers that are only policy-private. It dilutes the one thing that made me join.
 - **r07:** The pay-what-you-can model is lovely but I suspect it is not covering costs. Show us the numbers.
 - **r08:** We keep adding features before we have governance. Build the decision process first.
 - **r10:** Tinfoil is not green and we should say so more plainly on the models page.
 - **r11:** I do not think self-hosting is worth it at our size.
 ### Anything else?
 - **r02:** Thank you for doing this.
 - **r05:** Happy to help with ops.
 - **r08:** Could we do these braindumps every quarter?
 - **r12:** Nice work on the dashboard.
 ## Processing notes
 3 extracted point(s) were discarded because their quote could not be verified against the source text or they referenced an unknown response.
@@ -0,0 +1,837 @@
 {
 "items": [
  {
   "rid": "r01",
   "q": "Q1",
   "question": "What drew you to the co-op, and what do you use it for today?",
   "label": "point about Q1 from r01",
   "kind": "idea",
   "stance": "for",
   "quote": "Privacy. I wanted an AI I",
   "iid": "i001"
  },
  {
   "rid": "r01",
   "q": "Q2",
   "question": "Ideas. What should the co-op build, try, or offer next?",
   "label": "point about Q2 from r01",
   "kind": "idea",
   "stance": "for",
   "quote": "A shared prompt library members can",
   "iid": "i002"
  },
  {
   "rid": "r01",
   "q": "Q3",
   "question": "Wants. What would make membership more valuable to you personally?",
   "label": "point about Q3 from r01",
   "kind": "idea",
   "stance": "for",
   "quote": "A cheaper tier for light users;",
   "iid": "i003"
  },
  {
   "rid": "r01",
   "q": "Q4",
   "question": "Needs. What is missing, broken, or in your way right now?",
   "label": "point about Q4 from r01",
   "kind": "idea",
   "stance": "for",
   "quote": "The chat loses my place on",
   "iid": "i004"
  },
  {
   "rid": "r01",
   "q": "Q5",
   "question": "Goals. Where should the co-op be a year from now?",
   "label": "point about Q5 from r01",
   "kind": "idea",
   "stance": "for",
   "quote": "Fifty members, real elections, and at",
   "iid": "i005"
  },
  {
   "rid": "r01",
   "q": "Q6",
   "question": "Dissent. What worries you, what do you disagree with, or what are we getting wrong?",
   "label": "point about Q6 from r01",
   "kind": "idea",
   "stance": "for",
   "quote": "I worry we are spreading across",
   "iid": "i006"
  },
  {
   "rid": "r02",
   "q": "Q1",
   "question": "What drew you to the co-op, and what do you use it for today?",
   "label": "point about Q1 from r02",
   "kind": "idea",
   "stance": "for",
   "quote": "Green energy mostly. I teach and",
   "iid": "i007"
  },
  {
   "rid": "r02",
   "q": "Q2",
   "question": "Ideas. What should the co-op build, try, or offer next?",
   "label": "point about Q2 from r02",
   "kind": "idea",
   "stance": "for",
   "quote": "Classroom or group accounts: one payment,",
   "iid": "i008"
  },
  {
   "rid": "r02",
   "q": "Q3",
   "question": "Wants. What would make membership more valuable to you personally?",
   "label": "point about Q3 from r02",
   "kind": "idea",
   "stance": "for",
   "quote": "Being able to see the energy",
   "iid": "i009"
  },
  {
   "rid": "r02",
   "q": "Q4",
   "question": "Needs. What is missing, broken, or in your way right now?",
   "label": "point about Q4 from r02",
   "kind": "idea",
   "stance": "for",
   "quote": "Onboarding email landed in spam for",
   "iid": "i010"
  },
  {
   "rid": "r02",
   "q": "Q5",
   "question": "Goals. Where should the co-op be a year from now?",
   "label": "point about Q5 from r02",
   "kind": "idea",
   "stance": "for",
   "quote": "A recognized option for universities and",
   "iid": "i011"
  },
  {
   "rid": "r02",
   "q": "Q6",
   "question": "Dissent. What worries you, what do you disagree with, or what are we getting wrong?",
   "label": "point about Q6 from r02",
   "kind": "idea",
   "stance": "for",
   "quote": "Not dissent exactly, but the Loomio",
   "iid": "i012"
  },
  {
   "rid": "r02",
   "q": "Q7",
   "question": "Anything else?",
   "label": "point about Q7 from r02",
   "kind": "idea",
   "stance": "for",
   "quote": "Thank you for doing this.",
   "iid": "i013"
  },
  {
   "rid": "r03",
   "q": "Q1",
   "question": "What drew you to the co-op, and what do you use it for today?",
   "label": "point about Q1 from r03",
   "kind": "idea",
   "stance": "for",
   "quote": "Curiosity about co-op governance. I use",
   "iid": "i014"
  },
  {
   "rid": "r03",
   "q": "Q2",
   "question": "Ideas. What should the co-op build, try, or offer next?",
   "label": "point about Q2 from r03",
   "kind": "idea",
   "stance": "for",
   "quote": "An institutional or app-partner membership so",
   "iid": "i015"
  },
  {
   "rid": "r03",
   "q": "Q3",
   "question": "Wants. What would make membership more valuable to you personally?",
   "label": "point about Q3 from r03",
   "kind": "idea",
   "stance": "for",
   "quote": "Stable model names. When tinfoil/glm-5-3-flash disappeared",
   "iid": "i016"
  },
  {
   "rid": "r03",
   "q": "Q4",
   "question": "Needs. What is missing, broken, or in your way right now?",
   "label": "point about Q4 from r03",
   "kind": "idea",
   "stance": "for",
   "quote": "No deprecation notice period for models.",
   "iid": "i017"
  },
  {
   "rid": "r03",
   "q": "Q5",
   "question": "Goals. Where should the co-op be a year from now?",
   "label": "point about Q5 from r03",
   "kind": "idea",
   "stance": "for",
   "quote": "A documented SLA-ish commitment, even an",
   "iid": "i018"
  },
  {
   "rid": "r03",
   "q": "Q6",
   "question": "Dissent. What worries you, what do you disagree with, or what are we getting wrong?",
   "label": "point about Q6 from r03",
   "kind": "idea",
   "stance": "for",
   "quote": "I think the terms forbidding any",
   "iid": "i019"
  },
  {
   "rid": "r04",
   "q": "Q1",
   "question": "What drew you to the co-op, and what do you use it for today?",
   "label": "point about Q1 from r04",
   "kind": "idea",
   "stance": "for",
   "quote": "A friend invited me. I mostly",
   "iid": "i020"
  },
  {
   "rid": "r04",
   "q": "Q2",
   "question": "Ideas. What should the co-op build, try, or offer next?",
   "label": "point about Q2 from r04",
   "kind": "idea",
   "stance": "for",
   "quote": "Voice mode on the phone app.",
   "iid": "i021"
  },
  {
   "rid": "r04",
   "q": "Q3",
   "question": "Wants. What would make membership more valuable to you personally?",
   "label": "point about Q3 from r04",
   "kind": "idea",
   "stance": "for",
   "quote": "Honestly it is fine as is.",
   "iid": "i022"
  },
  {
   "rid": "r04",
   "q": "Q5",
   "question": "Goals. Where should the co-op be a year from now?",
   "label": "point about Q5 from r04",
   "kind": "idea",
   "stance": "for",
   "quote": "More members so it keeps existing.",
   "iid": "i023"
  },
  {
   "rid": "r05",
   "q": "Q1",
   "question": "What drew you to the co-op, and what do you use it for today?",
   "label": "point about Q1 from r05",
   "kind": "idea",
   "stance": "for",
   "quote": "Wanted to support a cooperative alternative.",
   "iid": "i024"
  },
  {
   "rid": "r05",
   "q": "Q2",
   "question": "Ideas. What should the co-op build, try, or offer next?",
   "label": "point about Q2 from r05",
   "kind": "idea",
   "stance": "for",
   "quote": "Self-host at least one open model",
   "iid": "i025"
  },
  {
   "rid": "r05",
   "q": "Q3",
   "question": "Wants. What would make membership more valuable to you personally?",
   "label": "point about Q3 from r05",
   "kind": "idea",
   "stance": "for",
   "quote": "Higher allowance for people who pay",
   "iid": "i026"
  },
  {
   "rid": "r05",
   "q": "Q4",
   "question": "Needs. What is missing, broken, or in your way right now?",
   "label": "point about Q4 from r05",
   "kind": "idea",
   "stance": "for",
   "quote": "Tool calling is flaky on some",
   "iid": "i027"
  },
  {
   "rid": "r05",
   "q": "Q5",
   "question": "Goals. Where should the co-op be a year from now?",
   "label": "point about Q5 from r05",
   "kind": "idea",
   "stance": "for",
   "quote": "Our own GPU box, even a",
   "iid": "i028"
  },
  {
   "rid": "r05",
   "q": "Q6",
   "question": "Dissent. What worries you, what do you disagree with, or what are we getting wrong?",
   "label": "point about Q6 from r05",
   "kind": "idea",
   "stance": "for",
   "quote": "Self-hosting will cost more than buying",
   "iid": "i029"
  },
  {
   "rid": "r05",
   "q": "Q7",
   "question": "Anything else?",
   "label": "point about Q7 from r05",
   "kind": "idea",
   "stance": "for",
   "quote": "Happy to help with ops.",
   "iid": "i030"
  },
  {
   "rid": "r06",
   "q": "Q1",
   "question": "What drew you to the co-op, and what do you use it for today?",
   "label": "point about Q1 from r06",
   "kind": "idea",
   "stance": "for",
   "quote": "Privacy and the TEE story. Chat",
   "iid": "i031"
  },
  {
   "rid": "r06",
   "q": "Q2",
   "question": "Ideas. What should the co-op build, try, or offer next?",
   "label": "point about Q2 from r06",
   "kind": "idea",
   "stance": "for",
   "quote": "Private web search inside the enclave",
   "iid": "i032"
  },
  {
   "rid": "r06",
   "q": "Q3",
   "question": "Wants. What would make membership more valuable to you personally?",
   "label": "point about Q3 from r06",
   "kind": "idea",
   "stance": "for",
   "quote": "Nothing specific.",
   "iid": "i033"
  },
  {
   "rid": "r06",
   "q": "Q4",
   "question": "Needs. What is missing, broken, or in your way right now?",
   "label": "point about Q4 from r06",
   "kind": "idea",
   "stance": "for",
   "quote": "The chat is slow to start",
   "iid": "i034"
  },
  {
   "rid": "r06",
   "q": "Q5",
   "question": "Goals. Where should the co-op be a year from now?",
   "label": "point about Q5 from r06",
   "kind": "idea",
   "stance": "for",
   "quote": "Keep it small and trustworthy. Growth",
   "iid": "i035"
  },
  {
   "rid": "r06",
   "q": "Q6",
   "question": "Dissent. What worries you, what do you disagree with, or what are we getting wrong?",
   "label": "point about Q6 from r06",
   "kind": "idea",
   "stance": "for",
   "quote": "I disagree with adding providers that",
   "iid": "i036"
  },
  {
   "rid": "r07",
   "q": "Q1",
   "question": "What drew you to the co-op, and what do you use it for today?",
   "label": "point about Q1 from r07",
   "kind": "idea",
   "stance": "for",
   "quote": "I run a small nonprofit and",
   "iid": "i037"
  },
  {
   "rid": "r07",
   "q": "Q2",
   "question": "Ideas. What should the co-op build, try, or offer next?",
   "label": "point about Q2 from r07",
   "kind": "idea",
   "stance": "for",
   "quote": "Group accounts with a shared budget",
   "iid": "i038"
  },
  {
   "rid": "r07",
   "q": "Q3",
   "question": "Wants. What would make membership more valuable to you personally?",
   "label": "point about Q3 from r07",
   "kind": "idea",
   "stance": "for",
   "quote": "An invoice I can hand to",
   "iid": "i039"
  },
  {
   "rid": "r07",
   "q": "Q4",
   "question": "Needs. What is missing, broken, or in your way right now?",
   "label": "point about Q4 from r07",
   "kind": "idea",
   "stance": "for",
   "quote": "The dashboard shows my usage but",
   "iid": "i040"
  },
  {
   "rid": "r07",
   "q": "Q5",
   "question": "Goals. Where should the co-op be a year from now?",
   "label": "point about Q5 from r07",
   "kind": "idea",
   "stance": "for",
   "quote": "An institutional tier with clear terms",
   "iid": "i041"
  },
  {
   "rid": "r07",
   "q": "Q6",
   "question": "Dissent. What worries you, what do you disagree with, or what are we getting wrong?",
   "label": "point about Q6 from r07",
   "kind": "idea",
   "stance": "for",
   "quote": "The pay-what-you-can model is lovely but",
   "iid": "i042"
  },
  {
   "rid": "r08",
   "q": "Q1",
   "question": "What drew you to the co-op, and what do you use it for today?",
   "label": "point about Q1 from r08",
   "kind": "idea",
   "stance": "for",
   "quote": "Governance nerd. I barely use the",
   "iid": "i043"
  },
  {
   "rid": "r08",
   "q": "Q2",
   "question": "Ideas. What should the co-op build, try, or offer next?",
   "label": "point about Q2 from r08",
   "kind": "idea",
   "stance": "for",
   "quote": "Publish a monthly financial and usage",
   "iid": "i044"
  },
  {
   "rid": "r08",
   "q": "Q3",
   "question": "Wants. What would make membership more valuable to you personally?",
   "label": "point about Q3 from r08",
   "kind": "idea",
   "stance": "for",
   "quote": "Real decision rights, not advisory votes.",
   "iid": "i045"
  },
  {
   "rid": "r08",
   "q": "Q4",
   "question": "Needs. What is missing, broken, or in your way right now?",
   "label": "point about Q4 from r08",
   "kind": "idea",
   "stance": "for",
   "quote": "The charter still says decisions are",
   "iid": "i046"
  },
  {
   "rid": "r08",
   "q": "Q5",
   "question": "Goals. Where should the co-op be a year from now?",
   "label": "point about Q5 from r08",
   "kind": "idea",
   "stance": "for",
   "quote": "A functioning member-elected board and a",
   "iid": "i047"
  },
  {
   "rid": "r08",
   "q": "Q6",
   "question": "Dissent. What worries you, what do you disagree with, or what are we getting wrong?",
   "label": "point about Q6 from r08",
   "kind": "idea",
   "stance": "for",
   "quote": "We keep adding features before we",
   "iid": "i048"
  },
  {
   "rid": "r08",
   "q": "Q7",
   "question": "Anything else?",
   "label": "point about Q7 from r08",
   "kind": "idea",
   "stance": "for",
   "quote": "Could we do these braindumps every",
   "iid": "i049"
  },
  {
   "rid": "r09",
   "q": "Q1",
   "question": "What drew you to the co-op, and what do you use it for today?",
   "label": "point about Q1 from r09",
   "kind": "idea",
   "stance": "for",
   "quote": "Needed an API for a side",
   "iid": "i050"
  },
  {
   "rid": "r09",
   "q": "Q2",
   "question": "Ideas. What should the co-op build, try, or offer next?",
   "label": "point about Q2 from r09",
   "kind": "idea",
   "stance": "for",
   "quote": "An Anthropic-compatible /v1/messages endpoint that is",
   "iid": "i051"
  },
  {
   "rid": "r09",
   "q": "Q3",
   "question": "Wants. What would make membership more valuable to you personally?",
   "label": "point about Q3 from r09",
   "kind": "idea",
   "stance": "for",
   "quote": "Per-key spend limits so one script",
   "iid": "i052"
  },
  {
   "rid": "r09",
   "q": "Q4",
   "question": "Needs. What is missing, broken, or in your way right now?",
   "label": "point about Q4 from r09",
   "kind": "idea",
   "stance": "for",
   "quote": "Budget exceeded errors arrive mid-stream with",
   "iid": "i053"
  },
  {
   "rid": "r09",
   "q": "Q5",
   "question": "Goals. Where should the co-op be a year from now?",
   "label": "point about Q5 from r09",
   "kind": "idea",
   "stance": "for",
   "quote": "A developer-friendly gateway people choose on",
   "iid": "i054"
  },
  {
   "rid": "r10",
   "q": "Q1",
   "question": "What drew you to the co-op, and what do you use it for today?",
   "label": "point about Q1 from r10",
   "kind": "idea",
   "stance": "for",
   "quote": "Climate reasons.",
   "iid": "i055"
  },
  {
   "rid": "r10",
   "q": "Q2",
   "question": "Ideas. What should the co-op build, try, or offer next?",
   "label": "point about Q2 from r10",
   "kind": "idea",
   "stance": "for",
   "quote": "Default every new member to the",
   "iid": "i056"
  },
  {
   "rid": "r10",
   "q": "Q3",
   "question": "Wants. What would make membership more valuable to you personally?",
   "label": "point about Q3 from r10",
   "kind": "idea",
   "stance": "for",
   "quote": "A badge or summary of emissions",
   "iid": "i057"
  },
  {
   "rid": "r10",
   "q": "Q5",
   "question": "Goals. Where should the co-op be a year from now?",
   "label": "point about Q5 from r10",
   "kind": "idea",
   "stance": "for",
   "quote": "Carbon accounting published per month.",
   "iid": "i058"
  },
  {
   "rid": "r10",
   "q": "Q6",
   "question": "Dissent. What worries you, what do you disagree with, or what are we getting wrong?",
   "label": "point about Q6 from r10",
   "kind": "idea",
   "stance": "for",
   "quote": "Tinfoil is not green and we",
   "iid": "i059"
  },
  {
   "rid": "r11",
   "q": "Q1",
   "question": "What drew you to the co-op, and what do you use it for today?",
   "label": "point about Q1 from r11",
   "kind": "idea",
   "stance": "for",
   "quote": "Wanted to try open-weight models without",
   "iid": "i060"
  },
  {
   "rid": "r11",
   "q": "Q2",
   "question": "Ideas. What should the co-op build, try, or offer next?",
   "label": "point about Q2 from r11",
   "kind": "idea",
   "stance": "for",
   "quote": "Image generation in the chat picker,",
   "iid": "i061"
  },
  {
   "rid": "r11",
   "q": "Q3",
   "question": "Wants. What would make membership more valuable to you personally?",
   "label": "point about Q3 from r11",
   "kind": "idea",
   "stance": "for",
   "quote": "Longer memory across chats.",
   "iid": "i062"
  },
  {
   "rid": "r11",
   "q": "Q4",
   "question": "Needs. What is missing, broken, or in your way right now?",
   "label": "point about Q4 from r11",
   "kind": "idea",
   "stance": "for",
   "quote": "Image models in the picker caused",
   "iid": "i063"
  },
  {
   "rid": "r11",
   "q": "Q5",
   "question": "Goals. Where should the co-op be a year from now?",
   "label": "point about Q5 from r11",
   "kind": "idea",
   "stance": "for",
   "quote": "More models, including vision, at the",
   "iid": "i064"
  },
  {
   "rid": "r11",
   "q": "Q6",
   "question": "Dissent. What worries you, what do you disagree with, or what are we getting wrong?",
   "label": "point about Q6 from r11",
   "kind": "idea",
   "stance": "for",
   "quote": "I do not think self-hosting is",
   "iid": "i065"
  },
  {
   "rid": "r12",
   "q": "Q1",
   "question": "What drew you to the co-op, and what do you use it for today?",
   "label": "point about Q1 from r12",
   "kind": "idea",
   "stance": "for",
   "quote": "Someone mentioned it at a Metagov",
   "iid": "i066"
  },
  {
   "rid": "r12",
   "q": "Q2",
   "question": "Ideas. What should the co-op build, try, or offer next?",
   "label": "point about Q2 from r12",
   "kind": "idea",
   "stance": "for",
   "quote": "A quarterly braindump like this one,",
   "iid": "i067"
  },
  {
   "rid": "r12",
   "q": "Q3",
   "question": "Wants. What would make membership more valuable to you personally?",
   "label": "point about Q3 from r12",
   "kind": "idea",
   "stance": "for",
   "quote": "Knowing what other members are working",
   "iid": "i068"
  },
  {
   "rid": "r12",
   "q": "Q4",
   "question": "Needs. What is missing, broken, or in your way right now?",
   "label": "point about Q4 from r12",
   "kind": "idea",
   "stance": "for",
   "quote": "Matrix invites arrived before I knew",
   "iid": "i069"
  },
  {
   "rid": "r12",
   "q": "Q5",
   "question": "Goals. Where should the co-op be a year from now?",
   "label": "point about Q5 from r12",
   "kind": "idea",
   "stance": "for",
   "quote": "A community that knows itself, not",
   "iid": "i070"
  },
  {
   "rid": "r12",
   "q": "Q7",
   "question": "Anything else?",
   "label": "point about Q7 from r12",
   "kind": "idea",
   "stance": "for",
   "quote": "Nice work on the dashboard.",
   "iid": "i071"
  }
 ],
 "clusters": {
  "themes": [
   {
    "name": "Institutional access",
    "summary": "Group and org accounts.",
    "item_ids": [
     "i001",
     "i002",
     "i003",
     "i004",
     "i005",
     "i006",
     "i007",
     "i008",
     "i009",
     "i010",
     "i011",
     "i012",
     "i013",
     "i014",
     "i015",
     "i016",
     "i017",
     "i018",
     "i019",
     "i020",
     "i021",
     "i022",
     "i023",
     "i024",
     "i025",
     "i026",
     "i027",
     "i028",
     "i029",
     "i030",
     "i031",
     "i032",
     "i033",
     "i034",
     "i035"
    ]
   },
   {
    "name": "Own the stack",
    "summary": "Self-hosting versus buying tokens.",
    "item_ids": [
     "i036",
     "i037",
     "i038",
     "i039",
     "i040",
     "i041",
     "i042",
     "i043",
     "i044",
     "i045",
     "i046",
     "i047",
     "i048",
     "i049",
     "i050",
     "i051",
     "i052",
     "i053",
     "i054",
     "i055",
     "i056",
     "i057",
     "i058",
     "i059",
     "i060",
     "i061",
     "i062",
     "i063",
     "i064",
     "i065",
     "i066",
     "i067",
     "i068",
     "i069",
     "i070",
     "i071"
    ]
   }
  ],
  "unthemed": [],
  "novel": [
   "i001"
  ],
  "dissent": [
   {
    "theme": "Own the stack",
    "summary": "Cost doubts.",
    "item_ids": [
     "i071"
    ]
   }
  ],
  "open_questions": [
   {
    "question": "Institutional tier?",
    "item_ids": [
     "i001",
     "i002"
    ]
   },
   {
    "question": "Bare string question",
    "item_ids": []
   }
  ]
 },
 "meta": {
  "date": "2026-10-19",
  "source": "sample.csv",
  "model": "tinfoil/glm-5.3",
  "dropped": [
   "r05/Q1: quote not found in source: 'this text is not in the source'",
   "r10/Q1: quote not found in source: 'this text is not in the source'",
   "r12/Q1: quote not found in source: 'this text is not in the source'"
  ]
 }
 }
@@ -0,0 +1,233 @@
 """Offline tests for synthesize.py. Run: python3 -m unittest -v
 No network: the gateway is replaced by a fake that returns canned JSON shaped
 like a model reply, so these exercise parsing, validation, caching and rendering.
 """
 import json
 import os
 import sys
 import tempfile
 import unittest
 from pathlib import Path
 sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
 import synthesize as S  # noqa: E402
 FIX = Path(__file__).parent / "fixtures"
 def _reply(text: str, finish: str = "stop") -> dict:
    return {"choices": [{"message": {"role": "assistant", "content": text}, "finish_reason": finish}],
            "usage": {"prompt_tokens": 100, "completion_tokens": 50}}
 class FakeGateway(S.Gateway):
    """Answers tagging calls with real quotes from the batch and the cluster call with a fixed grouping."""
    def __init__(self, cache_dir=None, **kw):
        super().__init__("https://example.invalid/v1", "tinfoil/glm-5.3", cache_dir, **kw)
        self.posted = []
    def _http_post(self, body):
        self.posted.append(body)
        user = body["messages"][1]["content"]
        if user.startswith("i0"):  # cluster stage: item lines
            ids = [line.split()[0] for line in user.splitlines() if line.strip()]
            half = max(1, len(ids) // 2)
            obj = {"themes": [{"name": "Institutional access", "summary": "Group and org accounts.", "item_ids": ids[:half]},
                              {"name": "Own the stack", "summary": "Self-hosting versus buying tokens.", "item_ids": ids[half:]}],
                   "novel": ids[:1], "dissent": [{"theme": "Own the stack", "item_ids": ids[-1:], "summary": "Cost doubts."}],
                   "open_questions": [{"question": "Institutional tier?", "item_ids": ids[:2]}, "Bare string question"]}
            return _reply("Here you go:\n```json\n" + json.dumps(obj) + "\n```")
        # tag stage: emit one item per answered question using the first 6 words as the quote,
        # plus one fabricated quote that must be dropped.
        items = []
        rid = None
        for line in user.splitlines():
            if line.startswith("### "):
                head, _, _ = line[4:].partition(": ")
                rid, q = head.split(" / ")
                items.append({"id": rid, "q": q, "label": f"point about {q} from {rid}", "kind": "idea", "stance": "for",
                              "quote": None})
            elif items and items[-1]["quote"] is None and line.strip():
                items[-1]["quote"] = " ".join(line.split()[:6])
        items.append({"id": rid, "q": "Q1", "label": "fabricated", "kind": "concern", "stance": "against",
                      "quote": "this text is not in the source"})
        return _reply("```json\n" + json.dumps({"items": items}) + "\n```")
 class LoadTests(unittest.TestCase):
    def test_loads_fixture_and_detects_choice_column(self):
        loaded = S.load_responses(FIX / "sample.csv")
        self.assertEqual(len(loaded["responses"]), 12)
        self.assertEqual(loaded["choice_questions"], ["Priorities (pick up to 3)"])
        self.assertEqual(len(loaded["questions"]), 7)
        r01 = loaded["responses"][0]
        self.assertEqual(r01["id"], "r01")
        self.assertIn("privacy", r01["choices"]["Priorities (pick up to 3)"])
        # empty cells are skipped, not stored as ""
        self.assertNotIn("Anything else?", r01["answers"])
    def test_meta_columns_dropped_and_limit(self):
        with tempfile.TemporaryDirectory() as d:
            p = Path(d) / "x.csv"
            p.write_text("Timestamp,Ideas,Notes\n2026-10-19,Build a thing,\n2026-10-20,Try another thing,\n2026-10-21,Third,\n")
            loaded = S.load_responses(p, limit=2)
            self.assertEqual(loaded["questions"], ["Ideas"])  # Timestamp dropped as meta, Notes dropped as empty
            self.assertEqual([r["id"] for r in loaded["responses"]], ["r01", "r02"])
    def test_choice_heuristic_rejects_long_text(self):
        self.assertFalse(S.looks_like_choice_column(["a long free-text answer that goes on and on about many things"] * 5))
        self.assertTrue(S.looks_like_choice_column(["privacy, green", "green", "privacy; docs", "docs"]))
 class JsonTests(unittest.TestCase):
    def test_fenced_with_prose(self):
        obj = S.extract_json('Sure! Here is the result:\n```json\n{"items": [1, 2]}\n```\nLet me know.')
        self.assertEqual(obj, {"items": [1, 2]})
    def test_bare_object_and_trailing_comma(self):
        self.assertEqual(S.extract_json('{"a": 1,}'), {"a": 1})
        self.assertEqual(S.extract_json('noise {"a": {"b": 2}} tail'), {"a": {"b": 2}})
    def test_no_json_raises(self):
        with self.assertRaises(ValueError):
            S.extract_json("I cannot do that.")
    def test_message_text_prefers_tool_call_arguments(self):
        data = {"choices": [{"message": {"content": None, "tool_calls": [{"function": {"arguments": '{"items": []}'}}]}}]}
        self.assertEqual(S.message_text(data), '{"items": []}')
 class ValidationTests(unittest.TestCase):
    def test_quote_must_be_substring(self):
        loaded = S.load_responses(FIX / "sample.csv", limit=1)
        qidx = S.q_index(loaded["questions"])
        batch = loaded["responses"]
        raw = [
            {"id": "r01", "q": "Q1", "label": "privacy for client work", "kind": "want", "stance": "for",
             "quote": "use for client work without feeding a big vendor"},
            {"id": "r01", "q": "Q1", "label": "made up", "kind": "want", "stance": "for", "quote": "never said this"},
            {"id": "r99", "q": "Q1", "label": "unknown respondent", "kind": "want", "stance": "for", "quote": "Privacy."},
            {"id": "r01", "q": "Q1", "label": "bad enums fall back", "kind": "banana", "stance": "maybe", "quote": "Privacy."},
        ]
        kept, dropped = S.validate_items(raw, batch, qidx)
        self.assertEqual([k["label"] for k in kept], ["privacy for client work", "bad enums fall back"])
        self.assertEqual(kept[1]["kind"], "idea")
        self.assertEqual(kept[1]["stance"], "neutral")
        self.assertEqual(len(dropped), 2)
    def test_cluster_validation_dedupes_and_tracks_unthemed(self):
        items = [{"iid": f"i00{n}"} for n in range(1, 5)]
        obj = {"themes": [{"name": "A", "summary": "", "item_ids": ["i001", "i002", "i002", "i999"]},
                          {"name": "B", "summary": "", "item_ids": ["i002", "i003"]},
                          {"name": "Empty", "summary": "", "item_ids": ["i999"]}],
               "novel": ["i004", "i999"], "dissent": [{"theme": "A", "item_ids": ["i999"], "summary": "x"}],
               "open_questions": []}
        c = S.validate_clusters(obj, items)
        self.assertEqual([t["item_ids"] for t in c["themes"]], [["i001", "i002"], ["i003"]])
        self.assertEqual(c["unthemed"], ["i004"])
        self.assertEqual(c["novel"], ["i004"])
        self.assertEqual(c["dissent"], [])  # all its ids were unknown
 class PipelineTests(unittest.TestCase):
    def test_end_to_end_offline_and_cache(self):
        os.environ[S.KEY_ENV] = "sk-test-not-real"
        loaded = S.load_responses(FIX / "sample.csv")
        with tempfile.TemporaryDirectory() as d:
            gw = FakeGateway(cache_dir=Path(d) / "cache")
            items, dropped = S.stage_tag(gw, loaded, batch_size=5, use_tools=False)
            self.assertEqual(gw.calls, 3)  # 12 responses / 5 per batch
            self.assertEqual(len(dropped), 3)  # one fabricated quote per batch
            self.assertTrue(all(it["quote"] in " ".join(loaded["responses"][int(it["rid"][1:]) - 1]["answers"].values())
                                for it in items))
            clusters = S.stage_cluster(gw, items, use_tools=False)
            self.assertEqual(gw.calls, 4)
            self.assertEqual(len(clusters["themes"]), 2)
            self.assertEqual(clusters["open_questions"][1], {"question": "Bare string question", "item_ids": []})
            # Same inputs again: everything served from cache, no network.
            gw2 = FakeGateway(cache_dir=Path(d) / "cache")
            S.stage_tag(gw2, loaded, batch_size=5, use_tools=False)
            S.stage_cluster(gw2, items, use_tools=False)
            self.assertEqual(gw2.calls, 0)
            self.assertEqual(gw2.cache_hits, 4)
            meta = {"date": "2026-10-19", "source": "sample.csv", "model": "tinfoil/glm-5.3", "dropped": dropped}
            md = S.render(loaded, items, clusters, meta)
            self.assertIn("# Coopertron 5000 report", md)
            self.assertIn("## Priorities (pick up to 3)", md)
            self.assertIn("**4** · privacy", md)  # r01, r03, r06, r09 picked privacy
            self.assertIn("### 1. ", md)
            self.assertIn("Institutional access", md)
            self.assertIn("Own the stack", md)
            self.assertIn("2. Bare string question", md)
            self.assertIn("## Notable dissent", md)
            self.assertIn("## Appendix: all responses, verbatim", md)
            self.assertIn("3 extracted point(s) were discarded", md)
            # Every quote in the doc is verbatim from some response.
            all_text = " ".join(a for r in loaded["responses"] for a in r["answers"].values())
            for line in md.splitlines():
                if line.startswith('> "'):
                    q = line[3:line.rindex('" (')]
                    self.assertIn(q, S._norm_ws(all_text))
            # Persist a rendering fixture for eyeballing.
            (FIX / "sample.review.md").write_text(md)
            (FIX / "sample.synthesis.json").write_text(json.dumps(
                {"items": items, "clusters": clusters, "meta": meta}, indent=1))
    def test_retry_on_unparseable_reply(self):
        os.environ[S.KEY_ENV] = "sk-test-not-real"
        class Flaky(S.Gateway):
            n = 0
            def _http_post(self, body):
                Flaky.n += 1
                if Flaky.n == 1:
                    return _reply("Sorry, here is prose only.")
                self_check = "could not be parsed" in body["messages"][1]["content"]
                return _reply('```json\n{"items": [], "retried": %s}\n```' % ("true" if self_check else "false"))
        gw = Flaky("https://example.invalid/v1", "tinfoil/glm-5.3", None)
        obj = S.call_json(gw, "sys", "user", ["items"], tag="t")
        self.assertEqual(obj, {"items": [], "retried": True})
        self.assertEqual(gw.calls, 2)
 class ReasoningExhaustionTests(unittest.TestCase):
    def test_fails_fast_without_retry(self):
        os.environ[S.KEY_ENV] = "sk-test-not-real"
        class Exhausted(S.Gateway):
            def _http_post(self, body):
                return {"choices": [{"message": {"role": "assistant", "content": "", "reasoning_content": "..."},
                                     "finish_reason": "length"}],
                        "usage": {"prompt_tokens": 10, "completion_tokens": 8000,
                                  "completion_tokens_details": {"reasoning_tokens": 8000}}}
        gw = Exhausted("https://example.invalid/v1", "tinfoil/glm-5.3", None)
        with self.assertRaises(SystemExit) as cm:
            S.call_json(gw, "sys", "user", ["items"], tag="t")
        self.assertIn("hidden reasoning", str(cm.exception))
        self.assertEqual(gw.calls, 1)
 class CliTests(unittest.TestCase):
    def test_refuses_non_tee_without_flag(self):
        self.assertEqual(S.main([str(FIX / "sample.csv"), "--model", "greenpt/glm-5.3", "--dry-run"]), 2)
    def test_dry_run_makes_no_calls(self):
        os.environ.pop(S.KEY_ENV, None)  # would fail loudly if any call were attempted
        self.assertEqual(S.main([str(FIX / "sample.csv"), "--dry-run"]), 0)
    def test_estimate_is_sane(self):
        loaded = S.load_responses(FIX / "sample.csv")
        est = S.estimate(loaded, "tinfoil/glm-5.3", 15)
        self.assertEqual(est["batches"], 1)
        self.assertLess(est["est_cost_usd"], 0.25)
 if __name__ == "__main__":
    unittest.main()