From 72361b2363fb916249768cc120382b9cd715b7ee Mon Sep 17 00:00:00 2001 From: Angel Marino Date: Mon, 26 Oct 2026 02:11:30 -0400 Subject: [PATCH] api-access, models: document gateway behaviour verified against the live API (fallbacks, reasoning defaults, usage, tool support) Checked on 2026-10-19 with a member key. api-access.md gains a closing section on what the gateway guarantees and what it doesn't: the fallback chain and the headers that reveal it, per-call cost headers, timeouts and retries, reasoning on by default on the Tinfoil models and how to turn it off, dropped parameters, usage fields, /model/info and /team/info being readable with a member key, the budget-exceeded shape, and the absence of rate limits, body caps and an uptime guarantee. models.md gains a Tools column (three of four tested models make tool calls; greenpt/glm-5.3 returns empty through the gateway, 3 of 3 runs), fallback and reasoning notes on the affected rows, and a quick-guide fix. --- api-access.md | 60 +++++++++++++++++++++++++++++++++++++++++++++++++++ models.md | 32 +++++++++++++++------------ 2 files changed, 78 insertions(+), 14 deletions(-) diff --git a/api-access.md b/api-access.md index acfcb7f..0457627 100644 --- a/api-access.md +++ b/api-access.md @@ -85,3 +85,63 @@ substitute our base URL and your member key for `apiKey` — the protocol value stays the same. Some tools append `/chat/completions` to the base URL themselves; if your base URL already ends in `/v1`, that's correct and you should not add `/chat/completions`. + +## What the gateway guarantees (and what it doesn't) + +Checked against `gateway.inference.coop` (LiteLLM 1.103.2) with a member key +on 2026-10-19. If any of this drifts, open a PR against this file. + +- **Supported route:** `POST /v1/chat/completions`. The audio, image and + embedding routes work for the models marked for them in the + [model list](models.md). `POST /v1/messages` (Anthropic format) also + answers: it is LiteLLM's built-in translation layer, not something the co-op + tests or supports. It carries tools and tool results correctly, but it drops + the `cost` field and any reasoning content, so prefer chat completions for + anything you meter. +- **Fallbacks are silent in the body, visible in the headers.** Some models + have a fallback chain (today `greenpt/deepseek-v4.1-flash` → + `tinfoil/deepseek-v4-1-flash` → `tinfoil/gpt-oss-120b`). If the first + provider fails, the next one serves your request at its own price and its + own privacy posture. Every response carries `x-litellm-model-group` (what + you asked for), `x-litellm-model-name` and `x-litellm-model-api-base` (what + answered), and `x-litellm-attempted-fallbacks` (how many hops). Read them + if the provider matters to you. +- **Cost per call is in the headers too:** `x-litellm-response-cost` in USD, + split into `-input`, `-output` and `-reasoning`, and `x-litellm-key-spend` + for your key's running total. With streaming, `usage.cost` arrives in the + final chunk instead. +- **Timeouts and retries.** 180 seconds per request. Connection errors are + retried up to twice; timeouts are never retried, because the provider has + already billed the generation. +- **Reasoning is on by default** for `tinfoil/deepseek-v4-1-flash` and + `tinfoil/glm-5.3`, and reasoning tokens count against `max_tokens`. A + 700-token `max_tokens` can come back as `finish_reason: "length"` with 700 + reasoning tokens and no content. To turn it off per request, send + `"reasoning_effort": "none"` (or + `"chat_template_kwargs": {"enable_thinking": false}`). Anthropic-style + `thinking: {...}` is dropped without an error. +- **Unknown parameters are dropped, not rejected** (`drop_params` is on). If + a parameter seems to have no effect, that is probably why. +- **Usage.** With `"stream_options": {"include_usage": true}` the last chunk + carries `prompt_tokens`, `completion_tokens`, + `completion_tokens_details.reasoning_tokens`, + `prompt_tokens_details.cached_tokens`, and `cost`. Some providers add an + `impact` block (energy in Wms, emissions in µgCO₂e) on non-streaming + responses. +- **Model metadata is readable.** `GET /model/info` with your key returns + every model's `model_info` (pricing, and whatever capability flags are + declared there), so a client can pick a model by what it supports. +- **Budget exhausted.** When your allowance is spent, the gateway returns + HTTP `400` with `error.type: "budget_exceeded"` before any stream starts. + Match on the type, not the status. Your allowance resets on your billing + date; the dashboard shows when. +- **Read your own usage.** Your key can read its own record + (`GET /key/info`) and its own budget container + (`GET /team/info?team_id=…`, with the `team_id` from `/key/info`): spend, + `max_budget`, and `budget_reset_at`. Handy for a client that wants to stop + before the gateway does. `/spend/logs` is admin-only. +- **No rate limits beyond the budget, and no request-size cap we could find + below 1.5 MB.** There is one shared container; please be a good neighbour. +- **No uptime guarantee** (Terms, "Service"). `GET /health/liveliness` and + `GET /health/readiness` answer without auth, so a client can check before + it falls back to something else. diff --git a/models.md b/models.md index be80255..e34411f 100644 --- a/models.md +++ b/models.md @@ -41,10 +41,10 @@ that **you choose** which axis matters more for a given task. - **Everyday chat and drafting** → a fast model: `greenpt/glm-5.3-flash`. These handle the bulk of what people use chat for, at a fraction of flagship cost. -- **Hard reasoning, coding, agentic work** → a flagship: `greenpt/glm-5.3` - (renewable) or `tinfoil/glm-5.3` (TEE-protected) — the GLM flagship on the - provider of your choice — or `tinfoil/deepseek-v4-1-flash` for long documents - and tool-driven tasks. +- **Hard reasoning, coding, agentic work** → `tinfoil/glm-5.3` (TEE-protected) + or `tinfoil/deepseek-v4-1-flash` for long documents and tool-driven tasks. + `greenpt/glm-5.3` is the renewable flagship for reasoning and drafting, but + tool calls currently return empty through the gateway (see the table). - **Images in, text out (vision)** → `tinfoil/deepseek-v4-1-flash` or the Apertus models — see the ✅ in the table below. - **Cheap and cheerful** → `tinfoil/gpt-oss-120b` or `greenpt/glm-5.3-flash`. @@ -63,6 +63,10 @@ cache-read price. **Vision** means the model accepts images as input available via the chat's image feature (the 🎨 button) using `regolo/qwen-image`. +**Tools** means the model made a valid streaming tool call through the co-op +gateway when we tested it (✅), failed to (⚠️, with the date), or hasn't been +tested yet (?). If you test one, add the result by PR. + The emoji in the model name marks its **value tier** (the same icons used in the chat picker): - 🔒 **Private** — Tinfoil: architecturally private (TEE enclaves) @@ -70,16 +74,16 @@ The emoji in the model name marks its **value tier** (the same icons used in the a zero-data-retention commitment). The 🌱 emoji marks renewable-powered providers, not a specific company. -| Model | Provider | Privacy | Green | Vision | Context | Pricing (in / out / cached) | Notes | -|---|---|---|---|---|---|---|---| -| `🌱 greenpt/deepseek-v4.1-flash` | GreenPT | policy | ✅ 100% renewable | ✅ | 1M | $0.22 / $1.10 / $0.011 | **Default.** DeepSeek's multimodal reasoning model, renewable energy; TEE twin below. | -| `🔒 tinfoil/deepseek-v4-1-flash` | Tinfoil | architectural | undisclosed | ✅ | 1M | $0.65 / $1.45 / $0.13 | TEE-protected version of the default; image input, tool calling, long docs/agents. | -| `🔒 tinfoil/glm-5.3` | Tinfoil | architectural | undisclosed | — | 1M | $1.80 / $5.75 / $0.45 | GLM flagship, TEE-protected; strongest private-tier model for agentic coding and long-horizon tasks. | -| `🔒 tinfoil/gpt-oss-120b` | Tinfoil | architectural | undisclosed | — | 131K | $0.15 / $0.60 / — | Lightweight open-weight reasoning; cheapest private tier. | -| `🌱 greenpt/glm-5.3-flash` | GreenPT | policy | ✅ 100% renewable | — | 1M | $0.11 / $0.44 / $0.022 | Fast GLM model, renewable energy. | -| `🌱 greenpt/glm-5.3` | GreenPT | policy | ✅ 100% renewable | — | 1M | $1.10 / $4.40 / $0.275 | GLM flagship — strongest agentic model in the green tier. | -| `🌱 regolo/apertus-70b` | Regolo | policy + ZDR | ✅ 100% renewable | ✅ | 65K | $0.46 / $2.44 / — | Fully open sovereign model (Apache-2.0), large/capable, image understanding. | -| `🌱 regolo/qwen3.5-122b` | Regolo | policy + ZDR | ✅ 100% renewable | ✅ | 262K | $1.16 / $4.87 / — | Large vision+reasoning MoE (Qwen 3.5, 122B-A10B); 262K context. | +| Model | Provider | Privacy | Green | Vision | Tools | Context | Pricing (in / out / cached) | Notes | +|---|---|---|---|---|---|---|---|---| +| `🌱 greenpt/deepseek-v4.1-flash` | GreenPT | policy | ✅ 100% renewable | ✅ | ✅ | 1M | $0.22 / $1.10 / $0.011 | **Default.** DeepSeek's multimodal reasoning model, renewable energy; TEE twin below. Falls back to the Tinfoil twin, then `tinfoil/gpt-oss-120b`, if GreenPT fails. | +| `🔒 tinfoil/deepseek-v4-1-flash` | Tinfoil | architectural | undisclosed | ✅ | ✅ | 1M | $0.65 / $1.45 / $0.13 | TEE-protected version of the default; image input, tool calling, long docs/agents. Reasoning on by default. | +| `🔒 tinfoil/glm-5.3` | Tinfoil | architectural | undisclosed | — | ✅ | 1M | $1.80 / $5.75 / $0.45 | GLM flagship, TEE-protected; strongest private-tier model for agentic coding and long-horizon tasks. Reasoning on by default; the fastest model we measured. | +| `🔒 tinfoil/gpt-oss-120b` | Tinfoil | architectural | undisclosed | — | ? | 131K | $0.15 / $0.60 / — | Lightweight open-weight reasoning; cheapest private tier. | +| `🌱 greenpt/glm-5.3-flash` | GreenPT | policy | ✅ 100% renewable | — | ? | 1M | $0.11 / $0.44 / $0.022 | Fast GLM model, renewable energy. | +| `🌱 greenpt/glm-5.3` | GreenPT | policy | ✅ 100% renewable | — | ⚠️ | 1M | $1.10 / $4.40 / $0.275 | GLM flagship, renewable energy. Tool calls currently return empty through the gateway (one reasoning token, then stop; 3 of 3 runs on 2026-10-19). Use `tinfoil/glm-5.3` for agentic work until this is fixed. | +| `🌱 regolo/apertus-70b` | Regolo | policy + ZDR | ✅ 100% renewable | ✅ | ? | 65K | $0.46 / $2.44 / — | Fully open sovereign model (Apache-2.0), large/capable, image understanding. | +| `🌱 regolo/qwen3.5-122b` | Regolo | policy + ZDR | ✅ 100% renewable | ✅ | ? | 262K | $1.16 / $4.87 / — | Large vision+reasoning MoE (Qwen 3.5, 122B-A10B); 262K context. Thinking is forced off in the gateway config. | Vision is verified working through the co-op gateway (base64 image input via `/v1/chat/completions`, standard OpenAI format). In the chat UI, just attach