api-access, models: apply review (greenpt/glm-5.3 tool calls work again, /v1/messages per model, /team/info 403, reasoning defaults re-checked 2026-10-27)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
1 parent
72361b2363
commit
730204bcd0
2 files changed
+25
-18
No files matched your search
+21
-14
@@ -88,16 +88,18 @@ if your base URL already ends in `/v1`, that's correct and you should not add
|
||||
|
||||
## What the gateway guarantees (and what it doesn't)
|
||||
|
||||
Checked against `gateway.inference.coop` (LiteLLM 1.103.2) with a member key
|
||||
on 2026-10-19. If any of this drifts, open a PR against this file.
|
||||
Checked against `gateway.inference.coop` (LiteLLM 1.103.2 at the time of
|
||||
writing) with a member key on 2026-10-19, re-checked 2026-10-26/27. If any of this drifts, open a PR against this file.
|
||||
|
||||
- **Supported route:** `POST /v1/chat/completions`. The audio, image and
|
||||
embedding routes work for the models marked for them in the
|
||||
[model list](models.md). `POST /v1/messages` (Anthropic format) also
|
||||
answers: it is LiteLLM's built-in translation layer, not something the co-op
|
||||
tests or supports. It carries tools and tool results correctly, but it drops
|
||||
the `cost` field and any reasoning content, so prefer chat completions for
|
||||
anything you meter.
|
||||
[model list](models.md). `POST /v1/messages` (Anthropic format) answers for
|
||||
some models (`greenpt/deepseek-v4.1-flash`, `tinfoil/deepseek-v4-1-flash`,
|
||||
`tinfoil/glm-5.3`, `tinfoil/gpt-oss-120b`) and returns 404 for others
|
||||
(`greenpt/glm-5.3-flash`): it is LiteLLM's built-in translation layer, not
|
||||
something the co-op tests or supports. Where it works it carries tools and
|
||||
tool results correctly, but it drops the `cost` field and any reasoning
|
||||
content, so prefer chat completions for anything you meter.
|
||||
- **Fallbacks are silent in the body, visible in the headers.** Some models
|
||||
have a fallback chain (today `greenpt/deepseek-v4.1-flash` →
|
||||
`tinfoil/deepseek-v4-1-flash` → `tinfoil/gpt-oss-120b`). If the first
|
||||
@@ -113,13 +115,17 @@ on 2026-10-19. If any of this drifts, open a PR against this file.
|
||||
- **Timeouts and retries.** 180 seconds per request. Connection errors are
|
||||
retried up to twice; timeouts are never retried, because the provider has
|
||||
already billed the generation.
|
||||
- **Reasoning is on by default** for `tinfoil/deepseek-v4-1-flash` and
|
||||
`tinfoil/glm-5.3`, and reasoning tokens count against `max_tokens`. A
|
||||
700-token `max_tokens` can come back as `finish_reason: "length"` with 700
|
||||
reasoning tokens and no content. To turn it off per request, send
|
||||
- **Reasoning is on by default** on every chat model we tested, and reasoning
|
||||
tokens count against `max_tokens`. How much varies: the GreenPT GLM models
|
||||
think for a dozen tokens or so and then answer, while the Tinfoil models and
|
||||
DeepSeek on either provider can think for hundreds. A 700-token `max_tokens`
|
||||
on `tinfoil/glm-5.3` or `tinfoil/gpt-oss-120b` can come back as
|
||||
`finish_reason: "length"` with 700 reasoning tokens and no content. To turn it off per request, send
|
||||
`"reasoning_effort": "none"` (or
|
||||
`"chat_template_kwargs": {"enable_thinking": false}`). Anthropic-style
|
||||
`thinking: {...}` is dropped without an error.
|
||||
`"chat_template_kwargs": {"enable_thinking": false}`); `"off"` is rejected
|
||||
with a 400, and `tinfoil/gpt-oss-120b` rejects `"none"` too, so thinking
|
||||
can't be disabled on that model. Anthropic-style `thinking: {...}` is
|
||||
dropped without an error.
|
||||
- **Unknown parameters are dropped, not rejected** (`drop_params` is on). If
|
||||
a parameter seems to have no effect, that is probably why.
|
||||
- **Usage.** With `"stream_options": {"include_usage": true}` the last chunk
|
||||
@@ -139,7 +145,8 @@ on 2026-10-19. If any of this drifts, open a PR against this file.
|
||||
(`GET /key/info`) and its own budget container
|
||||
(`GET /team/info?team_id=…`, with the `team_id` from `/key/info`): spend,
|
||||
`max_budget`, and `budget_reset_at`. Handy for a client that wants to stop
|
||||
before the gateway does. `/spend/logs` is admin-only.
|
||||
before the gateway does. The scoping is enforced: asking for another team's
|
||||
`team_id` returns 403. `/spend/logs` is admin-only.
|
||||
- **No rate limits beyond the budget, and no request-size cap we could find
|
||||
below 1.5 MB.** There is one shared container; please be a good neighbour.
|
||||
- **No uptime guarantee** (Terms, "Service"). `GET /health/liveliness` and
|
||||
|
||||
Reference in new issue
Block a user