api-access, models: document gateway behaviour verified against the live API (fallbacks, reasoning defaults, usage, tool support) #8

2026-10-26T06:12:19Z

solo.ist commented

2026-10-26 06:12:19 +00:00

Documents behaviour I verified against the live gateway on 2026-10-19 with my member key, so people pointing tools at the API don't have to rediscover it: the fallback chain and the response headers that reveal it, the per-call cost headers, reasoning on by default and how to turn it off per request, the usage fields, /model/info and /team/info being readable with a member key, and which models make tool calls. Three of four tested do; greenpt/glm-5.3 returns one reasoning token and stops, three runs, including at max_tokens: 2000, so the table and the quick guide now say so.

Also notes that /v1/messages answers but is unsupported. Nothing here changes a decision; it describes what the config already does (router_settings.fallbacks, drop_params, request_timeout). Happy to trim anything that's too much for this page or move the long section to its own file.

Documents behaviour I verified against the live gateway on 2026-10-19 with my member key, so people pointing tools at the API don't have to rediscover it: the fallback chain and the response headers that reveal it, the per-call cost headers, reasoning on by default and how to turn it off per request, the usage fields, `/model/info` and `/team/info` being readable with a member key, and which models make tool calls. Three of four tested do; `greenpt/glm-5.3` returns one reasoning token and stops, three runs, including at `max_tokens: 2000`, so the table and the quick guide now say so. Also notes that `/v1/messages` answers but is unsupported. Nothing here changes a decision; it describes what the config already does (`router_settings.fallbacks`, `drop_params`, `request_timeout`). Happy to trim anything that's too much for this page or move the long section to its own file.

solo.ist added 1 commit 2026-10-26 06:12:19 +00:00

api-access, models: document gateway behaviour verified against the live API (fallbacks, reasoning defaults, usage, tool support) 72361b2363

Checked on 2026-10-19 with a member key. api-access.md gains a closing
section on what the gateway guarantees and what it doesn't: the fallback
chain and the headers that reveal it, per-call cost headers, timeouts and
retries, reasoning on by default on the Tinfoil models and how to turn it
off, dropped parameters, usage fields, /model/info and /team/info being
readable with a member key, the budget-exceeded shape, and the absence of
rate limits, body caps and an uptime guarantee. models.md gains a Tools
column (three of four tested models make tool calls; greenpt/glm-5.3
returns empty through the gateway, 3 of 3 runs), fallback and
reasoning notes on the affected rows, and a quick-guide fix.

inference-bot commented

2026-10-26 14:58:22 +00:00

Re-checked every claim here today (2026-10-26) against the live gateway with a member-context key, and against the gateway's own config. Most hold exactly. Three need edits.

Verified as written:

Fallback chain and the headers that reveal it (x-litellm-model-group, -model-name, -model-api-base, -attempted-fallbacks) — and the chain itself matches router_settings.fallbacks: greenpt/deepseek-v4.1-flash → tinfoil/deepseek-v4-1-flash → tinfoil/gpt-oss-120b.
Cost headers, including the split into -input, -output, -reasoning, and x-litellm-key-spend.
request_timeout: 180, num_retries: 2, drop_params: true — all three as configured.
/model/info answers with a member key (12 models); /key/info reads your own record.
Health endpoints answer unauthenticated.
The 700-token claim reproduces exactly: tinfoil/deepseek-v4-1-flash, max_tokens: 700 → finish_reason: "length", 700 completion tokens, all 700 of them reasoning, zero content characters. Keep that detail; it is the single most useful line on the page.
The reasoning off-switch works: on tinfoil/glm-5.3, default gave no content and 40 reasoning tokens; with reasoning_effort: "none" it returned content with 0 reasoning tokens.
Version: responses now carry x-litellm-version: 1.103.2, so that pin is right (consider "at the time of writing" — this line will drift).

Worth adding: /team/info is not just readable — the scoping is enforced. A key asking for a team it does not belong to gets 403 (Team key for team=… not authorized to access this team). Saying so prevents the obvious worry that "readable with a member key" means readable for other members' teams.

Three edits:

/v1/messages is model-dependent, not "answers". It returned 404 for greenpt/glm-5.3-flash (No fallback model group found for lookup…), while answering 200 for greenpt/deepseek-v4.1-flash, tinfoil/deepseek-v4-1-flash, tinfoil/glm-5.3 and tinfoil/gpt-oss-120b — and your observation holds for those: no cost or usage field comes back. Suggest naming the models it works for, or noting that some 404.
The greenpt/glm-5.3 downgrade in models.md is not reproducible today. The PR says tool calls "currently return empty through the gateway" and the quick guide steers readers off it. Today I ran it four times (max_tokens: 400, one tool defined) and got a valid tool_calls response on all four, finish_reason: "tool_calls". Either it was fixed after 2026-10-19 or it is intermittent. As written it steers members away from a model that works, so it is worth re-testing before merge — and if it is intermittent, saying so with both dates is more useful than a flat "broken".
Minor: greenpt/glm-5.3-flash also emits reasoning tokens (1–5 on short calls; one run spent its whole 40-token budget thinking). Either widen the "reasoning is on by default" list or say "some models".

I can also offer the Tools column: today, valid tool calls from greenpt/glm-5.3-flash, greenpt/deepseek-v4.1-flash, greenpt/glm-5.3, tinfoil/deepseek-v4-1-flash and tinfoil/glm-5.3. Take the wording as you like.

Re-checked every claim here today (2026-10-26) against the live gateway with a member-context key, and against the gateway's own config. **Most hold exactly. Three need edits.** **Verified as written:** - Fallback chain and the headers that reveal it (`x-litellm-model-group`, `-model-name`, `-model-api-base`, `-attempted-fallbacks`) — and the chain itself matches `router_settings.fallbacks`: `greenpt/deepseek-v4.1-flash → tinfoil/deepseek-v4-1-flash → tinfoil/gpt-oss-120b`. - Cost headers, including the split into `-input`, `-output`, `-reasoning`, and `x-litellm-key-spend`. - `request_timeout: 180`, `num_retries: 2`, `drop_params: true` — all three as configured. - `/model/info` answers with a member key (12 models); `/key/info` reads your own record. - Health endpoints answer unauthenticated. - **The 700-token claim reproduces exactly**: `tinfoil/deepseek-v4-1-flash`, `max_tokens: 700` → `finish_reason: "length"`, 700 completion tokens, all 700 of them reasoning, zero content characters. Keep that detail; it is the single most useful line on the page. - The reasoning off-switch works: on `tinfoil/glm-5.3`, default gave no content and 40 reasoning tokens; with `reasoning_effort: "none"` it returned content with 0 reasoning tokens. - Version: responses now carry `x-litellm-version: 1.103.2`, so that pin is right (consider "at the time of writing" — this line will drift). **Worth adding:** `/team/info` is not just readable — the scoping is *enforced*. A key asking for a team it does not belong to gets `403` (`Team key for team=… not authorized to access this team`). Saying so prevents the obvious worry that "readable with a member key" means readable for other members' teams. **Three edits:** 1. **`/v1/messages` is model-dependent, not "answers".** It returned **404** for `greenpt/glm-5.3-flash` (`No fallback model group found for lookup…`), while answering 200 for `greenpt/deepseek-v4.1-flash`, `tinfoil/deepseek-v4-1-flash`, `tinfoil/glm-5.3` and `tinfoil/gpt-oss-120b` — and your observation holds for those: no cost or usage field comes back. Suggest naming the models it works for, or noting that some 404. 2. **The `greenpt/glm-5.3` downgrade in `models.md` is not reproducible today.** The PR says tool calls "currently return empty through the gateway" and the quick guide steers readers off it. Today I ran it four times (`max_tokens: 400`, one tool defined) and got a valid `tool_calls` response on all four, `finish_reason: "tool_calls"`. Either it was fixed after 2026-10-19 or it is intermittent. As written it steers members away from a model that works, so it is worth re-testing before merge — and if it is intermittent, saying so with both dates is more useful than a flat "broken". 3. **Minor:** `greenpt/glm-5.3-flash` also emits reasoning tokens (1–5 on short calls; one run spent its whole 40-token budget thinking). Either widen the "reasoning is on by default" list or say "some models". I can also offer the Tools column: today, valid tool calls from `greenpt/glm-5.3-flash`, `greenpt/deepseek-v4.1-flash`, `greenpt/glm-5.3`, `tinfoil/deepseek-v4-1-flash` and `tinfoil/glm-5.3`. Take the wording as you like.

inference-bot commented

2026-10-26 15:09:49 +00:00

Follow-up: I ran the mode question down, and I think this identifies the real cause.

Six configurations of greenpt/glm-5.3, one tool defined, asked to call it:

non-streaming, max_tokens 400 → tool_calls ✅
non-streaming, max_tokens 2000 → ✅
streaming, 400 → ✅ (5 tool-call deltas, finish_reason: "tool_calls")
streaming, 2000 → ✅
streaming, 2000, reasoning_effort: "none" → ✅
streaming, 2000, tool_choice: "required" → ✅

Controls behaved the same (greenpt/deepseek-v4.1-flash and tinfoil/glm-5.3 both streamed valid tool calls). So it is not streaming, not the token budget, not reasoning mode — and "tool calls currently return empty through the gateway" is not reproducible today in any mode I can reach.

The likely cause is the response shape. On a successful tool call this model returns finish_reason: "tool_calls", an empty content, and exactly one reasoning token — on all six runs above, without exception. That is precisely your signature: "returns one reasoning token and stops". A harness that reads content (or a client that only surfaces content) records an empty response and calls the model broken, while the tool call is sitting in tool_calls.

If that matches what your harness was reading, the fix is not to downgrade the model but to document the shape — something like: "a successful tool call from this model has empty content and one reasoning token; read tool_calls, not content." That trap is worth writing down, and it explains why three runs agreed with each other.

Caveats I cannot eliminate from here: GreenPT may have changed between 2026-10-19 and today, and if your tool schema, client or stream flag differed from mine the requests were not equivalent. If you can re-run with your original harness and still get no tool call, post the exact request — stream flag, tool schema, client — and I will pin the difference. I would rather find it than have the guide steer members off a model that works.

For the record on the steer itself, from /model/info today: greenpt/glm-5.3 is $1.10 in / $4.40 out per M tokens; tinfoil/glm-5.3 is $1.80 / $5.75. Tinfoil is the privacy upgrade — worth describing as a privacy trade-off rather than a capability one unless a tool-call failure can actually be reproduced.

Follow-up: I ran the mode question down, and I think this identifies the real cause. Six configurations of `greenpt/glm-5.3`, one tool defined, asked to call it: - non-streaming, `max_tokens` 400 → **tool_calls ✅** - non-streaming, `max_tokens` 2000 → **✅** - streaming, 400 → **✅** (5 tool-call deltas, `finish_reason: "tool_calls"`) - streaming, 2000 → **✅** - streaming, 2000, `reasoning_effort: "none"` → **✅** - streaming, 2000, `tool_choice: "required"` → **✅** Controls behaved the same (`greenpt/deepseek-v4.1-flash` and `tinfoil/glm-5.3` both streamed valid tool calls). So it is not streaming, not the token budget, not reasoning mode — and "tool calls currently return empty through the gateway" is not reproducible today in any mode I can reach. **The likely cause is the response shape.** On a successful tool call this model returns `finish_reason: "tool_calls"`, an **empty `content`**, and **exactly one reasoning token** — on all six runs above, without exception. That is precisely your signature: "returns one reasoning token and stops". A harness that reads `content` (or a client that only surfaces content) records an empty response and calls the model broken, while the tool call is sitting in `tool_calls`. If that matches what your harness was reading, the fix is not to downgrade the model but to document the shape — something like: *"a successful tool call from this model has empty content and one reasoning token; read `tool_calls`, not `content`."* That trap is worth writing down, and it explains why three runs agreed with each other. Caveats I cannot eliminate from here: GreenPT may have changed between 2026-10-19 and today, and if your tool schema, client or stream flag differed from mine the requests were not equivalent. **If you can re-run with your original harness and still get no tool call, post the exact request — stream flag, tool schema, client — and I will pin the difference.** I would rather find it than have the guide steer members off a model that works. For the record on the steer itself, from `/model/info` today: `greenpt/glm-5.3` is $1.10 in / $4.40 out per M tokens; `tinfoil/glm-5.3` is $1.80 / $5.75. Tinfoil is the privacy upgrade — worth describing as a privacy trade-off rather than a capability one unless a tool-call failure can actually be reproduced.

inference-bot referenced this pull request from code/litellm

2026-10-26 19:57:58 +00:00

Pull and serve model info automatically where possible #3

bo commented

2026-10-26 20:37:48 +00:00

First-time contributor

I was also looking into model capabilities and what information our gateway.inference.coop endpoint provides/doesn't. This change looks helpful. Can we also improve what info our endpoint provides? code/litellm#3 (comment)

I was also looking into model capabilities and what information our gateway.inference.coop endpoint provides/doesn't. This change looks helpful. Can we also improve what info our endpoint provides? https://git.inference.coop/code/litellm/issues/3#issue-57

solo.ist added 1 commit 2026-10-27 05:34:25 +00:00

api-access, models: apply review (greenpt/glm-5.3 tool calls work again, /v1/messages per model, /team/info 403, reasoning defaults re-checked 2026-10-27) 730204bcd0

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

solo.ist commented

2026-10-27 05:34:55 +00:00

Thanks for running this down, and for the price check at the end. Re-ran this morning with my original harness (same four-field suggest_edit schema, same system prompt, streaming with tool_choice: "auto" and include_usage): five of five valid tool calls from greenpt/glm-5.3, four streaming at max_tokens 400 and 2000 and one non-streaming, all with parseable arguments and finish_reason: "tool_calls". The control on greenpt/deepseek-v4.1-flash passed too. So it is not the schema, the client or the stream flag.

Your response-shape theory doesn't match what I captured on the 19th, though. Those three streams had no tool_calls delta anywhere. Each was one reasoning_content chunk, then finish_reason: "stop", then usage with one completion token:

data: {"choices":[{"index":0,"delta":{"reasoning_content":"!"}}], ...}
data: {"choices":[{"index":0,"delta":{},"finish_reason":"stop"}], ...}
data: {"usage":{"completion_tokens":1,"prompt_tokens":343,"completion_tokens_details":{"reasoning_tokens":1}, ...}}

That was a real failure on the wire, three times in a row, and it is gone today. Either GreenPT changed something between the 19th and the 26th or it was intermittent; from outside we can't tell which. Your point stands either way: the page shouldn't steer members off a model that works. Pushed:

models.md: greenpt/glm-5.3 is ✅ for Tools, with the note you suggested: a successful tool call comes back with empty content and one reasoning token, so read tool_calls. The dated history is kept in one sentence (failed 3 of 3 on 2026-10-19, 5 of 5 and your 10 on 2026-10-26/27). The quick-guide steer is gone and the two glm-5.3 rows are now described as a privacy trade-off, not a capability one. greenpt/glm-5.3-flash is ✅ on your run.
api-access.md: /v1/messages now names the models it answered for and says greenpt/glm-5.3-flash returned 404; the /team/info line says the scoping is enforced (403 for another team); the reasoning default now says every chat model we tested (re-checked today on six: all six return reasoning with a plain prompt, the GreenPT GLM models for a dozen tokens, the Tinfoil models and DeepSeek for hundreds, with tinfoil/glm-5.3 and tinfoil/gpt-oss-120b still spending all 700 of a 700-token budget on thinking); the version pin is "at the time of writing".

@bo, thanks. code/litellm#3 is the right fix and this column is the stopgap until then: once model info is pulled from upstream and served, the Tools column can just point at /model/info. One data point for your appendix: reasoning_effort: "none" (not off, which 400s) does disable thinking through the gateway. Re-checked today: 0 reasoning tokens on tinfoil/deepseek-v4-1-flash, tinfoil/glm-5.3 and greenpt/deepseek-v4.1-flash, one token on the two GreenPT GLM models. The exception is tinfoil/gpt-oss-120b, which rejects none with a 400 ("Harmony does not support reasoning_effort='none'"), so your "can't be disabled" finding is right for that one model.

Thanks for running this down, and for the price check at the end. Re-ran this morning with my original harness (same four-field `suggest_edit` schema, same system prompt, streaming with `tool_choice: "auto"` and `include_usage`): five of five valid tool calls from `greenpt/glm-5.3`, four streaming at `max_tokens` 400 and 2000 and one non-streaming, all with parseable arguments and `finish_reason: "tool_calls"`. The control on `greenpt/deepseek-v4.1-flash` passed too. So it is not the schema, the client or the stream flag. Your response-shape theory doesn't match what I captured on the 19th, though. Those three streams had no `tool_calls` delta anywhere. Each was one `reasoning_content` chunk, then `finish_reason: "stop"`, then usage with one completion token: ``` data: {"choices":[{"index":0,"delta":{"reasoning_content":"!"}}], ...} data: {"choices":[{"index":0,"delta":{},"finish_reason":"stop"}], ...} data: {"usage":{"completion_tokens":1,"prompt_tokens":343,"completion_tokens_details":{"reasoning_tokens":1}, ...}} ``` That was a real failure on the wire, three times in a row, and it is gone today. Either GreenPT changed something between the 19th and the 26th or it was intermittent; from outside we can't tell which. Your point stands either way: the page shouldn't steer members off a model that works. Pushed: * `models.md`: `greenpt/glm-5.3` is ✅ for Tools, with the note you suggested: a successful tool call comes back with empty `content` and one reasoning token, so read `tool_calls`. The dated history is kept in one sentence (failed 3 of 3 on 2026-10-19, 5 of 5 and your 10 on 2026-10-26/27). The quick-guide steer is gone and the two `glm-5.3` rows are now described as a privacy trade-off, not a capability one. `greenpt/glm-5.3-flash` is ✅ on your run. * `api-access.md`: `/v1/messages` now names the models it answered for and says `greenpt/glm-5.3-flash` returned 404; the `/team/info` line says the scoping is enforced (403 for another team); the reasoning default now says every chat model we tested (re-checked today on six: all six return reasoning with a plain prompt, the GreenPT GLM models for a dozen tokens, the Tinfoil models and DeepSeek for hundreds, with `tinfoil/glm-5.3` and `tinfoil/gpt-oss-120b` still spending all 700 of a 700-token budget on thinking); the version pin is "at the time of writing". @bo, thanks. code/litellm#3 is the right fix and this column is the stopgap until then: once model info is pulled from upstream and served, the Tools column can just point at `/model/info`. One data point for your appendix: `reasoning_effort: "none"` (not `off`, which 400s) does disable thinking through the gateway. Re-checked today: 0 reasoning tokens on `tinfoil/deepseek-v4-1-flash`, `tinfoil/glm-5.3` and `greenpt/deepseek-v4.1-flash`, one token on the two GreenPT GLM models. The exception is `tinfoil/gpt-oss-120b`, which rejects `none` with a 400 ("Harmony does not support reasoning_effort='none'"), so your "can't be disabled" finding is right for that one model.

This pull request has changes conflicting with the target branch.

models.md

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u https://git.inference.coop/solo.ist/docs docs/gateway-guarantees:solo.ist-docs/gateway-guarantees

git checkout solo.ist-docs/gateway-guarantees

Sign in to join this conversation.

3 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: co-op/docs#8

Allow edits from maintainers