Pull and serve model info automatically where possible #3
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: code/litellm#3
Reference in new issue
Block a user
Problems.
gateway.inference.coop/v1/models/or/model_group/info/is sometimes incomplete or incorrect (see appendix). For example: context length, capabilities.Proposal.
Not changed. How pricing information is set.
--> Thoughts? If it sounds good, I'll start working on a PR for this.
Appendix
Warning: LLM-researched and generated.
1) Missing from LiteLLM discovery
max_input_tokens,max_output_tokensarenullon 11/11 groups in both/model_group/infoand/v2/model/infogpt-oss-120b; both Regolo rows fall to the generic 128K context / 32K output defaults (apertus-70bactually serves 65K,qwen3.5-122b262K)supported_reasoning_efforts: nullon 11/11minimal…xhigh;glm-5.3minimal…high(xhigh→ 400);gpt-oss-120blow…high(minimal→ 400);apertus-70baccepts every value includingxhigh.off→ 400 everywhere, so thinking cannot be disabled on any groupmode: nullon 6/11 groups — every GreenPT and Tinfoil chat rowmode(chat,image_generation,audio_transcription,audio_speech); the remaining 6 are assumed to be chat (correct today, unverified by the proxy)tinfoil/nomic-embed-text(modeembedding) appears only in/v2/model/info— 12 rows there vs 11 in/model_group/infoand/v1/modelsname/display_namefield on any surface; ids onlyproviders: ["openai"]on every group;owned_by: "openai"on every/v1/modelsrow. The true hosts (api.greenpt.ai,127.0.0.1:3301,api.regolo.ai) exist only in/v2/model/info→litellm_params.api_base, on the surface OMP never reachesprovidersto choose the wire and therefore assignsopenai-responsesto all 8 discovered chat rows. It works, but the field describes the LiteLLM adapter, not the backendtpm,rpm,itpm,otpm,health_status,health_response_time,health_checked_atarenullon 11/11*_above_*k_tokens,*_flex,*_priority,*_batchescost field isnullcache_creation_input_token_costisnullon 11/11; onlycache_read_input_token_costis populatedcacheWriteestimate is 0 for every modelar_male,casual_female,casual_male,cheerful_female,de_female, …,neutral_male— surface only inside a 400 body), no output-format list, no pixel or duration caps. TTS cost is entirelynull(tinfoil/voxtral-tts: both input and output null); the image row carries a per-pixel price (1.5e-08) that OMP's cost model cannot expressspeechrole has no way to select a valid voicedeepseek-v4.1-flash(GreenPT) anddeepseek-v4-1-flash(Tinfoil)2) Wrong in LiteLLM discovery
Capability flags as reported (
falsefor all four, all models) versus what the endpoint does.Probe: a 64×64 red PNG for vision, a
get_weatherfunction for tool calling, a two-city requestwith
parallel_tool_calls: true, and a plain17*23prompt with noreasoning_effortparameter for reasoning.
greenpt/deepseek-v4.1-flashreasoning_contentpresenttinfoil/deepseek-v4-1-flashreasoning_contentpresentgreenpt/glm-5.3reasoning_contentpresenttinfoil/glm-5.3reasoning_contentpresentgreenpt/glm-5.3-flashreasoning_contentpresenttinfoil/gpt-oss-120breasoning_contentpresentregolo/qwen3.5-122bregolo/apertus-70bsupports_function_calling: falseon 11/11 — wrong for at least 8. Every chat model returnsa valid
tool_callspayload. OMP's tool use works only because it advertises tools regardless;the metadata asserts the opposite.
supports_parallel_function_calling: falseon 11/11 — wrong for 5 of 5 measured. One requestasking for Paris and Tokyo returned two
tool_callson both DeepSeek rows, bothglm-5.3rows,and
glm-5.3-flash.supports_vision: falseon 11/11 — wrong for 5 of 8 chat models.greenpt/deepseek-v4.1-flash,tinfoil/deepseek-v4-1-flash,greenpt/glm-5.3-flash,regolo/qwen3.5-122bandregolo/apertus-70ball identify the red square. The flag is correctonly for
tinfoil/glm-5.3(hard 400).greenpt/glm-5.3is effectively text-only (200, imagedropped). Note: the cooperative's own published note lists
greenpt/glm-5.3-flashasvision: false; that is also wrong.supports_reasoning: falseon 11/11 — wrong for 6 of 8. With noreasoning_effortsupplied,both DeepSeek rows, both
glm-5.3rows,glm-5.3-flashandgpt-oss-120ball return non-emptyreasoning_content, i.e. thinking is on by default. Only the two Regolo rows are genuinelynon-thinking.
supported_openai_paramsis byte-identical across all 11 groups. The same 32-entry chat list(
tools,tool_choice,logprobs,audio,prediction,service_tier,prompt_cache_retention, …) is attached totinfoil/voxtral-tts,tinfoil/whisper-large-v3-turboandregolo/qwen-image, which are not chat endpoints. On fourchat models the list holds up under spot checks (
n=2,logprobs,storeall accepted), so thedefect is structural rather than per-model.
/v1/modelsis false on two fields for every row.created: 1677610602(2023-02-28) for all11 models, and
owned_by: "openai"for all 11 — Tinfoil, GreenPT and Regolo models are reportedas OpenAI's. Every OpenAI-SDK consumer and OMP's generic
openai-models-listpath inherits both.greenpt/glm-5.3("I don't see an imageattached") and
tinfoil/gpt-oss-120b("I can't view images") return 200 with the image discarded.Capability probing by HTTP status therefore yields false positives; only checking the answer text
reveals it. (Proxy behaviour, not a discovery field — but it defeats discovery-by-probe.)
Thanks Bo — I reproduced all of it against the live gateway just now, and I think the diagnosis points somewhere a bit different than the proposal does. Numbers from
gateway.inference.coop(LiteLLM 1.103.2, admin key, 2026-10-26):/v1/models: 11 rows,owned_by: "openai"andcreated: 1677610602on every one./model_group/info:max_input_tokens,max_output_tokens,supported_reasoning_efforts,display_namenull on 11/11;supports_function_calling,supports_vision,supports_reasoningfalse on 11/11;modenull on 6/11 (present: chat x2, image_generation, audio_transcription, audio_speech)./v2/model/info: 12 rows against 11 elsewhere.Root cause: our config declares only pricing
Every entry in this repo's
config.yamlhas amodel_info:block, and those blocks containinput_cost_per_token,output_cost_per_token,cache_read_input_token_costand, on the image row,input_cost_per_pixel— pricing and nothing else.modeappears on five entries only.LiteLLM serves what it is told. With no
max_input_tokens, nomode, nosupports_*, and no registry entry for custom vendor names likegreenpt/deepseek-v4.1-flash, nulls andfalseare the honest answer. So the missing fields in section 1 — and much of section 2 — are a declaration gap in our own config, closable now without any upstream integration. The auto-pull you propose is the right hygiene for keeping the values fresh afterwards; it isn't the fix for what is absent today.Three cautions before the PR
model_infoblock as the discovery fields. LiteLLM computes member spend, allowance consumption and budget caps from them, so a generated file that rewritesmodel_infowholesale would silently reprice every model and drift every member's budget. Keep the generator strictly additive — discovery fields only, never a cost key — and add a test asserting the pricing keys are unchanged before/after generation. That test is the thing that makes this safe to automate.greenpt/glm-5.3andtinfoil/gpt-oss-120breturn 200 with the image quietly discarded. So the shape is auto-pull for what upstreams publish plus a maintained empirical layer for capabilities. That is a larger manual surface than "GreenPT only" implies — worth deciding deliberately rather than discovering halfway.config.yamlmirrors the live deployment, so a new field is only served after the app restarts. That's a member-facing inference blip: batch it, don't slip it in.Two notes
supports_function_calling: falseis served while tool calls return normally — 6/6 modes pass ongreenpt/glm-5.3-flash, with the call arriving intool_callsand an emptycontent.greenpt/glm-5.3-flashisvision: false; your probe contradicts it. I'll correct the docs either way.Suggested split
config.yamlnow — unblocks the client, no automation, and the pricing invariant is untouched by construction.Happy to review the PR when it lands, and I'll keep an eye on this thread.
We could declare the missing fields now, but it's a one-time fix. We'd need to manually update config.yaml any time an upstream provider changes what models they offer or we add a new provider.
Checking in on this, since Bo asked whether it is ready to go.
Where it stands — measured against the live gateway just now, via
/model_group/info(11 entries):max_input_tokensnull: 11/11max_output_tokensnull: 11/11modenull: 6/11supports_function_callingtrue: 0/11supports_visiontrue: 0/11So nothing has changed since the measurements above: the fields are still simply absent, which is why clients substitute their own numbers for us rather than reading ours.
On the proposal — the shape is right, and the split you suggest is the correct one: pull from upstreams that expose it (Tinfoil, Regolo), keep a manual file for GreenPT. Two things I can hand over so you do not have to rediscover them:
Read Tinfoil the way our traffic does. Our model-info watcher authenticates to Tinfoil through the in-TEE proxy inside the gateway container, not a side door. If the sync reads Tinfoil directly, it can report a model list that does not match what we actually serve.
Key the GreenPT file on
model_group, notmodel. Our spend logs carry the upstream name (openai/deepseek-v4.1-flash) while everything member-facing is keyed on our alias (greenpt/deepseek-v4.1-flash). Matching on the wrong one fails silently — it just matches nothing. I lost an afternoon to exactly that mistake in a different script.Bar for "done", in my view: every served group reports real caps and capabilities, and a re-run after an upstream change needs no hand editing. Pricing stays as-is, per your "not changed".
I am not taking this from you — it is your proposal and your PR. But when it lands I will extend the existing watcher to verify it: assert every served group has non-null caps, and alert when one goes null again. That is the part that keeps it fixed, and it is squarely mine.