155 lines
7.1 KiB
Markdown
155 lines
7.1 KiB
Markdown
# API access
|
|
|
|
Members can use the co-op's models programmatically through an OpenAI-compatible
|
|
API. It's the same infrastructure that powers the chat — any OpenAI-compatible
|
|
client works.
|
|
|
|
## Quick facts
|
|
|
|
- **Base URL:** `https://gateway.inference.coop/v1`
|
|
- **Auth:** a bearer API key (create one in the [Member Dashboard](https://dashboard.inference.coop))
|
|
- **Format:** OpenAI chat completions — `POST https://gateway.inference.coop/v1/chat/completions`
|
|
- **Models:** see the [model list](models.md).
|
|
|
|
API usage draws from the **same monthly allowance as chat** — there's no separate
|
|
quota.
|
|
|
|
## Get a key
|
|
|
|
Log in to the [Member Dashboard](https://dashboard.inference.coop) and create an
|
|
API key. The key is an `sk-…` string you pass as a bearer token.
|
|
|
|
## Plain `curl`
|
|
|
|
```bash
|
|
curl https://gateway.inference.coop/v1/chat/completions \
|
|
-H "Authorization: Bearer sk-your-key" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"model": "tinfoil/deepseek-v4-1-flash", "messages": [{"role": "user", "content": "Hello"}]}'
|
|
```
|
|
|
|
## OpenAI SDK (Python, Node, etc.)
|
|
|
|
Point the SDK at our base URL and use your member key:
|
|
|
|
```python
|
|
from openai import OpenAI
|
|
|
|
client = OpenAI(
|
|
base_url="https://gateway.inference.coop/v1",
|
|
api_key="sk-your-key",
|
|
)
|
|
resp = client.chat.completions.create(
|
|
model="tinfoil/deepseek-v4-1-flash",
|
|
messages=[{"role": "user", "content": "Hello"}],
|
|
)
|
|
```
|
|
|
|
Any OpenAI-compatible client works the same way: set the base URL to
|
|
`https://gateway.inference.coop/v1` and the API key to your member key.
|
|
|
|
## Coding agents and other OpenAI-compatible clients
|
|
|
|
Any tool that speaks the OpenAI API works. Where a tool asks for provider or
|
|
endpoint configuration:
|
|
|
|
- **Base URL / endpoint:** `https://gateway.inference.coop/v1`
|
|
- **API type / protocol:** "OpenAI compatible" (or `openai-completions` where a
|
|
protocol string must be named) — not "Ollama", "Anthropic", or "Responses"
|
|
- **API key:** your member key (the `sk-…` string)
|
|
- **Model:** one from the [model list](models.md), e.g. `tinfoil/deepseek-v4-1-flash` or `greenpt/glm-5.3-flash`
|
|
|
|
A typical provider entry looks like:
|
|
|
|
```json
|
|
{
|
|
"providers": {
|
|
"inference-coop": {
|
|
"baseUrl": "https://gateway.inference.coop/v1",
|
|
"api": "openai-completions",
|
|
"apiKey": "sk-your-key",
|
|
"models": [
|
|
{ "id": "tinfoil/deepseek-v4-1-flash" },
|
|
{ "id": "tinfoil/gpt-oss-120b" },
|
|
{ "id": "tinfoil/glm-5.3" },
|
|
{ "id": "greenpt/glm-5.3-flash" }
|
|
]
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
If a tool's own docs show a local-server example (e.g.
|
|
`baseUrl: http://localhost:11434/v1` with `api: "openai-completions"`),
|
|
substitute our base URL and your member key for `apiKey` — the protocol value
|
|
stays the same. Some tools append `/chat/completions` to the base URL themselves;
|
|
if your base URL already ends in `/v1`, that's correct and you should not add
|
|
`/chat/completions`.
|
|
|
|
## What the gateway guarantees (and what it doesn't)
|
|
|
|
Checked against `gateway.inference.coop` (LiteLLM 1.103.2 at the time of
|
|
writing) with a member key on 2026-10-19, re-checked 2026-10-26/27. If any of this drifts, open a PR against this file.
|
|
|
|
- **Supported route:** `POST /v1/chat/completions`. The audio, image and
|
|
embedding routes work for the models marked for them in the
|
|
[model list](models.md). `POST /v1/messages` (Anthropic format) answers for
|
|
some models (`greenpt/deepseek-v4.1-flash`, `tinfoil/deepseek-v4-1-flash`,
|
|
`tinfoil/glm-5.3`, `tinfoil/gpt-oss-120b`) and returns 404 for others
|
|
(`greenpt/glm-5.3-flash`): it is LiteLLM's built-in translation layer, not
|
|
something the co-op tests or supports. Where it works it carries tools and
|
|
tool results correctly, but it drops the `cost` field and any reasoning
|
|
content, so prefer chat completions for anything you meter.
|
|
- **Fallbacks are silent in the body, visible in the headers.** Some models
|
|
have a fallback chain (today `greenpt/deepseek-v4.1-flash` →
|
|
`tinfoil/deepseek-v4-1-flash` → `tinfoil/gpt-oss-120b`). If the first
|
|
provider fails, the next one serves your request at its own price and its
|
|
own privacy posture. Every response carries `x-litellm-model-group` (what
|
|
you asked for), `x-litellm-model-name` and `x-litellm-model-api-base` (what
|
|
answered), and `x-litellm-attempted-fallbacks` (how many hops). Read them
|
|
if the provider matters to you.
|
|
- **Cost per call is in the headers too:** `x-litellm-response-cost` in USD,
|
|
split into `-input`, `-output` and `-reasoning`, and `x-litellm-key-spend`
|
|
for your key's running total. With streaming, `usage.cost` arrives in the
|
|
final chunk instead.
|
|
- **Timeouts and retries.** 180 seconds per request. Connection errors are
|
|
retried up to twice; timeouts are never retried, because the provider has
|
|
already billed the generation.
|
|
- **Reasoning is on by default** on every chat model we tested, and reasoning
|
|
tokens count against `max_tokens`. How much varies: the GreenPT GLM models
|
|
think for a dozen tokens or so and then answer, while the Tinfoil models and
|
|
DeepSeek on either provider can think for hundreds. A 700-token `max_tokens`
|
|
on `tinfoil/glm-5.3` or `tinfoil/gpt-oss-120b` can come back as
|
|
`finish_reason: "length"` with 700 reasoning tokens and no content. To turn it off per request, send
|
|
`"reasoning_effort": "none"` (or
|
|
`"chat_template_kwargs": {"enable_thinking": false}`); `"off"` is rejected
|
|
with a 400, and `tinfoil/gpt-oss-120b` rejects `"none"` too, so thinking
|
|
can't be disabled on that model. Anthropic-style `thinking: {...}` is
|
|
dropped without an error.
|
|
- **Unknown parameters are dropped, not rejected** (`drop_params` is on). If
|
|
a parameter seems to have no effect, that is probably why.
|
|
- **Usage.** With `"stream_options": {"include_usage": true}` the last chunk
|
|
carries `prompt_tokens`, `completion_tokens`,
|
|
`completion_tokens_details.reasoning_tokens`,
|
|
`prompt_tokens_details.cached_tokens`, and `cost`. Some providers add an
|
|
`impact` block (energy in Wms, emissions in µgCO₂e) on non-streaming
|
|
responses.
|
|
- **Model metadata is readable.** `GET /model/info` with your key returns
|
|
every model's `model_info` (pricing, and whatever capability flags are
|
|
declared there), so a client can pick a model by what it supports.
|
|
- **Budget exhausted.** When your allowance is spent, the gateway returns
|
|
HTTP `400` with `error.type: "budget_exceeded"` before any stream starts.
|
|
Match on the type, not the status. Your allowance resets on your billing
|
|
date; the dashboard shows when.
|
|
- **Read your own usage.** Your key can read its own record
|
|
(`GET /key/info`) and its own budget container
|
|
(`GET /team/info?team_id=…`, with the `team_id` from `/key/info`): spend,
|
|
`max_budget`, and `budget_reset_at`. Handy for a client that wants to stop
|
|
before the gateway does. The scoping is enforced: asking for another team's
|
|
`team_id` returns 403. `/spend/logs` is admin-only.
|
|
- **No rate limits beyond the budget, and no request-size cap we could find
|
|
below 1.5 MB.** There is one shared container; please be a good neighbour.
|
|
- **No uptime guarantee** (Terms, "Service"). `GET /health/liveliness` and
|
|
`GET /health/readiness` answer without auth, so a client can check before
|
|
it falls back to something else.
|