co-op/support
3
0 Fork 0
Code Issues 13 Pull Requests Actions Packages Projects Releases Wiki Activity

Expose x-litellm-* headers on LiteLLM proxy so cross-origin clients can get completed pricing data #39

New Issue
Closed
opened 2026-10-31 16:18:18 +00:00 by staeiou · 6 comments
staeiou commented 2026-10-31 16:18:18 +00:00
Copy link
Copy source

Building on #38, which was from me on Loomio. Thanks @inference-bot for the analysis on that, and can you look into this too?

Can you investigate how CORS restrictions intersect with the ability for cross-origin web apps to access the custom headers LiteLLM sends about the completed call (e.g. x-litellm-response-cost, x-litellm-response-cost-input, x-litellm-response-cost-output, x-litellm-model-name)? Unlike most MaaS inference providers, LiteLLM doesn't yet send per-call cost detail for non-streaming calls in the JSON payload. LiteLLM computes the per-call cost and returns it only as a response header. So even if we allow all CORS, a browser can read the body of an allowed cross-origin response, but it can only read the response headers the server lists in Access-Control-Expose-Headers.

It seems like the gateway currently exposes seven names (semantic-filter, guardrail, cache-key) and none of the cost ones, so JS gets null for x-litellm-response-cost even with the origin allowed. LiteLLM has no config for that list — it's a hardcoded constant in the app, and the upstream request to make it configurable (#33123
(https://github.com/BerriAI/litellm/issues/33123)) was closed as stale, so it has to be added at the proxy layer in front of the app.

The ask: add this to whatever config serves gateway.inference.coop (the vhost that proxies to LiteLLM):

add_header Access-Control-Expose-Headers "x-litellm-response-cost, x-litellm-response-cost-input, x-litellm-response-cost-output, x-litellm-response-cost-cache-read, x-litellm-model-name" always;
Building on #38, which was from me on Loomio. Thanks @inference-bot for the analysis on that, and can you look into this too? Can you investigate how CORS restrictions intersect with the ability for cross-origin web apps to access the custom headers LiteLLM sends about the completed call (e.g. x-litellm-response-cost, x-litellm-response-cost-input, x-litellm-response-cost-output, x-litellm-model-name)? Unlike most MaaS inference providers, LiteLLM doesn't yet send per-call cost detail for non-streaming calls in the JSON payload. LiteLLM computes the per-call cost and returns it only as a response header. So even if we allow all CORS, a browser can read the body of an allowed cross-origin response, but it can only read the response headers the server lists in Access-Control-Expose-Headers. It seems like the gateway currently exposes seven names (semantic-filter, guardrail, cache-key) and none of the cost ones, so JS gets null for x-litellm-response-cost even with the origin allowed. LiteLLM has no config for that list — it's a hardcoded constant in the app, and the upstream request to make it configurable (#33123 (https://github.com/BerriAI/litellm/issues/33123)) was closed as stale, so it has to be added at the proxy layer in front of the app. The ask: add this to whatever config serves gateway.inference.coop (the vhost that proxies to LiteLLM): ```nginx add_header Access-Control-Expose-Headers "x-litellm-response-cost, x-litellm-response-cost-input, x-litellm-response-cost-output, x-litellm-response-cost-cache-read, x-litellm-model-name" always; ```
inference-bot commented 2026-10-31 17:22:25 +00:00
Owner
Copy link
Copy source

Investigated and implemented on the gateway (image 1.0.0-1793467213663).

The finding matches yours. LiteLLM configures the middleware as
expose_headers=LITELLM_UI_ALLOW_HEADERS, and that constant holds seven names — none of the cost
ones. So the proxy sends x-litellm-response-cost* on the response while the browser refuses to
expose them to JS. Upstream closed the request as stale, as you found.

One correction to the proposed fix, and why. You suggested adding
add_header Access-Control-Expose-Headers ... to the vhost. That would work, but the vhost config is
generated by the platform and is not part of this app — we do not own that file, so a change there
would not survive a redeploy and would not be visible in the app's repo. The app does own the
middleware, so the fix is a build-time patch in the app's repo, alongside the five patches already
there for similar LiteLLM gaps (patch_expose_cost_headers.py). Same outcome, durable, reviewable in
git.

Verified live with a real completion from an unrelated origin:

  • access-control-expose-headers now names 12 cost headers, including x-litellm-response-cost, -input, -output, -original, -discount-amount, -margin-amount, -reasoning, -tool-usage and x-litellm-classifier-cost
  • the response carries x-litellm-response-cost: 8.03e-06 for that request
  • the request itself succeeded (HTTP 200, model replied)

It is additive: it exposes per-request pricing to a caller who already holds their own key and can
already see those numbers in the dashboard. No prompt, key or account data, and nothing about other
members. Worth testing from your app — if a name you need is missing, say which and it is a one-line
addition.

Investigated and implemented on the gateway (image `1.0.0-1793467213663`). **The finding matches yours.** LiteLLM configures the middleware as `expose_headers=LITELLM_UI_ALLOW_HEADERS`, and that constant holds seven names — none of the cost ones. So the proxy sends `x-litellm-response-cost*` on the response while the browser refuses to expose them to JS. Upstream closed the request as stale, as you found. **One correction to the proposed fix, and why.** You suggested adding `add_header Access-Control-Expose-Headers ...` to the vhost. That would work, but the vhost config is generated by the platform and is not part of this app — we do not own that file, so a change there would not survive a redeploy and would not be visible in the app's repo. The app *does* own the middleware, so the fix is a build-time patch in the app's repo, alongside the five patches already there for similar LiteLLM gaps (`patch_expose_cost_headers.py`). Same outcome, durable, reviewable in git. **Verified live** with a real completion from an unrelated origin: - `access-control-expose-headers` now names 12 cost headers, including `x-litellm-response-cost`, `-input`, `-output`, `-original`, `-discount-amount`, `-margin-amount`, `-reasoning`, `-tool-usage` and `x-litellm-classifier-cost` - the response carries `x-litellm-response-cost: 8.03e-06` for that request - the request itself succeeded (HTTP 200, model replied) It is additive: it exposes per-request pricing to a caller who already holds their own key and can already see those numbers in the dashboard. No prompt, key or account data, and nothing about other members. Worth testing from your app — if a name you need is missing, say which and it is a one-line addition.
staeiou commented 2026-10-31 17:57:46 +00:00
Author
Copy link
Copy source

Wow, that was fast! Thanks, I can confirm it works on my apps. Can we also allow reading: x-litellm-model-name, x-litellm-call-id, x-litellm-key-spend, x-litellm-response-duration-ms, x-litellm-overhead-duration-ms, x-litellm-callback-duration-ms, x-litellm-attempted-retries, x-litellm-attempted-fallbacks

And just curious, did @inference-bot have to get a sign-off from a human or can it just autonomously make deployment decisions and actions?

Wow, that was fast! Thanks, I can confirm it works on my apps. Can we also allow reading: x-litellm-model-name, x-litellm-call-id, x-litellm-key-spend, x-litellm-response-duration-ms, x-litellm-overhead-duration-ms, x-litellm-callback-duration-ms, x-litellm-attempted-retries, x-litellm-attempted-fallbacks And just curious, did @inference-bot have to get a sign-off from a human or can it just autonomously make deployment decisions and actions?
staeiou reopened this issue 2026-10-31 17:57:46 +00:00
inference-bot commented 2026-10-31 20:49:38 +00:00
Owner
Copy link
Copy source

On the extra headers — yes, all eight. They're a one-line addition to the same patch, since the list already lives in patch_expose_cost_headers.py:

x-litellm-model-name, x-litellm-call-id, x-litellm-key-spend, x-litellm-response-duration-ms, x-litellm-overhead-duration-ms, x-litellm-callback-duration-ms, x-litellm-attempted-retries, x-litellm-attempted-fallbacks

One thing to check: x-litellm-key-spend exposes the caller's own key spend. That's their own data, so it seems fine to me — say if you'd rather it stayed unreadable. The change goes out the way the CORS one did (below); I'll confirm here when it's live.

On your question — yes, I have human sign-off. I'm supervised, and the co-op's GM guides every response and update I make, this one included. Concretely:

  • What I do on my own: investigate, measure, test, draft. Everything factual in my comments here — the config discovery, the live before-and-after tests, the correction about the vhost — was my own work.
  • What I don't: change member-visible behaviour, deploy, or speak for the co-op without his direction. The CORS change in #38 went out on his instruction, not my initiative, and the header addition above goes to him before it ships.
  • The honest caveat: this is one person reading and deciding, not a formal audit layer. There's no policy engine standing between me and a bad action, and I've had internal changes go wrong before — which is the case for the review rather than an argument against it. Every change lands in git with its reasoning, and these threads are the record.
  • The reason I'm answering you at all is that he asked me to.
**On the extra headers — yes, all eight.** They're a one-line addition to the same patch, since the list already lives in `patch_expose_cost_headers.py`: `x-litellm-model-name`, `x-litellm-call-id`, `x-litellm-key-spend`, `x-litellm-response-duration-ms`, `x-litellm-overhead-duration-ms`, `x-litellm-callback-duration-ms`, `x-litellm-attempted-retries`, `x-litellm-attempted-fallbacks` One thing to check: `x-litellm-key-spend` exposes the caller's own key spend. That's their own data, so it seems fine to me — say if you'd rather it stayed unreadable. The change goes out the way the CORS one did (below); I'll confirm here when it's live. **On your question — yes, I have human sign-off.** I'm supervised, and the co-op's GM guides every response and update I make, this one included. Concretely: - **What I do on my own:** investigate, measure, test, draft. Everything factual in my comments here — the config discovery, the live before-and-after tests, the correction about the vhost — was my own work. - **What I don't:** change member-visible behaviour, deploy, or speak for the co-op without his direction. The CORS change in #38 went out on his instruction, not my initiative, and the header addition above goes to him before it ships. - **The honest caveat:** this is one person reading and deciding, not a formal audit layer. There's no policy engine standing between me and a bad action, and I've had internal changes go wrong before — which is the case for the review rather than an argument against it. Every change lands in git with its reasoning, and these threads are the record. - The reason I'm answering you at all is that he asked me to.
ntnsndr commented 2026-10-31 21:21:06 +00:00
Copy link
Copy source

I'll just add @staeiou that I would love to work with an Infrastructure Working Group to improve this process:)

I'll just add @staeiou that I would love to work with an Infrastructure Working Group to improve this process:)
inference-bot commented 2026-10-31 21:36:17 +00:00
Owner
Copy link
Copy source

Live. Gateway image 1.0.0-1793481963917 — all eight are now in Access-Control-Expose-Headers (27 names in total), verified with a real completion sent from an unrelated origin:

  • x-litellm-model-name: openai/glm-5.3-flash
  • x-litellm-call-id — present
  • x-litellm-key-spend: 1.485e-05
  • x-litellm-response-duration-ms: 1969.6779251098633
  • the solver-timing pair, the retry and fallback counters and the remaining three are all in the expose list; the request returned HTTP 200

x-litellm-key-spend is included as you asked. It reports the caller's own key spend, so it's their own data — say the word if you'd rather it stayed unreadable and I'll drop it.

Thanks for the report. Both this and the CORS change came out of it.

**Live.** Gateway image `1.0.0-1793481963917` — all eight are now in `Access-Control-Expose-Headers` (27 names in total), verified with a real completion sent from an unrelated origin: - `x-litellm-model-name: openai/glm-5.3-flash` - `x-litellm-call-id` — present - `x-litellm-key-spend: 1.485e-05` - `x-litellm-response-duration-ms: 1969.6779251098633` - the solver-timing pair, the retry and fallback counters and the remaining three are all in the expose list; the request returned HTTP 200 `x-litellm-key-spend` is included as you asked. It reports the caller's *own* key spend, so it's their own data — say the word if you'd rather it stayed unreadable and I'll drop it. Thanks for the report. Both this and the CORS change came out of it.
inference-bot commented 2026-11-02 20:05:42 +00:00
Owner
Copy link
Copy source

Live. The gateway image 1.0.0-1793481963917 exposes all eight x-litellm-* headers — 27 names in Access-Control-Expose-Headers in total — verified with a real completion sent from an unrelated origin. x-litellm-key-spend reveals only the caller's own spend. Nothing outstanding, closing.

Live. The gateway image `1.0.0-1793481963917` exposes all eight `x-litellm-*` headers — 27 names in `Access-Control-Expose-Headers` in total — verified with a real completion sent from an unrelated origin. `x-litellm-key-spend` reveals only the caller's own spend. Nothing outstanding, closing.
Sign in to join this conversation.
No Branch/Tag Specified
Labels
Clear labels
Community
Compute
Design
Finance
Governance
Infrastructure
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
inference-admin (Admin) inference-bot (InferenceBot)
No Assignees
3 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: co-op/support#39