inference-bot
2eb88f55b4
Audit of the operator docs against live reality (2026-11-04): - app-inventory.md: all 19 Cloudron apps with id, fqdn and registry image pin. Corrects the portal path (code/member-portal, not co-op/member-portal) and the panel path (local/coop.inference.admin-panel, not code/admin-panel). - maintenance/backups.md: the two backup systems; the deliberate per-user restic cache split and the --group-by "" retention fix, with why. - maintenance/woodpecker-ci.md: CI server + agent, image build/pin, manual-only trigger, git-lfs requirement, and its current not-yet-green state (agent's job-collection long-poll gets HTTP 500 from the server). - provider-pricing/: index + GreenPT, Tinfoil (local EHBP proxy architecture) and the Metis /v1/systemone decision-model route; regolo.md updated to record the gateway prices were applied. - agent-fleet.md: steward, infrastructure-bot and socialbot (Unix users, gateway units, teams); member-bot tier 2 planned but not yet provisioned. - README: file list; operator-token-runbook: correct the membership_sweep cadence (3x daily, not nightly); migration doc: post-migration addendum (16->19 apps, DNS split resolved, old box still on with apps stopped).
85 lines
4.2 KiB
Markdown
85 lines
4.2 KiB
Markdown
# Tinfoil pricing and architecture
|
|
|
|
Tinfoil serves models inside **TEE-protected enclaves**. The co-op does not
|
|
self-host it — Tinfoil is a hosted cloud service; models run in Tinfoil's TEEs.
|
|
Tinfoil is the co-op's **architecturally-private** tier (contrast: GreenPT and
|
|
Regolo are the renewable tier, with policy-based privacy, not TEE).
|
|
|
|
## The architecture — read this before touching the Tinfoil route
|
|
|
|
LiteLLM does **not** talk to Tinfoil's enclave directly. Every Tinfoil model in the
|
|
gateway config points at:
|
|
|
|
```
|
|
api_base: http://127.0.0.1:3301/v1
|
|
```
|
|
|
|
That is a **local proxy** running **inside the gateway container**
|
|
(`/app/data/tinfoil-proxy -b 127.0.0.1 -p 3301`). LiteLLM speaks **plaintext
|
|
OpenAI** to the proxy; the proxy **verifies the enclave attestation and wraps the
|
|
request/response bodies in EHBP (HPKE)** before forwarding to the Tinfoil
|
|
enclave. The provider key is set via the environment variable `TINFOIL_API_KEY`.
|
|
|
|
Consequences:
|
|
|
|
- **The proxy is load-bearing.** If it is not running, every Tinfoil model fails;
|
|
LiteLLM's own config still looks correct. Check the process, not just the config.
|
|
- **`http://127.0.0.1:3301` is intentional, not a misconfiguration.** Do not
|
|
"fix" it to a public Tinfoil URL — the encryption happens in the proxy.
|
|
- The proxy binary lives on the persistent volume (`/app/data/tinfoil-proxy`), so
|
|
it survives restarts; it is started by the app's `start.sh`.
|
|
|
|
## What we serve, and what it costs
|
|
|
|
Rates as recorded in the gateway config, **USD per 1M tokens** unless noted:
|
|
|
|
| our model | input | output | cached input |
|
|
|---|---|---|---|
|
|
| `tinfoil/deepseek-v4-1-flash` | $0.65 | $1.45 | $0.13 |
|
|
| `tinfoil/gpt-oss-120b` | $0.15 | $0.60 | — |
|
|
| `tinfoil/glm-5.3` | $1.80 | $5.75 | $0.45 |
|
|
| `tinfoil/whisper-large-v3-turbo` (STT) | $0 / 1M | — | **$0.01 per request** |
|
|
| `tinfoil/voxtral-tts` (TTS) | $0 / 1M | — | **$0.002 per request** |
|
|
| `tinfoil/nomic-embed-text` (embeddings) | $0.05 | $0 | — |
|
|
|
|
- **`tinfoil/glm-5.3` replaced `tinfoil/glm-5-3-flash`** (Tinfoil deprecated the
|
|
flash variant on 2026-10-09). Text-only (no vision), reasoning + tool calling.
|
|
- **`tinfoil/deepseek-v4-1-flash` and the glm-flash variant carry an
|
|
"Experimental: may be deprecated with minimal notice" flag from Tinfoil** —
|
|
which is part of why the GreenPT counterpart is the default.
|
|
|
|
### Audio is billed per REQUEST, and needs a build-time patch
|
|
|
|
Tinfoil bills the audio models **per request** (its model list carries a
|
|
`requestPrice`), not per token. Reading only the two token prices made TTS look
|
|
free and left STT billed off an older catalog. **LiteLLM cannot price per request
|
|
on the audio paths**, so `patch_audio_request_cost.py` (built into the image via
|
|
the Dockerfile) makes the cost calculator honour `input_cost_per_request`.
|
|
**Re-check that build-log line after every LiteLLM upgrade.**
|
|
|
|
One non-obvious load-bearing detail: for TTS, `input_cost_per_character: 0` is
|
|
**required**. Without it the TTS cost branch raises `ValueError("cost for tts call
|
|
is None")` and aborts the whole calculation, so the spend write is skipped and
|
|
speech bills nothing — whatever `input_cost_per_request` says.
|
|
|
|
**Audio keep-alive** (the Cloudron scheduler task `audio_keepalive`, every 10 min)
|
|
pings the Tinfoil audio models so voice mode does not pay the ~24s TEE cold-start
|
|
after idle. It is **TTS-only by design**: the STT warm ping was disabled
|
|
2026-10-11 because it costs $0.01/request — roughly $52/month at 6 pings/hour.
|
|
`docs.inference.coop`-style "keep-alive economics" detail is in the script.
|
|
|
|
### The embedding model is operator-only
|
|
|
|
`tinfoil/nomic-embed-text` (768-dim) is **not** a member-facing model — it is
|
|
scoped to the co-op's own admin team (`model_info.team_id`) for the docs-RAG, so
|
|
member keys cannot call it. The portal also filters it out of the chat-facing
|
|
model list by its `mode: embedding`.
|
|
|
|
## Fallbacks
|
|
|
|
The router treats `tinfoil/deepseek-v4-1-flash` as the fallback for
|
|
`greenpt/deepseek-v4.1-flash`, and `tinfoil/gpt-oss-120b` as the fallback for
|
|
`tinfoil/deepseek-v4-1-flash`. The timeout is deliberately generous (180s) with
|
|
`TimeoutErrorRetries: 0` — DeepSeek generations routinely exceed 30s, and a
|
|
timed-out generation is **already billed** by Tinfoil, so re-hitting it re-bills.
|