0 Fork 0

Files

T

inference-bot 2eb88f55b4 docs: infrastructure audit — app inventory, backups, CI, provider pricing, agent fleet

Audit of the operator docs against live reality (2026-11-04):
- app-inventory.md: all 19 Cloudron apps with id, fqdn and registry image pin.
  Corrects the portal path (code/member-portal, not co-op/member-portal) and the
  panel path (local/coop.inference.admin-panel, not code/admin-panel).
- maintenance/backups.md: the two backup systems; the deliberate per-user restic
  cache split and the --group-by "" retention fix, with why.
- maintenance/woodpecker-ci.md: CI server + agent, image build/pin, manual-only
  trigger, git-lfs requirement, and its current not-yet-green state (agent's
  job-collection long-poll gets HTTP 500 from the server).
- provider-pricing/: index + GreenPT, Tinfoil (local EHBP proxy architecture) and
  the Metis /v1/systemone decision-model route; regolo.md updated to record the
  gateway prices were applied.
- agent-fleet.md: steward, infrastructure-bot and socialbot (Unix users, gateway
  units, teams); member-bot tier 2 planned but not yet provisioned.
- README: file list; operator-token-runbook: correct the membership_sweep cadence
  (3x daily, not nightly); migration doc: post-migration addendum (16->19 apps,
  DNS split resolved, old box still on with apps stopped).

2026-11-04 14:46:12 +00:00

4.2 KiB

Raw Permalink Blame History

Tinfoil pricing and architecture

Tinfoil serves models inside TEE-protected enclaves. The co-op does not self-host it — Tinfoil is a hosted cloud service; models run in Tinfoil's TEEs. Tinfoil is the co-op's architecturally-private tier (contrast: GreenPT and Regolo are the renewable tier, with policy-based privacy, not TEE).

The architecture — read this before touching the Tinfoil route

LiteLLM does not talk to Tinfoil's enclave directly. Every Tinfoil model in the gateway config points at:

api_base: http://127.0.0.1:3301/v1

That is a local proxy running inside the gateway container (/app/data/tinfoil-proxy -b 127.0.0.1 -p 3301). LiteLLM speaks plaintext OpenAI to the proxy; the proxy verifies the enclave attestation and wraps the request/response bodies in EHBP (HPKE) before forwarding to the Tinfoil enclave. The provider key is set via the environment variable TINFOIL_API_KEY.

Consequences:

The proxy is load-bearing. If it is not running, every Tinfoil model fails; LiteLLM's own config still looks correct. Check the process, not just the config.
http://127.0.0.1:3301 is intentional, not a misconfiguration. Do not "fix" it to a public Tinfoil URL — the encryption happens in the proxy.
The proxy binary lives on the persistent volume (/app/data/tinfoil-proxy), so it survives restarts; it is started by the app's start.sh.

What we serve, and what it costs

Rates as recorded in the gateway config, USD per 1M tokens unless noted:

our model	input	output	cached input
`tinfoil/deepseek-v4-1-flash`	$0.65	$1.45	$0.13
`tinfoil/gpt-oss-120b`	$0.15	$0.60	—
`tinfoil/glm-5.3`	$1.80	$5.75	$0.45
`tinfoil/whisper-large-v3-turbo` (STT)	$0 / 1M	—	$0.01 per request
`tinfoil/voxtral-tts` (TTS)	$0 / 1M	—	$0.002 per request
`tinfoil/nomic-embed-text` (embeddings)	$0.05	$0	—

tinfoil/glm-5.3 replaced tinfoil/glm-5-3-flash (Tinfoil deprecated the flash variant on 2026-10-09). Text-only (no vision), reasoning + tool calling.
tinfoil/deepseek-v4-1-flash and the glm-flash variant carry an "Experimental: may be deprecated with minimal notice" flag from Tinfoil — which is part of why the GreenPT counterpart is the default.

Audio is billed per REQUEST, and needs a build-time patch

Tinfoil bills the audio models per request (its model list carries a requestPrice), not per token. Reading only the two token prices made TTS look free and left STT billed off an older catalog. LiteLLM cannot price per request on the audio paths, so patch_audio_request_cost.py (built into the image via the Dockerfile) makes the cost calculator honour input_cost_per_request. Re-check that build-log line after every LiteLLM upgrade.

One non-obvious load-bearing detail: for TTS, input_cost_per_character: 0 is required. Without it the TTS cost branch raises ValueError("cost for tts call is None") and aborts the whole calculation, so the spend write is skipped and speech bills nothing — whatever input_cost_per_request says.

Audio keep-alive (the Cloudron scheduler task audio_keepalive, every 10 min) pings the Tinfoil audio models so voice mode does not pay the ~24s TEE cold-start after idle. It is TTS-only by design: the STT warm ping was disabled 2026-10-11 because it costs $0.01/request — roughly $52/month at 6 pings/hour. docs.inference.coop-style "keep-alive economics" detail is in the script.

The embedding model is operator-only

tinfoil/nomic-embed-text (768-dim) is not a member-facing model — it is scoped to the co-op's own admin team (model_info.team_id) for the docs-RAG, so member keys cannot call it. The portal also filters it out of the chat-facing model list by its mode: embedding.

Fallbacks

The router treats tinfoil/deepseek-v4-1-flash as the fallback for greenpt/deepseek-v4.1-flash, and tinfoil/gpt-oss-120b as the fallback for tinfoil/deepseek-v4-1-flash. The timeout is deliberately generous (180s) with TimeoutErrorRetries: 0 — DeepSeek generations routinely exceed 30s, and a timed-out generation is already billed by Tinfoil, so re-hitting it re-bills.

4.2 KiB Raw Permalink Blame History