docs: infrastructure audit — app inventory, backups, CI, provider pricing, agent fleet
Audit of the operator docs against live reality (2026-11-04): - app-inventory.md: all 19 Cloudron apps with id, fqdn and registry image pin. Corrects the portal path (code/member-portal, not co-op/member-portal) and the panel path (local/coop.inference.admin-panel, not code/admin-panel). - maintenance/backups.md: the two backup systems; the deliberate per-user restic cache split and the --group-by "" retention fix, with why. - maintenance/woodpecker-ci.md: CI server + agent, image build/pin, manual-only trigger, git-lfs requirement, and its current not-yet-green state (agent's job-collection long-poll gets HTTP 500 from the server). - provider-pricing/: index + GreenPT, Tinfoil (local EHBP proxy architecture) and the Metis /v1/systemone decision-model route; regolo.md updated to record the gateway prices were applied. - agent-fleet.md: steward, infrastructure-bot and socialbot (Unix users, gateway units, teams); member-bot tier 2 planned but not yet provisioned. - README: file list; operator-token-runbook: correct the membership_sweep cadence (3x daily, not nightly); migration doc: post-migration addendum (16->19 apps, DNS split resolved, old box still on with apps stopped).
This commit is contained in:
inference-bot committed
1 parent
8b33d823c1
commit
2eb88f55b4
12 files changed
+688
-3
No files matched your search
@@ -0,0 +1,44 @@
|
||||
# Provider pricing
|
||||
|
||||
What our model providers charge, and where the authoritative numbers live.
|
||||
|
||||
## The authority
|
||||
|
||||
**The gateway's own per-model costs are the authority** — they are what the
|
||||
response headers (`x-litellm-response-cost`) and member billing both read. They
|
||||
live in the LiteLLM config on the gateway's persistent volume
|
||||
(`/app/data/config.yaml`), under each model's `model_info`, as **USD per token**
|
||||
(the comments in that file quote per-1M rates). Read them live with
|
||||
`GET https://gateway.inference.coop/model/info` (master key), or read the config
|
||||
straight from the running app.
|
||||
|
||||
**The member-facing mirror** is `co-op/docs/models.md`. When a rate changes,
|
||||
update **both** the gateway config and `models.md` — the config is authoritative,
|
||||
`models.md` is the mirror.
|
||||
|
||||
A missing cost field makes a paid feature free; a stale one bills wrong; a missing
|
||||
`cache_read_input_token_cost` bills cache hits at the full input rate. All three
|
||||
have happened here.
|
||||
|
||||
## The pages
|
||||
|
||||
| File | Provider | Notes |
|
||||
|---|---|---|
|
||||
| `greenpt.md` | GreenPT | renewable energy; the default model lives here |
|
||||
| `tinfoil.md` | Tinfoil | TEE; the local-proxy architecture lives here |
|
||||
| `regolo.md` | Regolo | renewable + ZDR; includes our measured usage |
|
||||
| `decision-model.md` | GreenPT "Metis" via `/v1/systemone` | a pass-through **route**, not a chat model |
|
||||
|
||||
## The routing rule
|
||||
|
||||
Members reach models through one gateway, which fronts all providers. The
|
||||
per-request cost the gateway records is the number to trust; do not re-derive a
|
||||
price from a provider's public page (they drift, and some are login-walled).
|
||||
|
||||
## A note on currencies
|
||||
|
||||
Regolo quotes in **EUR**; our gateway figures convert at **EUR × 1.16 → USD**
|
||||
(the rate our numbers use — `regolo.md` documents the derivation). GreenPT's own
|
||||
listed rates appear in **EUR with the same numerals the gateway carries** —
|
||||
whether an EUR→USD conversion is intended there is **unverified**; confirm it
|
||||
against the GreenPT invoice before treating the figures as final.
|
||||
@@ -0,0 +1,58 @@
|
||||
# The decision-model route — GreenPT "Metis" (`/v1/systemone`)
|
||||
|
||||
The gateway exposes a **decision model** — GreenPT's **Metis** (System One API) —
|
||||
as a **pass-through route**, not a chat model.
|
||||
|
||||
## What it is, and why it is a route and not a model
|
||||
|
||||
Metis is a **decision** endpoint, not a chat completion: it takes a `state` plus up
|
||||
to 32 typed questions and returns **probabilities / choices / scores**. It
|
||||
generates no text (`output_tokens` is always 0), so **it cannot be a `model_list`
|
||||
entry** — LiteLLM's chat pipeline has nothing to parse. It is therefore exposed as
|
||||
a pass-through:
|
||||
|
||||
```
|
||||
path: /v1/systemone
|
||||
target: https://api.greenpt.ai/v1/systemone
|
||||
auth: true # members call it with their normal co-op API key
|
||||
```
|
||||
|
||||
- **It is API-only and deliberately NOT a chat-picker model.** It never appears in
|
||||
the chat model list. Do not add it there.
|
||||
- **Members authenticate with their own co-op key**, never the provider key — the
|
||||
route's `Authorization` header injects the provider key server-side.
|
||||
- Hosted by GreenPT (policy-based privacy, EU, 100% renewable), same
|
||||
`GREENPT_API_KEY` the green chat models use.
|
||||
|
||||
## Pricing and the metering fix
|
||||
|
||||
- **$0.12 per 1M input tokens; output is free.**
|
||||
|
||||
Metering is **exact, from the token count GreenPT reports back**. Because Metis is
|
||||
not a `model_list` entry, it is priced through the
|
||||
`coop_passthrough_cost_per_token` map, keyed by the route path
|
||||
(`/v1/systemone: 0.00000012`). `patch_passthrough_token_cost.py` (built into the
|
||||
image) reads `usage.input_tokens` from the Metis response body at logging time and
|
||||
writes the exact figure to member spend.
|
||||
|
||||
**Why this exists — record it, it will recur:** **generic pass-through routes
|
||||
originally logged nothing.** A route that is not a `model_list` entry gets no
|
||||
automatic pricing, so its calls were metered as $0 — a paid feature silently free.
|
||||
The pass-through accounting had to be fixed so spend lands against the **calling
|
||||
member's key**. If another pass-through route is ever added, it needs the same
|
||||
treatment or it bills nothing.
|
||||
|
||||
`cost_per_request: 0.0001` on the route is now only a **fallback** — it applies if
|
||||
a response arrives without a usable token count, so a call is never silently free.
|
||||
Keep the two in sync.
|
||||
|
||||
## Access control gotcha (community edition)
|
||||
|
||||
The route is opened to any authenticated member key via
|
||||
`coop_passthrough_open_to_keys` (consumed at build time by
|
||||
`patch_passthrough_key_access.py`). **LiteLLM's own gate for `auth=true`
|
||||
pass-through routes (`allowed_passthrough_routes`) is an Enterprise-only field**,
|
||||
so on this community-edition proxy it can never be granted to member keys. The
|
||||
only alternative, `auth: false`, would expose the route publicly and forward the
|
||||
co-op's GreenPT key to anonymous callers. **Authentication is unchanged** — the
|
||||
endpoint still requires a valid key.
|
||||
@@ -0,0 +1,44 @@
|
||||
# GreenPT pricing
|
||||
|
||||
GreenPT is OpenAI-compatible, EU-hosted on **100% renewable energy**. LiteLLM
|
||||
talks to it **directly** (no TEE proxy). Billing across the gateway:
|
||||
|
||||
- `api_base`: `https://api.greenpt.ai/v1`
|
||||
- key: `GREENPT_API_KEY` (gateway app env)
|
||||
|
||||
## What we serve, and what it costs
|
||||
|
||||
Rates as recorded in the gateway's authorititative config comment, **USD per 1M
|
||||
tokens** (the gateway's `model_info` carries the per-token equivalents):
|
||||
|
||||
| our model | input | output | cached input |
|
||||
|---|---|---|---|
|
||||
| `greenpt/deepseek-v4.1-flash` **(DEFAULT model)** | $0.22 | $1.10 | $0.011 |
|
||||
| `greenpt/glm-5.3-flash` | $0.11 | $0.44 | $0.022 |
|
||||
| `greenpt/glm-5.3` | $1.10 | $4.40 | $0.275 |
|
||||
|
||||
- **`greenpt/deepseek-v4.1-flash` is the gateway's default model.** It is the same
|
||||
model family as Tinfoil's `tinfoil/deepseek-v4-1-flash`, but on renewable energy
|
||||
and **not** marked "Experimental" by the provider (Tinfoil's counterpart is), so
|
||||
it is the default and Tinfoil's is its private counterpart. The policy trade is
|
||||
**policy-based privacy (GreenPT) vs TEE (Tinfoil)** — say so plainly rather than
|
||||
implying GreenPT is confidential computing.
|
||||
- `greenpt/glm-5.3` is the strongest **agentic** model in the green tier (1M context).
|
||||
- Note the **dotted slug** `glm-5.3-flash` on GreenPT, distinct from Tinfoil's
|
||||
`glm-5-3-flash` (hyphenated). Getting a model slug wrong is a silent 404 at call
|
||||
time.
|
||||
|
||||
## Historical
|
||||
|
||||
- **Removed 2026-10:** `green-l` / `green-r` — GreenPT now marks them "not
|
||||
recommended".
|
||||
- The config comment previously read "€0.22 in / €1.10 out" alongside the dollar
|
||||
figures — see the currency note in `README.md`; the intended FX is unverified.
|
||||
|
||||
## The fallback key
|
||||
|
||||
`GREENPT_FALLBACK_API_KEY` (agent host) is the **same value** as the gateway's
|
||||
`GREENPT_API_KEY` — verified by hash comparison. It powers the steward agent's
|
||||
direct-to-provider fallback during a gateway outage. **Rotating one without the
|
||||
other silently breaks the fallback** (a dedicated key is the cleaner end state).
|
||||
See the token runbook.
|
||||
@@ -97,3 +97,21 @@ At these volumes the mispricing is small: apertus's input roughly 2.5x-ing on a
|
||||
**$0.82 lifetime** is about **$1/month** of exposure, and qwen's cut moves ~$0.6/month in members'
|
||||
favour. So this is **billing accuracy, not urgency** — but accuracy still matters, because the same
|
||||
numbers feed the cost headers members can now read.
|
||||
|
||||
## UPDATE 2026-11-04 — the gateway was updated; both stale sections above are now resolved
|
||||
|
||||
Checked against the live gateway config (`/app/data/config.yaml`, `model_info`):
|
||||
|
||||
- `regolo/apertus-70b` = **1.16e-06 / 3.944e-06** per token
|
||||
($1.16 / $3.94 per 1M) — the new 2026-11-01 rate. The "~70 cent/M loss while the
|
||||
gateway still bills $0.46" paragraph above no longer applies.
|
||||
- `regolo/qwen3.5-122b` = **6.96e-07 / 3.944e-06** ($0.70 / $3.94 per 1M) — the new
|
||||
rate. The "over-bills members by ~46 cents/M" paragraph no longer applies.
|
||||
- `regolo/qwen-image` now carries **`input_cost_per_pixel: 1.74e-8`** — the EUR
|
||||
figure converted at ×1.16, i.e. the "image model is priced in EUR-as-USD" open
|
||||
inconsistency above is **fixed**. It is no longer priced at the raw EUR number.
|
||||
|
||||
So the two "Until the gateway is updated" tables and the image-model section above
|
||||
are **historical**. The usage measurement ($4.38/month, ~1.1% of inference spend)
|
||||
and the subscription analysis stand. Keep this file dated-section-per-change; do
|
||||
not delete the history, it shows the reasoning.
|
||||
@@ -0,0 +1,84 @@
|
||||
# Tinfoil pricing and architecture
|
||||
|
||||
Tinfoil serves models inside **TEE-protected enclaves**. The co-op does not
|
||||
self-host it — Tinfoil is a hosted cloud service; models run in Tinfoil's TEEs.
|
||||
Tinfoil is the co-op's **architecturally-private** tier (contrast: GreenPT and
|
||||
Regolo are the renewable tier, with policy-based privacy, not TEE).
|
||||
|
||||
## The architecture — read this before touching the Tinfoil route
|
||||
|
||||
LiteLLM does **not** talk to Tinfoil's enclave directly. Every Tinfoil model in the
|
||||
gateway config points at:
|
||||
|
||||
```
|
||||
api_base: http://127.0.0.1:3301/v1
|
||||
```
|
||||
|
||||
That is a **local proxy** running **inside the gateway container**
|
||||
(`/app/data/tinfoil-proxy -b 127.0.0.1 -p 3301`). LiteLLM speaks **plaintext
|
||||
OpenAI** to the proxy; the proxy **verifies the enclave attestation and wraps the
|
||||
request/response bodies in EHBP (HPKE)** before forwarding to the Tinfoil
|
||||
enclave. The provider key is set via the environment variable `TINFOIL_API_KEY`.
|
||||
|
||||
Consequences:
|
||||
|
||||
- **The proxy is load-bearing.** If it is not running, every Tinfoil model fails;
|
||||
LiteLLM's own config still looks correct. Check the process, not just the config.
|
||||
- **`http://127.0.0.1:3301` is intentional, not a misconfiguration.** Do not
|
||||
"fix" it to a public Tinfoil URL — the encryption happens in the proxy.
|
||||
- The proxy binary lives on the persistent volume (`/app/data/tinfoil-proxy`), so
|
||||
it survives restarts; it is started by the app's `start.sh`.
|
||||
|
||||
## What we serve, and what it costs
|
||||
|
||||
Rates as recorded in the gateway config, **USD per 1M tokens** unless noted:
|
||||
|
||||
| our model | input | output | cached input |
|
||||
|---|---|---|---|
|
||||
| `tinfoil/deepseek-v4-1-flash` | $0.65 | $1.45 | $0.13 |
|
||||
| `tinfoil/gpt-oss-120b` | $0.15 | $0.60 | — |
|
||||
| `tinfoil/glm-5.3` | $1.80 | $5.75 | $0.45 |
|
||||
| `tinfoil/whisper-large-v3-turbo` (STT) | $0 / 1M | — | **$0.01 per request** |
|
||||
| `tinfoil/voxtral-tts` (TTS) | $0 / 1M | — | **$0.002 per request** |
|
||||
| `tinfoil/nomic-embed-text` (embeddings) | $0.05 | $0 | — |
|
||||
|
||||
- **`tinfoil/glm-5.3` replaced `tinfoil/glm-5-3-flash`** (Tinfoil deprecated the
|
||||
flash variant on 2026-10-09). Text-only (no vision), reasoning + tool calling.
|
||||
- **`tinfoil/deepseek-v4-1-flash` and the glm-flash variant carry an
|
||||
"Experimental: may be deprecated with minimal notice" flag from Tinfoil** —
|
||||
which is part of why the GreenPT counterpart is the default.
|
||||
|
||||
### Audio is billed per REQUEST, and needs a build-time patch
|
||||
|
||||
Tinfoil bills the audio models **per request** (its model list carries a
|
||||
`requestPrice`), not per token. Reading only the two token prices made TTS look
|
||||
free and left STT billed off an older catalog. **LiteLLM cannot price per request
|
||||
on the audio paths**, so `patch_audio_request_cost.py` (built into the image via
|
||||
the Dockerfile) makes the cost calculator honour `input_cost_per_request`.
|
||||
**Re-check that build-log line after every LiteLLM upgrade.**
|
||||
|
||||
One non-obvious load-bearing detail: for TTS, `input_cost_per_character: 0` is
|
||||
**required**. Without it the TTS cost branch raises `ValueError("cost for tts call
|
||||
is None")` and aborts the whole calculation, so the spend write is skipped and
|
||||
speech bills nothing — whatever `input_cost_per_request` says.
|
||||
|
||||
**Audio keep-alive** (the Cloudron scheduler task `audio_keepalive`, every 10 min)
|
||||
pings the Tinfoil audio models so voice mode does not pay the ~24s TEE cold-start
|
||||
after idle. It is **TTS-only by design**: the STT warm ping was disabled
|
||||
2026-10-11 because it costs $0.01/request — roughly $52/month at 6 pings/hour.
|
||||
`docs.inference.coop`-style "keep-alive economics" detail is in the script.
|
||||
|
||||
### The embedding model is operator-only
|
||||
|
||||
`tinfoil/nomic-embed-text` (768-dim) is **not** a member-facing model — it is
|
||||
scoped to the co-op's own admin team (`model_info.team_id`) for the docs-RAG, so
|
||||
member keys cannot call it. The portal also filters it out of the chat-facing
|
||||
model list by its `mode: embedding`.
|
||||
|
||||
## Fallbacks
|
||||
|
||||
The router treats `tinfoil/deepseek-v4-1-flash` as the fallback for
|
||||
`greenpt/deepseek-v4.1-flash`, and `tinfoil/gpt-oss-120b` as the fallback for
|
||||
`tinfoil/deepseek-v4-1-flash`. The timeout is deliberately generous (180s) with
|
||||
`TimeoutErrorRetries: 0` — DeepSeek generations routinely exceed 30s, and a
|
||||
timed-out generation is **already billed** by Tinfoil, so re-hitting it re-bills.
|
||||
Reference in new issue
Block a user