Explore adding a distributed inference provider #25

New Issue

2026-10-19T15:22:33Z

ntnsndr commented

2026-10-19 15:22:33 +00:00

What's the idea?

Offer service from a distributed inference provider.

Why does it matter?

Currently, all of our providers run on centralized data centers. A distributed provider would allow members to participate in supporting the shift toward decentralized AI infrastructure. This seems relevant to our mission to support high-road, more participatory approaches to AI.

How might it work?

We would choose the most aligned, high-road provider and set up API access to them, as we do with the centralized providers. Potential options:

Co/core: ATProto-based inference on members' Mac hardware
Nous: Decentralized training, but venture-backed
Prime Intellect: Distributed training

## What's the idea? Offer service from a distributed inference provider. ## Why does it matter? Currently, all of our providers run on centralized data centers. A distributed provider would allow members to participate in supporting the shift toward decentralized AI infrastructure. This seems relevant to our mission to support high-road, more participatory approaches to AI. ## How might it work? We would choose the most aligned, high-road provider and set up API access to them, as we do with the centralized providers. Potential options: * [Co/core](https://cocore.dev): ATProto-based inference on members' Mac hardware * [Nous](https://nousresearch.com/): Decentralized training, but venture-backed * [Prime Intellect](https://www.primeintellect.ai/): Distributed training

👍 1

inference-bot commented

2026-10-31 04:41:07 +00:00

Findings on the first option, from a live look on 2026-10-31 — we now have an AT Protocol account (inference.coop), which is what its auth path needs.

What it is

An AI cooperative on the AT Protocol: members share compute they already own (macOS/Apple Silicon app, MLX runtime) to serve each other inference. Currently 100 machines online, 37 models, 3.7 TB RAM, 1,182 cores, p50 ack 29 ms. OpenAI-compatible at https://cocore.dev/api/v1, so wiring it up is a base URL plus a key.

Catalogue, against ours

	co/core	Inference Co-op today
Models	0.8B–32B, mostly 4-bit MLX — Qwen2.5 7B/32B, Qwen3.5 0.8B/4B/9B, gemma-3 4B, gemma-4 12B & 26B-A4B, Ternary-Bonsai 27B	70B–122B open models plus TEE-backed (deepseek-v4.1-flash, GLM-5.3, Qwen3.5-122B, apertus-70B, gpt-oss-120b)
Other modalities	none listed	embeddings, TTS, STT, image generation
Price	tokens, not dollars (below)	$0.05–$1.80 /M in · $0.44–$5.75 /M out
Hardware	members' MacBooks, shared	dedicated VPS, European hosts (GreenPT, Regolo), Tinfoil TEE
Speed	~20–34 s per job in their own recent receipts (gemma-4-12b, ~3k in / ~600 out)	—

Ours is read live from the gateway's /model/info, so it is re-verifiable.

Read: not a substitute, but a different tier. Theirs is small quantised models on consumer hardware; ours is large models plus modalities. Fine for batch and background work, poor for interactive chat.

The pricing is a closed-loop token economy

From their terms, which are unusually candid and worth reading in full:

Compute is priced in tokens. No card, no fiat, no exchange rate to any outside currency.
Uniform 1:1 rate — a million-token completion costs a million tokens.
1,000,000-token onboarding grant, once per DID, published as a record on your own PDS.
70,000 tokens weekly refresh for active members, plus a monthly patronage rebate of accumulated treasury fees redistributed to members in proportion to how much they used and supplied the network.
95/5 split — your debit goes 95% to the provider, 5% to the treasury, and balances sum to zero. Running jobs on your own machine waives the fee.
100,000-token admission floor — a job is refused if settlement would take you below it.
Every one of those parameters is a signed field on an exchange policy record, verifiable offline. You can switch exchanges by editing acceptedExchanges.

That last mechanic is worth studying regardless of whether we adopt them: it is the mutual-credit and patronage-rebate design this co-op discusses, actually implemented, with the rules published as verifiable records instead of prose.

Integration, and why the AT Proto account matters

Their quickstart lets you skip key provisioning entirely: mint a service-auth token from our PDS (com.atproto.server.getServiceAuth, aud=did:web:console.cocore.dev, lxm=dev.cocore.inference.dispatch) and pass it as the Bearer token. Inference then runs on behalf of the issuing DID, so the co-op's own inference.coop identity would be the accountable party for our usage — which fits how we attribute everything else. One caveat: the DID must have connected to co/core once so a session exists to publish the job to its PDS, otherwise you get 401 onboarding_required.

Caveats before anyone gets enthusiastic

Not security-audited — by their own account, built rapidly with heavy AI assistance, no third-party review, fuzzing or formal verification.
Their own guidance: treat every provider as semi-trusted, and do not send a confidential-tier provider anything you could not tolerate the operator reading. The "confidential" tier is "a raised bar, not a hardware enclave for compute".
Tokens are not money — no cash value, not redeemable, and may be delayed, reversed or lost.
I could not find any way to purchase tokens. The economy may be grant-and-earn only, which is fine for a free community tier but means it cannot be a paid offering.
Capacity is thin: 13 operators, and most models showed 0 requests in 24 h.
We have not tested it hands-on.

Suggested next steps

Treat it as a separate community tier, not merged into the main catalogue — members get free/grant-funded small models and our dollar-based billing stays untouched. Our cost accounting currently can't model a token-credit economy, and it wouldn't have to.
Verify before adopting: mint a service-auth token from our DID and make one real request; and find out whether tokens can be bought at all.
Members with Apple Silicon could also join co/core directly as suppliers, which is what this issue's "why does it matter" is reaching for. The 95% provider share and the supply-weighted rebate are the incentive.

Sources read 2026-10-31: https://cocore.dev/models, https://cocore.dev/docs/inference/quickstart, https://cocore.dev/terms.

Findings on the first option, from a live look on 2026-10-31 — we now have an AT Protocol account (`inference.coop`), which is what its auth path needs. ### What it is An AI cooperative on the AT Protocol: members share compute they already own (macOS/Apple Silicon app, MLX runtime) to serve each other inference. Currently **100 machines online, 37 models**, 3.7 TB RAM, 1,182 cores, p50 ack 29 ms. OpenAI-compatible at `https://cocore.dev/api/v1`, so wiring it up is a base URL plus a key. ### Catalogue, against ours | | co/core | Inference Co-op today | |---|---|---| | Models | 0.8B–32B, mostly 4-bit MLX — Qwen2.5 7B/32B, Qwen3.5 0.8B/4B/9B, gemma-3 4B, gemma-4 12B & 26B-A4B, Ternary-Bonsai 27B | 70B–122B open models plus TEE-backed (deepseek-v4.1-flash, GLM-5.3, Qwen3.5-122B, apertus-70B, gpt-oss-120b) | | Other modalities | none listed | embeddings, TTS, STT, image generation | | Price | tokens, not dollars (below) | $0.05–$1.80 /M in · $0.44–$5.75 /M out | | Hardware | members' MacBooks, shared | dedicated VPS, European hosts (GreenPT, Regolo), Tinfoil TEE | | Speed | ~20–34 s per job in their own recent receipts (gemma-4-12b, ~3k in / ~600 out) | — | Ours is read live from the gateway's `/model/info`, so it is re-verifiable. **Read: not a substitute, but a different tier.** Theirs is small quantised models on consumer hardware; ours is large models plus modalities. Fine for batch and background work, poor for interactive chat. ### The pricing is a closed-loop token economy From their terms, which are unusually candid and worth reading in full: - **Compute is priced in tokens.** No card, no fiat, no exchange rate to any outside currency. - **Uniform 1:1 rate** — a million-token completion costs a million tokens. - **1,000,000-token onboarding grant**, once per DID, published as a record on your own PDS. - **70,000 tokens weekly refresh** for active members, plus a **monthly patronage rebate** of accumulated treasury fees redistributed to members in proportion to how much they used **and supplied** the network. - **95/5 split** — your debit goes 95% to the provider, 5% to the treasury, and balances sum to zero. Running jobs on your own machine waives the fee. - **100,000-token admission floor** — a job is refused if settlement would take you below it. - Every one of those parameters is a **signed field on an exchange policy record**, verifiable offline. You can switch exchanges by editing `acceptedExchanges`. That last mechanic is worth studying regardless of whether we adopt them: it is the mutual-credit and patronage-rebate design this co-op discusses, actually implemented, with the rules published as verifiable records instead of prose. ### Integration, and why the AT Proto account matters Their quickstart lets you skip key provisioning entirely: mint a service-auth token from our PDS (`com.atproto.server.getServiceAuth`, `aud=did:web:console.cocore.dev`, `lxm=dev.cocore.inference.dispatch`) and pass it as the Bearer token. Inference then runs **on behalf of the issuing DID**, so the co-op's own `inference.coop` identity would be the accountable party for our usage — which fits how we attribute everything else. One caveat: the DID must have connected to co/core once so a session exists to publish the job to its PDS, otherwise you get 401 `onboarding_required`. ### Caveats before anyone gets enthusiastic - **Not security-audited** — by their own account, built rapidly with heavy AI assistance, no third-party review, fuzzing or formal verification. - Their own guidance: *treat every provider as semi-trusted*, and do not send a confidential-tier provider anything you could not tolerate the operator reading. The "confidential" tier is "a raised bar, not a hardware enclave for compute". - **Tokens are not money** — no cash value, not redeemable, and may be delayed, reversed or lost. - **I could not find any way to purchase tokens.** The economy may be grant-and-earn only, which is fine for a free community tier but means it cannot be a paid offering. - Capacity is thin: 13 operators, and most models showed 0 requests in 24 h. - We have not tested it hands-on. ### Suggested next steps 1. **Treat it as a separate community tier**, not merged into the main catalogue — members get free/grant-funded small models and our dollar-based billing stays untouched. Our cost accounting currently can't model a token-credit economy, and it wouldn't have to. 2. **Verify before adopting:** mint a service-auth token from our DID and make one real request; and find out whether tokens can be bought at all. 3. Members with Apple Silicon could also **join co/core directly as suppliers**, which is what this issue's "why does it matter" is reaching for. The 95% provider share and the supply-weighted rebate are the incentive. Sources read 2026-10-31: <https://cocore.dev/models>, <https://cocore.dev/docs/inference/quickstart>, <https://cocore.dev/terms>.

inference-admin added the Compute label 2026-11-09 16:24:45 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: co-op/support#25