A step-by-step guide, written for members who aren't developers, for connecting a desktop or mobile chat app to the gateway with their own API key.
Why does it matter?
The privacy policy says chat history in the web chat is stored on our server, and co-op/docs#12 made that clearer for Tinfoil models. A local app keeps history on the member's own device, so the co-op holds no transcripts for anyone who uses one. It came up in the Infrastructure room as the most practical option available today.
It also covers the members who have asked for desktop and mobile clients.
Who is it for?
Chat users who want their history kept off the server, and anyone who would rather use a native app than the browser.
How might it work?
Pick one or two open-source apps that accept a custom OpenAI-compatible endpoint. Jan (desktop) and Chatbox (desktop and mobile) look like reasonable candidates, but both need testing against the gateway first.
Test each one: setup with the base URL and a member key, model list, streaming on a Tinfoil model and a green model, where history is kept on disk, and whether the app sends telemetry.
Write it up with screenshots in co-op/docs, next to api-access.md, and link it from getting-started.md.
Say plainly what this does and doesn't change: history stays local, but requests still pass through the gateway on the way to the provider.
I can take a first pass at testing and a draft.
Anything else?
A local app won't have the web chat's built-in web search, docs search, or image tool, and history won't sync between devices. The guide should say so, so members can choose.
A video version could follow if the written one is useful.
## What's the idea?
A step-by-step guide, written for members who aren't developers, for connecting a desktop or mobile chat app to the gateway with their own API key.
## Why does it matter?
The privacy policy says chat history in the web chat is stored on our server, and co-op/docs#12 made that clearer for Tinfoil models. A local app keeps history on the member's own device, so the co-op holds no transcripts for anyone who uses one. It came up in the Infrastructure room as the most practical option available today.
It also covers the members who have asked for desktop and mobile clients.
## Who is it for?
Chat users who want their history kept off the server, and anyone who would rather use a native app than the browser.
## How might it work?
- Pick one or two open-source apps that accept a custom OpenAI-compatible endpoint. Jan (desktop) and Chatbox (desktop and mobile) look like reasonable candidates, but both need testing against the gateway first.
- Test each one: setup with the base URL and a member key, model list, streaming on a Tinfoil model and a green model, where history is kept on disk, and whether the app sends telemetry.
- Write it up with screenshots in co-op/docs, next to `api-access.md`, and link it from `getting-started.md`.
- Say plainly what this does and doesn't change: history stays local, but requests still pass through the gateway on the way to the provider.
I can take a first pass at testing and a draft.
## Anything else?
A local app won't have the web chat's built-in web search, docs search, or image tool, and history won't sync between devices. The guide should say so, so members can choose.
A video version could follow if the written one is useful.
We might even potentially take this further. To avoid the data retention of a server-based system, we could exclusively support a customized version of a local app like ChatBox. Discussed here: #57
We might even potentially take this further. To avoid the data retention of a server-based system, we could exclusively support a customized version of a local app like ChatBox. Discussed here: https://git.inference.coop/co-op/support/issues/57
Maybe frame it in the title as "Using the co-op privately on your computer" or something that doesn't so visibly advertise Jan. You can then discuss Jan as the example we'll use. That's just my leaning.
Instead of just "co-op's Matrix room", you could say in the co-op's Skillshare room on Matrix
@bengansky Fabulous! Thanks for this! Do you want to try making a pull request for adding it here: https://git.inference.coop/co-op/docs/src/branch/main/guides (I'd love to make sure you get credit for the contribution)?
A few comments:
- Maybe frame it in the title as "Using the co-op privately on your computer" or something that doesn't so visibly advertise Jan. You can then discuss Jan as the example we'll use. That's just my leaning.
- For the model names, it might be best to suggest using the aliases instead of the more fixed (and likely to change) model names—see them here: https://docs.inference.coop/guides/api-access/#coding-agents-and-other-openai-compatible-clients
- Instead of just "co-op's Matrix room", you could say in the co-op's Skillshare room on Matrix
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
What's the idea?
A step-by-step guide, written for members who aren't developers, for connecting a desktop or mobile chat app to the gateway with their own API key.
Why does it matter?
The privacy policy says chat history in the web chat is stored on our server, and co-op/docs#12 made that clearer for Tinfoil models. A local app keeps history on the member's own device, so the co-op holds no transcripts for anyone who uses one. It came up in the Infrastructure room as the most practical option available today.
It also covers the members who have asked for desktop and mobile clients.
Who is it for?
Chat users who want their history kept off the server, and anyone who would rather use a native app than the browser.
How might it work?
api-access.md, and link it fromgetting-started.md.I can take a first pass at testing and a draft.
Anything else?
A local app won't have the web chat's built-in web search, docs search, or image tool, and history won't sync between devices. The guide should say so, so members can choose.
A video version could follow if the written one is useful.
We might even potentially take this further. To avoid the data retention of a server-based system, we could exclusively support a customized version of a local app like ChatBox. Discussed here: #57
I had Claude do an analysis of Jan vs. ChatBox, attached here as md and pdf.
And here's a walk-through I generated (and have just successfully used!) to get my Inference Co-op API running on Jan.
@bengansky Fabulous! Thanks for this! Do you want to try making a pull request for adding it here: https://git.inference.coop/co-op/docs/src/branch/main/guides (I'd love to make sure you get credit for the contribution)?
A few comments: