Move inferencebot to our infrastructure safely #15

Closed
opened 2026-10-07 12:42:53 +00:00 by ntnsndr · 1 comment

What's the idea?

Most of our infrastructure has been built by an agent, inferencebot, running on our tokens but hosted on Nathan's Raspberry Pi. We should convert this key resource to a commons of the co-op by shifting it to our infrastructure.

Why does it matter?

To make the co-op more truly a co-op, it should not rely on a single member's personal tool.

How might it work?

The trick is we need to avoid two risks:

  • If our infrastructure breaks, the agent should be able to keep running to help fix it.
  • The agent should not have root access to its own environment, in case any rogue behavior emerges.

As a result, it is probably ideal to keep the agent(s) off our main Cloudron server, and with a fallback inference provider in case our systems fail.

Anything else?

Current setup:

  • Hermes agent profile on Raspberry Pi
  • Main gateway is a Matrix room
  • Main inference: co-op flash models
  • Fallback inference: Nathan's Ollama API key
  • API keys to: Cloudron, Gitea
## What's the idea? Most of our infrastructure has been built by an agent, inferencebot, running on our tokens but hosted on Nathan's Raspberry Pi. We should convert this key resource to a commons of the co-op by shifting it to our infrastructure. ## Why does it matter? To make the co-op more truly a co-op, it should not rely on a single member's personal tool. ## How might it work? The trick is we need to avoid two risks: * If our infrastructure breaks, the agent should be able to keep running to help fix it. * The agent should not have root access to its own environment, in case any rogue behavior emerges. As a result, it is probably ideal to keep the agent(s) off our main Cloudron server, and with a fallback inference provider in case our systems fail. ## Anything else? Current setup: * Hermes agent profile on Raspberry Pi * Main gateway is a Matrix room * Main inference: co-op flash models * Fallback inference: Nathan's Ollama API key * API keys to: Cloudron, Gitea
Author

Duplicative with #17

Duplicative with https://git.inference.coop/co-op/support/issues/17
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: co-op/support#15