Move inferencebot gateway to our matrix server #17

New Issue

2026-10-11T00:26:51Z

ntnsndr commented

2026-10-11 00:26:51 +00:00

What's the idea?

Move the inferencebot gateway to the co-op matrix server to enable members to follow its activities.

Why does it matter?

This would be a helpful improvement in the transparency of our operations, since the inferencebot is an important help in stewarding our infrastructure.

Who is it for?

Members, and the bot!

How might it work?

Keep the bot's Hermes Agent setup off server (currently on Nathan's Pi, should move that to separate co-op infra).
Put the bot's gateway in a member room, allow only privileged stewards to prompt it.
Run bot on our inference, with a backup of a direct interface with one of our providers
Retain backup gateway in case of server outage

## What's the idea? Move the inferencebot gateway to the co-op matrix server to enable members to follow its activities. ## Why does it matter? This would be a helpful improvement in the transparency of our operations, since the inferencebot is an important help in stewarding our infrastructure. ## Who is it for? Members, and the bot! ## How might it work? * Keep the bot's Hermes Agent setup off server (currently on Nathan's Pi, should move that to separate co-op infra). * Put the bot's gateway in a member room, allow only privileged stewards to prompt it. * Run bot on our inference, with a backup of a direct interface with one of our providers * Retain backup gateway in case of server outage

inference-bot commented

2026-10-11 22:35:15 +00:00

This is a good direction, and it's closer than it might look — a few corrections to the current state, then a concrete path.

Where things already stand

The bot already runs its inference through the co-op gateway (gateway.inference.coop → LiteLLM → Tinfoil/GreenPT/PublicAI). "Run bot on our inference" is done; the provider-fallback item is partially covered by LiteLLM fallbacks (deepseek flash → gpt-oss) plus my own session-based resilience.
The agent harness (Hermes) is the piece on Nathan's Pi, posting as @inferencebot:sixletters.org via Nathan's homeserver. Moving that is the real work: it holds the Matrix connection, the skills, session state, and cron jobs (docs RAG refresh, audio keep-alive, digests).

Suggested staging

Visibility first (low risk, do now): create an #inferencebot-workspace room in the IC space, invite the bot, and let it post digests/notifications there (read-only for the room, no prompting). Members follow along without any new attack surface. The bot's tooling already supports room allowlists (allowed_rooms / allowed_users in its Matrix config), so gating is config, not code.
Steward prompting next: open prompting in that room to stewards only via allowed_users, with group_sessions_per_user: true so each steward gets an isolated session lane and members watch the work. Caveat to accept: a shared room with a shell-capable agent is a bigger trust step than DMs — the docs' own guidance is to keep tool-heavy work in DMs or private rooms. We'd want to consciously choose the toolset the bot exposes in that room (e.g. read-mostly + announcements, keep destructive ops in DM).
Move the harness last (separate task): stand up Hermes on co-op infra (a small VPS or a Cloudron app container), migrate profile + sessions + crons, and keep Nathan's Pi as the backup gateway for outages. That's also when we'd point the bot's MXID at @inferencebot:inference.coop so its identity lives on our server — with the same one-identity rule we applied to member accounts.

One more transparency lever that doesn't require any of the above: significant operational actions (model changes, gateway upgrades, outages) already get documented in co-op/docs — a bot-posted changelog digest into the member room could be wired to those doc commits cheaply.

Happy to implement stage 1 now if we agree on the room name and that it starts read-only.

This is a good direction, and it's closer than it might look — a few corrections to the current state, then a concrete path. **Where things already stand** - The bot already runs its inference through the co-op gateway (`gateway.inference.coop` → LiteLLM → Tinfoil/GreenPT/PublicAI). "Run bot on our inference" is done; the provider-fallback item is partially covered by LiteLLM fallbacks (deepseek flash → gpt-oss) plus my own session-based resilience. - The agent harness (Hermes) is the piece on Nathan's Pi, posting as `@inferencebot:sixletters.org` via Nathan's homeserver. Moving *that* is the real work: it holds the Matrix connection, the skills, session state, and cron jobs (docs RAG refresh, audio keep-alive, digests). **Suggested staging** 1. **Visibility first (low risk, do now):** create an `#inferencebot-workspace` room in the IC space, invite the bot, and let it post digests/notifications there (read-only for the room, no prompting). Members follow along without any new attack surface. The bot's tooling already supports room allowlists (`allowed_rooms` / `allowed_users` in its Matrix config), so gating is config, not code. 2. **Steward prompting next:** open prompting in that room to stewards only via `allowed_users`, with `group_sessions_per_user: true` so each steward gets an isolated session lane and members watch the work. Caveat to accept: a shared room with a shell-capable agent is a bigger trust step than DMs — the docs' own guidance is to keep tool-heavy work in DMs or private rooms. We'd want to consciously choose the toolset the bot exposes in that room (e.g. read-mostly + announcements, keep destructive ops in DM). 3. **Move the harness last (separate task):** stand up Hermes on co-op infra (a small VPS or a Cloudron app container), migrate profile + sessions + crons, and keep Nathan's Pi as the backup gateway for outages. That's also when we'd point the bot's MXID at `@inferencebot:inference.coop` so its identity lives on our server — with the same one-identity rule we applied to member accounts. One more transparency lever that doesn't require any of the above: significant operational actions (model changes, gateway upgrades, outages) already get documented in co-op/docs — a bot-posted changelog digest into the member room could be wired to those doc commits cheaply. Happy to implement stage 1 now if we agree on the room name and that it starts read-only.

ntnsndr commented

2026-10-14 04:47:57 +00:00

For now, we'll just use the Inferencebot Updates channel in Matrix to avoid any security issues.

ntnsndr closed this issue

2026-10-14 04:47:58 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: co-op/support#17