Files

2.1 KiB

Page Fetcher

Internal page-fetch service for the co-op's LibreChat web search. It replaces the third-party Keenable fetch: LibreChat sends the result URL here, gets readable markdown back, and the model cites the page.

Contract

GET /v1/fetch?url=<page>          (X-API-Key required)
GET /v1/fetch/public?url=<page>   (same; path LibreChat uses keyless)
GET /health                       (no key — Cloudron health check)

Response (Keenable-compatible shape, so LibreChat needs no config fork):

{"content": "markdown…", "title": "…", "description": "…", "url": "https://final.example/page"}

Errors are JSON {"detail": "…"} with 4xx/5xx. FETCHER_KEY unset means the service refuses every fetch (503).

Where it sits

members → LibreChat → SearXNG (search) → Page Fetcher (this app) → page text
                     ↘ gateway (answer), with citations

LibreChat reaches it on the internal Cloudron network (http://fetch:8000). No member can reach it: the public app is restricted to the managers Cloudron group, there is no UI, and every fetch requires X-API-Key.

Privacy choices

  • No access log. The request path carries the fetched URL, which is member search output. Uvicorn runs with --no-access-log; errors log the host, never the full URL. Nothing about a fetch is stored.
  • SSRF guard. http/https only, ports 80/443 only, credentials in the URL refused, private/loopback/link-local/reserved/multicast addresses refused, redirects re-validated at every hop, 2 MB body cap, 10 s timeout.
  • No browser. JS-rendered pages yield little or no text; the fetch fails cleanly rather than returning junk.

Deploy

# build (Cloudron builder)
~/.hermes/scripts/cloudron_builder_build.sh coop.inference.fetcher <tag> .

# pin CloudronManifest.json dockerImage to the new tag, commit, push
# install once:
cloudron install --app fetch.inference.coop --location fetch

# configure: FETCHER_KEY (32+ random bytes), then restrict access to managers

Not in scope

  • Search itself stays on SearXNG.
  • No crawling, no index, no cache, no UI.