Page fetch service: Keenable-shaped API, SSRF guard, no access log
This commit is contained in:
commit
5f88cebede
6 files changed
+255
No files matched your search
@@ -0,0 +1,63 @@
|
||||
# Page Fetcher
|
||||
|
||||
Internal page-fetch service for the co-op's LibreChat web search. It replaces
|
||||
the third-party Keenable fetch: LibreChat sends the result URL here, gets
|
||||
readable markdown back, and the model cites the page.
|
||||
|
||||
## Contract
|
||||
|
||||
```
|
||||
GET /v1/fetch?url=<page> (X-API-Key required)
|
||||
GET /v1/fetch/public?url=<page> (same; path LibreChat uses keyless)
|
||||
GET /health (no key — Cloudron health check)
|
||||
```
|
||||
|
||||
Response (Keenable-compatible shape, so LibreChat needs no config fork):
|
||||
|
||||
```json
|
||||
{"content": "markdown…", "title": "…", "description": "…", "url": "https://final.example/page"}
|
||||
```
|
||||
|
||||
Errors are JSON `{"detail": "…"}` with 4xx/5xx. `FETCHER_KEY` unset means the
|
||||
service refuses every fetch (503).
|
||||
|
||||
## Where it sits
|
||||
|
||||
```
|
||||
members → LibreChat → SearXNG (search) → Page Fetcher (this app) → page text
|
||||
↘ gateway (answer), with citations
|
||||
```
|
||||
|
||||
LibreChat reaches it on the internal Cloudron network
|
||||
(`http://fetch:8000`). No member can reach it: the public app is restricted to
|
||||
the `managers` Cloudron group, there is no UI, and every fetch requires
|
||||
`X-API-Key`.
|
||||
|
||||
## Privacy choices
|
||||
|
||||
- **No access log.** The request path carries the fetched URL, which is member
|
||||
search output. Uvicorn runs with `--no-access-log`; errors log the host, never
|
||||
the full URL. Nothing about a fetch is stored.
|
||||
- **SSRF guard.** http/https only, ports 80/443 only, credentials in the URL
|
||||
refused, private/loopback/link-local/reserved/multicast addresses refused,
|
||||
redirects re-validated at every hop, 2 MB body cap, 10 s timeout.
|
||||
- **No browser.** JS-rendered pages yield little or no text; the fetch fails
|
||||
cleanly rather than returning junk.
|
||||
|
||||
## Deploy
|
||||
|
||||
```sh
|
||||
# build (Cloudron builder)
|
||||
~/.hermes/scripts/cloudron_builder_build.sh coop.inference.fetcher <tag> .
|
||||
|
||||
# pin CloudronManifest.json dockerImage to the new tag, commit, push
|
||||
# install once:
|
||||
cloudron install --app fetch.inference.coop --location fetch
|
||||
|
||||
# configure: FETCHER_KEY (32+ random bytes), then restrict access to managers
|
||||
```
|
||||
|
||||
## Not in scope
|
||||
|
||||
- Search itself stays on SearXNG.
|
||||
- No crawling, no index, no cache, no UI.
|
||||
Reference in new issue
Block a user