Welcome to TRAWL! 👋
Self-hosted web scraping for applications and AI agents. Fetch pages through the native API, or connect an MCP client to read clean Markdown, extract structured data, capture screenshots, and inspect browser activity.
TRAWL starts with a fast HTTP request, reuses solved browser sessions, and escalates to Camoufox and an optional residential proxy when a site requires more work. It handles supported JavaScript challenges and CAPTCHAs, and gives you a local dashboard to see what succeeded, failed, and took time.
For existing FlareSolverr users, the compatible /v1 endpoint works with Prowlarr and other *arr tools. TRAWL's adaptive routing and session reuse are designed for faster, more reliable requests; selected same-machine benchmarks show faster responses than FlareSolverr and Byparr, with results varying by site and session state.
- MCP tools for AI agents - read pages as Markdown or text, scrape HTML, extract JSON records, capture screenshots, and inspect browser diagnostics from known public URLs
- 4-tier execution - plain HTTP fetch → cached browser session → fresh challenge solve → optional residential proxy
- Structured extraction and screenshots - select fields from repeated elements, render JavaScript pages, and capture a viewport, full page, or specific element
- Challenge-aware HTTP/HTTPS proxy - direct forwarding for normal traffic, automatic tier escalation for detected walls, plus WebSockets, binary bodies, and Range/206 support
- Local metrics dashboard - persistent request history, tier outcomes, failure causes, and live updates
- Multi-WAF handling - dedicated Cloudflare, Akamai Bot Manager, and Imperva/Incapsula detection and browser flows
- Native captcha solving - CF Turnstile/Interstitial, reCAPTCHA v2 (free STT), hCaptcha, GeeTest v4 Slide, ALTCHA, and Friendly Captcha v1/v2
- Camoufox Firefox - fingerprint-patched at the C++/Juggler level to reduce automation signals
- Session cache - solved cookies and browser identity stored in Redis; accepted sessions can avoid a fresh solve
- FlareSolverr compatible - works with Prowlarr, Jackett, Sonarr, and the full *arr ecosystem out of the box
- No paid solver API required - reCAPTCHA audio can use Google's free STT endpoint or an optional local Whisper service
View/Collapse All
|
|
Bright Data - The most powerful platform for Web Unlocker, SERP API and web scraping tools. Why Bright Data? • Web Unlocker - bypass any anti-bot protection • SERP API - real-time Google, Bing & more results • Scraping Browser & dedicated scrapers • Massive residential proxy network • Built for scale and reliability Get started for free with Bright Data! |
|
|
NodeMaven - The most efficient proxy provider for Web Scrapping and Automation with the Highest Quality IP on the market. Why NodeMaven? • ZIP targeting • 99.9% uptime • IP filtering: all proxies have fraud score <97% • No KYC required • Unique free tools: Proxy Bandwidth Checker, Meta Tag Checker, IP Lookup and others! Special codes for TRAWL users: • TRAWL35 - 35% off to Mobile and Residential Proxies• TRAWL40 - 40% off to ISP (Static) Proxies
|
|
|
Thordata - Premium Residential Proxies for Data Collection. Thordata helps developers build reliable scraping, automation, and AI data workflows with high-quality residential IPs. 🌍 100M+ real IPs | 195+ countries 🔄 Rotating & sticky sessions | Precise geo-targeting ⚡ High concurrency | Stable connections Reduce blocks and collect data at scale with confidence. 🎁 TRAWL users: • 3-day free trial + 10% OFF Code: TRAWL10
|
The screenshot uses illustrative traffic. Your dashboard at http://localhost:8191/dashboard displays only requests handled by your TRAWL instance. See the metrics guide for setup.
# Clone and configure
git clone https://github.com/germondai/trawl
cd trawl
cp .env.example .env
# Start scraper + Redis
docker compose up -d
# Verify
curl http://localhost:8191/healthFirst boot takes 15–30s while the browser pool warms up. Subsequent starts are fast.
Prefer a one-click installation? TRAWL is available from the community app catalogs for both TrueNAS and Unraid:
- TrueNAS Community Apps — open Apps → Discover Apps and search for TRAWL.
- Unraid Community Apps — open the Apps tab and search for Trawl.
Thanks to the TrueNAS and Unraid community contributors who packaged and published these integrations.
curl -X POST http://localhost:8191/v1 \
-H 'Content-Type: application/json' \
-d '{"cmd":"request.get","url":"https://nowsecure.nl","maxTimeout":60000}'Returns richer metadata: tier, timings, sessionCached, full cookie list.
curl -X POST http://localhost:8191/scrape \
-H 'Content-Type: application/json' \
-d '{"url":"https://nowsecure.nl","maxTimeout":60000}'Set MCP_ENABLED=true to expose TRAWL's client-independent Streamable HTTP tools
for readable content, HTML, structured extraction, screenshots and browser diagnostics to any
MCP-compatible AI application or agent. They load known public URLs; TRAWL does not
provide web search or ranking. See the
MCP integration guide.
Set the FlareSolverr URL to:
http://localhost:8191 # running on the same host
http://trawl:8191 # running via Docker Compose on the same network
Some sites bind their Cloudflare clearance to the solving browser's full connection
fingerprint. The /v1 flow can't help there: Prowlarr keeps only the cookie + user-agent
and re-fetches the page with its own HTTP client, which Cloudflare re-challenges — the
cookie isn't portable. For those indexers, enable TRAWL's forward proxy and add it to
Prowlarr as an HTTP proxy:
MITM_ENABLED=true
MITM_PORT=8192
MITM_CA_DIR=/data/proxy-ca # persist the CA (mount a volume)
MITM_MAX_TIER=4 # cap escalation (e.g. 3 to stay off residential)
MITM_ALWAYS_SCRAPE=false # opt in to bypass the proxy's direct Tier 0 probeBy default the listener binds 0.0.0.0 so clients on a Docker bridge network can reach
it; set MITM_HOST=127.0.0.1 to restrict it to loopback on a bare-metal host.
- Install the proxy's CA into the client's trust store so it accepts the per-host certs:
curl http://<trawl-host>:8191/proxy-ca.crt→ add to the Prowlarr container's CA store (e.g. a linuxserver/custom-cont-init.dscript that copies it to/usr/local/share/ca-certificates/and runsupdate-ca-certificates). - Prowlarr → Settings → Indexer Proxies → HTTP, host
<trawl-host>, port8192. Give it a tag if only selected indexers should use it.
Ordinary requests use a direct HTTP/TLS path. Small HTML, JSON, and text responses are buffered
for challenge detection; detected challenges escalate through the same tier pipeline as
POST /scrape. Videos and large binary responses stream directly. Range requests are forwarded
end to end and can escalate when their response is a detected challenge; WebSocket upgrades use a
direct relay without browser escalation.
See the complete proxy documentation for routing details, supported traffic, limitations, CA installation, and client examples.
⚠️ A MITM proxy can impersonate any host to a client that trusts its CA. Only expose it on a private interface (localhost / a private Docker network), never publicly.
The proxy self-generates a root CA on first run. Its certificate and private key are persisted
under MITM_CA_DIR (default /data/proxy-ca). Per-host certificates are minted and cached
in memory while TRAWL runs; they do not need separate installation because they are signed by the
persistent root. Every client that uses the proxy must trust that root. Without it, HTTPS fails with
ERR_CERT_AUTHORITY_INVALID (browsers) or PKIX path building failed (Java).
Download the CA once per client:
curl http://<trawl-host>:8191/proxy-ca.crt -o trawl-ca.crt
# or in a Docker setup where the API isn't reachable from outside:
docker cp trawl:/data/proxy-ca/ca.crt ./trawl-ca.crtsudo security add-trusted-cert -d -r trustRoot \
-k /Library/Keychains/System.keychain ./trawl-ca.crt
# Verify
security find-certificate -c "TRAWL MITM Proxy CA"
# Remove later
sudo security delete-certificate -c "TRAWL MITM Proxy CA" \
/Library/Keychains/System.keychainsudo cp trawl-ca.crt /usr/local/share/ca-certificates/trawl-ca.crt
sudo update-ca-certificates
# Verify
awk '/BEGIN/{c++} c==2' /etc/ssl/certs/ca-certificates.crt | grep -c "TRAWL MITM"sudo cp trawl-ca.crt /etc/pki/ca-trust/source/anchors/trawl-ca.crt
sudo update-ca-trustFirefox installations that do not use operating-system roots need a per-profile NSS import:
# Firefox 115+ uses a file-backed NSS DB; older versions use the legacy libnssdb format.
# The certutil command is the same either way.
certutil -A -n "TRAWL MITM" -t "CT,C,C" -i trawl-ca.crt \
-d sql:$HOME/.mozilla/firefox/<profile-dir>
# Or via Firefox UI: Settings → Privacy & Security → Certificates → View Certificates →
# Authorities → Import… → check "Trust this CA to identify websites".
# Profile dir location: about:profiles in Firefox.Chrome uses the system trust store on macOS and Windows but has its own on Linux:
# Option A: launch Chrome with --user-data-dir + NSS DB update (same as Firefox).
# Option B: use Chrome's --ignore-certificate-errors-spki-list=<hash> (per-session, less safe).
# Option C: add the cert to the system store (above) — Chrome picks it up automatically on
# most Linux distros via the nss-tool lookup.# Find the JRE cacerts file for your client.
# JDownloader: <install>/jre/lib/security/cacerts
keytool -importcert -alias trawl -file trawl-ca.crt \
-keystore "<path-to-cacerts>" -storepass changeit
# If `keytool` reports "Certificate already exists in keystore", use -delete first:
# keytool -delete -alias trawl -keystore "<path-to-cacerts>" -storepass changeitProwlarr, Sonarr, and Radarr are .NET applications, not Java applications. For their
Docker-based installations, add the CA to the container's Linux system trust store. A common
LinuxServer pattern is a /custom-cont-init.d script:
# In the client's Compose service:
volumes:
- ./trawl-ca.crt:/config/trawl-ca.crt:ro
- ./install-trawl-ca.sh:/custom-cont-init.d/50-install-trawl-ca:ro#!/usr/bin/with-contenv bash
cp /config/trawl-ca.crt /usr/local/share/ca-certificates/trawl-ca.crt
update-ca-certificatesLinuxServer runs scripts in /custom-cont-init.d/ when the container starts. Java clients such as
JDownloader require the separate keytool import described above.
JDownloader bundles its own JRE; the CA must be imported into it.
- Find the JRE:
Settings → Advanced → Java Path(in JDownloader) or look in the install dir:- Windows:
C:\Program Files\JDownloader 2\jre\lib\security\cacerts - macOS:
/Applications/JDownloader 2.app/Contents/app/jre/lib/security/cacerts - Linux:
<install>/jre/lib/security/cacerts
- Windows:
- Run the
keytool -importcertcommand above against that file. - Restart JDownloader.
# Run PowerShell as Administrator.
Import-Certificate -FilePath .\trawl-ca.crt `
-CertStoreLocation Cert:\LocalMachine\Root
# Remove later
Get-ChildItem Cert:\LocalMachine\Root | Where-Object { $_.Subject -like "*TRAWL MITM*" } | Remove-ItemEvery installation method has a symmetric removal path. Search your trust store for
TRAWL MITM Proxy CA (the CA's CN) and delete that entry. The CA certificate and key also live at
<MITM_CA_DIR>/ca.crt and ca.key on the TRAWL host. Deleting either causes TRAWL to
generate a new root on its next start, so existing clients must install the new certificate.
Request
│
▼
Tier 1: Plain HTTP fetch ────── success ──→ return (< 100ms)
│ blocked
▼
Tier 2: Cached session ─────── success ──→ return (~500ms)
│ cache miss / expired
▼
Tier 3: Fresh challenge solve ─ success ──→ cache + return
│ IP flagged
▼
Tier 4: Residential proxy ──── success ──→ cache + return (15–45s)
│ failed
▼
error
| File | Description |
|---|---|
docker-compose.yml |
Scraper + Redis (default) |
docker-compose.minimal.yml |
Scraper only, no Redis |
docker-compose.prod.yml |
Production: restart: always, memory limit, healthcheck |
| Image tag | Built from | Runtime | Use case |
|---|---|---|---|
ghcr.io/germondai/trawl:latest |
apps/api/Dockerfile |
Bun 1.4.2 (modern, AVX2) | Compact default — Linux fingerprints |
ghcr.io/germondai/trawl:baseline |
apps/api/Dockerfile.baseline |
Bun 1.4.2 baseline (no AVX2) | Older CPUs / older kernels (Synology NAS, J4125, Atom-era) |
All tags live on the same ghcr.io/germondai/trawl package — they share the registry but differ in Dockerfile source or build arguments. Pick whichever tag fits your hardware and output:
# Modern hardware (most users)
image: ghcr.io/germondai/trawl:latest
# Older CPUs without AVX2 / Synology / older kernels
image: ghcr.io/germondai/trawl:baselineSynology note: many Synology NAS units (DSM 7.x on J4125 / older hardware) ship kernel 4.4.x, which Bun's modern runtime can't fully handle. Standard Bun requires kernel 5.1+ (5.6+ recommended); the baseline build degrades gracefully down to kernel 3.10. The :baseline tag is published for that case — confirmed working on a Synology DS920+ (Celeron J4125, DSM 7.3.2, kernel 4.4.302): the container starts cleanly, /health reports healthy, and it solves live Cloudflare challenges via /v1 (see #1). Published by independent GitHub Actions workflows: pushing a release tag such as v1.7.0 creates :1.7.0, :latest, :1.7.0-baseline, and :baseline; the daily 02:00 UTC nightly build creates :nightly and :nightly-<dev-sha> from the latest dev commit.
TRAWL follows Semantic Versioning. Pushing a v-prefixed git tag (e.g.
v1.0.0) triggers publish.yml, which builds and pushes the matching un-prefixed Docker tag
(ghcr.io/germondai/trawl:1.0.0) alongside :latest. The moving :nightly tag and immutable
:nightly-<shortsha> tags are built from the latest verified dev revision every day at 02:00 UTC
or on manual dispatch; branch pushes do not publish images.
See the Releases page for the full version
history and CHANGELOG.md for what changed in each one.
To publish a specific past commit that had a major fix without waiting for the next tip-of-main
release, tag that exact SHA and push it — publish.yml builds off the tag ref, not off main's
current tip:
git tag -a v1.0.1 <sha> -m "..."
git push origin v1.0.1TRAWL supports HTTP proxies, authenticated HTTP proxies, and SOCKS5 proxies. The standard Compose
files read proxy settings from the local .env file:
# Optional Tier 3 datacenter proxy
PROXY_URL=http://user:pass@datacenter.example.com:8080
# Optional Tier 4 residential proxy
RESIDENTIAL_PROXY_URL=socks5://user:pass@residential.example.com:1080docker compose up -dLeave either value empty to disable that proxy tier. Multiple endpoints can be separated with
commas; larger pools can use the corresponding *_LIST_FILE variable. See
Configuration → Proxies
for pool and mounted-file examples.
| Variable | Default | Description |
|---|---|---|
BROWSER_POOL_SIZE |
1 |
Warm Camoufox Firefox instances; raise for concurrent browser solves |
LOG_LEVEL |
info |
Operational logs: error, warn, info, debug, or silent |
METRICS_DASHBOARD_ENABLED |
false |
Explicitly enable the local dashboard without a token; bind its port to 127.0.0.1 |
METRICS_DASHBOARD_TOKEN |
— | Protect the dashboard, JSON endpoint, and live stream with a 32+ character token |
METRICS_DB_PATH |
/data/metrics/trawl.sqlite |
SQLite path for local metrics history (mount /data/metrics persistently) |
BROWSER_ACQUIRE_TIMEOUT_MS |
15000 |
How long acquire() polls for a free browser before HTTP 429 is returned |
BROWSER_RECYCLE_AFTER_CONTEXTS |
8 |
Rolling-replace after this many Tier 3/4 contexts; set 0 to disable |
BROWSER_MAX_CONTENT_PROCESSES |
2 |
Cap Firefox content processes per browser (dom.ipc.processCount); lowers RAM/CPU |
SCRAPE_MIN_TIER |
1 |
Lowest tier allowed globally (1 HTTP, 2 cached browser, 3 fresh, 4 residential) |
SESSION_CACHE_DRIVER |
redis |
Session cache backend: redis or single-process memory |
REDIS_SESSION_TTL_SECONDS |
3600 |
Redis or in-memory session TTL (seconds) |
MEMORY_SESSION_CACHE_MAX_ENTRIES |
1000 |
Maximum LRU-bounded entries for the memory driver |
REDIS_URL |
— | Redis connection string; empty or unset disables the Redis cache |
REDIS_CONNECT_TIMEOUT_MS |
5000 |
Maximum time for each Redis connection attempt |
REDIS_RETRY_DELAY_MS |
5000 |
Delay before reconnecting after startup failure; 0 disables retry |
SCRAPE_PROXY_SELECTION |
failover |
Pool policy: sticky failover, per-request roundrobin, or random |
PROXY_URL |
— | Optional Tier 3 HTTP or SOCKS5 proxy, or comma-separated pool |
PROXY_LIST_FILE |
— | File containing one Tier 3 proxy URL per line |
RESIDENTIAL_PROXY_URL |
— | Enables Tier 4 proxy escalation |
RESIDENTIAL_PROXY_LIST_FILE |
— | File containing one Tier 4 proxy URL per line |
STT_URL |
— | Local Whisper endpoint for reCAPTCHA (optional) |
PORT |
8191 |
API listen port |
MITM_ENABLED |
false |
Enable the challenge-bypassing HTTP/HTTPS proxy |
MITM_PORT |
8192 |
Forward-proxy listen port |
MITM_HOST |
0.0.0.0 |
Bind address; 127.0.0.1 for loopback-only |
MITM_CA_DIR |
/data/proxy-ca |
Persistent root CA certificate and private-key directory |
MITM_MAX_TIER |
4 |
Cap escalation used by the proxy (e.g. 3 to stay off residential) |
MITM_ALWAYS_SCRAPE |
false |
Skip proxy Tier 0; disables the direct media/large-file streaming path |
MITM_DEBUG |
false |
Log one line per proxied request (errors are always logged) |
Upgrading from an earlier release requires renaming several environment variables. See the configuration migration guide for the complete old-to-new mapping and Redis opt-in behavior.
Built on a modern, fast-by-default stack: Bun + Elysia for the API, Redis for caching, Camoufox (hardened Firefox) for browser automation, and Nuxt for the web UI — no legacy Node/Express baggage.
| Layer | Technology |
|---|---|
| Runtime | Bun |
| API | Elysia |
| Browser | Camoufox Firefox (via camoufox-js) |
| Session cache | Redis 8.8 |
| Landing page | Nuxt 4 |
| Documentation | VitePress |
Made with ❤️ by @germondai