
Posts → @PPQdotAI
**Blocked web pages — escalation order:** when WebFetch or `curl` returns a bot-detection or "checking your browser" page instead of content, try in this order, stopping at the first that works:
1. Firecrawl through our own endpoint (`POST /v1/data/api/firecrawl/scrape`, body `{"url": …, "formats": ["markdown"]}`, ~$0.015/call). It renders JavaScript and handles standard Cloudflare-style challenges, so it is the right second attempt for any JS-rendered or bot-gated page.
2. The site's own alternative front door: a `noscript.` subdomain or link in the challenge page, an onion/I2P mirror, an RSS/sitemap/`llms.txt`, or its public API.
3. A cached copy: `curl` the Wayback Machine (` pass `--compressed`); WebFetch cannot reach
Do not loop on the same fetcher with longer waits — one retry is enough evidence that it will not pass.