How to scrape a site for an AI agent
Trusted by750,000+ Techpresso subscribers·426 AI tools reviewed·Editorial team
An agent that "reads the web" with a raw HTTP get will choke on the first JS-rendered pricing table. You want markdown or JSON out, not a blob of nav chrome.
The two APIs we reviewed are Firecrawl (10% off via the partner link) and Context (25% off for six months). Devshot already ranks scrapers in best web scraping tools.
The short version
- Decide the job: one URL, a crawl of a docs site, or a search-then-scrape. Do not start with "the whole internet."
- Check robots.txt and the site's terms. If you would not email the owner, do not put it in a cron.
- Call Firecrawl or Context for markdown / JSON. Keep the raw HTML out of the prompt.
- Chunk, embed, or stuff the smallest useful piece into the agent. A 200-page crawl in one context window is how you waste money and get a confident wrong answer.
- Cache. Re-crawl on a schedule, not on every user message.
Firecrawl's free tier is 1,000 credits a month, no card. Hobby is $16/mo annual ($19 monthly) for 5,000 credits, per their pricing page as of September 2026. Start free.
When this is the wrong tool
- You need a human-readable briefing. Use Perplexity.
- You need to pay an affiliate when someone signs up. That is Dub.
- You need a browser the user drives. That is not a scrape API.
