How to scrape a site for an AI agent
Key factsScrape for an AI agent: get markdown or JSON, never raw HTML
- Check robots.txt and the site's terms before scraping; if you would not email the site owner, do not put the scrape in a cron job.
- Use an API like Firecrawl or Context to return markdown or JSON output instead of feeding raw HTML into the agent's prompt.
- Chunk, embed, or otherwise select only the smallest useful piece of scraped content rather than stuffing a 200-page crawl into one context window.
- Firecrawl's free tier is 1,000 credits a month with no card required, and Hobby costs $16/month billed annually or $19 monthly for 5,000 credits.
- Cache scraped content and re-crawl on a schedule rather than re-scraping on every user message.
An agent that "reads the web" with a raw HTTP get will choke on the first JS-rendered pricing table. You want markdown or JSON out, not a blob of nav chrome.
The two APIs we reviewed are Firecrawl (10% off via the partner link) and Context (25% off for six months). Devshot already ranks scrapers in best web scraping tools.
The short version
- Decide the job: one URL, a crawl of a docs site, or a search-then-scrape. Do not start with "the whole internet."
- Check robots.txt and the site's terms. If you would not email the owner, do not put it in a cron.
- Call Firecrawl or Context for markdown / JSON. Keep the raw HTML out of the prompt.
- Chunk, embed, or stuff the smallest useful piece into the agent. A 200-page crawl in one context window is how you waste money and get a confident wrong answer.
- Cache. Re-crawl on a schedule, not on every user message.
Firecrawl's free tier is 1,000 credits a month, no card. Hobby is $16/mo annual ($19 monthly) for 5,000 credits, per their pricing page as of September 2026. Start free.
When this is the wrong tool
- You need a human-readable briefing. Use Perplexity.
- You need to pay an affiliate when someone signs up. That is Dub.
- You need a browser the user drives. That is not a scrape API.
Related
This article may contain affiliate links and paid recommendations. How we make money
