---
title: "How to scrape a site for an AI agent | Dupple"
url: "https://dupple.com/learn/how-to-scrape-a-site-for-an-ai-agent"
description: "How to turn a website into clean data for an agent. Firecrawl (10% off) and Context (25% off for six months) are the two APIs we reviewed."
published: "2026-09-17"
updated: "2026-09-17"
llms_txt: "https://dupple.com/llms.txt"
---
# How to scrape a site for an AI agent

Trusted by

810,000+

Techpresso subscribers

·

Editorial policy

·

How we make money

Written by

Louis Corneloup

Founder at Dupple, covering AI tools and strategies for 810K+ readers.

How our guides are researched

.

September 17, 2026

· Updated September 2026

1 min read

**Key factsScrape for an AI agent: get markdown or JSON, never raw HTML**

- **Updated:** September 17, 2026

- Check robots.txt and the site's terms before scraping; if you would not email the site owner, do not put the scrape in a cron job.
- Use an API like Firecrawl or Context to return markdown or JSON output instead of feeding raw HTML into the agent's prompt.
- Chunk, embed, or otherwise select only the smallest useful piece of scraped content rather than stuffing a 200-page crawl into one context window.
- Firecrawl's free tier is 1,000 credits a month with no card required, and Hobby costs $16/month billed annually or $19 monthly for 5,000 credits.
- Cache scraped content and re-crawl on a schedule rather than re-scraping on every user message.

An agent that "reads the web" with a raw HTTP get will choke on the first JS-rendered pricing table. You want markdown or JSON out, not a blob of nav chrome.

The two APIs we reviewed are [Firecrawl](https://dupple.com/reviews/firecrawl) (10% off via the partner link) and [Context](https://dupple.com/reviews/context) (25% off for six months). Devshot already ranks scrapers in [best web scraping tools](https://devshot.co/reviews/best-web-scraping-tools).

## The short version

1. Decide the job: one URL, a crawl of a docs site, or a search-then-scrape. Do not start with "the whole internet."
2. Check robots.txt and the site's terms. If you would not email the owner, do not put it in a cron.
3. Call [Firecrawl](https://dupple.com/reviews/firecrawl) or [Context](https://dupple.com/reviews/context) for markdown / JSON. Keep the raw HTML out of the prompt.
4. Chunk, embed, or stuff the smallest useful piece into the agent. A 200-page crawl in one context window is how you waste money and get a confident wrong answer.
5. Cache. Re-crawl on a schedule, not on every user message.

[Firecrawl](https://dupple.com/reviews/firecrawl)'s free tier is 1,000 credits a month, no card. Hobby is $16/mo annual ($19 monthly) for 5,000 credits, per their [pricing page](https://www.firecrawl.dev/pricing) as of September 2026. Start free.

## When this is the wrong tool

- You need a human-readable briefing. Use [Perplexity](https://dupple.com/reviews/perplexity).
- You need to pay an affiliate when someone signs up. That is [Dub](https://dupple.com/reviews/dub).
- You need a browser the user drives. That is not a scrape API.

## Related

- [Firecrawl review](https://dupple.com/reviews/firecrawl)
- [Context review](https://dupple.com/reviews/context)
- [Developer AI stack](https://dupple.com/learn/developer-ai-stack-2026)
- [Current partner discounts](https://dupple.com/learn/ai-tool-discounts-2026)

Some offers on this page may be paid placements or contain affiliate links.

Related Articles

Tutorial

### How to Integrate AI into a Website (2026)

How to integrate AI into a website: chatbots, search, and personalization. Covers no-code tools, plugins, and custom API integration.

Tutorial

### How to Build an AI Agent: Beginner-Friendly Guide (2026)

Learn how to build an AI agent from scratch: frameworks, tools, and step-by-step instructions. Covers LangChain, CrewAI, AutoGen, and no-code options.

Blog Post

### What Is Hermes Agent? Nous Research's Self-Improving AI Agent (2026)

Hermes Agent is Nous Research's open-source AI agent with persistent memory across Telegram, Slack, email, and CLI. What it does, real pricing, and how to start.

Blog Post

### The Best AI Coding Agents in 2026 (Compared and Ranked)

I compared the best AI coding agents of 2026, from Claude Code and Cursor to Devin and Codex. Real pricing, benchmark scores, and where each one falls short.

Blog Post

### The 8 Best AI Agent Frameworks in 2026

I compared the best AI agent frameworks of 2026, from LangGraph and CrewAI to the OpenAI Agents SDK and Pydantic AI. Real pricing, honest downsides, and how to pick.

Article

### 10 Best AI Agent Observability Tools in 2026

The best AI agent observability tools in 2026: Arize AX Pro at $50/mo, LangSmith Plus at $39/seat, Logfire Team at $49/mo. Checked September 23, 2026.
