Crawl4AI Tutorial: Setup, Examples and a Hosted Alternative

Crawl4AI is an open-source Python web crawler that turns web pages into clean, LLM-ready Markdown and structured JSON. You install Crawl4AI with pip install -U crawl4ai, run crawl4ai-setup once to get a browser, and five lines of async Python give you a page as Markdown.

This tutorial walks through everything you need for real work: installation, your first crawl, filtered Markdown, structured extraction with and without an LLM, deep crawling, the crwl command line, Docker and MCP. We ran every Python example on 23 October 2026 with Crawl4AI 0.9.5 on Python 3.12, against the practice sites books.toscrape.com and quotes.toscrape.com, and show the real output. At the end you'll see the hosted route: the same crawl as a cloud job on Scrape.ly, with its exact cost.

Crawl4AI at a glance

How to install Crawl4AI

Install the package and the browser it drives. The second command downloads Chromium through Playwright, which the library needs for every crawl:

pip install -U crawl4ai
crawl4ai-setup        # installs the browser, once
crawl4ai-doctor       # checks the installation

If the browser step fails, the README's fallback installs Chromium and its system libraries by hand: python -m playwright install --with-deps chromium.

To try it without touching your global Python, uv gives you a throwaway environment. This is how we ran every example below:

uv run --python 3.12 --with crawl4ai==0.9.5 python first_crawl.py

Your first crawl: one page to Markdown

AsyncWebCrawler opens a browser, arun() loads the page, and the result carries the Markdown along with the HTML, links, media and status code:

import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url="https://books.toscrape.com/")
        print(result.success, result.status_code)
        print(len(result.markdown), "characters of Markdown")
        print(result.markdown[:300])

asyncio.run(main())

Our run printed True 200 and 13,476 characters of Markdown, starting like this:

[Books to Scrape](https://books.toscrape.com/index.html) We love being scraped!
  * [Home](https://books.toscrape.com/index.html)
  * All products
  * [ Books ](https://books.toscrape.com/catalogue/category/books_1/index.html)
    * [ Travel ](https://books.toscrape.com/catalogue/category/books/travel_2/index.html)

The raw HTML of that page is 51,294 bytes, so the Markdown is about a quarter of the size before any filtering. That matters when every character you send to a model costs tokens. The whole script, browser start-up included, took 3.5 seconds.

Clean Markdown with fit_markdown

The default Markdown keeps navigation and sidebars. Attach a content filter to the Markdown generator and you also get fit_markdown, which drops low-value blocks:

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode
from crawl4ai.content_filter_strategy import PruningContentFilterLXML
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator

async def main():
    config = CrawlerRunConfig(
        cache_mode=CacheMode.BYPASS,
        markdown_generator=DefaultMarkdownGenerator(
            content_filter=PruningContentFilterLXML(threshold=0.48, threshold_type="fixed")
        ),
    )
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url="https://quotes.toscrape.com/", config=config)
        print("raw:", len(result.markdown.raw_markdown))
        print("fit:", len(result.markdown.fit_markdown))

asyncio.run(main())

On quotes.toscrape.com the raw Markdown was 4,375 characters and the filtered version 3,663. Use PruningContentFilterLXML: in 0.9.5 the older PruningContentFilter still works but prints a deprecation warning saying the lxml version is about 10 times faster with identical output. For query-focused pages there is also BM25ContentFilter, and LLMContentFilter if you want a model to decide what stays.

CacheMode.BYPASS makes every run fetch the page fresh. Crawl4AI caches results by default, which is handy in development and confusing when you're checking whether a page changed.

Structured data without an LLM: CSS extraction

For pages with a stable layout, a CSS schema is the fastest and cheapest extractor: no model, no tokens, the same output every run. This schema pulls every book card from the books.toscrape.com home page:

import asyncio, json
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode, JsonCssExtractionStrategy

schema = {
    "name": "Books",
    "baseSelector": "article.product_pod",
    "fields": [
        {"name": "title", "selector": "h3 a", "type": "attribute", "attribute": "title"},
        {"name": "price", "selector": "p.price_color", "type": "text"},
        {"name": "availability", "selector": "p.availability", "type": "text"},
        {"name": "url", "selector": "h3 a", "type": "attribute", "attribute": "href"},
    ],
}

async def main():
    config = CrawlerRunConfig(
        extraction_strategy=JsonCssExtractionStrategy(schema),
        cache_mode=CacheMode.BYPASS,
    )
    async with AsyncWebCrawler(config=BrowserConfig(headless=True)) as crawler:
        result = await crawler.arun(url="https://books.toscrape.com/", config=config)
        books = json.loads(result.extracted_content)
        print(len(books), "books")
        print(json.dumps(books[:2], indent=2, ensure_ascii=False))

asyncio.run(main())

It returned all 20 books on the page in 1.6 seconds:

[
  {
    "title": "A Light in the Attic",
    "price": "£51.77",
    "availability": "In stock",
    "url": "catalogue/a-light-in-the-attic_1000/index.html"
  },
  {
    "title": "Tipping the Velvet",
    "price": "£53.74",
    "availability": "In stock",
    "url": "catalogue/tipping-the-velvet_999/index.html"
  }
]

baseSelector picks the repeating element, and each field reads text or an attribute inside it. JsonXPathExtractionStrategy and RegexExtractionStrategy work the same way, and generate_schema can ask an LLM once to write a schema you then reuse for free.

Structured data with an LLM

When layouts vary from site to site, LLMExtractionStrategy sends the page content to a model with your JSON schema and an instruction. The README says it works with any LLM provider. This is the README's own example; it needs an API key, so we didn't run it:

import os, asyncio
from pydantic import BaseModel, Field
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode, LLMConfig, LLMExtractionStrategy

class ModelFee(BaseModel):
    model_name: str = Field(..., description="Name of the model.")
    input_fee: str = Field(..., description="Fee for input tokens.")
    output_fee: str = Field(..., description="Fee for output tokens.")

async def main():
    run_config = CrawlerRunConfig(
        cache_mode=CacheMode.BYPASS,
        extraction_strategy=LLMExtractionStrategy(
            llm_config=LLMConfig(provider="openai/gpt-4o-mini", api_token=os.getenv("OPENAI_API_KEY")),
            schema=ModelFee.model_json_schema(),
            extraction_type="schema",
            instruction="Extract every model name with its input and output token fee.",
        ),
    )
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url="https://openai.com/api/pricing/", config=run_config)
        print(result.extracted_content)

asyncio.run(main())

LLM extraction costs tokens on every page and can return slightly different output between runs, so validate the JSON before you store it. A good pattern: use the LLM to write a CSS schema once, then run the schema at scale. Our guide to AI web scraping compares rule-based, LLM and agent extraction in more depth.

Deep crawling a whole site

To follow links beyond the first page, give CrawlerRunConfig a deep crawl strategy. Breadth-first is the simplest:

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode
from crawl4ai.deep_crawling import BFSDeepCrawlStrategy

async def main():
    config = CrawlerRunConfig(
        deep_crawl_strategy=BFSDeepCrawlStrategy(max_depth=1, include_external=False, max_pages=5),
        cache_mode=CacheMode.BYPASS,
    )
    async with AsyncWebCrawler() as crawler:
        results = await crawler.arun("https://quotes.toscrape.com/", config=config)
        for r in results:
            print(r.metadata.get("depth"), r.url, len(r.markdown))

asyncio.run(main())

It stopped at 5 pages as asked, in 2.2 seconds, and each result carried its depth in metadata. The parameters that keep a crawl under control:

• max_depth: levels beyond the start page.
• max_pages: a hard stop on the page count. The default is unlimited, so always set it.
• include_external: whether to leave the start domain.
• filter_chain and url_scorer: URL patterns, domains and keyword scores.

DFSDeepCrawlStrategy goes deep first, and BestFirstCrawlingStrategy with a KeywordRelevanceScorer visits the most relevant URLs first; the deep crawling docs recommend it. Pass stream=True to process pages as they arrive instead of waiting for the full list, and look at resume_state for crash recovery on long crawls.

The crwl command line

The package also installs crwl, which is handy for quick checks and shell scripts:

crwl https://books.toscrape.com/ -o markdown
crwl https://docs.crawl4ai.com --deep-crawl bfs --max-pages 10

The first command printed the same Markdown as our Python example. The README also shows -q "Extract all product prices" for asking a question about a page, which needs an LLM key set with crwl config.

Crawl4AI in Docker

For a shared service, run the Docker server. Since 0.9.0 it requires a token; without one it only answers inside its container. These are the README's commands:

export CRAWL4AI_API_TOKEN="$(openssl rand -hex 32)"
docker run -d -p 11235:11235 --name crawl4ai --shm-size=1g \
  -e CRAWL4AI_API_TOKEN="$CRAWL4AI_API_TOKEN" \
  unclecode/crawl4ai:latest

curl -s http://localhost:11235/md \
  -H "Authorization: Bearer $CRAWL4AI_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://news.ycombinator.com"}' | jq -r .markdown

The server exposes /md, /html, /crawl, /crawl/stream, /screenshot, /pdf and /execute_js, plus a dashboard at /dashboard and a playground at /playground. The self-hosting guide asks for at least 4 GB of RAM for the container. LLM keys go in a .llm.env file passed with --env-file.

Crawl4AI MCP: connect it to an AI agent

The Docker server also speaks MCP, over Server-Sent Events at http://localhost:11235/mcp/sse and WebSocket at ws://localhost:11235/mcp/ws. The self-hosting guide adds it to Claude Code like this:

claude mcp add --transport sse c4ai-sse http://localhost:11235/mcp/sse

The agent then gets the tools md, html, screenshot, pdf, execute_js, crawl and ask (which queries the Crawl4AI library docs). Crawl4AI Cloud has a hosted MCP endpoint too, at https://api.crawl4ai.com/mcp with your Cloud key.

For a wider view of the options, see our roundup of the best MCP servers for web scraping.

What to watch out for when you self-host

Crawl4AI hands you a capable crawler. Running it in production means running the parts around it too:

• The browser fleet. Every crawl drives Playwright browsers. You size the machines, watch memory and restart what crashes.
• Proxies and blocking. Crawl4AI supports proxies with rotation and a stealth mode, but you choose, buy and rotate the proxies.
• Scaling and scheduling. arun_many runs URLs in parallel on one machine. Queues, retries across machines and nightly schedules are your code.
• Storage. Results live in your process until you write them somewhere.
• Site changes. CSS schemas break when a layout changes. Re-check them, or let an LLM regenerate them.
• Rules and privacy. Crawl public pages, respect rate limits and terms, and treat personal data under the GDPR and CCPA.

A hosted alternative: Scrape.ly's Website Content Crawler

If you want the output of a Crawl4AI deep crawl without running browsers, proxies or servers, the Website Content Crawler on Scrape.ly does the same job as a cloud run. It follows same-site links from your start URLs, strips navigation and boilerplate with Mozilla Readability (or Extractus or Defuddle), and saves text and markdown for every page with its title, description, canonical URL, language and Open Graph data. Server-rendered pages go over HTTP with a Chrome TLS fingerprint; pages whose content only appears after JavaScript open in Chrome automatically. Proxies are included.

The Crawl4AI settings map over directly:

A real run: 10 pages of the Python tutorial

We ran it on 22 October 2026 from the Store preset Python tutorial as markdown, with the page limit raised to 10. It saved 10 pages, about 175,000 characters of Markdown, in 7.2 seconds and was charged $0.0058: $0.005 for 10 crawled pages at $0.50 per 1,000, plus $0.0008 of compute, proxy traffic and dataset writes.

Because the run took 7 seconds, the synchronous endpoint returns the pages directly:

curl -X POST "https://api.scrape.ly/v2/actors/querystone~website-content-crawler/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": [{"url": "https://docs.python.org/3/tutorial/"}], "maxCrawlPages": 10, "maxResults": 10, "proxyConfiguration": {"useScrapelyProxy": true}}'

For larger crawls, the Python client waits as long as the run takes:

import os
from scrapely_client import ScrapelyClient

client = ScrapelyClient(os.environ["SCRAPELY_TOKEN"], api_url="https://api.scrape.ly")

run = client.actor("querystone/website-content-crawler").call(
    run_input={
        "startUrls": [{"url": "https://docs.python.org/3/tutorial/"}],
        "maxCrawlPages": 500,
        "proxyConfiguration": {"useScrapelyProxy": True},
    }
)
print(run.status)  # SUCCEEDED, FAILED, ABORTED or TIMED-OUT
for page in client.dataset(run.default_dataset_id).iterate_items():
    print(page["url"], len(page.get("markdown") or ""))

Your agent can run it too. Add the hosted Scrape.ly MCP server to Claude Code with claude mcp add --transport http scrapely https://api.scrape.ly/mcp, sign in with /mcp, and ask: "Use scrapely to crawl docs.python.org/3/tutorial with querystone/website-content-crawler, 10 pages, and summarize each page." The agents page covers every client.

What you get on top of the crawl: a free plan with $5 of usage every month and no card (about 8,000 pages at this run's rate), a dollar cap per run with maxTotalChargeUsd, schedules, webhooks, exports to JSON, CSV or Excel, and 567 other Actors in the Store for sites that need a dedicated scraper. If you're weighing Crawl4AI against Firecrawl as well, read Crawl4AI vs Firecrawl.

FAQ

Is Crawl4AI free?

Yes. The library, CLI and Docker server are open source under the Apache 2.0 license, and you pay only for the machines, proxies and LLM tokens you use. Crawl4AI Cloud, the hosted version, includes 1,000 free credits a month, with paid plans from $10.

Does Crawl4AI need an API key or an LLM?

No. Crawling, Markdown, CSS, XPath and regex extraction and deep crawls all run without a key. You need an LLM key only for LLMExtractionStrategy, LLMContentFilter, schema generation or crwl -q; local models through Ollama work too.

Can Crawl4AI crawl JavaScript-heavy pages?

Yes. It drives real Playwright browsers, so it can run JavaScript, wait for elements and scroll the full page with scan_full_page for infinite scroll and lazy images.

Does Crawl4AI have an MCP server?

Yes. The self-hosted Docker server exposes MCP over SSE at /mcp/sse and WebSocket at /mcp/ws, and Crawl4AI Cloud has a hosted MCP endpoint. The Scrape.ly MCP server is hosted and signs in with OAuth, with a crawler plus 567 other Actors behind it.

What port does the Crawl4AI Docker image use?

Port 11235. Since version 0.9.0 you must pass CRAWL4AI_API_TOKEN, or the server only listens inside its container.

Is there a hosted alternative to Crawl4AI?

Scrape.ly's Website Content Crawler runs the same kind of crawl in the cloud for $0.50 per 1,000 pages plus usage. Our 10-page test cost $0.0058, proxy traffic included.