--- title: "Crawl4AI Tutorial: Setup, Examples, Deep Crawl | Scrape.ly" description: "Crawl4AI tutorial tested on v0.9.5: install, first crawl to Markdown, CSS and LLM extraction, deep crawling, Docker and MCP, plus a hosted alternative." url: https://scrape.ly/blog/crawl4ai-guide markdown: https://scrape.ly/blog/crawl4ai-guide.md --- [AI agents](https://scrape.ly/blog/category/ai-agents.md) # Crawl4AI Tutorial: Setup, Examples and a Hosted Alternative A tested Crawl4AI tutorial for v0.9.5: install it, crawl a page to Markdown, extract JSON with CSS selectors or an LLM, deep crawl a site, run it in Docker or over MCP, and see when a hosted crawler saves you the infrastructure. [Scrape.ly Team The team behind Scrape.ly](https://scrape.ly/blog/author/scrapely-team.md) - Published Oct 23, 2026 - Reading time 8 min read ![Pixel-art robot with an orange >\_ on its terminal head pulls a page of text out of a monitor showing the cyan Crawl4AI logo above lines of code, and drops it into a crate](https://scrape.ly/media/crawl4ai-guide.png?w=1080) Crawl4AI is an open-source Python web crawler that turns web pages into clean, LLM-ready Markdown and structured JSON. You install Crawl4AI with `pip install -U crawl4ai`, run `crawl4ai-setup` once to get a browser, and five lines of async Python give you a page as Markdown. This tutorial walks through everything you need for real work: installation, your first crawl, filtered Markdown, structured extraction with and without an LLM, deep crawling, the `crwl` command line, Docker and MCP. We ran every Python example on 23 October 2026 with Crawl4AI 0.9.5 on Python 3.12, against the practice sites books.toscrape.com and quotes.toscrape.com, and show the real output. At the end you'll see the hosted route: the same crawl as a cloud job on Scrape.ly, with its exact cost. ## Crawl4AI at a glance | | | | ---------- | ------------------------------------------------------------------------------ | | What it is | An async Python library, CLI and Docker server for crawling and scraping | | Version | 0.9.5, released 14 October 2026 ([PyPI](https://pypi.org/project/crawl4ai/)) | | License | Apache 2.0 | | Python | 3.10 or newer | | Browsers | Chromium, Firefox and WebKit through Playwright | | Output | Markdown (raw and filtered), cleaned HTML, links, media, screenshots, PDFs | | Extraction | CSS, XPath and regex schemas without an LLM; LLM extraction into a JSON schema | | Crawling | BFS, DFS and best-first deep crawls, sitemap seeding, adaptive crawling | | Server | Docker image with a REST API, dashboard, playground and MCP endpoints | | Hosted | Crawl4AI Cloud: 1,000 free credits a month, paid plans from $10 | ## How to install Crawl4AI Install the package and the browser it drives. The second command downloads Chromium through Playwright, which the library needs for every crawl: ```shell pip install -U crawl4ai crawl4ai-setup # installs the browser, once crawl4ai-doctor # checks the installation ``` If the browser step fails, the README's fallback installs Chromium and its system libraries by hand: `python -m playwright install --with-deps chromium`. To try it without touching your global Python, `uv` gives you a throwaway environment. This is how we ran every example below: ```shell uv run --python 3.12 --with crawl4ai==0.9.5 python first_crawl.py ``` ## Your first crawl: one page to Markdown `AsyncWebCrawler` opens a browser, `arun()` loads the page, and the result carries the Markdown along with the HTML, links, media and status code: ```python import asyncio from crawl4ai import AsyncWebCrawler async def main(): async with AsyncWebCrawler() as crawler: result = await crawler.arun(url="https://books.toscrape.com/") print(result.success, result.status_code) print(len(result.markdown), "characters of Markdown") print(result.markdown[:300]) asyncio.run(main()) ``` Our run printed `True 200` and 13,476 characters of Markdown, starting like this: ```plaintext [Books to Scrape](https://books.toscrape.com/index.html) We love being scraped! * [Home](https://books.toscrape.com/index.html) * All products * [ Books ](https://books.toscrape.com/catalogue/category/books_1/index.html) * [ Travel ](https://books.toscrape.com/catalogue/category/books/travel_2/index.html) ``` The raw HTML of that page is 51,294 bytes, so the Markdown is about a quarter of the size before any filtering. That matters when every character you send to a model costs tokens. The whole script, browser start-up included, took 3.5 seconds. ## Clean Markdown with fit_markdown The default Markdown keeps navigation and sidebars. Attach a content filter to the Markdown generator and you also get `fit_markdown`, which drops low-value blocks: ```python import asyncio from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode from crawl4ai.content_filter_strategy import PruningContentFilterLXML from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator async def main(): config = CrawlerRunConfig( cache_mode=CacheMode.BYPASS, markdown_generator=DefaultMarkdownGenerator( content_filter=PruningContentFilterLXML(threshold=0.48, threshold_type="fixed") ), ) async with AsyncWebCrawler() as crawler: result = await crawler.arun(url="https://quotes.toscrape.com/", config=config) print("raw:", len(result.markdown.raw_markdown)) print("fit:", len(result.markdown.fit_markdown)) asyncio.run(main()) ``` On quotes.toscrape.com the raw Markdown was 4,375 characters and the filtered version 3,663. Use `PruningContentFilterLXML`: in 0.9.5 the older `PruningContentFilter` still works but prints a deprecation warning saying the lxml version is about 10 times faster with identical output. For query-focused pages there is also `BM25ContentFilter`, and `LLMContentFilter` if you want a model to decide what stays. `CacheMode.BYPASS` makes every run fetch the page fresh. Crawl4AI caches results by default, which is handy in development and confusing when you're checking whether a page changed. ## Structured data without an LLM: CSS extraction For pages with a stable layout, a CSS schema is the fastest and cheapest extractor: no model, no tokens, the same output every run. This schema pulls every book card from the books.toscrape.com home page: ```python import asyncio, json from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode, JsonCssExtractionStrategy schema = { "name": "Books", "baseSelector": "article.product_pod", "fields": [ {"name": "title", "selector": "h3 a", "type": "attribute", "attribute": "title"}, {"name": "price", "selector": "p.price_color", "type": "text"}, {"name": "availability", "selector": "p.availability", "type": "text"}, {"name": "url", "selector": "h3 a", "type": "attribute", "attribute": "href"}, ], } async def main(): config = CrawlerRunConfig( extraction_strategy=JsonCssExtractionStrategy(schema), cache_mode=CacheMode.BYPASS, ) async with AsyncWebCrawler(config=BrowserConfig(headless=True)) as crawler: result = await crawler.arun(url="https://books.toscrape.com/", config=config) books = json.loads(result.extracted_content) print(len(books), "books") print(json.dumps(books[:2], indent=2, ensure_ascii=False)) asyncio.run(main()) ``` It returned all 20 books on the page in 1.6 seconds: ```json [ { "title": "A Light in the Attic", "price": "£51.77", "availability": "In stock", "url": "catalogue/a-light-in-the-attic_1000/index.html" }, { "title": "Tipping the Velvet", "price": "£53.74", "availability": "In stock", "url": "catalogue/tipping-the-velvet_999/index.html" } ] ``` `baseSelector` picks the repeating element, and each field reads text or an attribute inside it. `JsonXPathExtractionStrategy` and `RegexExtractionStrategy` work the same way, and `generate_schema` can ask an LLM once to write a schema you then reuse for free. ## Structured data with an LLM When layouts vary from site to site, `LLMExtractionStrategy` sends the page content to a model with your JSON schema and an instruction. The README says it works with any LLM provider. This is the README's own example; it needs an API key, so we didn't run it: ```python import os, asyncio from pydantic import BaseModel, Field from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode, LLMConfig, LLMExtractionStrategy class ModelFee(BaseModel): model_name: str = Field(..., description="Name of the model.") input_fee: str = Field(..., description="Fee for input tokens.") output_fee: str = Field(..., description="Fee for output tokens.") async def main(): run_config = CrawlerRunConfig( cache_mode=CacheMode.BYPASS, extraction_strategy=LLMExtractionStrategy( llm_config=LLMConfig(provider="openai/gpt-4o-mini", api_token=os.getenv("OPENAI_API_KEY")), schema=ModelFee.model_json_schema(), extraction_type="schema", instruction="Extract every model name with its input and output token fee.", ), ) async with AsyncWebCrawler() as crawler: result = await crawler.arun(url="https://openai.com/api/pricing/", config=run_config) print(result.extracted_content) asyncio.run(main()) ``` LLM extraction costs tokens on every page and can return slightly different output between runs, so validate the JSON before you store it. A good pattern: use the LLM to write a CSS schema once, then run the schema at scale. Our guide to [AI web scraping](https://scrape.ly/blog/ai-web-scraping.md) compares rule-based, LLM and agent extraction in more depth. ## Deep crawling a whole site To follow links beyond the first page, give `CrawlerRunConfig` a deep crawl strategy. Breadth-first is the simplest: ```python import asyncio from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode from crawl4ai.deep_crawling import BFSDeepCrawlStrategy async def main(): config = CrawlerRunConfig( deep_crawl_strategy=BFSDeepCrawlStrategy(max_depth=1, include_external=False, max_pages=5), cache_mode=CacheMode.BYPASS, ) async with AsyncWebCrawler() as crawler: results = await crawler.arun("https://quotes.toscrape.com/", config=config) for r in results: print(r.metadata.get("depth"), r.url, len(r.markdown)) asyncio.run(main()) ``` It stopped at 5 pages as asked, in 2.2 seconds, and each result carried its `depth` in `metadata`. The parameters that keep a crawl under control: - `max_depth`: levels beyond the start page. - `max_pages`: a hard stop on the page count. The default is unlimited, so always set it. - `include_external`: whether to leave the start domain. - `filter_chain` and `url_scorer`: URL patterns, domains and keyword scores. `DFSDeepCrawlStrategy` goes deep first, and `BestFirstCrawlingStrategy` with a `KeywordRelevanceScorer` visits the most relevant URLs first; the [deep crawling docs](https://docs.crawl4ai.com/core/deep-crawling/) recommend it. Pass `stream=True` to process pages as they arrive instead of waiting for the full list, and look at `resume_state` for crash recovery on long crawls. ## The crwl command line The package also installs `crwl`, which is handy for quick checks and shell scripts: ```shell crwl https://books.toscrape.com/ -o markdown crwl https://docs.crawl4ai.com --deep-crawl bfs --max-pages 10 ``` The first command printed the same Markdown as our Python example. The README also shows `-q "Extract all product prices"` for asking a question about a page, which needs an LLM key set with `crwl config`. ## Crawl4AI in Docker For a shared service, run the Docker server. Since 0.9.0 it requires a token; without one it only answers inside its container. These are the README's commands: ```shell export CRAWL4AI_API_TOKEN="$(openssl rand -hex 32)" docker run -d -p 11235:11235 --name crawl4ai --shm-size=1g -e CRAWL4AI_API_TOKEN="$CRAWL4AI_API_TOKEN" unclecode/crawl4ai:latest curl -s http://localhost:11235/md -H "Authorization: Bearer $CRAWL4AI_API_TOKEN" -H "Content-Type: application/json" -d '{"url": "https://news.ycombinator.com"}' jq -r .markdown ``` The server exposes `/md`, `/html`, `/crawl`, `/crawl/stream`, `/screenshot`, `/pdf` and `/execute_js`, plus a dashboard at `/dashboard` and a playground at `/playground`. The [self-hosting guide](https://docs.crawl4ai.com/core/self-hosting/) asks for at least 4 GB of RAM for the container. LLM keys go in a `.llm.env` file passed with `--env-file`. ## Crawl4AI MCP: connect it to an AI agent The Docker server also speaks MCP, over Server-Sent Events at `http://localhost:11235/mcp/sse` and WebSocket at `ws://localhost:11235/mcp/ws`. The self-hosting guide adds it to Claude Code like this: ```shell claude mcp add --transport sse c4ai-sse http://localhost:11235/mcp/sse ``` The agent then gets the tools `md`, `html`, `screenshot`, `pdf`, `execute_js`, `crawl` and `ask` (which queries the Crawl4AI library docs). Crawl4AI Cloud has a hosted MCP endpoint too, at `https://api.crawl4ai.com/mcp` with your Cloud key. For a wider view of the options, see our roundup of the [best MCP servers for web scraping](https://scrape.ly/blog/best-mcp-servers-for-web-scraping.md). ## What to watch out for when you self-host Crawl4AI hands you a capable crawler. Running it in production means running the parts around it too: - **The browser fleet.** Every crawl drives Playwright browsers. You size the machines, watch memory and restart what crashes. - **Proxies and blocking.** Crawl4AI supports proxies with rotation and a stealth mode, but you choose, buy and rotate the proxies. - **Scaling and scheduling.** `arun_many` runs URLs in parallel on one machine. Queues, retries across machines and nightly schedules are your code. - **Storage.** Results live in your process until you write them somewhere. - **Site changes.** CSS schemas break when a layout changes. Re-check them, or let an LLM regenerate them. - **Rules and privacy.** Crawl public pages, respect rate limits and terms, and treat personal data under the GDPR and CCPA. ## A hosted alternative: Scrape.ly's Website Content Crawler If you want the output of a Crawl4AI deep crawl without running browsers, proxies or servers, the [Website Content Crawler](https://scrape.ly/store/querystone/website-content-crawler.md) on Scrape.ly does the same job as a cloud run. It follows same-site links from your start URLs, strips navigation and boilerplate with Mozilla Readability (or Extractus or Defuddle), and saves `text` and `markdown` for every page with its title, description, canonical URL, language and Open Graph data. Server-rendered pages go over HTTP with a Chrome TLS fingerprint; pages whose content only appears after JavaScript open in Chrome automatically. Proxies are included. The Crawl4AI settings map over directly: | Crawl4AI | Website Content Crawler | | ------------------------------------ | -------------------------------------------------------------- | | `BFSDeepCrawlStrategy(max_pages=…)` | `maxCrawlPages` and `maxResults` | | `max_depth` | `maxCrawlDepth` | | `filter_chain` URL patterns | `includeUrlGlobs`, `excludeUrlGlobs` | | `fit_markdown` with a content filter | `htmlTransformer` (`readableText` by default) | | `AsyncUrlSeeder` for sitemaps | `useSitemaps`, `useLlmsTxt` | | Your proxy settings | `proxyConfiguration` (Scrape.ly proxy, residential by default) | ### A real run: 10 pages of the Python tutorial We ran it on 22 October 2026 from the Store preset [Python tutorial as markdown](https://scrape.ly/store/querystone/website-content-crawler/presets/crawl-5-pages.md), with the page limit raised to 10. It saved 10 pages, about 175,000 characters of Markdown, in 7.2 seconds and was charged $0.0058: $0.005 for 10 crawled pages at $0.50 per 1,000, plus $0.0008 of compute, proxy traffic and dataset writes. Because the run took 7 seconds, the synchronous endpoint returns the pages directly: ```shell curl -X POST "https://api.scrape.ly/v2/actors/querystone~website-content-crawler/run-sync-get-dataset-items" -H "Authorization: Bearer $SCRAPELY_TOKEN" -H "Content-Type: application/json" -d '{"startUrls": [{"url": "https://docs.python.org/3/tutorial/"}], "maxCrawlPages": 10, "maxResults": 10, "proxyConfiguration": {"useScrapelyProxy": true}}' ``` For larger crawls, the Python client waits as long as the run takes: ```python import os from scrapely_client import ScrapelyClient client = ScrapelyClient(os.environ["SCRAPELY_TOKEN"], api_url="https://api.scrape.ly") run = client.actor("querystone/website-content-crawler").call( run_input={ "startUrls": [{"url": "https://docs.python.org/3/tutorial/"}], "maxCrawlPages": 500, "proxyConfiguration": {"useScrapelyProxy": True}, } ) print(run.status) # SUCCEEDED, FAILED, ABORTED or TIMED-OUT for page in client.dataset(run.default_dataset_id).iterate_items(): print(page["url"], len(page.get("markdown") or "")) ``` Your agent can run it too. Add the hosted [Scrape.ly MCP server](https://scrape.ly/mcp.md) to Claude Code with `claude mcp add --transport http scrapely https://api.scrape.ly/mcp`, sign in with `/mcp`, and ask: "Use scrapely to crawl docs.python.org/3/tutorial with querystone/website-content-crawler, 10 pages, and summarize each page." The [agents page](https://scrape.ly/agents.md) covers every client. What you get on top of the crawl: a free plan with $5 of usage every month and no card (about 8,000 pages at this run's rate), a dollar cap per run with `maxTotalChargeUsd`, schedules, webhooks, exports to JSON, CSV or Excel, and 567 other Actors in the [Store](https://scrape.ly/store.md) for sites that need a dedicated scraper. If you're weighing Crawl4AI against Firecrawl as well, read [Crawl4AI vs Firecrawl](https://scrape.ly/blog/crawl4ai-vs-firecrawl.md). ## FAQ ### Is Crawl4AI free? Yes. The library, CLI and Docker server are open source under the Apache 2.0 license, and you pay only for the machines, proxies and LLM tokens you use. Crawl4AI Cloud, the hosted version, includes 1,000 free credits a month, with paid plans from $10. ### Does Crawl4AI need an API key or an LLM? No. Crawling, Markdown, CSS, XPath and regex extraction and deep crawls all run without a key. You need an LLM key only for `LLMExtractionStrategy`, `LLMContentFilter`, schema generation or `crwl -q`; local models through Ollama work too. ### Can Crawl4AI crawl JavaScript-heavy pages? Yes. It drives real Playwright browsers, so it can run JavaScript, wait for elements and scroll the full page with `scan_full_page` for infinite scroll and lazy images. ### Does Crawl4AI have an MCP server? Yes. The self-hosted Docker server exposes MCP over SSE at `/mcp/sse` and WebSocket at `/mcp/ws`, and Crawl4AI Cloud has a hosted MCP endpoint. The Scrape.ly MCP server is hosted and signs in with OAuth, with a crawler plus 567 other Actors behind it. ### What port does the Crawl4AI Docker image use? Port 11235. Since version 0.9.0 you must pass `CRAWL4AI_API_TOKEN`, or the server only listens inside its container. ### Is there a hosted alternative to Crawl4AI? Scrape.ly's Website Content Crawler runs the same kind of crawl in the cloud for $0.50 per 1,000 pages plus usage. Our 10-page test cost $0.0058, proxy traffic included. Scrape.ly Store Crawl a site to Markdown without servers The hosted Website Content Crawler: clean Markdown with source metadata for $0.50 per 1,000 pages, on your free $5 monthly credit. [Try it free](https://scrape.ly/store/querystone/website-content-crawler.md) [Browse the Store](https://scrape.ly/store.md) About the author [Scrape.ly Team](https://scrape.ly/blog/author/scrapely-team.md) The team behind Scrape.ly Scrape.ly is the web data platform from TeraShift GmbH in Switzerland, powered by Evomi: ready-made scrapers in the Scrape.ly Store, a cloud runtime, an API with JavaScript and Python SDKs, proxies, a hosted anti-bot browser and an MCP server for AI agents. - [Website](https://scrape.ly/index.md) Keep reading ## More in AI agents - [![Pixel-art robot with an orange >\_ on its terminal head carries a cable from an office tower with the blue LinkedIn logo on its sign into a server rack labelled MCP](https://scrape.ly/media/linkedin-mcp.png?w=750)](https://scrape.ly/blog/linkedin-mcp.md) ### [LinkedIn MCP: Give Claude, Cursor and ChatGPT LinkedIn Data Without Your Login](https://scrape.ly/blog/linkedin-mcp.md) [A LinkedIn MCP server lets your AI agent pull company pages, job listings, posts and ads into the chat. Set one up in Claude, Cursor or ChatGPT in a minute, without handing over your LinkedIn session, and see the exact tool calls and what came back.](https://scrape.ly/blog/linkedin-mcp.md) - [![Pixel-art robot with an orange >\_ on its terminal head carries a cable from a feed of posts headed by the orange Reddit logo into a server rack labelled MCP](https://scrape.ly/media/reddit-mcp.png?w=750)](https://scrape.ly/blog/reddit-mcp.md) ### [Reddit MCP: Connect Claude or Cursor to Reddit Data Without API Keys](https://scrape.ly/blog/reddit-mcp.md) [A Reddit MCP server lets Claude, Cursor or ChatGPT search Reddit, read subreddits and pull comment threads mid-conversation. Set one up in a minute without Reddit API credentials, and see the exact tool calls, a real run and what it cost.](https://scrape.ly/blog/reddit-mcp.md) - [![Pixel-art robot with an orange >\_ on its terminal head hands a data card from a web page on the left into a chat window whose header shows the ChatGPT logo and the orange Claude spark](https://scrape.ly/media/web-scraping-with-chatgpt.png?w=750)](https://scrape.ly/blog/web-scraping-with-chatgpt.md) ### [ChatGPT Web Scraping (and Claude): What Actually Works in 2026](https://scrape.ly/blog/web-scraping-with-chatgpt.md) [What ChatGPT and Claude can and can't do for web scraping today: search, code generation, code execution and MCP connectors. Plus a tested scraper you can ask either one to write, and how to connect them to Scrape.ly so they run real scrapers for you.](https://scrape.ly/blog/web-scraping-with-chatgpt.md) ## The internet is full of data. Start owning yours. Scrape.ly gives you the infrastructure, the tooling, and the team to extract, process, and operationalize web data at any scale. Setup takes minutes. The advantage lasts forever. [Get Started Free](https://my.scrape.ly/register) [Chat with Support](mailto:support@scrape.ly?subject=Question%20about%20Scrape.ly\&body=Hi%2C%20I%20have%20a%20question%20about%20Scrape.ly.) No credit card. No setup calls.