--- title: "RAG Web Scraping: Website to Markdown for LLMs | Scrape.ly" description: "RAG web scraping step by step: crawl a site to clean Markdown, keep code blocks, chunk by heading and load it into a vector store. Real crawl, tested code." url: https://scrape.ly/blog/website-to-markdown-for-llms markdown: https://scrape.ly/blog/website-to-markdown-for-llms.md --- [AI agents](https://scrape.ly/blog/category/ai-agents.md) # RAG Web Scraping: How to Turn Any Website Into Markdown for LLMs RAG web scraping turns a website into clean Markdown your model can retrieve from. A real 10-page crawl of the FastAPI docs shows why the HTML cleaner decides whether your code examples survive, plus tested chunking code, token counts and the cost per 1,000 pages. [Querystone Actor publisher on the Scrape.ly Store](https://scrape.ly/blog/author/querystone.md) - Published Oct 26, 2026 - Reading time 9 min read ![Pixel-art robot with an orange >\_ on its terminal head feeds a Markdown file with an orange # heading into the slot of a dark box labelled LLM, next to a web page and a small stack of Markdown files](https://scrape.ly/media/website-to-markdown-for-llms.png?w=1080) RAG web scraping means crawling a website, turning each page into clean Markdown, and splitting that Markdown into chunks a retrieval-augmented generation (RAG) system can search and quote. Done well, your model answers from the site's real content with a source URL for every claim; done badly, it retrieves menus, cookie banners and paragraphs whose code examples quietly went missing. This guide walks through the whole pipeline with a real crawl. On 26 October 2026 we crawled 10 pages of the FastAPI tutorial with the [Website Content Crawler](https://scrape.ly/store/querystone/website-content-crawler.md) twice, once with each of two HTML cleaners, and the difference was 78 code blocks. Below you'll find both runs, the Python that cleans and chunks the result (tested on that output), token counts, and what it costs per 1,000 pages. ## Why Markdown is the right format for RAG A model reads tokens, and you pay for every one you embed or put in a prompt. HTML spends most of them on markup. Plain text is barely smaller than Markdown and loses the structure a retriever needs. Markdown keeps headings, lists, tables and code fences while dropping almost everything else. | Format | Size of our 10 FastAPI pages | Keeps headings and code fences | Good for chunking | | -------------- | ---------------------------------------- | ------------------------------ | ------------------------------- | | Raw HTML | 1,471,033 bytes | Yes, buried in markup | No | | Plain text | 107,464 characters | No | Poorly: no headings to split on | | Clean Markdown | 114,940 characters (about 29,600 tokens) | Yes | Yes: split on `#`, `##`, `###` | The Markdown came to about 8% of the HTML's size, and the headings it kept are exactly what the chunker below splits on. ## The RAG web scraping pipeline Every website-to-RAG pipeline has the same five steps: 1. **Crawl.** Start from a URL, follow links under that path, stop at a page limit. 2. **Clean.** Strip navigation, footers and scripts; convert the main content to Markdown. 3. **Chunk.** Split each page into pieces of a few hundred tokens, keeping the heading path and URL with each piece. 4. **Embed and store.** Turn each chunk into a vector and save it with its metadata. 5. **Refresh.** Re-crawl on a schedule so answers don't go stale. Steps 1 and 2 are the hard part, because every site is built differently. That's where a hosted crawler saves you the work. ## A real crawl: 10 pages of the FastAPI tutorial The Website Content Crawler follows same-site links from your start URLs, fetches server-rendered pages over HTTP with a Chrome TLS fingerprint, opens pages that need JavaScript in Chrome, and saves `text` and `markdown` for each page with its title, description, canonical URL and language. This was our input: ```json { "startUrls": [{"url": "https://fastapi.tiangolo.com/tutorial/"}], "maxCrawlPages": 10, "maxResults": 10, "respectRobotsTxtFile": true, "saveMarkdown": true, "proxyConfiguration": {"useScrapelyProxy": true} } ``` It saved 10 pages in 9 seconds. The log tells you how the crawl went and why it stopped: ```plaintext 20261026T14:25:43.365Z [scrapely.actor] INFO Fetching with HTTP. Chrome is used when a page needs JavaScript or a browser input is set. 20261026T14:25:49.842Z [scrapely.actor] INFO Saved 10 content items. Timing: python 1.2s, imports 0.2s, start 0.3s, crawl 6.5s, 11 requests, median 0.9s, max 2.6s, hedged 0 (won 0). Proxy: datacenter first, 0 host(s) moved to residential 20261026T14:25:49.844Z [scrapely.actor] INFO [Scrapely] Saved 10 pages. Stopped at Max pages (10); raise it to crawl more. ``` ### The cleaner decides what survives That first run used the default HTML transformer, `readableText` (Mozilla Readability). The Markdown looked clean, and that's the trap. Here is the start of the "Path Parameters" page: ```plaintext # Path Parameters You can declare path "parameters" or "variables" with the same syntax used by Python format strings: The value of the path parameter `item_id` will be passed to your function as the argument `item_id`. ``` The sentence promises an example and the next paragraph explains it, but the example itself is gone. FastAPI shows code in tabbed widgets (one tab per Python version), and Readability dropped them as page furniture. A RAG system built on this would answer "how do I declare a path parameter?" without the one thing the reader needs. So we ran the same crawl again with one change, `"htmlTransformer": "defuddle"`. The Actor's input description says Defuddle keeps math, footnotes and code better. On the same page, the sentence is now followed by the tab's label ("Python 3.10+") and the example as a fenced code block in the Markdown: ```python from fastapi import FastAPI app = FastAPI() @app.get("/items/{item_id}") async def read_item(item_id): return {"item_id": item_id} ``` Across all 10 pages: | Page | Code blocks, Readability | Code blocks, Defuddle | | --------------------------------------- | ------------------------ | --------------------- | | Tutorial - User Guide | 2 | 3 | | First Steps | 12 | 21 | | Path Parameters | 2 | 15 | | Query Parameters | 13 | 19 | | Query Parameters and String Validations | 32 | 55 | | Path Parameters and Numeric Validations | 8 | 18 | | Query Parameter Models | 8 | 11 | | Request Body | 2 | 8 | | Body - Multiple Parameters | 11 | 16 | | Body - Fields | 2 | 4 | | **Total** | **92** | **170** | Defuddle kept 78 more code blocks, and the Markdown grew from 85,436 to 114,940 characters (21,516 to 29,587 tokens with OpenAI's `cl100k_base` encoding). The second run took 10 seconds. The lesson is general: for docs, help centers and anything with code, tabs or footnotes, try `defuddle` and compare a page by eye before you index a whole site. For articles and blogs, the default Readability output is usually what you want. Defuddle left two kinds of noise, which the cleaning step below removes: heading permalinks (`[¶](#path-parameters "Permanent link")`) and inline HTML from FastAPI's coloured terminal examples (``, ``). Scrape.ly Store Website Content Crawler Crawl docs, help centers and blogs into clean Markdown with source metadata. $0.50 per 1,000 pages plus platform usage. [Try it free](https://scrape.ly/store/querystone/website-content-crawler.md) [Browse the Store](https://scrape.ly/store.md) ## Run the crawl from code Create an API token under **Settings → API & Integrations** and export it as `SCRAPELY_TOKEN`. A 10-page crawl finishes in about 10 seconds, so the synchronous endpoint returns the pages directly: ```shell curl -X POST "https://api.scrape.ly/v2/actors/querystone~website-content-crawler/run-sync-get-dataset-items" -H "Authorization: Bearer $SCRAPELY_TOKEN" -H "Content-Type: application/json" -d '{"startUrls": [{"url": "https://fastapi.tiangolo.com/tutorial/"}], "maxCrawlPages": 10, "maxResults": 10, "htmlTransformer": "defuddle", "respectRobotsTxtFile": true, "proxyConfiguration": {"useScrapelyProxy": true}}' ``` For a whole site, use the Python client (`pip install scrapely-client-python`), which waits however long the run takes: ```python import os from scrapely_client import ScrapelyClient client = ScrapelyClient(os.environ["SCRAPELY_TOKEN"], api_url="https://api.scrape.ly") run = client.actor("querystone/website-content-crawler").call( run_input={ "startUrls": [{"url": "https://fastapi.tiangolo.com/tutorial/"}], "maxCrawlPages": 500, "htmlTransformer": "defuddle", "respectRobotsTxtFile": True, "proxyConfiguration": {"useScrapelyProxy": True}, } ) pages = list(client.dataset(run.default_dataset_id).iterate_items()) print(run.status, len(pages), "pages") ``` ## Clean and chunk the Markdown Chunking decides what the retriever can find. Split on headings so each chunk is about one thing, never cut through a code fence, and keep the URL and heading path with every chunk so the model can cite it. This script does all three (`pip install tiktoken`): ````python import re import tiktoken enc = tiktoken.get_encoding("cl100k_base") MAX_TOKENS = 800 def clean(md): md = re.sub(r'\[¶\]\(#[^)]*\)', '', md) # heading permalinks md = re.sub(r']*>', '', md) # inline HTML from terminal widgets return re.sub(r'\n{3,}', '\n\n', md).strip() def sections(md): """Split on #, ## and ### headings, never inside a ``` code fence.""" heading_path, buf, in_code = [], [], False for line in md.splitlines(): if line.startswith("```"): in_code = not in_code m = None if in_code else re.match(r'^(#{1,3}) (.+)', line) if m: if buf: yield heading_path[:], "\n".join(buf).strip() level = len(m.group(1)) heading_path = heading_path[:level 1] + [m.group(2).strip()] buf = [] buf.append(line) if buf: yield heading_path[:], "\n".join(buf).strip() def split_long(text): """Pack paragraphs (code blocks stay whole) into pieces under MAX_TOKENS.""" blocks, cur, in_code = [], [], False for line in text.splitlines(): if line.startswith("```"): in_code = not in_code cur.append(line) if not in_code and line.strip() == "": blocks.append("\n".join(cur)); cur = [] if cur: blocks.append("\n".join(cur)) piece = "" for b in blocks: if piece and len(enc.encode(piece + b)) > MAX_TOKENS: yield piece.strip(); piece = "" piece += b + "\n" if piece.strip(): yield piece.strip() def chunk_page(page): md = clean(page.get("markdown") or "") for path, body in sections(md): for i, piece in enumerate(split_long(body)): yield { "id": f'{page["url"]}#{"/".join(path)}:{i}', "url": page["url"], "title": page["metadata"].get("title"), "headings": path, "text": piece, "tokens": len(enc.encode(piece)), } chunks = [c for p in pages for c in chunk_page(p)] tokens = [c["tokens"] for c in chunks] print(len(pages), "pages ->", len(chunks), "chunks") print("tokens: total", sum(tokens), "median", sorted(tokens)[len(tokens) // 2], "max", max(tokens)) ```` We ran it on the 10 pages from the Defuddle crawl. It printed: ```plaintext 10 pages -> 107 chunks tokens: total 26954 median 221 max 662 ``` The chunks total about 2,600 tokens less than the raw Markdown, mostly the permalinks and terminal markup that `clean()` removed, and every chunk stayed well under the 800-token cap. Each one carries an ID like `https://fastapi.tiangolo.com/tutorial/#Tutorial - User Guide/AI Agent Skills:0`, which makes re-indexing idempotent: the same section gets the same ID on the next crawl. ### Chunking tips that matter more than the splitter - **Keep the heading path in the text you embed.** Prepend `"FastAPI > Path Parameters > Data validation"` to the chunk. A paragraph that says "this gives you editor support" means nothing without it. - **Size by tokens, not characters.** Code is token-dense. 200 to 800 tokens per chunk is a sensible range for docs; go smaller for FAQ-style content. - **Never split a code block.** Half a function retrieves worse than none. The splitter above treats a fence as one unit. - **Store the URL and title as metadata**, so answers can cite and link the source page. - **Deduplicate.** The crawler already skips pages that share a canonical URL. Hash chunk text too, because sites repeat the same notes on many pages. ## Load it into a vector store The `langchain-scrapely` package runs an Actor and returns its results as LangChain documents. This is the shape from the [LangChain integration docs](https://scrape.ly/docs/langchain.md), with the Markdown and source as metadata: ```python from langchain_core.documents import Document from langchain_scrapely import ScrapelyWrapper loader = ScrapelyWrapper().call_actor( actor_id="querystone/website-content-crawler", run_input={"startUrls": [{"url": "https://fastapi.tiangolo.com/tutorial/"}], "maxCrawlPages": 50, "htmlTransformer": "defuddle"}, dataset_mapping_function=lambda item: Document( page_content=item.get("markdown") or item.get("text") or "", metadata={"source": item.get("url"), "title": (item.get("metadata") or {}).get("title")}, ), ) docs = loader.load() ``` From there, pass `docs` (or the chunks from the script above) to any embedding model and vector store. LlamaIndex and LangGraph integrations are covered in the [docs](https://scrape.ly/docs/llama-index.md) too. AI agents can skip the pipeline for one-off questions. Add the hosted [Scrape.ly MCP server](https://scrape.ly/mcp.md) to your client (Claude Code: `claude mcp add --transport http scrapely https://mcp.scrape.ly`) and ask: "Use scrapely to crawl fastapi.tiangolo.com/tutorial with querystone/website-content-crawler, 10 pages, defuddle transformer, and tell me how path parameters are validated." For live search plus page content in one call, the [RAG Web Browser](https://scrape.ly/store/querystone/rag-web-browser.md) fetches the top Google results as Markdown. ## Keep the index fresh Docs change. Schedule the crawl in **Automations** (daily, weekly or a cron expression), and use **Only new items since the last run** with `url` as the key field to send only new pages onward, or re-index everything and let the stable chunk IDs overwrite old vectors. Two inputs make discovery more complete: `useSitemaps` queues the URLs a site lists in its sitemap, and `useLlmsTxt` reads the site's `/llms.txt` and the Markdown files it links. Our guide to [llms.txt](https://scrape.ly/blog/what-is-llms-txt.md) explains the format. ## What to watch out for - **Check one page by eye before you crawl a thousand.** As the FastAPI run shows, a page can look clean and still be missing its most important content. - **Set a page limit.** `maxCrawlPages` and `maxResults` stop a crawl of a large docs site from running away. Start small, then raise them. - **JavaScript-only pages.** The crawler opens a page in Chrome when its text only appears after scripts run. Chrome pages cost more platform usage than HTTP pages, so don't force a browser input you don't need. - **Logins.** Pages that redirect to a sign-in page aren't saved or charged; content behind a login can't be crawled. - **Blocking.** With Scrape.ly Proxy on, pages load through datacenter IPs first and a site that blocks them moves to residential IPs for the rest of the run. Our FastAPI run never needed to switch. - **Rights and privacy.** Crawl content you're allowed to use, honour robots.txt (`respectRobotsTxtFile`), and keep personal data out of your index unless you have a lawful basis. ## What it costs The Website Content Crawler charges $0.0005 per page saved with content, plus platform usage (compute, storage and proxy) at your plan's rates. The Store estimates usage at about $0.12 per 1,000 pages, so roughly $0.62 per 1,000 pages in total for server-rendered sites. Each of our 10-page runs came to 10 × $0.0005 = $0.005 in page fees plus about $0.0005 of platform usage; the account was billed $0.0045 because its plan has a Store discount. The Free plan's $5 of monthly credit covers about 8,000 pages at that rate, before you've paid anything. ## How Scrape.ly does RAG web scraping - **One Actor for the hard part.** Crawling, rendering when needed, a choice of HTML cleaners (Readability, Extractus, Defuddle), Markdown, metadata, sitemaps and llms.txt, with failed URLs in a separate `errors` dataset instead of a failed run. - **Pay per page.** A fixed page fee, usage at plan rates, and a maximum charge per run that stops the crawl cleanly when reached. - **Built for pipelines.** Schedules, "only new items" filters, webhooks, S3/R2, LangChain and LlamaIndex loaders, and an MCP server for agents. - **The rest of the web too.** 570+ Actors in the [Store](https://scrape.ly/store.md) for sites where you want fields, not Markdown. If you're comparing open-source crawlers, see the [Crawl4AI tutorial](https://scrape.ly/blog/crawl4ai-guide.md) and the [ScrapeGraphAI guide](https://scrape.ly/blog/scrapegraphai-guide.md). ## FAQ ### How do I convert a website to Markdown? Give a crawler the start URL and a page limit, and have it save Markdown for each page. With the Website Content Crawler that's `startUrls`, `maxCrawlPages` and the default `saveMarkdown: true`; switch `htmlTransformer` to `defuddle` for documentation with code. ### Can I convert a single URL to Markdown? Yes. Put the one URL in `startUrls` and set `maxCrawlDepth` to 0, so only that page is fetched. The run returns its Markdown, text and metadata in a few seconds. ### What chunk size should I use for RAG? For documentation, 200 to 800 tokens per chunk, split on headings, works well; our 10 FastAPI pages became 107 chunks with a median of 221 tokens. Keep code blocks whole and include the heading path in each chunk. ### Is Markdown better than HTML for LLMs? For retrieval, yes. Our 10 pages were 1,471,033 bytes of HTML and 114,940 characters of Markdown, with headings and code fences intact. Fewer tokens per page means cheaper embeddings and more room in the context window for the content that matters. ### How do I scrape a website for an LLM without writing a crawler? Run a hosted crawler such as the Website Content Crawler from the console, the API or your AI agent over MCP, then chunk and embed the Markdown it returns. You skip browsers, proxies and retry logic entirely. Scrape.ly Store Turn a website into RAG-ready Markdown Clean Markdown, titles and source URLs for every page, with schedules and a cost cap. $0.50 per 1,000 pages plus platform usage. [Try it free](https://scrape.ly/store/querystone/website-content-crawler.md) [Browse the Store](https://scrape.ly/store.md) About the author [Querystone](https://scrape.ly/blog/author/querystone.md) Actor publisher on the Scrape.ly Store Search results, AI answers, web pages and market feeds as structured data. For SEO teams, developers and AI builders, plus finance, crypto and sports data for analysts. A Scrapely studio. - [Website](https://scrape.ly/users/querystone.md) Keep reading ## More in AI agents - [![Pixel-art robot with an orange >\_ on its terminal head carries a cable from a tall Google Maps pin into a server rack labelled MCP](https://scrape.ly/media/google-maps-mcp.png?w=750)](https://scrape.ly/blog/google-maps-mcp.md) ### [Google Maps MCP: Places and Reviews for Claude, ChatGPT and Cursor](https://scrape.ly/blog/google-maps-mcp.md) [A Google Maps MCP server lets your AI agent find places, ratings, opening hours and reviews right in the chat. Connect Scrape.ly's hosted server to Claude, ChatGPT or Cursor without a Google Cloud project or Maps API key, and see real tool calls, output and costs.](https://scrape.ly/blog/google-maps-mcp.md) - [![Pixel-art robot with an orange >\_ on its terminal head carries a cable from a phone showing the Instagram logo and a grid of posts into a server rack labelled MCP](https://scrape.ly/media/instagram-mcp.png?w=750)](https://scrape.ly/blog/instagram-mcp.md) ### [Instagram MCP: Give Claude, ChatGPT and Cursor Instagram Data Without a Login](https://scrape.ly/blog/instagram-mcp.md) [An Instagram MCP server lets your AI agent read public profiles, posts and reels right in the chat. Connect Scrape.ly's hosted server to Claude, ChatGPT or Cursor in a minute, with no Instagram login or Business account, and see real prompts, tool calls, output and costs.](https://scrape.ly/blog/instagram-mcp.md) - [![Pixel-art robot with an orange >\_ on its terminal head plugs a cable into a server rack labelled MCP with the Scrape.ly logo on its front, while two more cables run into it from web pages on the left](https://scrape.ly/media/best-mcp-servers-for-web-scraping.png?w=750)](https://scrape.ly/blog/best-mcp-servers-for-web-scraping.md) ### [6 Best Web Scraping MCP Servers in 2026 (Setup for Claude Code and Cursor)](https://scrape.ly/blog/best-mcp-servers-for-web-scraping.md) [The 6 best MCP servers for web scraping in 2026, compared on hosting, sign-in, tools, free tiers and spending control, with copy-paste setup for Claude Code, Cursor and VS Code and a real agent run.](https://scrape.ly/blog/best-mcp-servers-for-web-scraping.md) ## The internet is full of data. Start owning yours. Scrape.ly gives you the infrastructure, the tooling, and the team to extract, process, and operationalize web data at any scale. Setup takes minutes. [Get Started Free](https://my.scrape.ly/register) [Chat with Support](mailto:hello@scrape.ly?subject=Question%20about%20Scrape.ly\&body=Hi%2C%20I%20have%20a%20question%20about%20Scrape.ly.) No credit card. No setup calls.