RAG Web Scraping: How to Turn Any Website Into Markdown for LLMs

RAG web scraping means crawling a website, turning each page into clean Markdown, and splitting that Markdown into chunks a retrieval-augmented generation (RAG) system can search and quote. Done well, your model answers from the site's real content with a source URL for every claim; done badly, it retrieves menus, cookie banners and paragraphs whose code examples quietly went missing.

This guide walks through the whole pipeline with a real crawl. On 26 October 2026 we crawled 10 pages of the FastAPI tutorial with the Website Content Crawler twice, once with each of two HTML cleaners, and the difference was 78 code blocks. Below you'll find both runs, the Python that cleans and chunks the result (tested on that output), token counts, and what it costs per 1,000 pages.

Why Markdown is the right format for RAG

A model reads tokens, and you pay for every one you embed or put in a prompt. HTML spends most of them on markup. Plain text is barely smaller than Markdown and loses the structure a retriever needs. Markdown keeps headings, lists, tables and code fences while dropping almost everything else.

The Markdown came to about 8% of the HTML's size, and the headings it kept are exactly what the chunker below splits on.

The RAG web scraping pipeline

Every website-to-RAG pipeline has the same five steps:

1. Crawl. Start from a URL, follow links under that path, stop at a page limit.
2. Clean. Strip navigation, footers and scripts; convert the main content to Markdown.
3. Chunk. Split each page into pieces of a few hundred tokens, keeping the heading path and URL with each piece.
4. Embed and store. Turn each chunk into a vector and save it with its metadata.
5. Refresh. Re-crawl on a schedule so answers don't go stale.

Steps 1 and 2 are the hard part, because every site is built differently. That's where a hosted crawler saves you the work.

A real crawl: 10 pages of the FastAPI tutorial

The Website Content Crawler follows same-site links from your start URLs, fetches server-rendered pages over HTTP with a Chrome TLS fingerprint, opens pages that need JavaScript in Chrome, and saves text and markdown for each page with its title, description, canonical URL and language. This was our input:

{
  "startUrls": [{"url": "https://fastapi.tiangolo.com/tutorial/"}],
  "maxCrawlPages": 10,
  "maxResults": 10,
  "respectRobotsTxtFile": true,
  "saveMarkdown": true,
  "proxyConfiguration": {"useScrapelyProxy": true}
}

It saved 10 pages in 9 seconds. The log tells you how the crawl went and why it stopped:

2026-10-26T14:25:43.365Z [scrapely.actor] INFO  Fetching with HTTP. Chrome is used when a page needs JavaScript or a browser input is set.
2026-10-26T14:25:49.842Z [scrapely.actor] INFO  Saved 10 content items. Timing: python 1.2s, imports 0.2s, start 0.3s, crawl 6.5s, 11 requests, median 0.9s, max 2.6s, hedged 0 (won 0). Proxy: datacenter first, 0 host(s) moved to residential
2026-10-26T14:25:49.844Z [scrapely.actor] INFO  [Scrapely] Saved 10 pages. Stopped at Max pages (10); raise it to crawl more.

The cleaner decides what survives

That first run used the default HTML transformer, readableText (Mozilla Readability). The Markdown looked clean, and that's the trap. Here is the start of the "Path Parameters" page:

# Path Parameters

You can declare path "parameters" or "variables" with the same syntax used by Python format strings:

The value of the path parameter `item_id` will be passed to your function as the argument `item_id`.

The sentence promises an example and the next paragraph explains it, but the example itself is gone. FastAPI shows code in tabbed widgets (one tab per Python version), and Readability dropped them as page furniture. A RAG system built on this would answer "how do I declare a path parameter?" without the one thing the reader needs.

So we ran the same crawl again with one change, "htmlTransformer": "defuddle". The Actor's input description says Defuddle keeps math, footnotes and code better. On the same page, the sentence is now followed by the tab's label ("Python 3.10+") and the example as a fenced code block in the Markdown:

from fastapi import FastAPI

app = FastAPI()


@app.get("/items/{item_id}")
async def read_item(item_id):
    return {"item_id": item_id}

Across all 10 pages:

Defuddle kept 78 more code blocks, and the Markdown grew from 85,436 to 114,940 characters (21,516 to 29,587 tokens with OpenAI's cl100k_base encoding). The second run took 10 seconds. The lesson is general: for docs, help centers and anything with code, tabs or footnotes, try defuddle and compare a page by eye before you index a whole site. For articles and blogs, the default Readability output is usually what you want.

Defuddle left two kinds of noise, which the cleaning step below removes: heading permalinks ([¶](#path-parameters "Permanent link")) and inline HTML from FastAPI's coloured terminal examples (<font color=…>, <span style=…>).

Run the crawl from code

Create an API token under Settings → API & Integrations and export it as SCRAPELY_TOKEN. A 10-page crawl finishes in about 10 seconds, so the synchronous endpoint returns the pages directly:

curl -X POST "https://api.scrape.ly/v2/actors/querystone~website-content-crawler/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": [{"url": "https://fastapi.tiangolo.com/tutorial/"}], "maxCrawlPages": 10, "maxResults": 10, "htmlTransformer": "defuddle", "respectRobotsTxtFile": true, "proxyConfiguration": {"useScrapelyProxy": true}}'

For a whole site, use the Python client (pip install scrapely-client-python), which waits however long the run takes:

import os
from scrapely_client import ScrapelyClient

client = ScrapelyClient(os.environ["SCRAPELY_TOKEN"], api_url="https://api.scrape.ly")

run = client.actor("querystone/website-content-crawler").call(
    run_input={
        "startUrls": [{"url": "https://fastapi.tiangolo.com/tutorial/"}],
        "maxCrawlPages": 500,
        "htmlTransformer": "defuddle",
        "respectRobotsTxtFile": True,
        "proxyConfiguration": {"useScrapelyProxy": True},
    }
)
pages = list(client.dataset(run.default_dataset_id).iterate_items())
print(run.status, len(pages), "pages")

Clean and chunk the Markdown

Chunking decides what the retriever can find. Split on headings so each chunk is about one thing, never cut through a code fence, and keep the URL and heading path with every chunk so the model can cite it. This script does all three (pip install tiktoken):

import re
import tiktoken

enc = tiktoken.get_encoding("cl100k_base")
MAX_TOKENS = 800

def clean(md):
    md = re.sub(r'\[¶\]\(#[^)]*\)', '', md)                 # heading permalinks
    md = re.sub(r'</?(font|span|u|b)\b[^>]*>', '', md)       # inline HTML from terminal widgets
    return re.sub(r'\n{3,}', '\n\n', md).strip()

def sections(md):
    """Split on #, ## and ### headings, never inside a ``` code fence."""
    heading_path, buf, in_code = [], [], False
    for line in md.splitlines():
        if line.startswith("```"):
            in_code = not in_code
        m = None if in_code else re.match(r'^(#{1,3}) (.+)', line)
        if m:
            if buf:
                yield heading_path[:], "\n".join(buf).strip()
            level = len(m.group(1))
            heading_path = heading_path[:level - 1] + [m.group(2).strip()]
            buf = []
        buf.append(line)
    if buf:
        yield heading_path[:], "\n".join(buf).strip()

def split_long(text):
    """Pack paragraphs (code blocks stay whole) into pieces under MAX_TOKENS."""
    blocks, cur, in_code = [], [], False
    for line in text.splitlines():
        if line.startswith("```"):
            in_code = not in_code
        cur.append(line)
        if not in_code and line.strip() == "":
            blocks.append("\n".join(cur)); cur = []
    if cur:
        blocks.append("\n".join(cur))
    piece = ""
    for b in blocks:
        if piece and len(enc.encode(piece + b)) > MAX_TOKENS:
            yield piece.strip(); piece = ""
        piece += b + "\n"
    if piece.strip():
        yield piece.strip()

def chunk_page(page):
    md = clean(page.get("markdown") or "")
    for path, body in sections(md):
        for i, piece in enumerate(split_long(body)):
            yield {
                "id": f'{page["url"]}#{"/".join(path)}:{i}',
                "url": page["url"],
                "title": page["metadata"].get("title"),
                "headings": path,
                "text": piece,
                "tokens": len(enc.encode(piece)),
            }

chunks = [c for p in pages for c in chunk_page(p)]
tokens = [c["tokens"] for c in chunks]
print(len(pages), "pages ->", len(chunks), "chunks")
print("tokens: total", sum(tokens), "median", sorted(tokens)[len(tokens) // 2], "max", max(tokens))

We ran it on the 10 pages from the Defuddle crawl. It printed:

10 pages -> 107 chunks
tokens: total 26954 median 221 max 662

The chunks total about 2,600 tokens less than the raw Markdown, mostly the permalinks and terminal markup that clean() removed, and every chunk stayed well under the 800-token cap. Each one carries an ID like https://fastapi.tiangolo.com/tutorial/#Tutorial - User Guide/AI Agent Skills:0, which makes re-indexing idempotent: the same section gets the same ID on the next crawl.

Chunking tips that matter more than the splitter

• Keep the heading path in the text you embed. Prepend "FastAPI > Path Parameters > Data validation" to the chunk. A paragraph that says "this gives you editor support" means nothing without it.
• Size by tokens, not characters. Code is token-dense. 200 to 800 tokens per chunk is a sensible range for docs; go smaller for FAQ-style content.
• Never split a code block. Half a function retrieves worse than none. The splitter above treats a fence as one unit.
• Store the URL and title as metadata, so answers can cite and link the source page.
• Deduplicate. The crawler already skips pages that share a canonical URL. Hash chunk text too, because sites repeat the same notes on many pages.

Load it into a vector store

The langchain-scrapely package runs an Actor and returns its results as LangChain documents. This is the shape from the LangChain integration docs, with the Markdown and source as metadata:

from langchain_core.documents import Document
from langchain_scrapely import ScrapelyWrapper

loader = ScrapelyWrapper().call_actor(
    actor_id="querystone/website-content-crawler",
    run_input={"startUrls": [{"url": "https://fastapi.tiangolo.com/tutorial/"}], "maxCrawlPages": 50, "htmlTransformer": "defuddle"},
    dataset_mapping_function=lambda item: Document(
        page_content=item.get("markdown") or item.get("text") or "",
        metadata={"source": item.get("url"), "title": (item.get("metadata") or {}).get("title")},
    ),
)
docs = loader.load()

From there, pass docs (or the chunks from the script above) to any embedding model and vector store. LlamaIndex and LangGraph integrations are covered in the docs too.

AI agents can skip the pipeline for one-off questions. Add the hosted Scrape.ly MCP server to your client (Claude Code: claude mcp add --transport http scrapely https://mcp.scrape.ly) and ask: "Use scrapely to crawl fastapi.tiangolo.com/tutorial with querystone/website-content-crawler, 10 pages, defuddle transformer, and tell me how path parameters are validated." For live search plus page content in one call, the RAG Web Browser fetches the top Google results as Markdown.

Keep the index fresh

Docs change. Schedule the crawl in Automations (daily, weekly or a cron expression), and use Only new items since the last run with url as the key field to send only new pages onward, or re-index everything and let the stable chunk IDs overwrite old vectors. Two inputs make discovery more complete: useSitemaps queues the URLs a site lists in its sitemap, and useLlmsTxt reads the site's /llms.txt and the Markdown files it links. Our guide to llms.txt explains the format.

What to watch out for

• Check one page by eye before you crawl a thousand. As the FastAPI run shows, a page can look clean and still be missing its most important content.
• Set a page limit. maxCrawlPages and maxResults stop a crawl of a large docs site from running away. Start small, then raise them.
• JavaScript-only pages. The crawler opens a page in Chrome when its text only appears after scripts run. Chrome pages cost more platform usage than HTTP pages, so don't force a browser input you don't need.
• Logins. Pages that redirect to a sign-in page aren't saved or charged; content behind a login can't be crawled.
• Blocking. With Scrape.ly Proxy on, pages load through datacenter IPs first and a site that blocks them moves to residential IPs for the rest of the run. Our FastAPI run never needed to switch.
• Rights and privacy. Crawl content you're allowed to use, honour robots.txt (respectRobotsTxtFile), and keep personal data out of your index unless you have a lawful basis.

What it costs

The Website Content Crawler charges $0.0005 per page saved with content, plus platform usage (compute, storage and proxy) at your plan's rates. The Store estimates usage at about $0.12 per 1,000 pages, so roughly $0.62 per 1,000 pages in total for server-rendered sites. Each of our 10-page runs came to 10 × $0.0005 = $0.005 in page fees plus about $0.0005 of platform usage; the account was billed $0.0045 because its plan has a Store discount. The Free plan's $5 of monthly credit covers about 8,000 pages at that rate, before you've paid anything.

How Scrape.ly does RAG web scraping

• One Actor for the hard part. Crawling, rendering when needed, a choice of HTML cleaners (Readability, Extractus, Defuddle), Markdown, metadata, sitemaps and llms.txt, with failed URLs in a separate errors dataset instead of a failed run.
• Pay per page. A fixed page fee, usage at plan rates, and a maximum charge per run that stops the crawl cleanly when reached.
• Built for pipelines. Schedules, "only new items" filters, webhooks, S3/R2, LangChain and LlamaIndex loaders, and an MCP server for agents.
• The rest of the web too. 570+ Actors in the Store for sites where you want fields, not Markdown. If you're comparing open-source crawlers, see the Crawl4AI tutorial and the ScrapeGraphAI guide.

FAQ

How do I convert a website to Markdown?

Give a crawler the start URL and a page limit, and have it save Markdown for each page. With the Website Content Crawler that's startUrls, maxCrawlPages and the default saveMarkdown: true; switch htmlTransformer to defuddle for documentation with code.

Can I convert a single URL to Markdown?

Yes. Put the one URL in startUrls and set maxCrawlDepth to 0, so only that page is fetched. The run returns its Markdown, text and metadata in a few seconds.

What chunk size should I use for RAG?

For documentation, 200 to 800 tokens per chunk, split on headings, works well; our 10 FastAPI pages became 107 chunks with a median of 221 tokens. Keep code blocks whole and include the heading path in each chunk.

Is Markdown better than HTML for LLMs?

For retrieval, yes. Our 10 pages were 1,471,033 bytes of HTML and 114,940 characters of Markdown, with headings and code fences intact. Fewer tokens per page means cheaper embeddings and more room in the context window for the content that matters.

How do I scrape a website for an LLM without writing a crawler?

Run a hosted crawler such as the Website Content Crawler from the console, the API or your AI agent over MCP, then chunk and embed the Markdown it returns. You skip browsers, proxies and retry logic entirely.