Skip to content

Website Content Crawler

Try for free

Crawl whole websites and extract clean text and Markdown for AI models, vector databases, and RAG pipelines. Removes navigation and boilerplate and keeps titles, metadata, and links.

📄 Website Content Crawler

Turn public docs, help centers, and blogs into clean Markdown and text your RAG pipeline can index.

Built for teams who put customer-facing sites into a vector store or agent knowledge base and need a crawl that finishes, even when one URL fails.

✨ Features

  • 🔗 Start from startUrls and follow same-site links under that path, up to maxCrawlDepth, maxCrawlPages, and maxResults.
  • 🧭 Widen or narrow the crawl with includeUrlGlobs and excludeUrlGlobs. Start URLs are still fetched when an exclude glob matches them.
  • 📝 Save text and, when saveMarkdown is on, markdown from the page you actually downloaded. A JSON or plain-text response is saved as text. A page whose sentences exist only after JavaScript is opened in Chrome, and the saved text is the text that page shows. Server-rendered pages stay on HTTP unless you set a browser input.
  • 🧹 Clean the DOM with removeElementsCssSelector and keepElementsCssSelector, then run htmlTransformer: readableText, readableTextIfPossible, extractus, defuddle, or none.
  • 🏷️ Keep real page metadata: title, description, author, keywords, languageCode, canonicalUrl, openGraph, jsonLd, and response headers (cookie and authorization headers are left out).
  • 🗺️ Optionally enqueue pages from sitemaps (useSitemaps) and from /llms.txt plus same-host Markdown links (useLlmsTxt).
  • 🤖 Optionally skip paths disallowed by robots.txt (respectRobotsTxtFile).
  • 📎 With saveFiles or saveContentTypes, download linked files and store them only after the magic bytes match. PDF text and DOCX text are extracted when the file contains them. CSV is stored as text.
  • 🧾 A failed URL is written to the errors dataset. The rest of the crawl continues.
  • 🌍 proxyConfiguration uses Scrapely Proxy. When proxy is on and no group is set, the run uses the residential group. A proxy country you pick is passed through.

Server-rendered pages are fetched with HTTP and a Chrome TLS fingerprint. When the HTML is only a script that writes the sentences a visitor would see, that URL is loaded in Chrome and the dataset stores that rendered text. Chrome is also used when you set clickElementsCssSelector, dynamicContentWaitSecs, maxScrollHeightPixels, removeCookieWarnings, or blockMedia. Those inputs are applied on that page: matching elements are clicked, Chrome waits the number of seconds you set, the page is scrolled up to the pixel limit, cookie and consent banners are removed, and images, fonts, stylesheets, and videos are left unloaded when you ask for that. The form does not include saveScreenshots, summarize, or pageFunction. This actor does not save screenshots, write summaries, or run a page function.

🚀 Why teams use this

  • Production caps. maxCrawlPages and maxResults stop a runaway crawl of a large docs site.
  • One bad URL does not kill the run. You get the pages that loaded, and the failures sit in a separate dataset with errorCode and errorExplanation.
  • Output is ready for a vector index: plain text, Markdown when you ask for it, and source metadata you can cite.
  • Duplicate pages that share a canonical URL are not saved twice. Set ignoreCanonicalUrl when the site publishes a canonical you do not want to trust.
  • Scheduling, dataset export, and proxy are handled by scrape.ly. You do not have to host the crawler.

💡 Use cases

  • Turn documentation or a knowledge base into Markdown for a vector database.
  • Feed a whole site into a RAG pipeline or a custom GPT.
  • Audit a site's content and page titles for SEO.

🧪 Example input

{
  "blockMedia": false,
  "proxyConfiguration": {
    "useScrapelyProxy": true
  },
  "respectRobotsTxtFile": true,
  "startUrls": [
    {
      "url": "https://docs.python.org/3/tutorial/"
    }
  ],
  "storeSkippedUrls": false,
  "useLlmsTxt": false,
  "useSitemaps": false
}

🧾 Sample output

One item from a real run of the example input, shortened. Long text is cut and lists show their first entries.

{
  "url": "https://docs.python.org/3/tutorial/",
  "crawl": {
    "loadedUrl": "https://docs.python.org/3/tutorial/",
    "loadedTime": "2026-10-13T19:23:30.526Z",
    "depth": 0,
    "httpStatusCode": 200,
    "contentLengthBytes": 37315,
    "contentType": "text/html"
  },
  "metadata": {
    "canonicalUrl": "https://docs.python.org/3/tutorial/index.html",
    "title": "The Python Tutorial — Python 3.15.0 documentation",
    "description": "Python is an easy to learn, powerful programming language. It has efficient high-level data structures and a simple but effective approach to object-oriented…",
    "languageCode": "en",
    "openGraph": [
      {
        "property": "…",
        "content": "…"
      },
      {
        "property": "…",
        "content": "…"
      }
    ],
    "headers": {
      "server": "nginx",
      "content-type": "text/html",
      "last-modified": "Tue, 13 Oct 2026 15:50:36 GMT",
      "etag": "\"6ace534c-91c3\"",
      "x-clacks-overhead": "GNU Terry Pratchett",
      "strict-transport-security": "max-age=315360000; includeSubDomains; preload",
      "content-encoding": "gzip",
      "via": "1.1 varnish, 1.1 varnish",
      "accept-ranges": "bytes",
      "age": "1527",
      "date": "Tue, 13 Oct 2026 19:23:30 GMT",
      "x-served-by": "cache-lga-kjfk8660037-LGA, cache-pdk-kpdk2140044-PDK",
      "x-cache": "HIT, HIT",
      "x-cache-hits": "62, 0",
      "x-timer": "S1791919410.165256,VS0,VE1",
      "vary": "Accept-Encoding",
      "content-length": "7713"
    }
  },
  "text": "The Python Tutorial\n\nTip\n\nThis tutorial is designed for\n\nprogrammers\nthat are new to the Python language,\n\nnot\n\nbeginners\nwho are new to programming.\n\nPython…",
  "markdown": "# The Python Tutorial\n\nTip\n\nThis tutorial is designed for\n*programmers* that are new to the Python language,\n**not** *beginners* who are new to programming.\n…"
}

💰 Pricing

You pay $0.0005 per crawled page ($0.50 per 1,000 pages) plus platform usage. Only pages saved with content (text, Markdown or HTML) are charged. Skipped URLs, failed URLs (they go to the separate errors dataset) and pages without content are not charged.

Worked example: 1,000 pages ≈ $0.50 in event fees plus platform usage. On our test runs at the default 1024 MB, usage was about $0.002 for the 17-page example input and about $0.014 for 100 large documentation pages, so 1,000 pages come to roughly $0.65 in total.

Set a maximum charge for the run if you want a hard cap. The run then stops cleanly once it is reached and keeps every page saved so far.

📥 Input

Required: startUrls, proxyConfiguration.

FieldWhat it does
startUrlsPages to crawl, or a requestsFromUrl list of URLs.
includeUrlGlobs, excludeUrlGlobsExtra pages to allow, or linked pages to skip.
maxCrawlDepth, maxCrawlPages, maxResultsDepth, fetch count, and saved-item count.
useSitemaps, useLlmsTxt, respectRobotsTxtFileSitemap discovery, /llms.txt, robots.txt.
keepUrlFragments, ignoreCanonicalUrlKeep #fragments in the URL, or save pages even when the canonical was already saved.
proxyConfigurationScrapely Proxy on or off, with optional proxy groups and country, or your own proxy URLs.
initialCookies, customHttpHeadersSent on requests. A custom User-Agent is not applied over the TLS profile.
initialConcurrency, maxConcurrencyParallel fetches, capped at 8.
requestTimeoutSecs, maxRequestRetries, maxSessionRotationsTimeout and retry limits.
minFileDownloadSpeedKBpsAbort a file whose average speed stays under this for more than 10 seconds.
ignoreHttpsErrorsContinue when the TLS certificate does not verify.
dynamicContentWaitSecsAbove 0, open the page in Chrome and wait that many seconds before saving. 0 does not open Chrome by itself.
maxScrollHeightPixelsAbove 0, open the page in Chrome and scroll down to that height. 0 does not scroll.
removeCookieWarningsWhen on, open the page in Chrome and remove short cookie, consent, or GDPR banners.
blockMediaWhen on, open the page in Chrome and do not load images, fonts, stylesheets, or videos. Scripts still load.
clickElementsCssSelectorWhen set, open the page in Chrome and click matching elements that do not leave the page.
waitForSelectorFail that URL if the selector is missing from the fetched HTML. In Chrome, a non-zero dynamic wait also waits for this selector. softWaitForSelector does not fail the URL.
expandIframesFetch up to five same-host iframes and inline them.
keepElementsCssSelector, removeElementsCssSelectorDOM keep and remove lists.
htmlTransformer, readableTextCharThresholdWhich cleaner runs, and when readable text is long enough to keep.
aggressivePruneDrop long lines that already appeared on an earlier page.
saveMarkdown, saveHtml, saveHtmlAsFileMarkdown field, HTML field, or an HTML snapshot in the key-value store.
saveFiles, saveContentTypesDownload linked documents or responses whose content type matches.
debugMode, debugLog, storeSkippedUrlsTransformer lengths, verbose logs, and a SKIPPED_URLS record.

saveScreenshots, summarize, and pageFunction are not on the form. A customer cannot request a screenshot, a summary, or a page function.

crawlerType, signHttpRequests, clientSideMinChangePercentage, renderingTypeDetectionPercentage, reuseStoredDetectionResults, and stickyContainerCssSelector are not on the form. A run that sends one of them is rejected and does not start. This actor does not sign Web Bot Auth requests and does not guess whether a page is client-rendered.

Bodies larger than 20 MB are not stored.

📤 Output

Each saved page is one default-dataset item. Empty fields are omitted. false and 0 are kept when they are real.

  • url
  • crawl.loadedUrl, crawl.loadedTime, crawl.referrerUrl, crawl.depth, crawl.httpStatusCode, crawl.contentLengthBytes, crawl.contentType
  • metadata.canonicalUrl, metadata.title, metadata.description, metadata.author, metadata.keywords, metadata.languageCode, metadata.openGraph, metadata.jsonLd, metadata.headers
  • text
  • markdown when saveMarkdown produced Markdown
  • html when saveHtml is on
  • htmlUrl when a snapshot was stored
  • debug when debugMode is on

Failed URLs go to the errors dataset: url, attempts, httpStatus, errorCode, retryable, errorExplanation, advice, and sometimes debug.contentType. The response body is not stored there.

Downloaded files use key-value keys prefixed with file-. HTML snapshots use html-. This actor does not write screenshotUrl or summary.

Related actors

G2
4.7/5
Excellent
Trustpilot

The internet is full of data.
Start owning yours.

Scrape.ly gives you the infrastructure, the tooling, and the team to extract, process, and operationalize web data at any scale. Setup takes minutes. The advantage lasts forever.

Get Started FreeBook a Demo
No credit card. No setup calls.
Scrape.ly Icon
Scrape.lyby Evomi
The web's data, in your hands.
Powered by Evomi infrastructure.
SOC 2 Type IIISO 27001EWDCIGDPR
Products
Scrape.ly StoreProxyAnti-blockingAPIPricingCompare
Developers
DocsTemplatesMonetizeIntegrationsMCP ServerFor AI agentsStatus
Company
About UsAffiliate ProgramCareersContactSupport
Legal
Terms of ServicePrivacy PolicyCookie PolicyAcceptable UseRefund PolicyAll legal documents
Social Media
YouTubeLinkedInGitHubTelegram
© 2026 TeraShift GmbH
All services online