
Website Content Crawler
scrape.ly/store/querystone/website-content-crawlerAIDeveloper tools
1 monthly user

Convert Word, PowerPoint, Excel, CSV, HTML, EPUB, RTF, OpenDocument, JSON, XML and PDF files to clean, LLM-ready Markdown with metadata, per-slide/sheet sections and RAG chunks.
Turn Word, PowerPoint, Excel, CSV, web pages, e-books, RTF, OpenDocument, JSON, XML and PDF files into clean, LLM-ready Markdown. Every document comes back with its title, author, dates and page, slide or sheet counts, and you can get one row per slide, sheet or page, or chunks ready for a vector store. No login, no browser, no AI API: files are parsed directly, in seconds.
| Field | Description |
|---|---|
urls | Document URLs, one per line. |
outputMode | document (default), sections or chunks. |
chunkSize / chunkOverlap | Chunk size and overlap in approximate tokens (default 1000 / 100). |
keepImages | Keep image tags with URLs (default off: images become their alt text). |
htmlMainContentOnly | Web pages: keep the main content only (default on). |
maxRowsPerSheet | Rows per spreadsheet tab (default 1000). |
maxPagesPerDocument | PDF page cap (default 500). |
keyValueStoreId / keyValueStoreKeys | Convert files from one of your key-value stores. |
requestHeaders | Extra headers for protected downloads you have access to. |
{
"urls": [
"https://example.com/files/quarterly-report.docx",
"https://example.com/files/board-deck.pptx",
"https://example.com/files/budget.xlsx"
],
"outputMode": "document"
}
| Field | Description |
|---|---|
url, fileName, format | Source, file name and detected format (docx, pptx, xlsx, pdf, html, ...). |
title, author, description, language | Document metadata (the title falls back to the first heading or the file name). |
createdAt, modifiedAt, lastModifiedBy, application | Dates and authoring details when the file has them. |
pageCount, slideCount, sheetCount, rowCount, sectionCount | Structure counts. |
wordCount, charCount, tokenEstimate | Size of the Markdown. |
billedPages | Pages charged for this document. |
markdown | The converted Markdown. |
sectionType, sectionNumber, sectionTitle | In section mode: slide, sheet, page or heading section. |
chunkIndex, chunkCount, headings | In chunk mode: position and heading path of the chunk. |
warnings | Notes, for example a shortened sheet. Files that fail are not saved; they are listed with the reason in the status message and the OUTPUT record (failedInputs). |
Example (one row per document, Markdown trimmed):
{
"url": "https://example.com/files/quarterly-report.odt",
"fileName": "quarterly-report.odt",
"format": "odt",
"title": "Quarterly Report",
"author": "Jane Example",
"sectionCount": 1,
"wordCount": 23,
"charCount": 208,
"tokenEstimate": 52,
"billedPages": 1,
"fileSizeBytes": 1897,
"markdown": "# Quarterly Report\n\nRevenue grew strongly. See [the website](https://example.com) for details.\n\n## Highlights\n\n- New office opened\n- Hired 12 people\n\n| Region | Sales |\n| --- | --- |\n| EU | 120 |\n| US | 200 |"
}
A slide deck comes back as ## Slide 1: <title> blocks, a workbook as one ## <sheet name> block with a Markdown table per tab.
You pay $2.50 per 1,000 pages ($0.0025 per page), plus a small amount of platform usage (compute and datacenter proxy traffic, usually well under a cent per run).
Example: 100 Word documents of about 1,000 words each are 200 pages, about $0.50. Failed, empty and unsupported files are never charged.
Do I need an account on the source site, an API key or a proxy?
No. Public URLs download directly through the built-in datacenter proxy. For private files you can access, add an Authorization header in requestHeaders, or upload the files to a key-value store.
Is it legal? Converting documents you own or are allowed to read is fine. You are responsible for having the right to process the files you submit.
Which formats are not supported?
Legacy binary Word and PowerPoint files (.doc, .ppt), images and scanned PDFs without a text layer (there is no OCR), audio and ZIP archives. They are not saved as rows and cost nothing; the run's status message and the OUTPUT record (failedInputs) list each one with a clear reason.
What if a field is empty?
Fields are left out when the file does not contain them (many files have no author or dates). title falls back to the first heading or the file name.
How large can documents be?
Up to 50 MB by default (maxFileSizeMb, up to 500 MB). Very long documents are cut to fit one dataset row in document mode, with a warning; use section or chunk mode to keep all text.
Can I use it from an AI agent, the API or on a schedule? Yes. Call it through the MCP server, the REST API or the API clients, or schedule it in the console. Each run returns its rows in the default dataset.
Every field below can be set in the run form, passed through the API, or saved as a task. Unset fields fall back to the defaults shown.
urlsarrayOptionalDocument URLs
Links to the documents to convert, one per line: Word (.docx), PowerPoint (.pptx), Excel (.xlsx, .xls), CSV/TSV, web pages (HTML), e-books (.epub), RTF, OpenDocument (.odt, .ods, .odp), JSON, XML/RSS, Jupyter notebooks, Outlook .msg and PDFs with a text layer. Google Docs/Sheets/Slides, Google Drive, Dropbox and GitHub links are turned into direct downloads. Key-value store record URLs of your account work too.
outputModeenumOptionaldefault "document"Output rows
One row per document with the full Markdown (default); one row per slide, sheet, PDF page or top-level heading section; or ready-to-embed chunks for RAG. The price is per page either way.
Chunking for RAG and what the Markdown keeps.
chunkSizeintegerOptionaldefault 1000Chunk size (tokens)
Target chunk size for 'Chunks for RAG', in approximate tokens (1 token is about 4 characters). Chunks split at headings and paragraphs; tables and code blocks stay whole when they fit.
chunkOverlapintegerOptionaldefault 100Chunk overlap (tokens)
How many tokens of the previous chunk to repeat at the start of the next one, so context is not lost at the boundary.
keepImagesbooleanOptionaldefault falseKeep image links
Keep Markdown image tags with their URLs. Off by default: images become their alt text (when it says something), which keeps the output compact for LLMs.
htmlMainContentOnlybooleanOptionaldefault trueWeb pages: main content only
For HTML pages, drop navigation, headers, footers, sidebars, cookie banners and similar page chrome, and keep the main content.
maxRowsPerSheetintegerOptionaldefault 1000Max rows per sheet
Spreadsheets and CSV files: rows to include per sheet. Longer sheets are cut with a note in the Markdown.
maxPagesPerDocumentintegerOptionaldefault 500Max PDF pages per document
PDFs: convert at most this many pages (protects against huge files). 0 means no limit.
Convert files you uploaded to a key-value store instead of (or as well as) URLs.
keyValueStoreIdstringOptionalKey-value store with files
Convert files stored in one of your key-value stores. Pick a store; every record is converted unless you list keys below.
keyValueStoreKeysarrayOptionalRecord keys
Only convert these record keys from the store. Leave empty to convert every record except INPUT and OUTPUT.
requestHeadersobjectOptionalRequest headers
Extra HTTP headers for the downloads, for example {"Authorization": "Bearer ..."} for files you have access to. Not sent to key-value store URLs.
maxDocumentsintegerOptionaldefault 0Max documents
Convert at most this many documents in the run. 0 means no limit.
maxFileSizeMbintegerOptionaldefault 50Max file size (MB)
Larger files are skipped and reported as failed rows (not charged).
timeoutSecsintegerOptionaldefault 60Download timeout (seconds)
How long to wait for one download before retrying.
maxConcurrencyintegerOptionaldefault 5Parallel documents
How many documents to download and convert at the same time.
proxyConfigurationobjectOptionaldefault {"useScrapelyProxy":true,"scrapelyProxyGroups":["DATACENTER…Proxy configuration
Downloads go through datacenter proxies by default. Switch to residential only if a host refuses datacenter IPs.
Every run writes its results to a dataset. Preview it in the dashboard, then export it or read it through the API.
| Field | Type | Description |
|---|---|---|
| url | any | Document URL |
| format | any | Format |
| title | any | Title |
| author | any | Author |
| pageCount | any | Pages |
| slideCount | any | Slides |
| sheetCount | any | Sheets |
| wordCount | any | Words |
| billedPages | any | Billed pages |
| markdown | any | Markdown |
You pay for the events the actor reports, such as each result or page it produces. Platform usage is included in the event prices.
| Event | Charged when | Price |
|---|---|---|
| Page converted | One PDF page or slide, or 500 words of Markdown for other formats (at least one per document). Failed, empty and unsupported files are free. | $0.0025 |
Run this actor from your own code. Swap in your API token and the examples work as-is.
import { ScrapelyClient } from 'scrapely-client'
const client = new ScrapelyClient({ token: process.env.SCRAPELY_TOKEN })
// Start the actor and wait for the run to finish.
const run = await client.actor('querystone/document-to-markdown').call({
"urls": [
"https://raw.githubusercontent.com/microsoft/markitdown/4cc9fa17653d695d64fb9eee5b33d4de55ff84e8/packages/markitdown/tests/test_files/test.docx",
"https://raw.githubusercontent.com/microsoft/markitdown/4cc9fa17653d695d64fb9eee5b33d4de55ff84e8/packages/markitdown/tests/test_files/test.pptx",
"https://raw.githubusercontent.com/microsoft/markitdown/4cc9fa17653d695d64fb9eee5b33d4de55ff84e8/packages/markitdown/tests/test_files/test.xlsx"
],
"proxyConfiguration": {
"useScrapelyProxy": true,
"scrapelyProxyGroups": [
"DATACENTER"
]
}
})
// Read the dataset the run produced.
const { items } = await client.dataset(run.defaultDatasetId).listItems()
console.log(items)Tokens live in your account settings. Keep them server-side: a token carries the full permissions of your account.
Connect the Scrape.ly MCP server once and your agent can run this actor as a tool. Pick your tool:
Fetch scrape.ly/agents.md and run querystone/document-to-markdownPaste into Claude Code, Codex, Cursor or any AI agent. It signs you in once in the browser.
{
"mcpServers": {
"scrapely": {
"url": "https://api.scrape.ly/mcp?tools=actors,runs,storage,docs&actors=querystone/document-to-markdown"
}
}
}No key in the file: your client opens the browser to sign you in the first time it connects.
Inputs the developer prepared for common jobs. Open one to see its input, then copy it to your account to run it as-is or adjust it.
Word, PowerPoint and more converted into Markdown sections.
Documents split into chunks ready for search or AI.
Main content of a PDF and an HTML page as Markdown.
Scrape.ly gives you the infrastructure, the tooling, and the team to extract, process, and operationalize web data at any scale. Setup takes minutes. The advantage lasts forever.