
Website Content Crawler
scrape.ly/store/querystone/website-content-crawlerAIDeveloper tools
1 monthly user

Extract text, Markdown, tables and metadata from PDF URLs or from every PDF linked on a web page. Per-page text, page ranges, password-protected PDFs. Fast, no login needed.
Turn any PDF link into clean text, LLM-ready Markdown, tables and metadata in seconds. Paste PDF URLs, or a web page that links to PDFs, and get one structured row per document, ready to export as JSON, CSV or Excel. No login, no uploads, no setup.
--- between pages.| Field | What it does |
|---|---|
urls | PDF links, or web pages whose PDF links should be extracted. Google Drive / Dropbox share links and arXiv abstract pages work too. |
maxPdfsPerPage | Max PDFs taken from each web page (default 20). |
includeMarkdown / includeFullText / includePages | Which text fields each row contains. |
extractTables | Detect ruled tables (off by default; slower on long documents). |
outputFormat | document (one row per PDF) or page (one row per page). |
pageRange | e.g. 1-5, 8, 12-. Empty means all pages. |
password | Password for protected PDFs. |
maxDocuments, maxFileSizeMb, timeoutPerPdfSecs, maxConcurrency | Limits and download settings. |
{
"urls": [
"https://www.irs.gov/pub/irs-pdf/fw9.pdf",
"https://arxiv.org/pdf/1706.03762"
],
"includeMarkdown": true,
"extractTables": true,
"pageRange": "1-10"
}
| Field | Description |
|---|---|
url, finalUrl, sourcePageUrl, fileName | Where the PDF came from (and the page it was found on). |
title, author, subject, keywords, creator, producer | Document metadata. |
creationDate, modificationDate | ISO 8601, converted to UTC. |
pageCount, extractedPageCount, pdfVersion, pageWidthPt, pageHeightPt, fileSizeBytes, isEncrypted | File facts. |
fullText | The whole document as plain text, line breaks kept. |
markdown | Markdown rendition (headings, paragraphs, lists, tables). |
pages | [{pageNumber, text}] for every extracted page. |
tables, tableCount | [{pageNumber, tableNumber, rows, markdown}] when table detection is on. |
charCount, wordCount, emptyPageNumbers | Size of the text and pages without a text layer. |
textExtractionStatus, warning, recommendedAction | text_extracted or no_text (scanned or blank), with a plain-language hint. URLs that fail (download errors, web pages without PDFs, damaged files) are not saved as rows: they are listed with the reason in the run's OUTPUT record (failedInputs) and in the status message. |
Example row (text shortened):
{
"url": "https://www.irs.gov/pub/irs-pdf/fw9.pdf",
"fileName": "fw9.pdf",
"title": "Form W-9 (Rev. March 2024)",
"author": "SE:W:CAR:MP",
"subject": "Request for Taxpayer Identification Number and Certification",
"keywords": "Fillable",
"creator": "Designer 6.5",
"producer": "Designer 6.5",
"creationDate": "2024-03-06T13:18:13.000Z",
"modificationDate": "2024-03-06T13:18:13.000Z",
"pdfVersion": "1.7",
"pageCount": 6,
"pageWidthPt": 612.0,
"pageHeightPt": 792.0,
"isEncrypted": false,
"fileSizeBytes": 140815,
"extractedPageCount": 6,
"charCount": 37689,
"wordCount": 6272,
"textExtractionStatus": "text_extracted",
"warning": null,
"error": null,
"fullText": "Form W-9\n(Rev. March 2024)\nRequest for Taxpayer\nIdentification Number and Certification\n...",
"markdown": "Form W-9 (Rev. March 2024)\n\n# Request for Taxpayer Identification Number and Certification\n\n...",
"pages": [{ "pageNumber": 1, "text": "Form W-9\n(Rev. March 2024)\n..." }]
}
You pay $1.50 per 1,000 extracted PDFs ($0.0015 per document), whatever the page count, plus normal platform usage: compute time and the datacenter traffic of the downloads ($1 per GB). There is no start fee. Failed downloads, invalid URLs and files that are not PDFs are free.
Example from a test run: 10 PDFs (5.5 MB, 100 pages) cost $0.024 in total: $0.015 in document fees plus about $0.009 of compute and download traffic. 1,000 typical PDFs cost roughly $2–3 in total.
Is it legal? Extracting text from PDFs you are allowed to access is legal. Respect copyright and the terms of the sites you download from.
Do I need a login or proxy? No. Downloads use datacenter proxies by default. Files behind a login are not supported.
Does it read scanned PDFs? It reads the embedded text layer. Scanned (image-only) pages are flagged with textExtractionStatus: "no_text" or listed in emptyPageNumbers so you can route them to OCR.
How big can a PDF be? Up to 100 MB by default (raise maxFileSizeMb up to 500 MB). Very long documents are split across fields so each row stays under the dataset size limit; use per-page rows or a page range for huge files.
Why is a field empty? Many PDFs have no title, author or keywords in their metadata; those fields are then null.
Can I schedule it or use the API? Yes. Schedule runs, call it through the API, or connect it to your workflows with webhooks.
Every field below can be set in the run form, passed through the API, or saved as a task. Unset fields fall back to the defaults shown.
urlsarrayOptionalPDF URLs or web pages
Direct links to PDF files (one per line). You can also paste a web page: the extractor finds the PDFs it links to. Google Drive and Dropbox share links and arXiv abstract pages are converted to direct downloads.
maxPdfsPerPageintegerOptionaldefault 20Max PDFs per web page
When a URL is a web page, extract at most this many PDFs linked from it. Direct PDF URLs are always processed.
Choose what each row contains.
includeMarkdownbooleanOptionaldefault trueMarkdown output
Add a 'markdown' field with headings, paragraphs and bullet lists rebuilt from the PDF layout. Ready for LLMs and RAG pipelines.
includeFullTextbooleanOptionaldefault truePlain text
Add a 'fullText' field with the whole document as plain text (line breaks kept).
includePagesbooleanOptionaldefault truePer-page text
Add a 'pages' array with the text of every page. Turn off to keep rows small for very long PDFs.
extractTablesbooleanOptionaldefault falseDetect tables
Find ruled tables and return them as rows of cells and as Markdown tables (also inserted into the Markdown output). Slower on long documents.
outputFormatenumOptionaldefault "document"Rows
One row per document (default), or one row per page (handy for chunking and search indexes). The price is per document either way.
Limits, passwords and download settings.
pageRangestringOptionalPage range
Extract only these pages, e.g. 1-5, 8, 12- (12- means page 12 to the end). Leave empty for all pages.
followPdfLinksbooleanOptionaldefault trueFind PDF links on web pages
When a URL returns a web page instead of a PDF, extract the PDFs linked from that page.
maxDocumentsintegerOptionaldefault 0Max documents per run
Stop after this many PDFs in total. 0 means no limit.
maxFileSizeMbintegerOptionaldefault 100Max file size (MB)
Larger PDFs are skipped and reported as failed rows (not charged).
timeoutPerPdfSecsintegerOptionaldefault 90Download timeout per PDF (seconds)
How long to wait for one download before retrying.
maxConcurrencyintegerOptionaldefault 5Parallel downloads
How many PDFs to download at the same time.
proxyConfigurationobjectOptionaldefault {"useScrapelyProxy":true,"scrapelyProxyGroups":["DATACENTER…Proxy configuration
Downloads go through datacenter proxies by default. Switch to residential only if a host refuses datacenter IPs.
Every run writes its results to a dataset. Preview it in the dashboard, then export it or read it through the API.
| Field | Type | Description |
|---|---|---|
| url | any | PDF URL |
| fileName | any | File |
| title | any | Title |
| author | any | Author |
| pageCount | any | Pages |
| wordCount | any | Words |
| textExtractionStatus | any | Status |
| creationDate | any | Created |
| fileSizeBytes | any | Size (bytes) |
| sourcePageUrl | any | Found on page |
| warning | any | Warning |
You pay for the events the actor reports, such as each result or page it produces. Platform usage is included in the event prices.
| Event | Charged when | Price |
|---|---|---|
| PDF extracted | Charged once for each PDF document extracted, whatever its page count. Failed downloads, invalid URLs and non-PDF files are free. | $0.0015 |
Run this actor from your own code. Swap in your API token and the examples work as-is.
import { ScrapelyClient } from 'scrapely-client'
const client = new ScrapelyClient({ token: process.env.SCRAPELY_TOKEN })
// Start the actor and wait for the run to finish.
const run = await client.actor('querystone/pdf-text-extractor').call({
"urls": [
"https://www.irs.gov/pub/irs-pdf/fw9.pdf",
"https://arxiv.org/pdf/1706.03762"
],
"maxPdfsPerPage": 20,
"proxyConfiguration": {
"useScrapelyProxy": true,
"scrapelyProxyGroups": [
"DATACENTER"
]
}
})
// Read the dataset the run produced.
const { items } = await client.dataset(run.defaultDatasetId).listItems()
console.log(items)Tokens live in your account settings. Keep them server-side: a token carries the full permissions of your account.
Connect the Scrape.ly MCP server once and your agent can run this actor as a tool. Pick your tool:
Fetch scrape.ly/agents.md and run querystone/pdf-text-extractorPaste into Claude Code, Codex, Cursor or any AI agent. It signs you in once in the browser.
{
"mcpServers": {
"scrapely": {
"url": "https://api.scrape.ly/mcp?tools=actors,runs,storage,docs&actors=querystone/pdf-text-extractor"
}
}
}No key in the file: your client opens the browser to sign you in the first time it connects.
Inputs the developer prepared for common jobs. Open one to see its input, then copy it to your account to run it as-is or adjust it.
Full text, Markdown, pages and tables from PDF links.
One result per page with text and tables.
Finds the PDF on a page and reads only the chosen pages.
Scrape.ly gives you the infrastructure, the tooling, and the team to extract, process, and operationalize web data at any scale. Setup takes minutes. The advantage lasts forever.