
Website Content Crawler
scrape.ly/store/querystone/website-content-crawlerAIDeveloper tools
1 monthly user

Crawl any website and get a 0-100 SEO score for every page and the whole site, with prioritized issues and fix hints: meta tags, headings, canonicals, hreflang, broken links, redirects, images, structured data, speed, robots.txt, sitemaps, AI crawler rules and llms.txt.
Crawl any website and get a 0-100 SEO score for every page and for the whole site, with every issue explained, ranked by priority and paired with a fix. Titles, meta tags, headings, canonicals, hreflang, broken links, redirects, images, structured data, speed, robots.txt, sitemaps, AI crawler rules and llms.txt, all in one run. No login, no browser extension, results as JSON, CSV or Excel.
topIssues) that tells you what to fix first.https://example.com).| Field | Default | What it does |
|---|---|---|
startUrls | (required) | Websites or pages to audit. Each one gets its own crawl and site summary. |
maxPagesPerSite | 20 | HTML pages audited (and charged) per site. |
crawlPages | true | Follow internal links and sitemap URLs. Off = audit only the listed URLs. |
useSitemap | true | Also queue URLs from the XML sitemap. |
maxDepth | 10 | Maximum clicks from the start URL. |
respectRobotsTxt | true | Skip URLs that robots.txt disallows for Googlebot. |
includeSubdomains | false | Crawl blog.example.com etc. too. |
includeUrlPatterns / excludeUrlPatterns | empty | Text or regex filters for discovered URLs. |
checkExternalLinks / checkImages | true | Request outbound links and images to find broken and heavy ones. |
maxLinkChecksPerSite / maxImageChecksPerSite | 300 / 200 | Caps on extra requests per site. |
renderJavaScript | auto | Render JavaScript apps in a browser: auto, never or always. |
{
"startUrls": ["https://books.toscrape.com/"],
"maxPagesPerSite": 20,
"excludeUrlPatterns": ["/tag/", "?page="]
}
The dataset holds two kinds of rows:
page: one per audited page, with score, grade, categoryScores (meta, technical, content, headings, links, images, performance, structuredData), issues (each with id, severity, category, message, fix, details), title and description with length and pixel width, headings, word count, readability, keywords, indexability, canonical, hreflang, Open Graph, Twitter card, structured data types, links (broken, redirecting), images (missing alt, broken, large, sizes), performance (response time, HTML size, compression, scripts), security (HTTPS, HSTS, mixed content) and accessibility basics.site-summary: one per website, with the site score, averagePageScore, siteChecksScore, category averages, score distribution, topIssues ranked by priority, siteIssues, robots.txt, aiCrawlers, llmsTxt, sitemap, host checks, duplicates, broken links, error pages and crawl statistics. All summaries are also saved in the key-value store record SITE_SUMMARIES.| Field | Meaning |
|---|---|
score / grade | 0-100 and excellent (90+), good (75+), needs improvement (50+), poor |
issues[].severity | critical, warning or notice |
topIssues[].priority | Higher = fix first (severity x share of pages x impact) |
indexability | indexable, noindex, canonicalized or blocked by robots.txt |
links.broken | Broken links on the page with status code and anchor text |
aiCrawlers[] | Each AI bot with allowed / blocked / partial |
Example page row (trimmed):
{
"type": "page",
"site": "books.toscrape.com",
"url": "https://books.toscrape.com/",
"finalUrl": "https://books.toscrape.com/",
"statusCode": 200,
"score": 74,
"grade": "needs improvement",
"categoryScores": {
"meta": 55,
"technical": 77,
"content": 45,
"headings": 92,
"links": 100,
"images": 95,
"performance": 100,
"structuredData": 50
},
"issueCounts": {
"critical": 0,
"warning": 6,
"notice": 6
},
"issues": [
{
"id": "THIN_CONTENT",
"severity": "warning",
"category": "content",
"message": "Only 135 words of main content",
"fix": "Add useful, original text; merge or noindex very thin pages."
},
{
"id": "DUPLICATE_CONTENT",
"severity": "warning",
"category": "content",
"message": "Same main text as 1 other page(s)",
"fix": "Consolidate duplicates or point a canonical to the main version.",
"details": [
"https://books.toscrape.com/index.html"
]
},
{
"id": "DESCRIPTION_MISSING",
"severity": "warning",
"category": "meta",
"message": "Missing meta description",
"fix": "Add a meta description of 70-160 characters that sells the page."
}
],
"title": "All products | Books to Scrape - Sandbox",
"titleLength": 40,
"titlePixels": 345,
"metaDescription": "",
"metaDescriptionLength": 0,
"h1": "All products",
"h1Count": 1,
"headingCounts": {
"h1": 1,
"h2": 0,
"h3": 20,
"h4": 0,
"h5": 0,
"h6": 0
},
"headings": [
{
"level": 1,
"text": "All products"
},
{
"level": 3,
"text": "A Light in the ..."
},
{
"level": 3,
"text": "Tipping the Velvet"
}
],
"wordCount": 135,
"contentArea": "body",
"textToHtmlRatio": 1.9,
"readability": {
"score": 79.2,
"level": "easy",
"formula": "Flesch reading ease (English)",
"averageWordsPerSentence": 9.6
},
"topKeywords": [
{
"keyword": "stock",
"count": 20,
"density": 25.97
},
{
"keyword": "little",
"count": 2,
"density": 2.6
},
{
"keyword": "products",
"count": 1,
"density": 1.3
}
],
"indexable": true,
"indexability": "indexable",
"metaRobots": "NOARCHIVE,NOCACHE",
"xRobotsTag": null,
"canonical": null,
"canonicalStatus": "missing",
"canonicalTargetStatus": null,
"inSitemap": null,
"redirectChain": [],
"hreflang": [],
"lang": "en-us",
"charset": "UTF-8",
"viewport": "width=device-width",
"openGraph": {},
"twitterCard": {},
"structuredData": {
"jsonLdBlocks": 0,
"types": [],
"microdataTypes": [],
"rdfaTypes": [],
"errors": []
},
"links": {
"internal": 73,
"external": 0,
"externalNofollow": 0,
"emptyAnchors": 0,
"broken": [],
"brokenInternal": 0,
"brokenExternal": 0,
"redirecting": [],
"unverified": 0
},
"images": {
"total": 20,
"missingAlt": 0,
"missingDimensions": 20,
"lazyLoaded": 0,
"checked": 20,
"broken": [],
"large": [],
"totalKB": 173.1,
"items": [
{
"url": "https://books.toscrape.com/media/cache/2c/da/2cdad67c44b002e7ead0cc35693c0e8b.jpg",
"alt": "A Light in the Attic",
"hasAlt": true,
"decorative": false,
"hasDimensions": false,
"lazy": false,
"sizeKB": 9.6,
"format": "jpeg"
}
]
},
"performance": {
"responseTimeMs": 382.0,
"downloadTimeMs": 498.9,
"renderTimeMs": null,
"htmlSizeKB": 50.1,
"compression": "br",
"scripts": 5,
"inlineScripts": 2,
"stylesheets": 3,
"iframes": 0,
"images": 20
},
"security": {
"https": true,
"hsts": true,
"mixedContent": [
"http://ajax.googleapis.com/ajax/libs/jquery/1.9.1/jquery.min.js"
]
},
"accessibility": {
"ariaLabels": 0,
"landmarks": 2,
"formInputsWithoutLabel": 0,
"skipLink": false
},
"renderedWithBrowser": false,
"rawHtmlWordCount": null,
"crawlDepth": 0,
"foundVia": "start",
"auditedAt": "2026-10-15T23:14:48+00:00"
}
Example site-summary row (trimmed):
{
"type": "site-summary",
"site": "books.toscrape.com",
"startUrl": "https://books.toscrape.com/",
"finalUrl": "https://books.toscrape.com/",
"startRedirects": [],
"score": 77,
"grade": "good",
"averagePageScore": 78,
"siteChecksScore": 75,
"pagesAudited": 5,
"categoryAverages": {
"meta": 64,
"technical": 76,
"content": 57,
"headings": 92,
"links": 100,
"images": 95,
"performance": 100,
"structuredData": 50
},
"scoreDistribution": {
"excellent": 0,
"good": 3,
"needsImprovement": 2,
"poor": 0
},
"issueCounts": {
"critical": 1,
"warning": 23,
"notice": 33
},
"topIssues": [
{
"id": "VERY_THIN_CONTENT",
"severity": "critical",
"category": "content",
"title": "Almost no text content (under 100 words)",
"fix": "Add real content or noindex the page.",
"pages": 1,
"samplePages": [
"https://books.toscrape.com/catalogue/category/books/travel_2/index.html"
],
"share": 0.2,
"priority": 9.0,
"rank": 1
},
{
"id": "DESCRIPTION_MISSING",
"severity": "warning",
"category": "meta",
"title": "Missing meta description",
"fix": "Add a meta description of 70-160 characters that sells the page.",
"pages": 5,
"samplePages": [
"https://books.toscrape.com/",
"https://books.toscrape.com/index.html"
],
"share": 1.0,
"priority": 7.28,
"rank": 2
},
{
"id": "MIXED_CONTENT",
"severity": "warning",
"category": "technical",
"title": "Mixed content (HTTP resources on an HTTPS page)",
"fix": "Load every script, style, image and frame over HTTPS.",
"pages": 5,
"samplePages": [
"https://books.toscrape.com/",
"https://books.toscrape.com/index.html"
],
"share": 1.0,
"priority": 6.76,
"rank": 3
}
],
"siteIssues": [
{
"id": "ROBOTS_TXT_MISSING",
"severity": "warning",
"category": "site",
"message": "/robots.txt returns HTTP 404",
"fix": "Add /robots.txt with your Sitemap line."
},
{
"id": "SITEMAP_MISSING",
"severity": "warning",
"category": "site",
"message": "No XML sitemap found",
"fix": "Publish /sitemap.xml and list it in robots.txt."
}
],
"robotsTxt": {
"url": "https://books.toscrape.com/robots.txt",
"found": false,
"status": 404,
"sitemaps": [],
"errors": [],
"googlebotAllowed": "allowed"
},
"aiCrawlers": [
{
"bot": "GPTBot",
"vendor": "OpenAI",
"purpose": "model training",
"status": "allowed",
"explicit": false,
"rule": null
},
{
"bot": "OAI-SearchBot",
"vendor": "OpenAI",
"purpose": "ChatGPT search index",
"status": "allowed",
"explicit": false,
"rule": null
}
],
"aiCrawlersBlocked": [],
"llmsTxt": {
"found": false,
"url": "https://books.toscrape.com/llms.txt",
"status": 404,
"llmsFullTxt": false
},
"sitemap": {
"found": false,
"declaredInRobotsTxt": false,
"sitemaps": [],
"urlCount": 0,
"valid": null,
"errors": [],
"truncated": false,
"filesRead": 4
},
"hosts": {
"https": true,
"httpRedirectsToHttps": null,
"httpStatus": 401,
"alternateHost": "www.books.toscrape.com",
"alternateHostStatus": 404,
"alternateHostRedirects": null,
"missingPageStatus": 404,
"soft404": false
},
"duplicates": {
"titles": [
{
"value": "all products | books to scrape - sandbox",
"pages": [
"https://books.toscrape.com/",
"https://books.toscrape.com/index.html"
],
"count": 2
}
],
"descriptions": [],
"content": [
{
"pages": [
"https://books.toscrape.com/",
"https://books.toscrape.com/index.html"
],
"count": 2
}
]
},
"brokenLinks": {
"internal": 0,
"external": 0,
"list": []
},
"errorPages": [],
"crawl": {
"pagesAudited": 5,
"indexable": 5,
"noindex": 0,
"canonicalized": 0,
"errors4xx": 0,
"errors5xx": 0,
"unreachable": 0,
"nonHtmlSkipped": 0,
"blockedByRobotsTxt": 0,
"blockedByRobotsSample": [],
"indexableNotInSitemap": null,
"indexableNotInSitemapSample": [],
"maxDepth": 1,
"queuedNotAudited": 100,
"linkChecks": 100,
"imageChecks": 49,
"renderedWithBrowser": 0,
"residentialRetries": 0,
"stopReason": "page limit reached",
"durationSecs": 6.2
},
"averageResponseTimeMs": 188,
"auditedAt": "2026-10-15T23:14:48+00:00"
}
You pay $0.03 per audited page ($30 per 1,000 pages), plus a small amount of platform usage (compute and proxy traffic, typically about $0.01 for a 20-page audit). A 20-page audit costs about $0.61; a 100-page audit about $3.05. Site summaries, robots.txt, sitemap, llms.txt and link checks are included. Pages that fail to load, redirects to pages already audited, files and blocked pages are not charged. Set a maximum charge per run and the audit stops cleanly when it is reached.
Is it legal to audit a website? The tool reads public pages the way a search engine does and respects robots.txt by default. Audit sites you own or have permission to analyze, and keep the crawl rate reasonable.
Do I need a login or proxy? No. Datacenter proxies are used by default; pages that block them are retried once over a residential proxy automatically.
How many pages can I audit? Up to 5,000 per site per run, because the audit keeps every page of a site in memory to compare them. Duplicate titles, descriptions and content are compared across all audited pages of a site.
Does it handle JavaScript sites? Yes. In auto mode, pages whose HTML is an empty JavaScript shell are rendered in a real browser and the page is flagged when its content needs JavaScript (which many crawlers and AI bots do not run).
Why is a field empty? Empty means the page does not have it (for example no canonical) or the check was skipped (for example image sizes when the server sends no file size).
Can I schedule it or use the API? Yes. Schedule weekly audits and compare scores over time, or start runs and read the dataset through the API.
Every field below can be set in the run form, passed through the API, or saved as a task. Unset fields fall back to the defaults shown.
startUrlsarrayRequiredWebsites to audit
One website or page per line, e.g. https://example.com. Each one is crawled on its own site (same domain) and gets its own site summary. A missing https:// is added automatically.
maxPagesPerSiteintegerOptionaldefault 20Max pages per site
How many HTML pages are audited (and charged) per website. Pages that fail to load, redirects to pages already audited and files are not counted or charged. At most 5,000: every page of a site is kept in memory until the end of the audit, because titles, descriptions and content are compared across all pages.
Which pages of each website are audited.
crawlPagesbooleanOptionaldefault trueCrawl the site
Follow internal links and sitemap URLs to audit more pages of each site. Turn off to audit only the listed URLs (site checks still run).
useSitemapbooleanOptionaldefault trueAdd pages from the sitemap
Queue URLs from the site's XML sitemap (found through robots.txt or the usual locations) next to the pages found through links. The sitemap is always checked either way.
maxDepthintegerOptionaldefault 10Max link depth
How many clicks away from the start URL links are followed.
respectRobotsTxtbooleanOptionaldefault trueRespect robots.txt
Skip discovered URLs that robots.txt disallows for Googlebot (they are listed in the site summary). Start URLs are always audited.
includeSubdomainsbooleanOptionaldefault falseInclude subdomains
Also crawl subdomains such as blog.example.com when starting from example.com.
includeUrlPatternsarrayOptionalOnly URLs matching
Only audit discovered URLs that contain one of these texts or regular expressions, e.g. /blog/. Leave empty for the whole site.
excludeUrlPatternsarrayOptionalSkip URLs matching
Never audit discovered URLs that contain one of these texts or regular expressions, e.g. /tag/ or ?page=.
Link, image and JavaScript checks.
checkExternalLinksbooleanOptionaldefault trueCheck external links
Request outbound links to find broken ones. Internal links are always checked.
checkImagesbooleanOptionaldefault trueCheck images
Request image URLs to find broken and oversized images (file size from the server).
maxLinkChecksPerSiteintegerOptionaldefault 300Max link checks per site
Upper limit of linked URLs (not audited themselves) that are requested to find broken links and redirects. Most-linked URLs first.
maxImageChecksPerSiteintegerOptionaldefault 200Max image checks per site
Upper limit of image URLs requested per site to measure file size and find broken images.
renderJavaScriptenumOptionaldefault "auto"Render JavaScript
Auto renders only pages whose HTML is an empty JavaScript shell (single-page apps) in a real browser and reports how much content needs JavaScript. Always renders every page (slower). Never audits the raw HTML only, as most crawlers and AI bots see it.
Speed and timeouts.
maxConcurrencyintegerOptionaldefault 8Parallel requests per site
How many pages of one site are downloaded at the same time. Lower it for small or slow servers.
requestTimeoutSecsintegerOptionaldefault 20Request timeout (seconds)
Give up on a request after this many seconds.
proxyConfigurationobjectOptionaldefault {"useScrapelyProxy":true,"scrapelyProxyGroups":["DATACENTER…Proxy configuration
Datacenter proxies work for most sites; blocked pages are retried once over a residential proxy automatically. Note that response times are measured through the proxy.
Every run writes its results to a dataset. Preview it in the dashboard, then export it or read it through the API.
| Field | Type | Description |
|---|---|---|
| type | string | page or site-summary |
| site | string | null | |
| url | string | null | |
| finalUrl | string | null | |
| startUrl | string | null | |
| statusCode | number | null | |
| score | number | null | |
| grade | string | null | |
| categoryScores | object | null | |
| issueCounts | object | null | |
| issues | array | null | |
| title | string | null | |
| metaDescription | string | null | |
| h1 | string | null | |
| wordCount | number | null | |
| indexability | string | null | |
| canonical | string | null | |
| links | object | null | |
| images | object | null | |
| performance | object | null | |
| pagesAudited | number | null | |
| topIssues | array | null | |
| error | string | null |
You pay for the events the actor reports, such as each result or page it produces. Platform usage is included in the event prices.
| Event | Charged when | Price |
|---|---|---|
| Audited page | One HTML page crawled, checked and scored (failed pages, duplicates, files and blocked pages are free). Site summaries and link checks are included. | $0.03 |
Run this actor from your own code. Swap in your API token and the examples work as-is.
import { ScrapelyClient } from 'scrapely-client'
const client = new ScrapelyClient({ token: process.env.SCRAPELY_TOKEN })
// Start the actor and wait for the run to finish.
const run = await client.actor('querystone/seo-audit-tool').call({
"startUrls": [
"https://books.toscrape.com/"
],
"maxPagesPerSite": 20,
"proxyConfiguration": {
"useScrapelyProxy": true,
"scrapelyProxyGroups": [
"DATACENTER"
]
}
})
// Read the dataset the run produced.
const { items } = await client.dataset(run.defaultDatasetId).listItems()
console.log(items)Tokens live in your account settings. Keep them server-side: a token carries the full permissions of your account.
Connect the Scrape.ly MCP server once and your agent can run this actor as a tool. Pick your tool:
Fetch scrape.ly/agents.md and run querystone/seo-audit-toolPaste into Claude Code, Codex, Cursor or any AI agent. It signs you in once in the browser.
{
"mcpServers": {
"scrapely": {
"url": "https://api.scrape.ly/mcp?tools=actors,runs,storage,docs&actors=querystone/seo-audit-tool"
}
}
}No key in the file: your client opens the browser to sign you in the first time it connects.
Inputs the developer prepared for common jobs. Open one to see its input, then copy it to your account to run it as-is or adjust it.
Crawls the site and checks titles, links, images and more.
Quick audit of a single web app.
Audit that also follows subdomains of the site.
Scrape.ly gives you the infrastructure, the tooling, and the team to extract, process, and operationalize web data at any scale. Setup takes minutes. The advantage lasts forever.