
Website Content Crawler
scrape.ly/store/querystone/website-content-crawlerAIDeveloper tools
1 monthly user

Detect the CMS, frameworks, analytics, CDN, ecommerce platform and 7,600 other technologies a website uses. One JSON row per URL with categories, versions and confidence.
Find out which CMS, JavaScript frameworks, analytics, CDN, ecommerce platform and hosting a website runs on. Paste a list of URLs or domains and get one clean JSON row per site, with categories, versions where the site shows them, and a confidence score. It uses plain HTTP and about 7,600 community fingerprints, so there is no browser, no login and no per-site setup.
https://, trailing slashes and odd input are cleaned up for you. Bad rows are returned with a clear reason.url, status_code, tech_count, technologies, categories, tech_names, category_list, final_url, scan_time_ms) match the common Wappalyzer-style actors.CMS, Analytics, Ecommerce) or set a minimum confidence.| Field | What it does |
|---|---|
urls | Websites to scan. Bare domains get https:// added. Duplicates are scanned once. |
urlsFileUrl | Link to a text or CSV file with one URL or domain per line, for long lists. |
categories_filter | Only return technologies in these categories. Not case sensitive. |
min_confidence | Drop technologies below this score (0–100). Default 0. |
include_confidence | Add the confidence score to each technology. Default on. |
include_versions | Report versions when visible. Default on. |
dns_lookups | Also read MX, TXT, NS and CNAME records (email provider, DNS host, verified SaaS tools). Default off. |
timeout_seconds | Per-request timeout, 3–60. Default 15. |
proxyConfiguration | Datacenter proxies by default. Residential or your own proxies are optional. |
{
"urls": ["https://stripe.com", "shopify.com", "wordpress.org"],
"categories_filter": [],
"include_confidence": true,
"include_versions": true
}
| Field | Description |
|---|---|
url, final_url, status_code | Address scanned, address after redirects and the HTTP status. |
scan_status | Always ok. Sites that could not be scanned are not saved; they are listed in the status message and in the OUTPUT record (failedInputs, with the reason). |
tech_count | Number of technologies returned. |
technologies | List of name, category, categories, version, confidence, evidence, website. |
categories | Technology names grouped by category. |
tech_names, category_list | Comma separated summaries, handy for CSV. |
scan_time_ms | Time spent on this site. |
Real example (technology list trimmed):
{
"url": "https://shopify.com/",
"final_url": "https://www.shopify.com/",
"status_code": 200,
"scan_status": "ok",
"tech_count": 12,
"categories": {
"CDN": ["Cloudflare"],
"Ecommerce": ["Cart Functionality", "Shopify"],
"Programming languages": ["Liquid", "Ruby"],
"Security": ["HSTS"]
},
"technologies": [
{
"name": "Cloudflare",
"category": "CDN",
"categories": ["CDN"],
"version": null,
"evidence": "header: cf-cache-status",
"website": "https://www.cloudflare.com",
"confidence": 100
}
],
"tech_names": "Cart Functionality, Cloudflare, HSTS, HTTP/3, Liquid, Open Graph, Priority Hints, Ruby, Schema.org, Shopify, Tailwind CSS, Twitter Card",
"category_list": "CDN, Ecommerce, Programming languages, Security",
"scan_time_ms": 1850
}
You pay $0.07 per scanned site (one tech-detection event). Sites that cannot be reached, return an HTTP error, are blocked or are not valid URLs are free: they are not saved as rows and are listed under failedInputs in the OUTPUT record. Example: 1,000 working sites cost about $70, and 100 sites about $7. Platform usage (compute and proxy traffic) is small, usually well under $0.001 per site, and is billed separately at the normal rates.
Is it legal? It requests each site's public home page once, like a normal visitor, and reads the response. Respect the terms of the sites you scan.
Do I need a login or a proxy? No login. The actor routes requests through datacenter proxies by default. You can switch to residential or use your own.
How accurate is it? Detection is based on what the server sends: headers, cookies, HTML, script addresses, meta tags and inline script text. It does not run JavaScript, so technologies that only appear after the page runs, and very new or rare tools, can be missed. Matches from inline script text are scored lower.
Why is a version empty? Many sites hide versions. The field is null rather than guessed.
What happens to sites that fail? Sites that answer with an error such as 403 or 404, cannot be reached or are not valid URLs are not saved and not charged. The run's status message and the OUTPUT record (failedInputs) list each one with the reason.
Can I schedule it or call it by API? Yes, like any actor: schedules, webhooks, the API and integrations all work.
Fingerprint data: community Wappalyzer technology definitions from the enthec/webappanalyzer project (GPL-3.0), plus a few extra rules written for this actor. Migration note: output fields follow the common Wappalyzer-style actors.
Every field below can be set in the run form, passed through the API, or saved as a task. Unset fields fall back to the defaults shown.
urlsarrayOptionalWebsites to scan
Website URLs or bare domains, one per line. A missing https:// is added automatically. Duplicates are scanned once.
urlsFileUrlstringOptionalURL list file
Optional link to a text or CSV file with one URL or domain per line. Use it for long lists: run inputs larger than about 8 KB cannot be submitted directly.
Narrow down what is returned for each website.
categories_filterarrayOptionaldefault []Only these categories
Return only technologies in these categories, for example CMS, Analytics, CDN, Ecommerce, JavaScript frameworks or Payment processors. Names are not case sensitive. Leave empty for everything.
min_confidenceintegerOptionaldefault 0Minimum confidence
Drop technologies with a confidence score below this value (0 to 100). Implied technologies score up to 60.
Control the detail of each result and how the sites are checked.
include_confidencebooleanOptionaldefault trueInclude confidence scores
Add a 0 to 100 confidence score to every detected technology.
include_versionsbooleanOptionaldefault trueExtract version numbers
Report the version of a technology when the site exposes it, for example WordPress 6.5 or jQuery 3.7.1. The version is null when it is not visible.
dns_lookupsbooleanOptionaldefault falseCheck DNS records
Also read the domain's MX, TXT, NS and CNAME records. This reveals email providers, DNS and CDN services, and SaaS tools the company has verified its domain with (these appear as extra technologies). Adds a fraction of a second per site.
timeout_secondsintegerOptionaldefault 15Timeout per website (seconds)
Give up on a website after this many seconds per request attempt. Raise it for slow sites.
proxyConfigurationobjectOptionaldefault {"useScrapelyProxy":true,"scrapelyProxyGroups":["DATACENTER…Proxy configuration
Datacenter proxies are the default. Switch to residential only if sites block datacenter addresses, or supply your own proxy URLs.
Every run writes its results to a dataset. Preview it in the dashboard, then export it or read it through the API.
| Field | Type | Description |
|---|---|---|
| url | string | Normalized address that was scanned |
| final_url | string | null | Address after redirects |
| status_code | integer | null | HTTP status of the final response |
| scan_status | string | Always ok (sites that could not be scanned are listed in the OUTPUT record, not saved) |
| tech_count | integer | Number of technologies returned |
| categories | object | Technology names grouped by category |
| technologies | array | Detected technologies |
| tech_names | string | Comma separated technology names |
| category_list | string | Comma separated categories found |
| scan_time_ms | integer | Time spent on this site in milliseconds |
You pay for the events the actor reports, such as each result or page it produces. Platform usage is included in the event prices.
| Event | Charged when | Price |
|---|---|---|
| Scanned website | One website scanned successfully for its technologies. Unreachable, blocked and invalid sites are not charged. | $0.07 |
Run this actor from your own code. Swap in your API token and the examples work as-is.
import { ScrapelyClient } from 'scrapely-client'
const client = new ScrapelyClient({ token: process.env.SCRAPELY_TOKEN })
// Start the actor and wait for the run to finish.
const run = await client.actor('querystone/tech-stack-detector').call({
"urls": [
"https://stripe.com",
"https://shopify.com",
"https://wordpress.org"
],
"categories_filter": [],
"min_confidence": 0,
"include_confidence": true,
"include_versions": true,
"dns_lookups": false,
"timeout_seconds": 15,
"proxyConfiguration": {
"useScrapelyProxy": true,
"scrapelyProxyGroups": [
"DATACENTER"
]
}
})
// Read the dataset the run produced.
const { items } = await client.dataset(run.defaultDatasetId).listItems()
console.log(items)Tokens live in your account settings. Keep them server-side: a token carries the full permissions of your account.
Connect the Scrape.ly MCP server once and your agent can run this actor as a tool. Pick your tool:
Fetch scrape.ly/agents.md and run querystone/tech-stack-detectorPaste into Claude Code, Codex, Cursor or any AI agent. It signs you in once in the browser.
{
"mcpServers": {
"scrapely": {
"url": "https://api.scrape.ly/mcp?tools=actors,runs,storage,docs&actors=querystone/tech-stack-detector"
}
}
}No key in the file: your client opens the browser to sign you in the first time it connects.
Inputs the developer prepared for common jobs. Open one to see its input, then copy it to your account to run it as-is or adjust it.
Detected technologies with versions and confidence scores.
Only these categories, with at least 50% confidence.
Scrape.ly gives you the infrastructure, the tooling, and the team to extract, process, and operationalize web data at any scale. Setup takes minutes. The advantage lasts forever.