
Website Content Crawler
scrape.ly/store/querystone/website-content-crawlerAIDeveloper tools
1 monthly user

Get Similarweb traffic data for any website: global, country and category rank, monthly visits, engagement, traffic sources, top countries, keywords, competitors, audience, technologies and company info. No login needed.
Get Similarweb traffic intelligence for any website in seconds: global, country and category rank, monthly visits for the last three months, engagement, traffic sources, top countries, keywords, competitors, audience demographics, technologies and company details. Paste a list of domains, get one clean row per website. No Similarweb account or login needed.
github.com), full URLs (https://www.bbc.co.uk/news) and Similarweb page URLs all work.| Field | Type | Description |
|---|---|---|
domains | array of strings | Domains or website URLs. Duplicates and www. variants are merged. |
includeSimilarSites | boolean | Adds similar sites, topic tags and related apps. Default true. |
proxy | object | Proxy for the similar-sites lookups. Default: datacenter. |
{
"domains": ["github.com", "amazon.com", "notion.so"],
"includeSimilarSites": true
}
One row per website. Shares are fractions (0.18 = 18%).
| Field | Description |
|---|---|
domain, url | The website and its Similarweb page |
title, description, category | Site title, description and Similarweb category |
globalRank, countryRank, categoryRank | Ranks with their change since last month |
totalVisits, totalVisitsChange | Visits in the latest month and the change versus the month before |
monthlyVisits, estimatedMonthlyVisits | Visits for each of the last three months |
bounceRate, pagesPerVisit, avgVisitDuration | Engagement |
trafficSources, topTrafficSource | Marketing channels ranked, with the share of the top channel |
topCountries, countriesTotal | Top five countries by share of visits |
organicSearchShare, paidSearchShare, keywordsTotal, topKeywords | Search traffic and the top keywords with CPC |
competitors, globalRankCompetitors, countryRankCompetitors, categoryRankCompetitors | Competitors and rank neighbours |
globalRankHistory, countryRankHistory, categoryRankHistory | Rank for each of the last three months |
socialNetworks, topReferringSites, topReferringCategories, topOutgoingSites, topOutgoingCategories, topAdSites | Where visitors come from and go to |
technologies, technologiesTotal | Technology categories and the top technology in each |
ageDistribution, maleShare, femaleShare, audienceInterests | Audience |
company | Name, parent domain, year founded, headquarters, employee and revenue range |
similarSites, tags, relatedApps | Similar websites with visits and similarity score, topic tags, related mobile apps |
icon, previewDesktop, previewMobile, snapshotDate, scrapedAt | Images and dates |
Example (trimmed):
{
"domain": "github.com",
"url": "https://www.similarweb.com/website/github.com/",
"title": "GitHub · Build and ship software on a single, collaborative platform",
"category": "computers_electronics_and_technology/programming_and_developer_software",
"snapshotDate": "2026-09-01",
"globalRank": 45,
"countryRank": { "countryCode": "US", "rank": 77, "change": 4 },
"categoryRank": 4,
"totalVisits": 645466516,
"totalVisitsChange": -0.0059,
"monthlyVisits": [
{ "date": "2026-07-01", "visits": 637885711, "month": "July 2026" },
{ "date": "2026-08-01", "visits": 649321442, "month": "August 2026" },
{ "date": "2026-09-01", "visits": 645466516, "month": "September 2026" }
],
"bounceRate": 0.3644,
"pagesPerVisit": 5.69,
"avgVisitDuration": "00:06:27",
"avgVisitDurationSeconds": 387,
"topTrafficSource": "direct",
"trafficSources": [{ "source": "direct", "rank": 1, "share": 0.5331 }],
"topCountries": [
{ "countryCode": "US", "share": 0.1873, "shareChange": 0.0062 },
{ "countryCode": "CN", "share": 0.1183, "shareChange": 0.0137 }
],
"organicSearchShare": 0.9996,
"keywordsTotal": 2300400,
"topKeywords": [{ "keyword": "github", "cpc": 1.62 }],
"competitors": [{ "domain": "stackoverflow.com", "categoryRank": 94, "affinity": 1 }],
"maleShare": 0.6978,
"femaleShare": 0.3022,
"company": { "name": "GitHub", "yearFounded": 2008, "headquartersCity": "San Francisco", "employeesMin": 501, "employeesMax": 1000 },
"similarSites": [{ "domain": "stackoverflow.com", "similarityScore": 0.94, "totalVisits": 69124392 }],
"tags": ["git", "programming", "open source"]
}
You pay $1.50 per 1,000 websites that return data ($0.0015 each), plus the platform usage of the run (the browser session and compute). Domains with no Similarweb data and invalid inputs are never charged.
Example: 1,000 websites cost about $1.50 in website fees plus roughly $0.70 to $1.00 of platform usage, about $2.20 to $2.50 in total.
Is it legal to scrape Similarweb? The actor only reads data that Similarweb shows publicly to every visitor without logging in. Check that your use of the data complies with Similarweb's terms and the laws that apply to you.
Do I need a Similarweb account, cookies or a proxy? No. The actor uses a built-in browser to load the public pages. The proxy setting only affects the optional similar-sites lookups.
How many websites can I check in one run? Hundreds to thousands. In our tests the actor processed 200 domains in under 3 minutes.
Why is a field empty? Similarweb shows some numbers only to paying users (for example the share of every traffic channel except the top one, keyword volumes, and the names of some referring sites). Small websites often have no data at all: they are listed in the run summary and not charged.
Can I schedule it or use it from the API? Yes. Schedule runs in the console to track traffic every month, or start runs and download results through the API.
snapshotDate).Every field below can be set in the run form, passed through the API, or saved as a task. Unset fields fall back to the defaults shown.
domainsarrayRequiredWebsites
Domains or website URLs to look up, one per line (e.g. github.com, https://www.bbc.co.uk/news). Similarweb page URLs work too. Each website becomes one result.
includeSimilarSitesbooleanOptionaldefault trueInclude similar sites, tags and related apps
Adds up to 40 similar websites (with visits and similarity score), topic tags and related mobile apps to each result. Free of extra charge.
proxyobjectOptionaldefault {"useScrapelyProxy":true,"scrapelyProxyGroups":["DATACENTER…Proxy
Proxy for the similar-sites lookups. Similarweb pages always load through the built-in browser.
Every run writes its results to a dataset. Preview it in the dashboard, then export it or read it through the API.
| Field | Type | Description |
|---|---|---|
| domain | string |
You pay for the events the actor reports, such as each result or page it produces. Platform usage is included in the event prices.
| Event | Charged when | Price |
|---|---|---|
| Website | One website with Similarweb data saved to the dataset. Websites without data are free. | $0.0015 |
Run this actor from your own code. Swap in your API token and the examples work as-is.
import { ScrapelyClient } from 'scrapely-client'
const client = new ScrapelyClient({ token: process.env.SCRAPELY_TOKEN })
// Start the actor and wait for the run to finish.
const run = await client.actor('querystone/similarweb-scraper').call({
"domains": [
"github.com",
"amazon.com",
"notion.so"
]
})
// Read the dataset the run produced.
const { items } = await client.dataset(run.defaultDatasetId).listItems()
console.log(items)Tokens live in your account settings. Keep them server-side: a token carries the full permissions of your account.
Connect the Scrape.ly MCP server once and your agent can run this actor as a tool. Pick your tool:
Fetch scrape.ly/agents.md and run querystone/similarweb-scraperPaste into Claude Code, Codex, Cursor or any AI agent. It signs you in once in the browser.
{
"mcpServers": {
"scrapely": {
"url": "https://api.scrape.ly/mcp?tools=actors,runs,storage,docs&actors=querystone/similarweb-scraper"
}
}
}No key in the file: your client opens the browser to sign you in the first time it connects.
Inputs the developer prepared for common jobs. Open one to see its input, then copy it to your account to run it as-is or adjust it.
Traffic, ranks and audience data for one website.
Traffic data plus similar sites, tags and apps.
Scrape.ly gives you the infrastructure, the tooling, and the team to extract, process, and operationalize web data at any scale. Setup takes minutes. The advantage lasts forever.