
Website Content Crawler
scrape.ly/store/querystone/website-content-crawlerAIDeveloper tools
1 monthly user

Turn any list of URLs into link previews: title, description, image, favicon, logo, site name, Open Graph, Twitter cards, oEmbed, author, dates, canonical URL and redirect chain. Fast HTTP, no browser needed.
Turn any list of URLs into rich link previews: title, description, preview image, favicon, logo, site name, Open Graph and Twitter card data, oEmbed embeds, author, publish date, canonical URL and the full redirect chain. Built for chat apps, CMSs, bookmarking tools and AI agents that need to "unfurl" links quickly and cheaply. No login, no browser by default, many URLs in parallel.
<head> is complete.<link> tags are merged into one clean record, with fallbacks for the image (Twitter, JSON-LD, image_src, first large image), the logo and the favicon.error and cost nothing.og:image, wrong canonical, broken redirects or no Twitter card.https:// is fine).| Field | Description |
|---|---|
urls | URLs to unfurl, one per line. Duplicates and tracking parameters (utm_*, fbclid, ...) are removed. |
maxItems | Process at most this many URLs (0 = all). |
includeOembed | Fetch oEmbed data (default on). |
includeJsonLdTypes | Add schema.org types from JSON-LD (default on). |
includeRawMeta | Add every Open Graph, Twitter and meta tag as objects (default off). |
maxConcurrency | Parallel requests (default 20, up to 100). |
requestTimeoutSecs | Timeout per request (default 15 s). |
residentialRetry | Retry blocked sites over a residential IP (default on). |
browserFallback | Load still-blocked pages in a hosted browser (default off). |
{
"urls": ["github.com", "https://www.youtube.com/watch?v=dQw4w9WgXcQ", "https://stripe.com"],
"includeOembed": true,
"maxConcurrency": 20
}
| Field | Description |
|---|---|
url, finalUrl, status, ok, redirectChain | What was requested, where it ended up and how. |
title, description, image, imageWidth, imageHeight, imageAlt | The preview card. imageSource says where the image came from. |
siteName, type, canonicalUrl, language | Site and page identity. |
favicon, appleTouchIcon, logo, themeColor | Branding. |
author, publishedAt, modifiedAt, keywords | Article details when the page has them. |
twitterCard, twitterSite, twitterCreator | Twitter / X card data. |
video, audio, feedUrl | Media and RSS/Atom feed links. |
oembed | type, title, author_name, provider_name, thumbnail_url, html, width, height, ... |
jsonLdTypes, jsonLdPrimaryType | Schema.org types found on the page. |
contentType, fetchedVia, responseTimeMs, fetchedAt, error | Fetch details. |
Fields are left out when a page does not have them. Example:
{
"url": "https://stripe.com",
"finalUrl": "https://stripe.com/en-ch",
"ok": true,
"status": 200,
"redirectChain": [{ "url": "https://stripe.com", "status": 307 }],
"contentType": "text/html",
"title": "Stripe | Financial Infrastructure to Grow Your Revenue",
"description": "Stripe is a financial services platform that helps all types of businesses accept payments, build flexible billing models and manage money movement.",
"image": "https://b.stripecdn.com/mkt-ssr-statics/assets/_next/static/media/stripe.37119086.jpg",
"imageSource": "og",
"siteName": "Stripe",
"type": "website",
"canonicalUrl": "https://stripe.com/en-ch",
"favicon": "https://images.stripeassets.com/fzn2n1nzq965/4vVgZi0ZMoEzOhkcv7EVwK/8cce6fdcf2733b2ec8e99548908847ed/favicon.png?w=96&h=96",
"logo": "https://images.stripeassets.com/fzn2n1nzq965/1hgcBNd12BfT9VLgbId7By/01d91920114b124fb4cf6d448f9f06eb/favicon.svg",
"language": "en-CH",
"twitterCard": "summary_large_image",
"twitterSite": "@stripe",
"jsonLdTypes": ["WebSite", "Organization"],
"jsonLdPrimaryType": "Organization",
"responseTimeMs": 579,
"error": null
}
You pay $1.50 per 1,000 link previews ($0.0015 per successful preview), plus platform usage: a little compute, datacenter proxy traffic, and residential traffic only for sites that block the datacenter proxy. A preview usually transfers well under 100 KB because the actor stops after the page <head>.
Example: 1,000 URLs with 950 successful previews cost $1.43 in previews plus usage. Failed, blocked and invalid URLs are free.
Do I need a login, API key or my own proxy? No. Pages are fetched through the built-in datacenter proxy, with a residential retry for sites that block it.
Is it legal? The actor reads the public metadata websites publish for exactly this purpose (link previews and social cards). Respect each site's terms and use the data responsibly.
Why is a field empty?
Many pages do not publish every tag. Fields a page does not have are left out of its row. The image falls back to the Twitter card, JSON-LD, image_src or the first large image; the favicon falls back to /favicon.ico.
What about X/Twitter, TikTok or Reddit posts? Their oEmbed endpoints give a title, author, text and embed HTML even when the page itself shows a login or bot wall.
How many URLs can I send?
There is no fixed limit. Runs stream results and keep memory flat; use maxItems and the run's spending limit to cap a job.
Can I call it from the API, MCP or on a schedule? Yes. Each run writes one row per URL to its default dataset.
ok: false and are not charged.Every field below can be set in the run form, passed through the API, or saved as a task. Unset fields fall back to the defaults shown.
urlsarrayOptionalURLs
Pages to unfurl, one per line. A missing https:// is added. Duplicates (including tracking parameters such as utm_*) are removed.
maxItemsintegerOptionaldefault 0Max URLs
Process at most this many URLs. 0 means all of them.
Choose what each preview contains.
includeOembedbooleanOptionaldefault trueoEmbed data
Fetch the oEmbed record (embed HTML, author, thumbnail, size) for pages that offer one, such as YouTube, Vimeo, X/Twitter, TikTok, Spotify, SoundCloud and Reddit. Also fills previews for posts whose pages are behind a wall.
includeJsonLdTypesbooleanOptionaldefault trueSchema.org types
Add the schema.org types found in the page's JSON-LD (for example Article, Product, VideoObject).
includeRawMetabooleanOptionaldefault falseRaw Open Graph, Twitter and meta tags
Add every Open Graph, Twitter and meta tag as objects. Off by default to keep rows small.
Speed, timeouts and fallbacks for blocked sites.
maxConcurrencyintegerOptionaldefault 20Parallel requests
How many URLs to fetch at the same time.
requestTimeoutSecsintegerOptionaldefault 15Request timeout (seconds)
Maximum time for one request.
residentialRetrybooleanOptionaldefault trueRetry blocked sites over residential proxy
When a site blocks the datacenter proxy (HTTP 403/429 or a bot check), try once more through a residential IP. Residential traffic is billed per GB as platform usage.
browserFallbackbooleanOptionaldefault falseBrowser fallback
If a page is still blocked after the residential retry, load it in a hosted browser. Slower and uses browser time, so it is off by default.
proxyConfigurationobjectOptionaldefault {"useScrapelyProxy":true,"scrapelyProxyGroups":["DATACENTER…Proxy configuration
Pages are fetched through datacenter proxies by default, with an automatic residential retry for blocked sites.
Every run writes its results to a dataset. Preview it in the dashboard, then export it or read it through the API.
| Field | Type | Description |
|---|---|---|
| image | any | Image |
| url | any | URL |
| title | any | Title |
| description | any | Description |
| siteName | any | Site |
| favicon | any | Favicon |
| type | any | Type |
| status | any | HTTP status |
| error | any | Error |
You pay for the events the actor reports, such as each result or page it produces. Platform usage is included in the event prices.
| Event | Charged when | Price |
|---|---|---|
| Link preview | Charged once per URL that returns a preview. Dead links, blocked pages and invalid URLs are free. | $0.0015 |
Run this actor from your own code. Swap in your API token and the examples work as-is.
import { ScrapelyClient } from 'scrapely-client'
const client = new ScrapelyClient({ token: process.env.SCRAPELY_TOKEN })
// Start the actor and wait for the run to finish.
const run = await client.actor('querystone/link-preview-scraper').call({
"urls": [
"https://github.com",
"https://www.youtube.com/watch?v=dQw4w9WgXcQ",
"https://www.theverge.com",
"https://stripe.com",
"https://en.wikipedia.org/wiki/Open_Graph_protocol",
"https://vimeo.com/76979871",
"https://www.bbc.com/news",
"https://www.nytimes.com",
"https://medium.com",
"https://x.com/jack/status/20"
],
"proxyConfiguration": {
"useScrapelyProxy": true,
"scrapelyProxyGroups": [
"DATACENTER"
]
}
})
// Read the dataset the run produced.
const { items } = await client.dataset(run.defaultDatasetId).listItems()
console.log(items)Tokens live in your account settings. Keep them server-side: a token carries the full permissions of your account.
Connect the Scrape.ly MCP server once and your agent can run this actor as a tool. Pick your tool:
Fetch scrape.ly/agents.md and run querystone/link-preview-scraperPaste into Claude Code, Codex, Cursor or any AI agent. It signs you in once in the browser.
{
"mcpServers": {
"scrapely": {
"url": "https://api.scrape.ly/mcp?tools=actors,runs,storage,docs&actors=querystone/link-preview-scraper"
}
}
}No key in the file: your client opens the browser to sign you in the first time it connects.
Inputs the developer prepared for common jobs. Open one to see its input, then copy it to your account to run it as-is or adjust it.
Title, description, image, oEmbed and all meta tags for each URL.
Title, description, image and raw meta tags for each URL.
Scrape.ly gives you the infrastructure, the tooling, and the team to extract, process, and operationalize web data at any scale. Setup takes minutes. The advantage lasts forever.