Wayback Machine Scraper

🏛️ Wayback Machine Scraper

Look up any URL or domain in the Internet Archive's Wayback Machine and get its full capture history as clean, structured rows: capture date, archived HTTP status, content type, size, content digest and direct snapshot links. List every URL ever archived on a domain, find the snapshot closest to a date, and optionally get each archived page as clean Markdown or plain text, ready for LLMs, RAG pipelines and AI agents. No login, no API key.

✨ Why use Wayback Machine Scraper?

• 🕰️ Full capture history: every snapshot of a page or a whole site, with date ranges, status and MIME filters, and collapsing to one capture per day, month, year or content change.
• 🌐 All archived URLs of a domain: each distinct URL ever captured, once, with its first capture. Great for finding old pages, removed files and forgotten subdomains.
• 📅 Closest snapshot to a date: see what a page said on a specific day, with the distance in days.
• 🤖 LLM-ready content: archived pages come back as Markdown (headings, lists, tables) or plain text, with menus and footers removed. Original HTML is available too.
• 🔁 Change flags: every snapshot row says whether the content changed since the previous capture, and repeat captures ("revisits") inherit the real status and type.
• 💸 Simple pricing: one flat price per row, no start fee.

🎯 Use cases

• 🔎 SEO and site migrations: recover old URLs, redirects and 404s from a domain's history before or after a relaunch.
• ⚖️ Compliance and evidence: get the archived text of terms, privacy policies or claims as they stood on a given date.
• 🕵️ Competitor research: track how pricing pages, product pages and messaging changed over the years.
• 🧠 AI and RAG: feed historical page text into LLM pipelines, agents and vector stores.
• 🛡️ Security research: list archived endpoints, files and subdomains of a target domain.
• 📚 Research and journalism: cite and quote pages that no longer exist.

🚀 How to use

1. Paste one or more URLs or domains (for example python.org, https://example.com/pricing or *.example.com).
2. Pick what to get: snapshots, archived URLs of a domain, or the closest snapshot to a date.
3. Optionally set a date range, status codes, content types and a result limit, and turn on Include page content for Markdown text.
4. Run it and export the results as JSON, CSV, Excel or HTML, or read them through the API.

📥 Input

{
  "urls": ["python.org"],
  "mode": "snapshots",
  "collapse": "year",
  "statusCodes": "200",
  "maxResults": 20,
  "includeContent": true,
  "contentFormat": "markdown"
}

📤 Output

{
  "input": "python.org",
  "matchType": "exact",
  "originalUrl": "http://www.python.org:80/",
  "timestamp": "19970501011626",
  "archivedAt": "1997-05-01T01:16:26Z",
  "archiveUrl": "https://web.archive.org/web/19970501011626/http://www.python.org:80/",
  "rawArchiveUrl": "https://web.archive.org/web/19970501011626id_/http://www.python.org:80/",
  "mimeType": "text/html",
  "statusCode": 200,
  "isOk": true,
  "isRedirect": false,
  "isRevisit": false,
  "digest": "6NXXB545DYQEJSYQFULNPIVRRHBMOWZC",
  "contentLength": 3366,
  "urlKey": "org,python)/",
  "changed": true,
  "title": "Python Language Home Page",
  "markdown": "Come to the Sixth Int'l Python Conference PythonWin 1.0 is out! (Python for Win 95, NT) The final version of Grail 0.3 is out Prior news. News and Announcements\n\nPython is an *interpreted, interactive, object-oriented, extensible* programming language. It provides an extraordinary combination of clarity and versatility...",
  "contentChars": 2424,
  "wordCount": 373,
  "contentTruncated": false,
  "extraction": "main",
  "contentError": null
}

💰 Pricing

You pay $3.00 per 1,000 rows ($0.003 per snapshot or archived URL), plus a small amount of platform usage (compute and proxy traffic). Page content is included in the row price. Example: 1,000 snapshots cost about $3.00 in row fees; 100 snapshots with Markdown content cost about $0.30 plus a few cents of usage. There is no start fee, and inputs with no captures cost nothing.

❓ FAQ

Is it legal? The tool reads public data from the Internet Archive's public CDX and playback services, the same data anyone can see on web.archive.org. Respect the Internet Archive's terms and the copyright of the archived sites when you reuse content.

Do I need an account, API key or proxy? No. Requests go through datacenter proxies by default and are spread out politely.

How many results can I get? As many as the archive holds. Large domains can have millions of captures; set maxResults to 0 for no limit. Newest-first mode returns up to 10,000 captures per URL.

Why is a field empty? Repeat captures ("revisits") have no status of their own in the archive; the tool fills it from the earlier identical capture when that capture is in the same result set. Content is only loaded for successful text pages; contentError says why it was skipped.

Can I use it from an AI agent, the API or a schedule? Yes. Inputs are plain JSON with clear field names, outputs are flat rows with Markdown text, and the tool works through MCP, the REST API, webhooks and schedules.

Is it fast? Metadata queries take a few seconds to about a minute depending on the archive's load. Page content adds roughly one download per snapshot, run in parallel.

⚠️ Limits

• The Internet Archive rate-limits heavy users and its index can be slow at peak times; the tool retries with backoff, but very large domain exports take a while.
• Content is the page's original HTML converted to text. Pages that need JavaScript to show their text (single-page apps) may come back nearly empty.
• Only the archived HTML document is converted; images, PDFs and other files are listed but not converted.
• Snapshots excluded from the Wayback Machine at the site owner's request are not available.