Stack Exchange Scraper

💬 Stack Exchange Scraper

Get clean Q&A data from Stack Overflow and every other Stack Exchange site (Super User, Server Fault, Ask Ubuntu, Math, Unix & Linux and 170+ more). Each question comes with its Markdown body, the accepted and top-voted answers, optional comments, author details and related or linked questions, plus a ready-to-embed Markdown document for RAG pipelines and AI agents. Uses the official Stack Exchange API: fast, stable, no login.

✨ Why use Stack Exchange Scraper?

• 🧠 RAG-ready — every question includes ragMarkdown: title, metadata, question and answers (accepted first) in one Markdown document with source and license attribution.
• ✍️ Original Markdown — question, answer and comment bodies exactly as the authors wrote them, code blocks included, not scraped HTML.
• 🔎 Advanced search — free text, tags, excluded tags, accepted-only, minimum score and answers, date ranges, title text, sort by relevance, votes, newest or activity.
• 🔗 Any input — question URLs, answer URLs, tag pages, search pages or bare ids, from any Stack Exchange site in one run.
• 🧩 Context extras — top comments, author profiles, related and linked (duplicate) questions on request.
• ⚡ Fast and quota-aware — up to 100 questions per API call with all their answers; proxy exits rotate automatically when the keyless API quota runs low.

🎯 Use cases

• 🤖 RAG and AI agents — build a knowledge base of real developer Q&A for a framework, library or error message.
• 🏋️ Fine-tuning datasets — question/accepted-answer pairs with scores for quality filtering.
• 📣 DevRel and docs — find the most-viewed unanswered questions about your product and fix your docs.
• 📊 Tech trend research — track question volume and scores for tags over time.
• 🛠️ Support content — collect solutions for known errors on Server Fault, Super User or Ask Ubuntu.

🚀 How to use

1. Enter a search query and/or tags, or paste question, tag or search URLs.
2. Pick the site (stackoverflow by default) and set Max questions.
3. Optionally add filters (accepted only, minimum score, dates) and extras (comments, related questions, author details).
4. Run it and export the dataset as JSON, CSV or Excel, or read it through the API.

📥 Input

{
  "query": "web scraping python",
  "site": "stackoverflow",
  "maxQuestions": 20,
  "acceptedOnly": true,
  "minScore": 10,
  "answersMode": "acceptedAndTop",
  "maxAnswers": 3
}

📤 Output

{
  "questionId": 2081586,
  "site": "stackoverflow",
  "url": "https://stackoverflow.com/questions/2081586/web-scraping-with-python",
  "title": "Web scraping with Python",
  "tags": ["python", "screen-scraping"],
  "score": 197,
  "answerCount": 10,
  "hasAcceptedAnswer": true,
  "creationDate": "2010-01-17T16:06:13Z",
  "author": {"displayName": "Jane Example", "reputation": 1520, "profileUrl": "https://stackoverflow.com/users/1/example"},
  "questionMarkdown": "I'd like to grab daily sunrise/sunset times from a web site. Is it possible to scrape web content with Python? ...",
  "acceptedAnswer": {"answerId": 2081600, "isAccepted": true, "score": 192, "markdown": "Use urllib2 in combination with the brilliant BeautifulSoup library: ..."},
  "ragMarkdown": "# Web scraping with Python\n\nSource: https://stackoverflow.com/questions/2081586/... | Site: stackoverflow | Score: 197 ...",
  "license": "CC BY-SA 2.5"
}

💰 Pricing

$0.004 per question saved ($4 per 1,000), with its answers, comments, related questions and RAG document included, plus a small amount of platform usage. Example: 1,000 questions with accepted and top answers ≈ $4 plus a few cents of usage.

❓ FAQ

Do I need an API key or login? No. The scraper uses the public Stack Exchange API without a key.

What about the API quota? Keyless access allows 300 requests per day per IP. One request returns up to 100 questions with all their answers, and the scraper rotates proxy exits when an exit runs low. If every exit is used up, the run stops cleanly with the questions saved so far.

How many questions can I get? As many as the listing has; set maxQuestions to 0 for no limit. Keyless access allows 300 API requests per day per IP, so very large runs rely on the proxy exits rotating. Comments, related and linked questions cost extra API calls (related and linked: one per question), so turn them on only when you need them.

Is it legal to use this data? Stack Exchange content is licensed under CC BY-SA. Keep the attribution (url, license, and the footer in ragMarkdown) when you reuse it.

Why is a field empty? Many questions have no accepted answer, comments or bounty; those fields are null or empty.

Can I schedule it? Yes. Use fromDate with a relative value such as 7 days to fetch new questions every week.

⚠️ Limits

• Deleted questions and answers are not available.
• Search follows the API's rules: up to 5 tags, and relevance sort needs a search query.
• Comments are the top-voted comments per post (default 5), not the full thread.
• Private Stack Overflow for Teams content is not covered.