Hacker News Scraper

🟧 Hacker News Scraper

Export Hacker News stories, comments, front-page rankings and user profiles to JSON, CSV or Excel. Search the full HN archive with points, comments, date and author filters, pull whole comment threads with depth and parent links, or snapshot the live front page. No login, no browser, a few seconds per run, and $0.15 per 1,000 results.

✨ Why use Hacker News Scraper?

• ⚡ Fast: reads the official Hacker News APIs directly. A 20-story search finishes in about 2 seconds and 3,000 date-sorted results in about 6 seconds (local timing).
• 🔎 Full-archive search: every story and comment since 2007, sorted by relevance or newest first, with no 1,000-result cap when you sort by date.
• 💬 Complete comment trees: every reply as its own row with depth, parentId and the story it belongs to, ready for sentiment analysis or LLM pipelines.
• 📰 Live rankings: top (front page), new, best, Ask HN, Show HN and job lists with their current rank.
• 👤 User profiles: karma, account age, about text and recent stories or comments for any username.
• 💸 Cheap: one flat price per row, and no proxy needed (it connects directly and only falls back to datacenter proxies if rate-limited).

🎯 Use cases

• 📈 Brand and product monitoring: track every mention of your company, product or competitor, newest first, on a schedule.
• 🚀 Launch tracking: follow your Show HN or Launch HN post's points, comments and rank on the front page.
• 🧠 Developer sentiment research: collect full discussions about a framework, language or tool for analysis.
• 🤖 AI and RAG datasets: clean plain-text stories and comments with thread structure.
• 🧑‍💼 Hiring and lead generation: pull "Who is hiring?" threads and job posts with all replies.
• 📰 Newsletters and dashboards: daily top stories above a points threshold.

🚀 How to use

1. Enter one or more search queries, pick front-page lists, paste Hacker News URLs or add usernames.
2. Set Max results per input (default 20).
3. Optionally narrow with Filters (content type, points, comments, dates, author) and turn on Include comments.
4. Click Start and download the results as JSON, CSV, Excel or HTML, or read them through the API.

📥 Input

{
  "searchQueries": ["sqlite", "postgres"],
  "storyLists": ["top"],
  "startUrls": ["https://news.ycombinator.com/item?id=8863"],
  "maxResults": 50,
  "contentTypes": ["story"],
  "sortBy": "date",
  "minPoints": 100,
  "dateFrom": "30 days",
  "includeComments": true,
  "maxCommentsPerItem": 100
}

📤 Output

Each row has a recordType: item (a story, job, poll or comment matched by your input), comment (a reply from a comment tree) or user.

Example (sample values):

[
  {
    "recordType": "item",
    "id": 41234567,
    "type": "story",
    "title": "Show HN: A tiny SQLite-backed job queue",
    "url": "https://example.com/blog/tiny-queue",
    "domain": "example.com",
    "author": "sample_dev",
    "points": 312,
    "numComments": 87,
    "createdAt": "2025-08-14T15:02:11Z",
    "createdAtUnix": 1755183731,
    "text": null,
    "textHtml": null,
    "storyId": 41234567,
    "storyTitle": "Show HN: A tiny SQLite-backed job queue",
    "storyUrl": "https://example.com/blog/tiny-queue",
    "parentId": null,
    "depth": null,
    "rank": 1,
    "tags": ["story", "show_hn"],
    "hnUrl": "https://news.ycombinator.com/item?id=41234567",
    "source": "search:sqlite",
    "scrapedAt": "2026-10-14T18:50:24Z"
  },
  {
    "recordType": "comment",
    "id": 41234890,
    "type": "comment",
    "author": "another_hacker",
    "createdAt": "2025-08-14T15:40:02Z",
    "text": "We run something similar in production. WAL mode made a big difference.",
    "storyId": 41234567,
    "storyTitle": "Show HN: A tiny SQLite-backed job queue",
    "parentId": 41234567,
    "depth": 1,
    "hnUrl": "https://news.ycombinator.com/item?id=41234890",
    "source": "search:sqlite"
  }
]

💰 Pricing

You pay $0.15 per 1,000 results ($0.00015 per row): every story, comment and user profile row counts as one result. 1,000 results ≈ $0.15 in result fees, plus well under a cent of platform usage for compute and data traffic (billed separately; a 10,000-result run used about $0.006 in our tests). Set a maximum charge on the run and the actor stops cleanly when it is reached.

❓ FAQ

Is it legal to scrape Hacker News? The actor only reads public data through Hacker News' official public APIs. Check the terms that apply to your use and handle personal data responsibly.

Do I need a login or my own proxy? No. No account and no proxy are needed. The run connects directly to the public Hacker News APIs and switches to datacenter proxies on its own if it gets rate-limited.

How many results can I get? Front-page lists hold up to 500 stories each. Relevance search returns up to 1,000 results per query; with Sort by date you can page through the whole archive (tens of thousands of results).

Why are some fields empty? Hacker News leaves them empty: link posts have no text, Ask HN posts have no url, comments have no points (HN hides them), and deleted comments are skipped.

Can I schedule it or use the API? Yes. Schedule runs to monitor keywords (sort by date with a dateFrom like 1 day), and start runs or download datasets through the API.

⚠️ Limits

• Comment scores are not public on Hacker News, so points is empty for comments.
• front_page in Content types only matches stories that are on the front page right now; use the top list for the live ranking.
• Search results and comment counts come from the Hacker News search index, which can lag the live site by a few minutes.
• Item URLs you paste directly are returned even if they do not match the filters.