RSS Feed Scraper

📰 RSS Feed Scraper

Turn any RSS, Atom or JSON feed, or any website that has one, into clean, structured rows: titles, links, publish dates, authors, summaries, categories, images and podcast files. Filter by date and keyword, remove duplicates across feeds, and optionally pull the full article text as Markdown for RAG pipelines, summaries and AI agents. No login, no API key, priced per feed.

✨ Why use RSS Feed Scraper?

• 🔍 Paste a website, get its feed: feeds advertised on a page are discovered automatically, with common paths like /feed and /rss.xml as a fallback.
• 🧩 Every format: RSS 2.0, RSS 1.0 (RDF), Atom and JSON Feed, mixed freely in one run.
• 🤖 LLM-ready: feed HTML is converted to Markdown, and the full article behind each link can be extracted as clean Markdown or text with menus, ads and footers removed.
• 📅 Monitoring-friendly filters: "published in the last 24 hours", date ranges, keyword filters and cross-feed deduplication that ignores tracking parameters.
• 🎙️ Podcasts and media: enclosure URLs, types, sizes and episode durations.
• 💸 Cheap: one small fee per feed read, however many items it has; full articles are an optional extra.

🎯 Use cases

• 🧠 RAG and AI agents: feed fresh news, blog posts and release notes into vector stores and LLM workflows.
• 📊 Media monitoring: track news sites, competitors' blogs and press rooms for keywords.
• 📰 News aggregation: merge many feeds into one deduplicated stream for a newsletter or dashboard.
• 🎧 Podcast catalogs: collect episodes, audio file links and durations.
• 🔔 Alerts: schedule a run every hour with "Published after: 1 hour" and send new items to Slack, email or a webhook.
• 🔬 Research: build datasets of articles from a list of publications.

🚀 How to use

1. Paste feed URLs or website URLs (for example https://hnrss.org/frontpage or https://www.theverge.com).
2. Set Max items per feed and, if you like, a date range or keywords.
3. Turn on Get full article text if you need the whole article as Markdown.
4. Run it and export the items as JSON, CSV, Excel or HTML, or read them through the API.

📥 Input

{
  "urls": ["https://feeds.bbci.co.uk/news/rss.xml", "https://www.theverge.com"],
  "maxItemsPerFeed": 20,
  "dateFrom": "24 hours",
  "fetchFullArticle": true
}

📤 Output

{
  "sourceUrl": "https://feeds.bbci.co.uk/news/rss.xml",
  "feedUrl": "https://feeds.bbci.co.uk/news/rss.xml",
  "feedTitle": "BBC News",
  "feedType": "rss",
  "feedLanguage": "en-gb",
  "position": 1,
  "title": "Bus hits train stopped on French side of Channel Tunnel, with 19 injured",
  "link": "https://www.bbc.co.uk/news/articles/c6wyrnmgzn01o?at_medium=RSS&at_campaign=rss",
  "guid": "https://www.bbc.co.uk/news/articles/c6wyrnmgzn01o#1",
  "publishedAt": "2026-10-15T21:12:14Z",
  "summary": "Two people were severely injured and evacuated by helicopter but no one's life appeared to be in danger, officials said.",
  "categories": [],
  "imageUrl": "https://ichef.bbci.co.uk/ace/standard/240/cpsprodpb/c978/live/dfe9a6d0-c8df-11f1-9a39-31b474234113.png",
  "enclosures": [],
  "contentMarkdown": null,
  "scrapedAt": "2026-10-15T21:54:26Z",
  "articleTitle": "Bus hits train stopped on French side of Channel Tunnel, with 19 injured - BBC News",
  "articleMarkdown": "# Bus hits train stopped on French side of Channel Tunnel, with 19 injured\n\n- Published\n\n**A bus has hit a stationary freight train near the French entrance to the Channel Tunnel, injuring the driver and 18 passengers, the French authorities say.**\n\nThe bus, which was carrying lorry drivers due to take a shuttle rail service, struck the train...",
  "articleWordCount": 215,
  "articleError": null
}

💰 Pricing

• $1.50 per 1,000 feeds ($0.0015 for each feed read, whatever its number of items). A feed with no new items after your filters is free.
• $1.00 per 1,000 full articles ($0.001 each), only when Get full article text is on. Articles that fail to load are free.
• Plus a small amount of platform usage (compute and proxy traffic).

Example: reading 100 feeds costs about $0.15 plus usage. Adding full text for 1,000 articles adds $1.00 plus the proxy traffic of downloading the pages.

❓ FAQ

Is it legal? Feeds are published so that people and apps can read them. Respect each publisher's terms and copyright when you reuse full articles.

Do I need a login, API key or proxy? No. Requests go through datacenter proxies by default; sites that block them are retried through residential proxies (you can turn this off).

Why is a field empty? Feeds differ a lot: many omit authors, images or full content. Empty fields mean the feed did not provide them. articleError explains why an article could not be extracted (for example a paywall or a page that needs JavaScript).

How many items can I get? Everything the feed contains. Most feeds list their latest 10 to 100 items; feeds do not include older history.

Can I use it for monitoring, from an AI agent or via the API? Yes. Schedule it with a relative "Published after" such as 1 hour, call it through the REST API, or connect it to AI agents over MCP. Inputs are simple JSON and every row is flat, with Markdown text.

⚠️ Limits

• Only what the feed publishes: feeds rarely go back more than a few weeks.
• Full-article extraction works on normal article pages. Paywalled pages, pages behind a login and pages that need JavaScript to show their text may return no article.
• At most 3 requests at a time go to the same website, so very large single-site jobs take a little longer.