How to Scrape News Articles: Headlines, Full Text and Dates

Web scraping news is a two-step job: first find the articles about your topic, then extract each one's full text, author and publish date. The fastest way to do both is with two ready-made scrapers you run in the cloud: one that reads Google News for a query, and one that turns any article URL into clean text and metadata.

This guide walks through both steps without code and with code (curl, Python, Node.js and an AI agent over MCP). Everything comes from two real runs on 22 October 2026: five Federal Reserve headlines from Google News, then three of those articles extracted in full. You get the exact inputs, the outputs, the logs and what each step cost, plus the copyright rules that decide what you may store.

What data you can get from news sites

Each step gives you a different layer. Google News gives you the list; the publisher's page gives you the article.

Comments, live-blog updates that load later and image galleries are not extracted as separate items. Content behind a login or a hard paywall stays behind it.

How web scraping news works: find, then extract

A news crawler has to solve two different problems. Finding articles means watching sources: a search engine, a site's home page, its RSS feed or its sitemap. Extracting an article means removing menus, ads and related-story boxes, and reading the author and date from the page's structured data. A single script that does both for every site needs a selector set per publisher, which is exactly what breaks when a site redesigns.

Splitting the job keeps each part simple:

• The Google News Scraper answers "what was published about X?" for any query, country and date range.
• The Article Extractor is an article scraper that answers "what does this article say, who wrote it, and when?" for any news site, with no per-site setup. It can also crawl a home page, an RSS feed or a sitemap directly when you follow fixed sources.

Step 1: find headlines on Google News

1. Open the Google News Scraper in the Scrape.ly Store.
2. Pick a ready-made input. We used the preset Federal Reserve news in a date range and lowered it from 10 to 5 articles.
3. Set your query, edition and dates. This is the exact input of our run:

{
  "query": "Federal Reserve",
  "maxItems": 5,
  "gl": "us",
  "hl": "en",
  "time_period": "custom",
  "time_period_min": "09/01/2026",
  "time_period_max": "10/01/2026",
  "exactPublishTimes": false
}

4. Start the run and export the list as JSON, CSV or Excel, or pass the links to step 2.

The fields that matter: query (one per run), maxItems (up to 100), gl and hl for the country and language edition (de and de for German news), and time_period for the last hour, day, week, month, year or a custom range. Turn on exactPublishTimes when you need publish times to the second; it reads the Google News feed instead and is slower.

Step 2: extract full text, author and date

1. Open the Article Extractor.
2. Start from the 2 articles by URL preset and paste the links from step 1 into articleUrls. We used three:

{
  "articleUrls": [
    "https://apnews.com/article/federal-reserve-interest-rates-inflation-a633de46f84a3487f3bdc03f788188fa",
    "https://www.politico.com/news/2026/10/01/powell-federal-reserve-trump-01102917",
    "https://www.atlantafed.org/news-and-events/2026/09/16/fomc-statement"
  ]
}

3. Start the run. Every listed URL is extracted; no other pages are crawled from them.

To follow sources rather than single links, put home pages, section pages, RSS feeds or sitemaps in startUrls, set maxArticles, and add dateFrom (for example 24 hours) and keywords. With onlyNewArticles, a scheduled run returns each article once. The Guardian world news from the last 7 days preset is a working example.

How to scrape news articles with the API

Create an API token under Settings → API & Integrations and export it as SCRAPELY_TOKEN. This Python script (pip install scrapely-client-python) chains both steps and keeps a tidy record per article:

import os
from scrapely_client import ScrapelyClient

client = ScrapelyClient(os.environ["SCRAPELY_TOKEN"], api_url="https://api.scrape.ly")

news = client.actor("querystone/google-news-scraper").call(
    run_input={"query": "Federal Reserve", "maxItems": 5, "gl": "us", "hl": "en", "time_period": "last_week"}
)
headlines = list(client.dataset(news.default_dataset_id).iterate_items())

articles = client.actor("querystone/article-extractor").call(
    run_input={"articleUrls": [h["link"] for h in headlines[:3]]}
)
for a in client.dataset(articles.default_dataset_id).iterate_items():
    print(a["status"], a.get("publishedAt"), a.get("wordCount"), a.get("isPaywalled"), a["url"])

Both of our runs finished in under 20 seconds (5.8 and 17.0), so the synchronous endpoint works for each step. It waits up to 45 seconds and returns the items as JSON:

curl -X POST "https://api.scrape.ly/v2/actors/querystone~google-news-scraper/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"query": "Federal Reserve", "maxItems": 5, "gl": "us", "hl": "en", "time_period": "last_week"}'

Crawling whole sites with the Article Extractor takes longer. If a call answers 408, the run is still going: don't call again, use a client instead.

Node.js (npm install scrapely-client), extracting the latest articles from a site's start page:

import { ScrapelyClient } from 'scrapely-client';

const client = new ScrapelyClient({ token: process.env.SCRAPELY_TOKEN, baseUrl: 'https://api.scrape.ly' });

const run = await client.actor('querystone/article-extractor').call({
  startUrls: ['https://techcrunch.com/category/artificial-intelligence/'],
  maxArticles: 5,
  dateFrom: '3 days',
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items.map((a) => [a.publishedAt, a.wordCount, a.title]));

For AI agents, add the MCP server at https://api.scrape.ly/mcp to Cursor (~/.cursor/mcp.json):

{
  "mcpServers": {
    "scrapely": {
      "url": "https://api.scrape.ly/mcp",
      "headers": { "Authorization": "Bearer YOUR_TOKEN" }
    }
  }
}

In Claude Code, run claude mcp add --transport http scrapely https://api.scrape.ly/mcp and sign in with /mcp. Then ask: "Use scrapely to find this week's Google News articles about the Federal Reserve, extract the top three with querystone/article-extractor, and summarize each in two sentences with its publish date." The markdown field is ready for the model. See the MCP docs.

A real run: Federal Reserve news, from headlines to full text

Step 1 ran on 22 October 2026 and returned 5 articles in 5.8 seconds, charged $0.0208:

2026-10-22T18:25:56.528Z [scrapely.actor] INFO  Using fast search mode. The run switches to feed mode only if fast search mode fails.
2026-10-22T18:25:58.960Z [scrapely.actor] INFO  Fast search mode found 5 article(s) in 2432 ms (1 API call(s), 0 retried, slowest 2383 ms).
2026-10-22T18:25:59.253Z [scrapely.actor] INFO  Saved 5 article(s) in fast search mode.

Step 2 extracted rows 5, 2 and 3 in 17.0 seconds, charged $0.0042:

2026-10-22T18:26:55.592Z [scrapely.actor] INFO  Extracting articles from 0 start URL(s) and 3 article URL(s) via datacenter proxy.
2026-10-22T18:27:09.089Z [scrapely.actor] INFO  3 article(s) extracted. Downloaded 1.0 MB; 2 page(s) retried over residential proxy, 1 page(s) rendered in the browser.
2026-10-22T18:27:09.868Z [Scrapely] Container state: status=exited exitCode=0 oomKilled=false

We show whether a byline was present rather than the journalists' names. The comparison of the two steps teaches three things:

• Dates from step 1 are approximate. Google News showed the AP story as "4 weeks ago", which became 22 September. The article's own metadata says 20 September, 11:59 UTC. Use publishedAt from step 2, or turn on exactPublishTimes in step 1.
• Headlines and titles can differ. AP's page metadata carries a different title from the headline on Google News and on the page itself. Store both if you match stories across sources.
• Paywalls show up as flags, not errors. Politico's page needed a browser, came back with 262 words and isPaywalled: true: you get what a normal visitor sees. The Atlanta Fed press release is simply short, at 41 words.

What it costs

Step 1, the Google News Scraper, charges $0.004 per article: 5 × $0.004 = $0.02 in result fees. The run was charged $0.0208 in total, because it also used $0.0008 of compute; fast search mode needs no proxy.

Step 2, the Article Extractor, charges $0.0009 per extracted article: 3 × $0.0009 = $0.0027. The total was $0.0042, because it also used $0.0011 of compute, $0.0003 of proxy traffic for the retried pages, and a little storage.

The whole pipeline cost $0.025. For 1,000 headlines and 1,000 full articles, the fees are $4.00 plus $0.90, so $4.90 plus usage. Pages that turn out not to be articles, articles outside your date or keyword filters, and failed URLs are not charged. The free plan includes $5 of usage every month with no credit card.

What to watch out for

• Strong anti-bot walls. Some publishers block automated visits even after the residential retry and the browser fallback; the Article Extractor names Reuters as an example. Those pages are reported in the run and not charged.
• Paywalls and logins. The extractor reads what a normal visitor gets. On a hard paywall that's often the teaser. Don't use tricks to get around a paywall; license the content instead.
• One query per Google News run. Run several queries in parallel, or use a schedule per topic. Each run returns up to 100 articles.
• Live blogs and galleries. Updates loaded later by JavaScript and image galleries aren't separate items. Pick standard articles when you need full text.
• Duplicates across runs. For monitoring, use onlyNewArticles with a date window so each article arrives once.
• Personal data. Bylines name journalists in their published role, but articles can name private people too. If you store full text, you store whatever personal data it contains.

Is it legal to scrape news articles?

Two rights matter here: copyright and the publisher's terms. Links, publisher names, publish dates and bylines are factual metadata, and headlines are short. The body of an article is the publisher's copyrighted work. Collecting it for internal analysis, monitoring or research is a different act from republishing it, and the publisher's terms of use may restrict automated collection either way.

A safe default: store the link, headline, source, date, author and your own analysis (topics, sentiment, a short summary), and keep full text only where you need it and are allowed to. Respect robots.txt and paywalls, and keep request rates moderate. Scrape.ly Store Actors collect only publicly available pages that any visitor can see. This is not legal advice; check each publisher's terms and ask a lawyer before you republish anything. Our web scraping vs. API guide covers when a licensed news API is the better fit.

Why Scrape.ly for news scraping

• No per-site selectors. The Article Extractor reads structured data, Open Graph and meta tags on any news site, so a redesign doesn't break your pipeline the way a hand-written BeautifulSoup script does.
• Discovery built in. Google News search, home pages, RSS feeds and sitemaps, with date and keyword filters and an "only new" mode for scheduled monitoring.
• Real costs, per article. Our five headlines and three full articles cost $0.025 together, and filtered or failed pages are free.
• LLM-ready. Markdown output, an MCP server for agents, and integrations with LangChain and LlamaIndex. For the endpoints in one place, see the Google News API page.

FAQ

What is a news scraper?

A news scraper collects articles from news sites: headlines, links, publish dates, authors and the article text. On Scrape.ly it's two scrapers that work together: the Google News Scraper finds articles for a query, and the Article Extractor pulls the full text and metadata.

How do I build a news crawler?

Point the Article Extractor at the sites you follow (home pages, RSS feeds or sitemaps) with startUrls, add a date window like 24 hours, turn on onlyNewArticles, and schedule the run every hour. Each run returns only articles you haven't seen yet.

Can you scrape Google News?

Yes. The Google News Scraper reads public Google News results for any query, country and language edition, with no login and no API key, up to 100 articles per run.

Can I scrape paywalled news articles?

You get what the site sends to a normal visitor, which on a hard paywall is usually a teaser, and the row is flagged isPaywalled. Getting past a paywall would breach the publisher's terms; license the content if you need it.

How accurate are the publish dates?

Step 2's publishedAt comes from the article's own metadata and was exact in our run. Step 1's date_utc is worked out from the age Google News shows, so it's only as precise as "4 weeks ago" unless you turn on exactPublishTimes.