How to Scrape Google AI Overviews and AI Mode Answers

To scrape Google AI Overviews, send your keywords to a Google AI Overview scraper that opens Google in a real browser from the country you choose and saves one row per keyword: whether an overview appeared, its full text and every source it cites. On Scrape.ly that's the Google AI Overviews Scraper, at $1 per 1,000 overviews, and keywords without an overview cost no event fee. For Google's separate AI Mode tab, a second Actor returns the full answer with sources, follow-up questions and brand mentions.

This guide covers the fields, the no-code steps, the API (curl, Python, Node.js and MCP), costs and pitfalls. It uses two real runs from 23 October 2026: three US keywords for AI Overviews in 26 seconds, and one AI Mode answer with brand tracking.

What data you can get from Google AI Overviews

The Google AI Overviews Scraper writes one row per keyword:

Source cards come first in sources, then sites linked only from inside the answer. The text behind Google's "Show more" is included; images, videos and widgets inside an overview are not.

How to run the Google AI Overview scraper without code

1. Open the Actor in the Scrape.ly Store and sign in.
2. Pick the preset AI Overviews for 3 US searches. "Try this input" opens it in the console.
3. Check the input. This is the exact JSON of our run:

{
  "queries": "what is python programming\nhow to choose a heat pump\nkubernetes pros and cons",
  "countryCode": "us",
  "languageCode": "en",
  "includeQueriesWithoutOverview": false
}

4. Click Start. Our run took 26 seconds.
5. Open the AI Overviews view for one row per keyword, or Cited sources for one row per source, and export as JSON, CSV, Excel or HTML.

The fields that matter most:

• queries (required): one keyword per line, up to 5,000 per run. Duplicates are removed.
• countryCode and languageCode: the market (default us, en). Google is opened from a residential exit in that country, because overviews differ by market. The preset AI Overviews for 3 German searches shows another market.
• includeQueriesWithoutOverview: on by default. It saves a free row with hasAiOverview: false for every keyword without an overview, which is how you measure coverage. Turn it off to keep only overviews.

How to scrape Google AI Overviews with the API

Create an API token under Settings → API & Integrations and export it as SCRAPELY_TOKEN.

curl. Our three keywords took 26 seconds, and the README puts 10 keywords at about 45 seconds, the limit of the synchronous endpoint. Use the asynchronous flow: start the run, wait, then read the rows.

# 1. start the run; the response's data.id is the run ID
curl -X POST "https://api.scrape.ly/v2/actors/querystone~google-ai-overviews-scraper/runs" \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"queries": "what is python programming\nhow to choose a heat pump\nkubernetes pros and cons", "countryCode": "us", "languageCode": "en", "includeQueriesWithoutOverview": false}'
# 2. wait (up to 60 s per call; repeat until status is SUCCEEDED)
curl "https://api.scrape.ly/v2/actor-runs/RUN_ID?waitForFinish=60" \
  -H "Authorization: Bearer $SCRAPELY_TOKEN"
# 3. read the rows (CSV here)
curl "https://api.scrape.ly/v2/actor-runs/RUN_ID/dataset/items?format=csv" \
  -H "Authorization: Bearer $SCRAPELY_TOKEN"

Python (pip install scrapely-client-python). call() waits however long the run takes. This example counts how often each domain is cited across your keywords, the core number of AI search visibility:

import os
from collections import Counter
from scrapely_client import ScrapelyClient

client = ScrapelyClient(os.environ["SCRAPELY_TOKEN"], api_url="https://api.scrape.ly")

run = client.actor("querystone/google-ai-overviews-scraper").call(
    run_input={"queries": "what is python programming\nhow to choose a heat pump\nkubernetes pros and cons",
               "countryCode": "us", "languageCode": "en"}
)
print(run.status)  # SUCCEEDED, FAILED, ABORTED or TIMED-OUT
cited = Counter()
for row in client.dataset(run.default_dataset_id).iterate_items():
    print(row["query"], row["hasAiOverview"], row["sourcesCount"])
    cited.update({s["domain"] for s in row["sources"]})
print(cited.most_common(10))

Node.js (npm install scrapely-client):

import { ScrapelyClient } from 'scrapely-client';

const client = new ScrapelyClient({ token: process.env.SCRAPELY_TOKEN, baseUrl: 'https://api.scrape.ly' });

const run = await client.actor('querystone/google-ai-overviews-scraper').call({
  queries: 'what is python programming\nhow to choose a heat pump\nkubernetes pros and cons',
  countryCode: 'us',
  languageCode: 'en',
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items.map((r) => [r.query, r.hasAiOverview, r.sources.map((s) => s.domain).join(', ')]));

AI agents (MCP). Add the hosted Scrape.ly MCP server. In Cursor, put this in ~/.cursor/mcp.json:

{
  "mcpServers": {
    "scrapely": {
      "url": "https://api.scrape.ly/mcp",
      "headers": { "Authorization": "Bearer YOUR_TOKEN" }
    }
  }
}

In Claude Code, run claude mcp add --transport http scrapely https://api.scrape.ly/mcp and sign in with /mcp. Then ask: "Use scrapely to run querystone/google-ai-overviews-scraper for my 20 keywords in the US and tell me which domains are cited most and whether mysite.com appears." Runs longer than 45 seconds keep going; the agent polls get-actor-run, then reads the rows with get-actor-output. The Google AI Overviews API page shows the same Actor as an API product.

A real run: three US keywords

We ran the preset on 23 October 2026. All three keywords had an AI Overview. The run saved 3 rows in 26 seconds and was charged $0.0259.

What the rows show:

• Reddit is everywhere. reddit.com was cited for all three keywords, first for two of them. YouTube was cited twice for both the heat pump and the Python query.
• Two of three overviews were streamed in. Google added them after the page loaded (type: live). A plain HTTP scraper that reads only the first HTML would have missed both.
• The answers are long. The overview texts ran from 2,596 to 3,466 characters, with headings and bullet lists kept as lines.
• Official sources still count. python.org appeared twice in the Python overview's sources, as a source card and as a link inside the answer.

The log, without the platform's start-up lines:

2026-10-23T03:02:32.904Z [scrapely.actor] INFO  Checking 3 queries for AI Overviews (country=US, language=en, concurrency=3).
2026-10-23T03:02:38.040Z [scrapely.actor] INFO  browser_session attempt=1 state=sorry latency_ms=4644 connect=1574 commit=2819 ready=4533
2026-10-23T03:02:41.983Z [scrapely.actor] INFO  browser_session attempt=2 state=serp latency_ms=3835 connect=1508 commit=2711 ready=3707
2026-10-23T03:02:41.984Z [scrapely.actor] INFO  session_ready live=1 after_ms=9011
2026-10-23T03:02:49.042Z [scrapely.actor] INFO  Done: 3 AI Overviews, 0 queries without one, 0 failed, 3/3 queries in 16 s. (deferred overviews: 2, deferred without overview: 0, partial: 0)
2026-10-23T03:02:49.694Z [Scrapely] Container state: status=exited exitCode=0 oomKilled=false

The first browser session landed on Google's "sorry" page, its block page for automated traffic. The Actor opened a second session, got a normal results page, and read all three overviews in 16 seconds. "deferred overviews: 2" are the two live rows.

Plus one Google AI Mode answer

AI Mode is a separate tab with chat-style answers, and the AI Overviews Actor doesn't cover it. The Google AI Mode Scraper does. We asked it one question from its preset 2 CRM questions with brand tracking:

{
  "queries": "best crm for startups",
  "brands": ["HubSpot", "Salesforce", "Pipedrive"],
  "domains": ["hubspot.com"],
  "countryCode": "us",
  "languageCode": "en"
}

The answer came back in 12 seconds with 20 sources, 10 of them cited inside the answer text. The brand fields did the counting: mentionedBrands listed HubSpot (5 mentions, first) and Pipedrive (3 mentions, second). Salesforce wasn't named in the answer, although salesforce.com was cited as source 4. domainCitations showed hubspot.com cited at positions 15 and 17, both inside the answer. The row also carried 3 follow-up questions Google asked back and no product cards. The answer named Attio, Zoho CRM and Close as well, so add every competitor you care about to brands.

What it costs

AI Overviews: one event, $0.001 per keyword with an overview, so $1.00 per 1,000 overviews. Keywords without an overview and failed checks pay no event fee. Platform usage (the hosted browser that opens Google, plus compute) is billed on top.

For our run: 3 overviews × $0.001 = $0.003 in event fees. The run was charged $0.0259 in total: $0.021 of hosted browser time, $0.0017 of compute and $0.0002 of proxy traffic and storage. Browser time weighs more on a tiny run like this one ($0.007 per keyword). In the publisher's own test of 1,000 keywords, usage came to about $2.10, and the run totaled about $3.08 with $0.98 in events.

AI Mode: $0.15 per answer, plus usage. Our single answer was charged $0.1746: $0.15 for the answer, $0.024 of browser time and under $0.001 of compute and storage. Queries without an AI Mode answer are free and aren't saved.

Set a maximum charge per run for a hard cap; both Actors stop cleanly before they would go over it.

What to watch out for when scraping AI Overviews

• Overviews come and go. Google shows one only when it decides it adds something. Navigational, local and very short keywords rarely get one, and the same keyword can get one today and not tomorrow. Track coverage with the free hasAiOverview: false rows instead of assuming every keyword has an answer.
• Many overviews load late. Two of our three were streamed in after the page loaded. The Actor waits for them; DIY scrapers that read the first HTML miss them.
• Google blocks automated sessions. Our log shows a "sorry" page on the first attempt. The Actor retries on a fresh session, so you don't write that logic.
• Answers differ by market and over time. Overviews vary with location, language and day. Fix countryCode and languageCode per market and schedule the same list weekly to see real changes.
• AI Mode is a different product. Our AI Mode answer cited 20 sources against 7 to 10 for the overviews, and it's priced per answer. Use it for the questions where buyers ask Google for recommendations, and AI Overviews for broad keyword coverage.

Is it legal to scrape Google AI Overviews?

The Actors read the same public results page any logged-out visitor sees, without a Google account. Google's rules on automated access are specific, though. Its Terms of Service list "using automated means to access content from any of our services in violation of the machine-readable instructions on our web pages (for example, robots.txt files that disallow crawling, training, or other activities)" as abuse, and Google's robots.txt disallows /search for all crawlers. Read both before you build a product on this data.

Overview text is generated by Google and summarizes publishers' pages, so use it for analysis rather than republishing it. Overviews rarely contain personal data; if a source or answer names a person and you store it, the GDPR and CCPA apply. For visibility research on public results at modest volumes, those are the habits that matter. This is general information, not legal advice: for commercial use, ask a lawyer about your case.

Why Scrape.ly for AI Overview data

• Google only reports your own clicks. Google's own guidance says sites appearing in AI Overviews and AI Mode are included in Search Console's overall traffic, reported within the "Web" search type (Google Search Central). That covers only your own site and doesn't show the answer or who else was cited.
• No selectors to babysit. Hand-written guides drive Selenium, click "Show more" and parse Google's generated class names, then add retries for missing overviews. The Actor returns text and sources as fields and handles late-loading overviews and block pages.
• You pay for answers, not attempts. $1 per 1,000 overviews, nothing for keywords without one, plus a maximum charge per run.
• One platform for all AI search. Pair this Actor with the AI Mode Scraper, the ChatGPT Search Scraper and the Google Search Scraper's live AI overview preset, which puts the overview next to organic rankings. Our AI search visibility tracking guide turns these into a weekly brand report, and scraping Google search results covers the classic blue links.

FAQ

Does Google offer an API for AI Overviews?

Google's documentation for site owners points to Search Console, which counts traffic from AI features inside its Web search totals for your own site only; it doesn't return the overview text or the other cited sites. To get the overview text and cited sources for any keyword, use a scraper such as the Google AI Overviews Scraper, which you can also call as a REST API.

How do I know which keywords trigger an AI Overview?

Run your keyword list with includeQueriesWithoutOverview on (the default). Every keyword gets a row, and hasAiOverview is true or false. Rows without an overview carry no event fee, so measuring coverage is cheap.

Can I scrape Google AI Mode answers too?

Yes, with the Google AI Mode Scraper. It returns the answer as text and Markdown, every source with its real URL and whether it's cited inside the answer, product cards, follow-up questions, and mentions of the brands and domains you list. It costs $0.15 per answer.

How much does it cost to scrape Google AI Overviews?

$1.00 per 1,000 overviews in event fees, plus hosted-browser usage. Our 3-keyword run cost $0.0259 in total; the publisher measured about $3.08 for 1,000 keywords. Keywords without an overview pay no event fee.

Can I scrape AI Overviews for another country?

Yes. Set countryCode (for example gb, de or in) and languageCode. The Actor opens Google from a residential exit in that country, so you see the overview local users see.