How to Scrape Google Scholar: Papers, Citations and Author Profiles

To scrape Google Scholar, send keywords, a "Cited by" ID or an author profile to a scraper that reads Scholar's public results and saves each paper with its title, authors, venue, year, citation count and free full-text link. On Scrape.ly the Google Scholar Scraper does this from a search box or one API call, at $1.05 per 1,000 papers, with no Google account, API key or proxy to manage.

Below you'll find the fields it returns, the no-code steps, code for curl, Python and Node.js, the AI-agent route over MCP, and a real run from 26 October 2026: the top 10 papers for "large language models", with the log and the exact cost. More Google how-tos: Google Shopping and Google Flights.

What data you can get from Google Scholar

The Google Scholar Scraper has three modes you can mix in one run: keyword searches, "cited by" lists and author profiles. Paper rows (type: "paper") hold:

Author rows (type: "author") add a researcher's affiliation, interests, all-time and recent citations, h-index, i10-index, citations per year, co-authors and up to 5,000 articles. They're charged separately, so a keyword search never pays for profiles you didn't ask for.

How to scrape Google Scholar without code

1. Open the Actor's page in the Scrape.ly Store.
2. Pick the preset Large language model papers, 10 results. "Try this input" loads it in the console. This is the exact input of our run:

{
  "queries": ["large language models"],
  "maxResults": 10,
  "sortBy": "relevance",
  "includeCitations": false,
  "includePatents": false,
  "reviewArticlesOnly": false,
  "maxArticlesPerAuthor": 5,
  "articleSort": "citations",
  "language": "en"
}

3. Type your own topic and click Start. Ten papers took under 6 seconds in our run.
4. Open the Papers view and export JSON, CSV, Excel or HTML.

The fields that matter most:

• queries: keywords, one per line. Scholar operators work: author:"hinton", source:nature, intitle: and "exact phrases". Pasted Scholar search links work too.
• citedBy: a clusterId from any paper row, or a full "Cited by" link, to list the papers citing it. The Papers citing an article preset shows the shape.
• authors: Scholar author IDs or profile links, one profile row each, as in the author profile preset.
• maxResults: papers per search or cited-by list, up to 1,000.
• yearFrom, yearTo, sortBy, resultLanguages: filters. sortBy: "date" returns papers added in the last year, newest first.

Scrape Google Scholar with the API

Create a token under Settings → API & Integrations in the dashboard and export it as SCRAPELY_TOKEN. Searches run in fast mode and finish in seconds, so the synchronous endpoint returns the papers in one call:

curl -X POST "https://api.scrape.ly/v2/actors/querystone~google-scholar-scraper/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"queries": ["large language models"], "maxResults": 10, "includeCitations": false}'

The call waits up to 45 seconds and returns a JSON array; add ?format=csv for CSV. Author profiles and authorProfileLinks read Scholar's own pages and take longer. If a run passes 45 seconds, the call answers 408 while the run keeps going, so don't call again: start it with POST /v2/actors/querystone~google-scholar-scraper/runs, poll /v2/actor-runs/RUN_ID?waitForFinish=60 and read /v2/actor-runs/RUN_ID/dataset/items. The Actors API docs show each call.

Python (pip install scrapely-client-python). This script finds the most-cited paper for a topic, then lists the newest papers that cite it, which is the core loop of a citation tracker:

import os
from scrapely_client import ScrapelyClient

client = ScrapelyClient(os.environ["SCRAPELY_TOKEN"], api_url="https://api.scrape.ly")
scholar = client.actor("querystone/google-scholar-scraper")

search = scholar.call(run_input={"queries": ["large language models"], "maxResults": 10, "includeCitations": False})
papers = list(client.dataset(search.default_dataset_id).iterate_items())
top = max(papers, key=lambda p: p["citedByCount"] or 0)
print(f'{top["citedByCount"]:>6} citations  {top["title"]} ({top["year"]})')

citing = scholar.call(run_input={"citedBy": [top["clusterId"]], "maxResults": 20, "yearFrom": 2026})
for p in client.dataset(citing.default_dataset_id).iterate_items():
    print(f'{p["year"]}  {p["citedByCount"] or 0:>4}  {p["title"][:70]}  {p.get("fullTextUrl") or ""}')

Node.js (npm install scrapely-client):

import { ScrapelyClient } from 'scrapely-client';

const client = new ScrapelyClient({ token: process.env.SCRAPELY_TOKEN, baseUrl: 'https://api.scrape.ly' });

const run = await client.actor('querystone/google-scholar-scraper').call({
  queries: ['large language models'],
  maxResults: 10,
  includeCitations: false,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items.map((p) => [p.position, p.citedByCount, p.year, p.title]));

The Python client docs cover paging through large datasets. If you think of this data as an endpoint, the Google Scholar API page is where we cover the Google Scholar scraper as an API product.

With an AI agent (MCP)

Scrape.ly's hosted MCP server is at https://mcp.scrape.ly. In Cursor, add it to ~/.cursor/mcp.json:

{
  "mcpServers": {
    "scrapely": {
      "url": "https://mcp.scrape.ly",
      "headers": { "Authorization": "Bearer YOUR_TOKEN" }
    }
  }
}

In Claude Code, run claude mcp add --transport http scrapely https://mcp.scrape.ly and sign in with /mcp. Use https://mcp.scrape.ly?actors=querystone/google-scholar-scraper to give the agent only this tool. Then ask: "Use scrapely to find the 10 most relevant Google Scholar papers on retrieval-augmented generation since 2024, and give me title, year, citations and the free PDF link." Every row is flat JSON, which suits literature-review agents and RAG pipelines. The run in this article was started over MCP too. The MCP docs list the other clients.

A real run: the top 10 papers on large language models

We ran the preset on 26 October 2026. It saved 10 papers in 5.7 seconds, and its list price was $0.0105. The first five rows (authors are shown as Scholar prints them; they appear in their published capacity, and we list the first author only):

2026-10-26T14:25:23.694Z [Scrapely] Container started (setup 0.9 s)
2026-10-26T14:25:25.440Z [scrapely.actor] INFO  Using fast search mode. A search switches to direct mode (Scholar pages) only if fast search mode fails or the search needs it.
2026-10-26T14:25:25.440Z [scrapely.actor] INFO  Reading 1 search(es) and 0 author profile(s), up to 10 results per search.
2026-10-26T14:25:28.180Z [scrapely.actor] INFO  Searches in fast search mode: 1, in direct mode: 0. Fast search mode: 3 API pages, 3 calls, 0 retried, 2 short pages asked again, slowest 1082 ms.
2026-10-26T14:25:28.180Z [scrapely.actor] INFO  Loaded 0 pages (0 through the browser, 0 incomplete). DATACENTER: 0 requests, 0.00 MB, 0 throttled or blocked; RESIDENTIAL: 0 requests, 0.00 MB, 0 throttled or blocked.
2026-10-26T14:25:28.180Z [scrapely.actor] INFO  Saved 10 papers.

What the ten papers show:

• Relevance isn't citation order. The survey ranked third held 3,864 citations, more than the one ranked second (3,639). The paper with the second-highest count overall, "Large language models in medicine" in Nature Medicine (6,334), sat at position 10. Sort by citedByCount yourself if impact is what you're after.
• 27,412 citations across ten papers. The most-cited was the evaluation survey at 7,002; the least, a paper in Critical AI at 331.
• Free full text for all ten. Eight had a PDF link and two an HTML one, hosted on the ACM Digital Library, arXiv, MDPI, Springer, Elsevier, Duke University Press and nature.com.
• Recent work dominates. Five papers were from 2024, three from 2023, one from 2025 and one from 2026.
• Scholar shortens things. Seven author lists were cut (authorsTruncated: true) and five venue names ended in "…". That's how Scholar prints them; the Actor flags it instead of guessing.
• No throttling. The log shows fast search mode, 3 calls, zero retries and zero Scholar pages loaded, which is why no proxy was needed.

What it costs

Two charge events, platform usage (compute, proxy, browser) included: $0.00105 per paper ($1.05 per 1,000), from searches and cited-by lists, and $0.01049 per author profile ($10.49 per 1,000), including the articles you ask for. Retries, throttled pages and failed inputs aren't charged. Prices checked on 26 October 2026.

• Our run: 10 papers × $0.00105 = $0.0105 at list price. The account was billed $0.0084 after its plan's 20% Store discount.
• A literature review: 20 queries × 50 papers = 1,000 papers, about $1.05.
• A hiring shortlist: 30 author profiles × $0.01049 = $0.31, each with citations per year and top articles.

The Free plan's $5 of monthly credit covers about 4,760 papers. See pricing for plans.

What to watch out for

• 1,000 results per search is Scholar's ceiling. Deep pages also throttle: on the publisher's test runs, fast mode saved 41 to 50 papers when 50 were asked for and 75 of 100. For more, split a topic into narrower searches, for example by year range. The status message says when a search stopped early.
• Snippets, not abstracts. snippet is the text Scholar shows, not the full abstract. For full abstracts by DOI, pair this Actor with the OpenAlex Research Papers Scraper.
• Author profile IDs cost time. In fast mode authorProfiles is null. Turn on authorProfileLinks to fill it; the Actor then reads Scholar's own pages, which is slower and subject to throttling. Author profile rows themselves always work.
• Citation-only records are a choice. includeCitations: false (as in the preset) drops [CITATION] entries Scholar knows only from reference lists. Keep them for exhaustive citation counts.
• No export formats. BibTeX, APA and MLA "Cite" formats aren't exported; build them from the fields.
• Profiles are personal data. Author rows hold a person's name, affiliation and verified email domain. Collect them only when you need them, as in the legal section below.

Is it legal to scrape Google Scholar?

The Actor reads the public pages and results anyone can open without an account, and paper metadata (titles, venues, years, citation counts) is bibliographic fact. Google is direct about automated access, though. Scholar's help page answers a researcher whose script got blocked: "please respect our robots.txt when you access Google Scholar using automated software", and it states "we're unable to provide bulk access". Scholar's robots.txt disallows /scholar for all crawlers, and Google's Terms of Service (effective July 30, 2026) list automated access that violates robots.txt instructions as abuse. Read them before you build a product on this data.

Author profiles add privacy law. A researcher's name, affiliation and email domain are personal data under the GDPR and the CCPA, even when published, so collect profiles for a clear purpose such as citation analysis, keep only the fields you need, and don't use them for unsolicited marketing. Listing authors of published papers in their published capacity, as our table does, is ordinary bibliographic practice. Our guide on whether web scraping is legal covers the general rules. This is not legal advice: for commercial use, ask a lawyer about your case.

Why Scrape.ly for Google Scholar data

• No official API, and no bulk access. Google offers no Scholar API and says on its help page it can't provide bulk access. The Actor turns the public results into rows.
• No blocks to fight. The open-source scholarly library warns that search_pubs and citedby queries "will lead to Google Scholar blocking your requests and may eventually block your IP address" (its README). Fast search mode served our run with zero Scholar page loads, and direct-mode pages are retried on fresh IPs automatically.
• Fields, not selectors. Crawlbase's tutorial parses div.gs_r and h3.gs_rt and splits the byline by hand. Here authors come as a list, year and venue split out, citation counts as numbers.
• Cheap at research scale. $1.05 per 1,000 papers, platform usage included, with a hard cap per run.

For the wider family of Google results, see our guides to scraping Google search results and the best SERP APIs.

FAQ

Is there an official Google Scholar API?

No. Google doesn't publish one, and Scholar's help page says it can't provide bulk access to records. Third-party scrapers like this one read the public results and return them as JSON over a normal REST API.

Can you scrape Google Scholar with Python?

Yes. Install scrapely-client-python, call querystone/google-scholar-scraper with your queries, and iterate over the dataset, as in the citation-tracker example above. The scraping runs in the cloud, so your script needs no proxy, browser or parser.

How do I scrape the papers that cite an article?

Copy the clusterId from a paper row (or the paper's "Cited by" link) into citedBy. The Actor saves the citing papers as normal paper rows, with the cited paper's ID in citedById.

Can I scrape a researcher's h-index and citations?

Yes. Put the Scholar author ID or profile link in authors. Each profile row has all-time and recent citations, h-index, i10-index, citations per year, co-authors and up to 5,000 articles, at $0.01049 per profile.

Why does Google Scholar block my scraper?

Scholar throttles signed-out automated traffic, especially on deep result pages, and asks scripts to respect its robots.txt. The Actor's fast search mode avoids Scholar's pages for searches, and pages it must read are retried on fresh IPs. Retries are never charged.

How much does it cost to scrape Google Scholar?

$0.00105 per paper and $0.01049 per author profile, platform usage included. Our 10-paper run cost $0.0105 at list price.