
Website Content Crawler
scrape.ly/store/querystone/website-content-crawlerAIDeveloper tools
1 monthly user

Search 300M+ scholarly works in OpenAlex with filters (year, type, open access, topic, institution, author, language, citations), look up DOIs, list citing works and references, and fetch author and institution profiles. Full abstracts as text, open-access PDF links, clean JSON for research and AI agents.
Search and export scholarly papers from OpenAlex, the open catalogue of 300M+ works from every field. Get full abstracts as plain text, DOIs, authors with ORCIDs and institutions, venues, citation counts, topics and open-access PDF links, plus citing works, reference lists, author profiles and institution profiles. No login and no API key needed; 200 results per request, so thousands of papers take seconds.
| Field | What it does |
|---|---|
searchQueries | Keyword searches over titles, abstracts and full text |
works | DOIs, OpenAlex ids, PubMed ids or arXiv links to look up |
citingWorksOf | Papers that cite these DOIs/ids |
referencesOf | Papers these DOIs/ids cite |
authorWorks | Papers by these authors (name, ORCID or id) |
authorSearch, institutionSearch | Author and institution profile rows |
maxResults | Rows per list (default 20, 0 = no limit) |
yearFrom, yearTo, workTypes, openAccessOnly, oaStatus | Paper filters |
topics, institutions, authorFilter, countries, languages | Who and what filters (names or ids) |
minCitations, hasAbstract, hasPdf, excludeRetracted | Quality filters |
sortBy | relevance, citations, newest, oldest |
includeAbstract, includeReferenceLists | Output options |
apiUrls | Advanced: OpenAlex API or web links |
apiKey | Optional: your own free OpenAlex key |
{
"searchQueries": ["retrieval augmented generation"],
"yearFrom": 2023,
"openAccessOnly": true,
"workTypes": ["article", "preprint"],
"sortBy": "citations",
"maxResults": 200
}
Paper rows (type: "work"): id, doi, title, publicationYear, publicationDate, workType, language, abstract, authors (name, id, ORCID, position, corresponding, institutions), authorNames, institutions, countries, venue, venueIssn, publisher, volume, issue, firstPage, lastPage, isOpenAccess, oaStatus, oaUrl, pdfUrl, license, landingPageUrl, citedByCount, referencedWorksCount, fwci, citationPercentile, primaryTopic, subfield, field, domain, topics, keywords, concepts, meshTerms, funders, sdgs, isRetracted, pmid, pmcid, countsByYear, optional referencedWorks and relatedWorks, openalexUrl, and sourceType / sourceInput (which input produced the row).
Author rows (type: "author"): id, name, orcid, alternativeNames, worksCount, citedByCount, hIndex, i10Index, twoYearMeanCitedness, lastKnownInstitution(s), countryCode, affiliations (with years), topics, countsByYear.
Institution rows (type: "institution"): id, name, ror, institutionType, countryCode, city, region, country, latitude, longitude, homepageUrl, imageUrl, acronyms, worksCount, citedByCount, hIndex, i10Index, parentInstitutions, topics, countsByYear.
{
"type": "work",
"sourceType": "lookup",
"sourceInput": "https://arxiv.org/abs/1706.03762",
"id": "W2626778328",
"title": "Attention Is All You Need",
"publicationYear": 2025,
"workType": "preprint",
"language": "en",
"abstract": "The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. …",
"authors": [{"name": "Jane Example", "id": "A5000000001", "orcid": null, "position": "first", "isCorresponding": false, "institutions": ["Example University"]}],
"authorNames": ["Jane Example", "John Sample"],
"isOpenAccess": true,
"oaStatus": "gold",
"pdfUrl": "https://example.org/preprint.pdf",
"citedByCount": 24501,
"referencedWorksCount": 28,
"primaryTopic": "Natural Language Processing Techniques",
"field": "Computer Science",
"keywords": ["transformer models", "attention mechanism", "machine translation"],
"openalexUrl": "https://openalex.org/W2626778328"
}
Is this legal? Yes. OpenAlex is an open catalogue; its data is released under CC0. The actor uses the public OpenAlex API.
Do I need an API key? No. The actor works without one. For very large jobs you can add your own free OpenAlex key in Advanced.
How many results can I get? Up to 10,000 per list (search, citing list, filter set) without an API key, which is OpenAlex's own paging limit. Add your free OpenAlex key in Advanced to page through everything, or split a big list by year.
Why is the abstract empty for some papers? OpenAlex has no abstract for those papers (common for older works and some publishers). Use Only papers with an abstract to skip them.
How do I get the papers citing a DOI? Put the DOI in Papers citing. Filters and sort apply to that list too.
Can I call it from an AI agent? Yes: the inputs are simple strings and numbers, every row has clear field names, and it runs as a tool over MCP or the API.
pdfUrl and oaUrl point to where it can be read.Every field below can be set in the run form, passed through the API, or saved as a task. Unset fields fall back to the defaults shown.
searchQueriesarrayOptionalSearch papers
Keywords to search in titles, abstracts and full text, one per line. Results are ranked by relevance unless you pick another sort.
worksarrayOptionalLook up papers (DOI or id)
DOIs (10.1038/nature14539 or https://doi.org/...), OpenAlex work ids (W2919115771), PubMed ids (pmid:26017442) or arXiv links. One row per paper.
citingWorksOfarrayOptionalPapers citing
DOIs or OpenAlex work ids. Saves the papers that cite each one (filters and sort apply).
referencesOfarrayOptionalReferences of
DOIs or OpenAlex work ids. Saves the papers each one cites.
authorWorksarrayOptionalPapers by author
Author names, ORCIDs or OpenAlex author ids (A5108093963). Saves each author's papers. Names use the best match.
authorSearcharrayOptionalAuthor profiles
Author names, ORCIDs or ids. Saves author rows: works, citations, h-index, i10-index, institutions, topics, yearly counts.
institutionSearcharrayOptionalInstitution profiles
Institution names, ROR ids or OpenAlex ids. Saves institution rows: location, type, works, citations, h-index, topics.
maxResultsintegerOptionaldefault 20Maximum results per list
Most rows to save for each search, citing list, reference list, author or institution search. 0 means no limit.
yearFromintegerOptionalPublished from (year)
Only papers published in or after this year.
yearTointegerOptionalPublished until (year)
Only papers published in or before this year.
workTypesarrayOptionalPaper types
Only these types: article, preprint, review, book, book-chapter, dataset, dissertation, letter, editorial, report, ...
openAccessOnlybooleanOptionaldefault falseOpen access only
Only papers that are free to read somewhere.
oaStatusarrayOptionalOpen access status
Only these statuses: diamond, gold, green, hybrid, bronze, closed.
topicsarrayOptionalTopics
Topic names (e.g. Topic Modeling) or OpenAlex topic ids (T10181). Names use the best match.
institutionsarrayOptionalInstitutions
Only papers with an author from these institutions: names, ROR ids or OpenAlex ids.
authorFilterarrayOptionalAuthors
Only papers by these authors: names, ORCIDs or OpenAlex ids.
countriesarrayOptionalAuthor country
Only papers with an author in this country (two-letter code, e.g. US, DE).
languagesarrayOptionalLanguages
Only papers in these languages (two-letter codes, e.g. en, de, fr).
minCitationsintegerOptionalMinimum citations
Only papers cited at least this many times.
hasAbstractbooleanOptionaldefault falseOnly papers with an abstract
hasPdfbooleanOptionaldefault falseOnly papers with a full-text PDF in OpenAlex
excludeRetractedbooleanOptionaldefault trueExclude retracted papers
sortByenumOptionaldefault "relevance"Sort by
Relevance applies to searches; lists without a search are sorted by citations unless you choose newest or oldest.
includeAbstractbooleanOptionaldefault trueInclude abstracts
Full abstract as plain text (rebuilt from OpenAlex's word index).
includeReferenceListsbooleanOptionaldefault falseInclude reference and related id lists
Add the ids of every cited work and of related works to each paper row (larger rows).
apiUrlsarrayOptionalOpenAlex API or web links
Advanced: paste api.openalex.org or openalex.org links for works, authors or institutions; their filter, search and sort are used.
proxyConfigurationobjectOptionaldefault {"useScrapelyProxy":true,"scrapelyProxyGroups":["DATACENTER…Proxy configuration
Datacenter proxy (default) is the fastest and cheapest option for this API.
Every run writes its results to a dataset. Preview it in the dashboard, then export it or read it through the API.
| Field | Type | Description |
|---|---|---|
| type | string | work, author or institution |
| sourceType | string | null | Which input produced the row |
| sourceInput | string | null | The input value |
| id | string | null | OpenAlex id |
| doi | string | null | DOI |
| title | string | null | Title |
| publicationYear | number | null | Year |
| publicationDate | string | null | Date |
| workType | string | null | article, preprint, review, ... |
| language | string | null | Language |
| abstract | string | null | Full abstract as text |
| authors | array | null | Authors with ids, ORCID, institutions |
| authorNames | array | null | Author names |
| institutions | array | null | Institutions |
| countries | array | null | Author countries |
| venue | string | null | Journal or source |
| venueType | string | null | Source type |
| venueIssn | string | null | ISSN-L |
| publisher | string | null | Publisher |
| volume | string | null | Volume |
| issue | string | null | Issue |
| firstPage | string | null | First page |
| lastPage | string | null | Last page |
| isOpenAccess | boolean | null | Free to read |
| oaStatus | string | null | OA status |
| oaUrl | string | null | Best open-access link |
| pdfUrl | string | null | PDF link |
| license | string | null | License |
| landingPageUrl | string | null | Publisher page |
| citedByCount | number | null | Times cited |
| referencedWorksCount | number | null | References |
| fwci | number | null | Field-weighted citation impact |
| citationPercentile | number | null | Citation percentile |
| primaryTopic | string | null | Primary topic |
| subfield | string | null | Subfield |
| field | string | null | Field |
| domain | string | null | Domain |
| topics | array | null | Topics |
| keywords | array | null | Keywords |
| concepts | array | null | Concepts |
| meshTerms | array | null | MeSH terms |
| funders | array | null | Funders |
| sdgs | array | null | UN SDGs |
| isRetracted | boolean | null | Retracted |
| pmid | string | null | PubMed id |
| pmcid | string | null | PMC id |
| countsByYear | object | null | Citations (or works) per year |
| referencedWorks | array | null | Ids of cited works |
| relatedWorks | array | null | Ids of related works |
| openalexUrl | string | null | OpenAlex link |
| name | string | null | Author or institution name |
| orcid | string | null | ORCID |
| alternativeNames | array | null | Other names |
| worksCount | number | null | Works |
| hIndex | number | null | h-index |
| i10Index | number | null | i10-index |
| twoYearMeanCitedness | number | null | 2-year mean citedness |
| lastKnownInstitutions | array | null | Last known institutions |
| lastKnownInstitution | string | null | Last known institution |
| countryCode | string | null | Country code |
| affiliations | array | null | Affiliation history |
| ror | string | null | ROR id |
| institutionType | string | null | Institution type |
| city | string | null | City |
| region | string | null | Region |
| country | string | null | Country |
| latitude | number | null | Latitude |
| longitude | number | null | Longitude |
| homepageUrl | string | null | Homepage |
| imageUrl | string | null | Image |
| acronyms | array | null | Acronyms |
| parentInstitutions | array | null | Parent institutions |
You pay for the events the actor reports, such as each result or page it produces. Platform usage is included in the event prices.
| Event | Charged when | Price |
|---|---|---|
| Result | One saved OpenAlex paper, author or institution. | $0.0032 |
Run this actor from your own code. Swap in your API token and the examples work as-is.
import { ScrapelyClient } from 'scrapely-client'
const client = new ScrapelyClient({ token: process.env.SCRAPELY_TOKEN })
// Start the actor and wait for the run to finish.
const run = await client.actor('querystone/openalex-scraper').call({
"searchQueries": [
"retrieval augmented generation"
],
"maxResults": 20,
"proxyConfiguration": {
"useScrapelyProxy": true,
"scrapelyProxyGroups": [
"DATACENTER"
]
}
})
// Read the dataset the run produced.
const { items } = await client.dataset(run.defaultDatasetId).listItems()
console.log(items)Tokens live in your account settings. Keep them server-side: a token carries the full permissions of your account.
Connect the Scrape.ly MCP server once and your agent can run this actor as a tool. Pick your tool:
Fetch scrape.ly/agents.md and run querystone/openalex-scraperPaste into Claude Code, Codex, Cursor or any AI agent. It signs you in once in the browser.
{
"mcpServers": {
"scrapely": {
"url": "https://api.scrape.ly/mcp?tools=actors,runs,storage,docs&actors=querystone/openalex-scraper"
}
}
}No key in the file: your client opens the browser to sign you in the first time it connects.
Inputs the developer prepared for common jobs. Open one to see its input, then copy it to your account to run it as-is or adjust it.
Open-access papers with 50+ citations, most cited first.
Works that cite a given OpenAlex paper.
Keyword search for research papers.
Author profiles found by name.
Scrape.ly gives you the infrastructure, the tooling, and the team to extract, process, and operationalize web data at any scale. Setup takes minutes. The advantage lasts forever.