arXiv Papers Scraper

📚 arXiv Papers Scraper

Search arXiv the way researchers do (keywords, field syntax, categories, authors, submission dates), pull each day's new submissions for any category, or look up papers by ID, and get clean JSON back in seconds. Every paper comes with its abstract, authors, categories, DOI, journal reference, code links and a ready BibTeX entry. Switch on full text and you also get the whole paper as Markdown, ready for RAG pipelines and AI agents. No login, no API key.

✨ Why use arXiv Papers Scraper?

• ⚡ Fast — the official arXiv API, up to 500 papers per request; a 20-paper search finishes in about 10 seconds and 1,000 papers in under 20.
• 🤖 Built for AI and RAG — flat, predictable fields, plain-text abstracts and optional full text as Markdown with LaTeX math kept as $...$.
• 🗂️ Three ways in — search queries, daily new-submission listings per category, and lists of IDs or arXiv links.
• 📑 Citation-ready — DOI, journal reference, publication status (preprint, accepted, published) and a BibTeX entry for every paper.
• 🤝 Polite by design — follows arXiv's API guidance of one request every three seconds over a single connection, so your runs never get blocked.
• 💸 Cheap — pay per paper, nothing for empty searches.

🎯 Use cases

• 🧠 RAG corpora — collect papers on a topic with their full text and feed them into a vector store.
• 🕵️ Agent tools — let an AI agent search arXiv, read abstracts and fetch the full paper when it needs details.
• 📰 Daily monitoring — schedule a run on cs.CL, cs.LG or q-bio.GN and get every new paper each morning.
• 📊 Research analytics — track publication trends by category, author or keyword over time.
• 📚 Literature reviews — export abstracts and BibTeX for a topic to CSV or Excel in one go.
• 👩‍🔬 Author tracking — list everything a researcher or lab has posted.

🚀 How to use

1. Enter one or more search queries (plain keywords or arXiv syntax such as ti:transformer AND au:vaswani), and/or paste paper IDs or arXiv URLs, and/or list categories for new submissions.
2. Optionally narrow the search with categories, authors and a submission date range, and pick the sort order.
3. Set Max papers per query or category.
4. Turn on Include full text if you need the whole paper.
5. Run it and export the results as JSON, CSV, Excel or HTML, or read them through the API.

📥 Input

{
  "searchQueries": ["retrieval augmented generation"],
  "categories": ["cs.CL", "cs.IR"],
  "dateFrom": "2025-01-01",
  "maxResults": 50,
  "sortBy": "submittedDate",
  "includeFullText": true
}

📤 Output

Example from a real run (abstract, full text and BibTeX shortened):

{
  "arxivId": "1706.03762",
  "arxivIdVersioned": "1706.03762v7",
  "version": 7,
  "title": "Attention Is All You Need",
  "abstract": "The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performi...",
  "authors": [
    "Ashish Vaswani",
    "Noam Shazeer",
    "Niki Parmar",
    "..."
  ],
  "authorCount": 8,
  "firstAuthor": "Ashish Vaswani",
  "published": "2017-06-12T17:57:34Z",
  "updated": "2023-08-02T00:41:18Z",
  "publishedDate": "2017-06-12",
  "primaryCategory": "cs.CL",
  "primaryCategoryName": "Computation and Language",
  "categories": [
    "cs.CL",
    "cs.LG"
  ],
  "comment": "15 pages, 5 figures",
  "journalRef": null,
  "doi": null,
  "arxivDoi": "10.48550/arXiv.1706.03762",
  "publicationStatus": "preprint",
  "hasCode": false,
  "codeUrls": [],
  "absUrl": "https://arxiv.org/abs/1706.03762v7",
  "pdfUrl": "https://arxiv.org/pdf/1706.03762v7",
  "htmlUrl": "https://arxiv.org/html/1706.03762v7",
  "bibtex": "@misc{vaswani2017attention,\n  title={{Attention Is All You Need}},\n  author={Ashish Vaswan...",
  "input": "1706.03762",
  "inputType": "paper",
  "scrapedAt": "2026-10-15T22:54:24Z",
  "fullText": "Provided proper attribution is provided, Google hereby grants permission to reproduce the tables and figures in this paper solely for use in journalistic or scholarly works.\n\n# Attention Is All You Need\n\n## Abstract\n\nThe dominant sequence transduction models a...",
  "fullTextSource": "html",
  "fullTextFormat": "markdown",
  "fullTextWords": 6053,
  "fullTextError": null
}

💰 Pricing

• $1.50 per 1,000 papers with metadata and abstract.
• $3.00 per 1,000 papers when full text is switched on and found (papers whose full text can't be loaded are billed at the normal rate).
• Example: 1,000 papers ≈ $1.50, or $3.00 with full text, plus a few cents of platform usage (compute and proxy traffic), which is billed separately.
• Empty searches and invalid inputs cost nothing.

❓ FAQ

Is it legal to scrape arXiv? The actor uses arXiv's public API and export mirror and follows arXiv's rate guidance. Paper metadata is free to reuse; full texts keep their authors' licenses, so check each paper's license before redistributing it.

Do I need an arXiv account, API key or proxy? No. The default datacenter proxy is enough.

How many papers can I get? Up to 30,000 per search query (an arXiv API limit). Split bigger jobs by date range or category.

Why is full text slower? arXiv asks clients to wait three seconds between requests, so each full text adds about 3 to 6 seconds. Metadata comes in pages of up to 500 papers, so it stays fast.

What if a field is empty? It is null (or an empty list) when arXiv has no value, for example doi for papers that were never published in a journal, or authorDetails when no affiliations were submitted.

Can I use it from an AI agent or on a schedule? Yes. Call it through the API or MCP with a query and read the dataset, or schedule it with newSubmissionsCategories for a daily feed.

⚠️ Limits

• arXiv's relevance ranking is keyword based; sort by submission date for the newest papers.
• Affiliations exist only for papers whose authors entered them.
• Full text comes from arXiv's HTML version (most papers) or the PDF as plain text; figures are not included (their captions are), and complex PDF layouts can come out imperfect.
• Citation counts are not part of arXiv's data, so the output has none.
• At most 30,000 papers per search query or category (the arXiv API does not page further); split bigger jobs by date range or category.
• Daily listings follow arXiv's announcement schedule (Sunday to Thursday evenings US Eastern time); on other days the newest submissions are returned instead.