arXiv Paper Search Scraper API + AI Summaries

arXiv Paper Scraper at a glance: data from the official arXiv API · no login, no browser, no proxies · $1 per 1,000 papers · optional AI with your own key · JSON, CSV, Excel export, API and MCP ready

What does the arXiv Paper Scraper do?

Search arXiv research papers by keyword, category and submission date and export clean metadata: title, abstract, categories, publish/update dates, PDF link, DOI, journal reference and comments. Turn on the optional AI summaries to get a plain-English TL;DR, key contributions, methods, practical applications and difficulty level for every paper, perfect for literature reviews, research newsletters and AI research agents.

The Actor uses the official arXiv API (export.arxiv.org/api) and follows arXiv's polite-use rules (one request every 3 seconds). No browser, no proxies, no login.

Why this arXiv scraper?

• Keyword search in all fields, title only or abstract only, or exact phrases with quotes
• Category filters: cs.AI, cs.CL, cs.LG, stat.ML, q-bio, physics... (OR-ed)
• Date range filter on submission date, sort by newest, last updated or relevance
• Fetch specific papers by arXiv ID or URL
• Thousands of papers per run, paginated automatically
• AI summaries (optional, your own OpenAI/OpenRouter key): TL;DR, contributions, methods, applications, field, level
• Agent and RAG friendly: clean JSON, one paper per row, stable arxivId

Use cases

• Daily/weekly paper digests for a research topic (schedule the Actor)
• Literature reviews and related-work sections
• AI research agents and MCP tools that need fresh papers
• RAG knowledge bases of abstracts and summaries
• Trend tracking of methods and fields over time
• Tech scouting for startups, VCs and R&D teams

How to use the arXiv Paper Scraper

1. Click Try for free (or Start) on this page and sign in to Apify.
2. Enter keywords (or an exact phrase), and optionally arXiv categories such as cs.AI or cs.CL.
3. Optionally set a submission date range, sort order and maximum number of papers, or list specific arXiv IDs.
4. Optionally enable AI summaries and paste your OpenAI or OpenRouter key.
5. Click Start, then download results as JSON, CSV or Excel, or fetch them via the API.

Input

Field	Description
query	Keywords (all must match), "exact phrase", or raw arXiv syntax such as ti:transformer AND abs:vision
searchIn	all (default), ti (title) or abs (abstract)
categories	Optional arXiv categories, e.g. ["cs.AI", "cs.CL"]
arxivIds	Or fetch specific papers: 1706.03762, https://arxiv.org/abs/2005.14165
dateFrom / dateTo	Submission date range (YYYY-MM-DD)
sortBy / sortOrder	submittedDate (default), lastUpdatedDate or relevance; descending or ascending
maxResults	Maximum papers (default 100)
enableAi, llmProvider, llmApiKey, llmModel	Optional AI summaries with your own key

{ "query": "retrieval augmented generation", "categories": ["cs.CL", "cs.AI"], "dateFrom": "2026-10-01", "maxResults": 200 }

Output example

One row per paper, straight from the official arXiv API:

{
  "arxivId": "2610.40454",
  "version": "v1",
  "title": "Glance: The Generator Is Its Own Reranker",
  "abstract": "Retrieval-augmented generation commonly serves separate models for reranking and answering, duplicating model residency and passage processing. We introduce Glance...",
  "primaryCategory": "cs.CL",
  "categories": ["cs.CL", "cs.IR"],
  "publishedAt": "2026-10-29T17:01:22Z",
  "updatedAt": "2026-10-29T17:01:22Z",
  "absUrl": "https://arxiv.org/abs/2610.40454v1",
  "pdfUrl": "https://arxiv.org/pdf/2610.40454v1",
  "doi": null,
  "journalRef": null,
  "comment": "12 pages, 2 figures, 6 tables.",
  "authorCount": 2,
  "ai": {
    "tldr": "One 9B model both reranks retrieved passages and answers, beating a separate cross-encoder reranker.",
    "contributions": ["Decision adapter + pointer head for reranking from a shared prefill", "~0.1 GiB extra memory per skill", "Better nDCG@10 than bge-reranker-v2-m3"],
    "methods": ["adapter fine-tuning", "MS MARCO", "SciFact", "HotpotQA"],
    "applications": ["cheaper RAG serving", "single-model search assistants"],
    "field": "Retrieval-augmented generation",
    "difficulty": "expert"
  },
  "scrapedAt": "2026-10-31T20:04:24.565Z"
}

The ai object only appears when AI summaries are enabled with a key (values shown are illustrative).

How much does it cost?

This Actor uses pay per event pricing: you pay only for results saved to your dataset, with no monthly fee. Apify platform compute is included.

Event	Price
Paper scraped	$0.001 per paper ($1 per 1,000)
AI paper summary	$0.003 per paper (plus your own LLM tokens)

Examples

• 1,000 papers: $1
• 10,000 papers: $10
• 1,000 papers with AI: $4 plus your LLM tokens (typically cents with gpt-4o-mini)

Apify's Free plan includes $5 of monthly usage, enough for about 5,000 papers without AI. Set Maximum cost per run in the run options to cap spend; the Actor stops cleanly at the limit.

Use it via API, Python, n8n, Make or AI agents

Run the Actor and get results in one call (replace <APIFY_TOKEN> with your token):

curl -X POST "https://api.apify.com/v2/acts/data-drip~arxiv-papers-ai/run-sync-get-dataset-items?token=<APIFY_TOKEN>" \
  -H 'Content-Type: application/json' \
  -d '{"query": "retrieval augmented generation", "categories": ["cs.CL", "cs.AI"], "dateFrom": "2026-10-01", "maxResults": 200}'

from apify_client import ApifyClient

client = ApifyClient('<APIFY_TOKEN>')
run = client.actor('data-drip/arxiv-papers-ai').call(run_input={"query": "retrieval augmented generation", "categories": ["cs.CL", "cs.AI"], "dateFrom": "2026-10-01", "maxResults": 200})
for item in client.dataset(run['defaultDatasetId']).iterate_items():
    print(item)

• Schedules: run it hourly, daily or weekly from Apify Schedules.
• Integrations: send results to Google Sheets, Slack, Zapier, Make, n8n or a webhook.
• AI agents (MCP): add data-drip/arxiv-papers-ai as a tool in the Apify MCP server (https://mcp.apify.com) for Claude, Cursor and other MCP clients.

Limitations

• Returns abstracts and PDF links, not parsed full text.
• Author names are replaced by an author count (no personal data).
• arXiv asks for one request every 3 seconds, so very large runs (10,000+ papers) take a while.

FAQ

Is it allowed? Yes. The Actor uses arXiv's official public API, which is meant for programmatic access, and keeps to arXiv's rate guidance of one request every 3 seconds. arXiv metadata is openly licensed. Thank you to arXiv for use of its open access interoperability.

Why are author names not included? We keep the output free of personal data, so you get authorCount instead of names. The absUrl links to the full paper page with all bylines.

Can I get the full text? The Actor returns the abstract and a direct pdfUrl. Full-text PDF parsing is not included.

How fresh is the data? Queries hit arXiv live, so new submissions appear as soon as arXiv announces them. Schedule a daily run with dateFrom for a digest.

Do I need an AI key? No. Without a key the Actor is a fast metadata scraper and you pay only for papers. To enable AI, paste your own OpenAI or OpenRouter key into LLM API key (stored as an encrypted secret). LLM tokens are billed by your provider, not by us.

Can I schedule it? Yes. Use Apify Schedules (for example daily) and connect the dataset to Google Sheets, Slack or a webhook to get only fresh papers.

How do I get help or request a feature? Open an issue on the Actor's Issues tab.

---

Related searches: arxiv scraper, arxiv api, research paper search, academic papers scraper, paper summaries, literature review, ai research digest, rag dataset, scientific papers, machine learning papers.