arXiv Papers Scraper — Research to LLM-Ready Text for RAG

Search arXiv (or fetch specific paper IDs) and get clean, structured records — title, authors, abstract, categories, dates, DOI, PDF & abstract URLs — and optionally the full paper text as LLM-ready Markdown. Built for feeding research into ChatGPT / Claude or a RAG pipeline.

Why this one

Researchers and AI builders constantly pull papers into LLMs to summarise, compare, and ground answers. This Actor turns an arXiv search into a clean dataset in one run, using arXiv's official Atom API — no key, no scraping tricks. Turn on fetchFullText and it also returns each paper's body as Markdown from arXiv's HTML render.

What it does

• Search by keywords, categories (cs.CL, cs.AI, …), authors, or a raw arXiv query.
• Or fetch exact papers by arXiv ID (2005.11401).
• Sort by relevance / last updated / submitted.
• Optional full-text Markdown with a token count per paper.
• Export to JSON, CSV, or Markdown, or pull via the Apify API.

Input

Field	Description
searchTerms	Keywords (combined with OR).
categories	arXiv categories, e.g. cs.CL.
authors	Author filters.
searchQuery	Raw arXiv query (advanced; overrides the above).
ids	Specific arXiv IDs.
maxResults	Number of papers (≤200).
sortBy / sortOrder	Ordering.
fetchFullText	Also fetch each paper's full text as Markdown.
maxCharsPerPaper	Truncate full text (0 = no limit).

Example

{
  "searchTerms": ["retrieval augmented generation"],
  "categories": ["cs.CL"],
  "maxResults": 25,
  "sortBy": "submitted",
  "fetchFullText": false
}

Output

One record per paper:

{
  "resultType": "paper",
  "arxivId": "2005.11401",
  "title": "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks",
  "authors": ["Patrick Lewis", "Ethan Perez", "..."],
  "summary": "Large pre-trained language models ...",
  "primaryCategory": "cs.CL",
  "published": "2020-05-22T...",
  "pdfUrl": "https://arxiv.org/pdf/2005.11401"
}

Use cases

• RAG over research — chunk abstracts (or full text) into a vector DB.
• Literature review — pull a topic's latest papers and summarise with an LLM.
• Trend monitoring — schedule daily runs on a category and feed new papers to a model.
• Citation / metadata harvesting — DOIs, journal refs, authors at scale.

FAQ

Do I need an API key? No — arXiv's API is open. The Actor sends a descriptive User-Agent and paces requests per arXiv etiquette.

Can I get the full paper, not just the abstract? Yes — enable fetchFullText (from arXiv's HTML render). Some older papers may not have an HTML version.

Please use arXiv data in line with arXiv's API terms.

🤖 Use with Claude or ChatGPT (MCP)

Run this actor from Claude, ChatGPT, Cursor or any MCP client via the Apify MCP server. In Claude Desktop: Settings → Connectors → Add custom connector → https://mcp.apify.com. Or expose just this tool:

{ "mcpServers": { "apify": { "url": "https://mcp.apify.com?tools=get_anything/arxiv-papers-scraper" } } }

Full guide: Connect Apify actors to Claude & ChatGPT.

⭐ Found this useful?

If this Actor saved you time, please leave a rating on its Apify page — reviews genuinely help others discover it and help me keep improving it.