arXiv Scraper

Research papers from arXiv — by category, search term, author or link — with abstracts, authors, DOIs and PDF links, as clean JSON.

arXiv is where physics, mathematics, computer science, AI, quantitative biology, finance, statistics and economics research appears first, often months before journal publication. This Actor turns arXiv into a dataset: paste a category listing, a search, an author page or a paper link, or type your own search terms, and get every matching paper with its title, abstract, authors and their affiliations, primary and cross-listed categories, submission and update dates, version, DOI, journal reference, author comments, and abstract and PDF links. Filter by submission date and sort by relevance or date. There is no login, no API key, and no proxy to configure. Built for research monitoring, literature reviews, AI and ML trend tracking, academic talent scouting, and feeding papers into search, RAG and summarisation pipelines.

⚡ Quick start

{
  "startUrls": [{ "url": "https://arxiv.org/list/cs.AI/new" }],
  "maxItems": 5
}

Or search by keyword within a category and date range:

{
  "searchTerms": ["retrieval augmented generation"],
  "category": "cs.CL",
  "dateFrom": "2026-01-01",
  "sortBy": "submittedDate",
  "maxItems": 50
}

🧩 Input

Field	Type	Required	Default	Description
startUrls	array	no	—	arXiv URLs: category listings (arxiv.org/list/cs.AI/new, /recent, or a month such as /list/cs.CL/2024-02), papers (arxiv.org/abs/<id> or PDF links), author pages (arxiv.org/a/<author-id>), or arXiv search results pages.
maxItems	integer	no	—	Maximum number of papers per start URL or search term. The input form starts at 5; leave it empty to get every match (arXiv caps one search at 30,000 papers).
searchTerms	array	no	—	Search terms, one search per line. Every word must match; quotes keep a phrase together. arXiv query syntax works too, e.g. ti:transformer AND au:vaswani.
category	string	no	—	Limit the search terms to an arXiv category such as cs.CL or astro-ph.GA, or a whole archive such as math. With no search terms, returns that category's papers.
sortBy	string	no	"relevance"	Sort search-term results by relevance, submittedDate or lastUpdatedDate.
sortOrder	string	no	"descending"	descending (newest or best first) or ascending.
dateFrom	string	no	—	Only search-term or category results submitted on or after this date (YYYY-MM-DD).
dateTo	string	no	—	Only search-term or category results submitted on or before this date (YYYY-MM-DD).
proxyConfiguration	object	no	{ "useApifyProxy": false }	Proxy settings. arXiv needs no proxy.

Add at least one start URL, one search term, or a category. Category listings return the newest submissions first. sortBy, sortOrder, dateFrom and dateTo apply to the search terms and category; start URLs keep their own order.

Finding an author page. Open any paper by the author on arXiv, click their name, and look for a link to arxiv.org/a/<author-id> on their page. Only authors who have claimed their papers on arXiv have one; for anyone else, search with au:<surname> instead.

📤 Output

Each paper is one dataset row.

Fields

Field	Type	Description
id	string	arXiv ID without the version, such as 2610.10726 or hep-th/9711200.
version	number	Version number of the returned paper (3 for v3).
url	string	The paper's abstract page, always pointing to the latest version.
title	string	Paper title.
summary	string	The abstract. LaTeX math is kept as the authors wrote it.
authors	array	Authors in order, each with name and affiliation (null when arXiv has none).
primaryCategory	string	Primary arXiv category, such as cs.CL.
categories	array	Every category the paper is listed in, including cross-lists.
publishedAt	string	When version 1 was submitted (ISO 8601).
updatedAt	string	When the returned version was submitted (ISO 8601).
doi	string	DOI of the published version, when the authors added one.
journalRef	string	Journal reference, when the authors added one.
comment	string	The authors' comment: page counts, figures, conference or acceptance notes.
absUrl	string	Abstract page of this exact version.
pdfUrl	string	PDF of this exact version.
searchQuery	string	The arXiv query that found the paper; null for papers added by link or from an author page.

💡 Use cases

• Research monitoring — a daily feed of new papers in the categories, topics or labs you follow.
• Literature reviews — pull every paper matching a query and date range into a spreadsheet or reference manager.
• AI and ML trend tracking — count submissions by topic and category over time.
• Talent and partner scouting — find active authors and their institutions in a field.
• RAG and summarisation — feed abstracts and PDF links into search, embedding or LLM pipelines.

❓ FAQ

• Do I need an API key or an arXiv account? No. The Actor reads arXiv's public listings and paper records.
• Do I need a proxy? No. It runs without one by default.
• Why is it not faster? arXiv asks automated tools to wait three seconds between requests, and the Actor keeps to that. Each request returns up to 500 papers, so large searches still finish in minutes.
• Does it download the PDFs? No. Every row carries pdfUrl, so you can fetch the PDFs you need.
• Why are doi, journalRef or affiliations empty? The authors add these themselves and many do not. Fresh preprints rarely have a DOI or journal reference yet.
• Can I search one author? Yes. Use the search term au:<surname> (for example au:hinton), or the author's arXiv page if they have one.
• How do I get only the papers from one day or month? Set dateFrom and dateTo, or use a monthly listing such as https://arxiv.org/list/cs.CL/2024-02.
• Why did a search return fewer papers than expected? Every word in a search term must match. Use OR between alternatives, or arXiv query syntax for finer control. The log shows arXiv's message when it rejects a query.

🔗 More scrapers by Anyx

• Google Search Scraper
• Web Content Scraper
• Substack Scraper
• Reddit Scraper

🤝 Anyx Solutions

We build custom scrapers and data-extraction pipelines.

• Email: tantosthor@gmail.com

🖼 Image credit

Image credit: arxiv.org