arXiv Scraper

Extract research papers from arXiv via the official Atom API. 2M+ preprints: title, abstract, authors, categories, DOI, comments, journal refs. Full arXiv query syntax. No API key, no browser, no proxy.

What does arXiv Scraper do?

Scrape arXiv, the world's largest preprint repository for physics, mathematics, computer science, quantitative biology, and economics, in two modes:

• Search by query — full arXiv query syntax: field prefixes (ti:, au:, abs:, cat:, co:, jr:, rn:), Boolean operators (AND, OR, ANDNOT), and category filters. Sort by submission date, last-updated date, or relevance.
• Fetch by arXiv ID — pass exact IDs (2401.12345, 1706.03762v5), get full records in batches of up to 100.

Every record includes arXiv ID with version, title, abstract, authors with affiliations, primary category, all cross-listed categories, publication and update timestamps, author comments, journal reference, DOI, and direct links to the abstract page and PDF.

arXiv exposes all of this through its free, public, unauthenticated Atom API at export.arxiv.org/api/query. No key, no login, no registration.

Output fields

Field	Description
arxivId	arXiv identifier without version
arxivVersion	Version suffix (v1, v2, …)
arxivIdFull	Full identifier with version
title	Paper title
abstract	Full abstract text
abstractLength	Character count
authors	Array of { name, affiliation }
authorCount	Number of authors
firstAuthor / lastAuthor	Author summary
primaryCategory	Primary arXiv category (e.g. cs.LG)
categories	All cross-listed categories
publishedDate	First submission date
updatedDate	Last revision date
comment	Author comment (e.g. "10 pages, 5 figures")
journalRef	Journal reference if published
doi	DOI if assigned
doiUrl	DOI resolver link
abstractUrl	arXiv abstract page
pdfUrl	Direct PDF download

Who is it for?

• AI/ML researchers building training corpora from preprint literature
• Quantitative finance analysts tracking new methods in q-fin and stat.ML
• Academic librarians enriching institutional repositories
• R&D teams monitoring competitor publication activity
• Systematic review teams sourcing preprints before journal publication
• Science journalists tracking emerging research trends by category

Pricing

$0.90 per 1,000 results. No subscription.

Results	Cost
100	$0.09
1,000	$0.90
10,000	$9.00

How to use it

1. Pick a Mode.
2. For search: enter a Search Query and optionally pick a Search Field and Categories.
3. For fetch: enter arXiv IDs.
4. Set Sort By and Sort Order.
5. Set Max Items (default 2000).
6. Click Start.

Output example

{
  "recordType": "paper",
  "arxivId": "1706.03762",
  "arxivVersion": "v7",
  "arxivIdFull": "1706.03762v7",
  "title": "Attention Is All You Need",
  "abstract": "The dominant sequence transduction models are based on complex recurrent or convolutional neural networks...",
  "abstractLength": 1224,
  "authors": [
    { "name": "Ashish Vaswani", "affiliation": "Google Brain" },
    { "name": "Noam Shazeer", "affiliation": "Google Brain" },
    { "name": "Niki Parmar", "affiliation": "Google Research" }
  ],
  "authorCount": 8,
  "firstAuthor": "Ashish Vaswani",
  "lastAuthor": "Illia Polosukhin",
  "primaryCategory": "cs.CL",
  "categories": ["cs.CL", "cs.LG", "cs.AI"],
  "publishedDate": "2017-06-12T17:57:34Z",
  "updatedDate": "2023-08-02T00:41:18Z",
  "comment": "15 pages, 5 figures",
  "journalRef": "Advances in Neural Information Processing Systems 30 (NIPS 2017)",
  "doi": "10.48550/arXiv.1706.03762",
  "doiUrl": "https://doi.org/10.48550/arXiv.1706.03762",
  "abstractUrl": "https://arxiv.org/abs/1706.03762v7",
  "pdfUrl": "https://arxiv.org/pdf/1706.03762v7",
  "scrapedAt": "2026-09-25T12:00:00.000Z"
}

Technical details

• Official arXiv Atom API — http://export.arxiv.org/api/query. Free, public, unauthenticated.
• Rate limit: ~1 request every 3 seconds on the legacy API. The actor defaults to 3000ms between requests. Each request returns up to 2,000 papers, so the limit rarely bottlenecks.
• Query syntax — field prefixes (ti:, au:, abs:, cat:, co:, jr:, rn:), Boolean operators, and quoted phrases. Plain-text queries are prefixed with the selected Search Field.
• XML parsing via cheerio in xmlMode, including the arxiv: namespace for comment, journal ref, and DOI fields.
• Batch fetch — up to 100 arXiv IDs per request via id_list.
• No browser, no proxy, no API key — pure HTTP.

Known limits

• Rate limit is 1 request every ~3 seconds. The actor respects this. Since each request returns up to 2,000 papers, most runs need only 1–2 requests.
• Full text is not included. arXiv provides metadata and abstracts through the API. Full PDFs are linked via pdfUrl — download separately.
• Abstracts are normalized. arXiv flattens multi-line abstracts into a single string. LaTeX markup is preserved as-is.
• Author affiliations are often missing. Many submissions don't include per-author affiliation in the Atom feed. Where present, they appear as semicolon-joined strings.
• Versioning is per-entry. arxivId is the base ID, arxivVersion is the version suffix. Revisions create new versions, not new papers.

FAQ

Do I need an API key? No. The arXiv API is fully public and unauthenticated.

Do I need a proxy? No. Datacenter IPs work at the documented rate limit.

What's the difference between search and fetch mode? Search queries the full corpus by keyword or structured query. Fetch retrieves specific papers by ID.

How do I search a specific category? Use the Categories input (e.g. cs.LG), or add cat:cs.LG to the query.

How do I search by author? Set Search Field to Author, or write au:del_maestro in the query directly.

How do I export data? After a run, go to Storage → Export as JSON, CSV, Excel.

Support

Open an issue on the Actor's page for bugs or feature requests.