arXiv Papers Scraper - search the API or bulk-harvest whole categories

Get arXiv paper metadata as clean JSON: title, full abstract, authors, all categories and the primary one, submission and update dates, DOI, journal reference, author comments and direct PDF links. Built on arXiv's official API and its OAI-PMH bulk interface, with no key and no HTML scraping. US$1.00 per 1,000 papers.

Two modes

Search - arXiv API queries. Combine free text, field searches (ti:, au:, abs:, cat:, all:), category filters, a submission date range and a sort order. Up to 30,000 results per query (an arXiv API limit; split by dates for more). You can also fetch specific papers by arXiv ID.

Harvest - OAI-PMH bulk download of one or more categories for a date range. Built for recurring exports ("every new cs.LG paper since yesterday") and large pulls; ~1,300 records per request.

The Actor follows arXiv's etiquette of at most one request every 3 seconds, whatever your settings.

Input examples

Newest 200 papers in two categories:

{ "mode": "search", "categories": ["cs.AI", "cs.CL"], "sortBy": "submittedDate", "maxItems": 200 }

Keyword search inside a category and period:

{ "mode": "search", "searchQueries": ["retrieval augmented generation"], "categories": ["cs.CL"], "dateFrom": "2025-01-01", "dateTo": "2025-12-31", "maxItems": 1000 }

Author and title searches:

{ "mode": "search", "searchQueries": ["au:lecun AND cat:cs.LG", "ti:\"attention is all you need\""], "categories": [] }

Specific papers:

{ "mode": "search", "categories": [], "ids": ["1706.03762", "hep-th/9901001"] }

Daily export of all new machine learning papers:

{ "mode": "harvest", "categories": ["cs.LG", "stat.ML"], "dateFrom": "2026-10-12", "dateTo": "2026-10-12", "maxItems": 5000 }

Output

{
  "id": "2610.12823",
  "version": 1,
  "title": "When Charts Deceive Web Agents: A Paired Benchmark for Misleading Visualization-Grounded Tasks",
  "abstract": "Web agents increasingly browse, interpret, and act on web content ...",
  "authors": ["Yisheng Li", "Dongming Han", "Wei Tong"],
  "categories": ["cs.CL"],
  "primaryCategory": "cs.CL",
  "published": "2026-10-09T00:03:13.000Z",
  "updated": "2026-10-09T00:03:13.000Z",
  "doi": null,
  "journalRef": null,
  "comment": "Accepted to EMNLP 2026. 19 pages",
  "pdfUrl": "https://arxiv.org/pdf/2610.12823v1",
  "url": "https://arxiv.org/abs/2610.12823",
  "source": "harvest:cs.CL",
  "scrapedAt": "2026-10-13T17:16:25.192Z"
}

Authors are the public authorship listed on each paper (bibliographic metadata). The Actor does not download or parse PDFs; pdfUrl points to the file.

Pricing

Event	Price
Paper saved	US$1.00 per 1,000 (US$0.001 each)

Use Max papers in total to cap the bill.

FAQ

How fast is it? Search mode returns 200 papers per request, i.e. about 65 papers per second including the mandatory 3 second pause; harvest mode returns up to ~1,300 records per request.

Why harvest as well as search? The search API cannot go past 30,000 results per query and is optimised for lookups. OAI-PMH is arXiv's bulk interface, ideal for category-wide, date-based exports.

Harvest returned fewer papers than I expected. With only new papers on (default), revised older papers are skipped and the window is matched against the first submission date. Turn it off to get every paper whose metadata changed in the window.

Does it need a proxy? No. It starts with direct requests and switches to a residential proxy by itself if arXiv rate-limits the IP.

Use cases: literature monitoring, research trend analysis, building paper corpora for RAG and fine-tuning, citation and author networks, newsletters, dataset creation.

En español

Scraper de arXiv con la API oficial y OAI-PMH: busca por palabras clave, autor, título o categoría, o descarga en bloque categorías completas por fecha. Devuelve título, resumen, autores, categorías, fechas, DOI, referencia de revista y enlace al PDF. Respeta el límite de 1 petición cada 3 segundos. Precio: US$1.00 por cada 1.000 papers.