arXiv Papers Scraper — search research papers by topic, author, category & date

Structured metadata for any slice of arXiv (2.5M+ preprints in CS, math, physics, statistics, quantitative finance, biology, economics): full-text keyword search with field prefixes, author lookup, the complete category taxonomy (~155 codes), submission-date windows, server-side sorting and fetching by id. Every item carries title, authors, affiliations, abstract, categories, DOI, journal reference, dates and PDF/HTML links. Built for research agents, tech-scouting teams, patent analysts and newsletter authors.

No API key, no proxy, no browser — the public arXiv API over plain HTTP. Two modes: search (the whole archive via the arXiv API) and latest (today's announced papers per category via arXiv's RSS feed — fast and not rate-limited, ideal for daily digests).

Use cases

• Daily digest of new papers — mode: "latest", categories: ["cs.AI","cs.LG"], announceTypes: ["new"] → exactly what arXiv announced today, in seconds, on a daily schedule.
• Tech scouting / weekly digest — categories: ["cs.AI","cs.MA"], submittedFrom: <last Monday>, sortBy: submittedDate → every new agent paper of the week, straight into a newsletter or Slack.
• Competitor & lab monitoring — authors: ["Vaswani"] or a query with au: per researcher; run daily with sortBy: lastUpdatedDate to catch new versions too.
• Prior-art search for patents / due diligence — searchIn: "title", matchMode: "phrase", submittedTo: <priority date> returns the exact-phrase hits older than the filing.
• RAG / knowledge-base ingestion — paginate with limit: 2000 + startOffset and keep fields: ["id","title","abstract","pdfUrl"] for a lean corpus.
• Trend analysis — count papers per month for a topic (submittedFrom/submittedTo windows) and by primaryCategory.
• Citation enrichment — feed a list of arXiv ids from a bibliography via idList to get DOIs, versions and journal references back.

Input

All fields are optional, but in search mode at least one of query, authors, category/categories, a date window or idList is required; latest mode needs categories. All conditions are ANDed.

Field	Type	Default	Allowed values / notes
mode	string	search	search — whole archive via the arXiv API. latest — the current daily announcement for categories from the RSS feed (rss.arxiv.org/rss/cs.AI+cs.LG); query, searchIn, matchMode, authors, excludeCategories are applied locally; idList, dates, sorting and startOffset are ignored.
announceTypes	string[]	[]	latest mode only: new, cross (cross-listed new papers), replace, replace-cross (new versions). Empty = all.
query	string	–	Keywords (whole-word match, no stemming). In latest mode plain keywords only. Raw arXiv syntax (search mode) is passed through: prefixes ti: abs: au: co: jr: cat: rn: all:, operators AND OR ANDNOT, parentheses, quotes.
searchIn	string	all	all, title, abstract, author, comment, journalRef, reportNumber — field for plain-keyword queries (ignored for raw syntax).
matchMode	string	all	all (every word), any (OR), phrase (exact phrase).
authors	string[]	[]	Author names, all must match (au:). Best recall: Surname or Surname_I.
category	string	–	One category code (legacy field, same as one entry in categories).
categories	string[]	[]	Paper matches if listed in ANY of them (cross-lists count). Case-insensitive; archive names cs, math, q-fin… expand to all sub-categories. See Reference.
excludeCategories	string[]	[]	ANDNOT cat: filter. Needs at least one positive condition.
submittedFrom / submittedTo	string	–	Original submission date (v1), inclusive. YYYY-MM-DD, YYYY-MM, YYYY or ISO datetime (GMT).
idList	string[]	[]	2301.00001, 2301.00001v2, arXiv:…, https://arxiv.org/abs/…, hep-th/9901001. Combined with a query = intersection.
sortBy	string	relevance	relevance, submittedDate, lastUpdatedDate.
sortOrder	string	descending	descending, ascending.
limit (alias maxItems)	integer	25	1–2000. Search: pages of 100 with a 3 s pause between them (arXiv rule). Latest: a busy category announces 200–600 papers a day — raise it for a full digest.
startOffset	integer	0	Skip the first N results — manual paging across runs (max 30 000).
includeAbstract	boolean	true	false = compact metadata only.
maxAbstractChars	integer	0	Truncate abstracts (0 = full).
fields	string[]	[]	Keep only these output fields.

Reference

Query syntax cheatsheet

Prefix	Field	Example
ti:	title	ti:"large language model" AND ti:agents
abs:	abstract	abs:tokenization ANDNOT abs:BPE
au:	author	au:Hinton_G
cat:	category	cat:q-fin.TR OR cat:q-fin.PM
co: / jr: / rn:	comment / journal ref / report number	jr:Nature
submittedDate:[A TO B]	added automatically from submittedFrom/submittedTo	submittedDate:[202601010000 TO 202601312359]

Categories

Top-level archives (accepted as shorthand for all their sub-categories): cs, econ, eess, math, astro-ph, cond-mat, gr-qc, hep-ex, hep-lat, hep-ph, hep-th, math-ph, nlin, nucl-ex, nucl-th, physics, quant-ph, q-bio, q-fin, stat. Official taxonomy: https://arxiv.org/category_taxonomy

cs (40)

Code	Name
cs.AI	Artificial Intelligence
cs.AR	Hardware Architecture
cs.CC	Computational Complexity
cs.CE	Computational Engineering, Finance, and Science
cs.CG	Computational Geometry
cs.CL	Computation and Language
cs.CR	Cryptography and Security
cs.CV	Computer Vision and Pattern Recognition
cs.CY	Computers and Society
cs.DB	Databases
cs.DC	Distributed, Parallel, and Cluster Computing
cs.DL	Digital Libraries
cs.DM	Discrete Mathematics
cs.DS	Data Structures and Algorithms
cs.ET	Emerging Technologies
cs.FL	Formal Languages and Automata Theory
cs.GL	General Literature
cs.GR	Graphics
cs.GT	Computer Science and Game Theory
cs.HC	Human-Computer Interaction
cs.IR	Information Retrieval
cs.IT	Information Theory
cs.LG	Machine Learning
cs.LO	Logic in Computer Science
cs.MA	Multiagent Systems
cs.MM	Multimedia
cs.MS	Mathematical Software
cs.NA	Numerical Analysis
cs.NE	Neural and Evolutionary Computing
cs.NI	Networking and Internet Architecture
cs.OH	Other Computer Science
cs.OS	Operating Systems
cs.PF	Performance
cs.PL	Programming Languages
cs.RO	Robotics
cs.SC	Symbolic Computation
cs.SD	Sound
cs.SE	Software Engineering
cs.SI	Social and Information Networks
cs.SY	Systems and Control

econ (3)

Code	Name
econ.EM	Econometrics
econ.GN	General Economics
econ.TH	Theoretical Economics

eess (4)

Code	Name
eess.AS	Audio and Speech Processing
eess.IV	Image and Video Processing
eess.SP	Signal Processing
eess.SY	Systems and Control

math (32)

Code	Name
math.AC	Commutative Algebra
math.AG	Algebraic Geometry
math.AP	Analysis of PDEs
math.AT	Algebraic Topology
math.CA	Classical Analysis and ODEs
math.CO	Combinatorics
math.CT	Category Theory
math.CV	Complex Variables
math.DG	Differential Geometry
math.DS	Dynamical Systems
math.FA	Functional Analysis
math.GM	General Mathematics
math.GN	General Topology
math.GR	Group Theory
math.GT	Geometric Topology
math.HO	History and Overview
math.IT	Information Theory
math.KT	K-Theory and Homology
math.LO	Logic
math.MG	Metric Geometry
math.MP	Mathematical Physics
math.NA	Numerical Analysis
math.NT	Number Theory
math.OA	Operator Algebras
math.OC	Optimization and Control
math.PR	Probability
math.QA	Quantum Algebra
math.RA	Rings and Algebras
math.RT	Representation Theory
math.SG	Symplectic Geometry
math.SP	Spectral Theory
math.ST	Statistics Theory

astro-ph (6)

Code	Name
astro-ph.CO	Cosmology and Nongalactic Astrophysics
astro-ph.EP	Earth and Planetary Astrophysics
astro-ph.GA	Astrophysics of Galaxies
astro-ph.HE	High Energy Astrophysical Phenomena
astro-ph.IM	Instrumentation and Methods for Astrophysics
astro-ph.SR	Solar and Stellar Astrophysics

cond-mat (9)

Code	Name
cond-mat.dis-nn	Disordered Systems and Neural Networks
cond-mat.mes-hall	Mesoscale and Nanoscale Physics
cond-mat.mtrl-sci	Materials Science
cond-mat.other	Other Condensed Matter
cond-mat.quant-gas	Quantum Gases
cond-mat.soft	Soft Condensed Matter
cond-mat.stat-mech	Statistical Mechanics
cond-mat.str-el	Strongly Correlated Electrons
cond-mat.supr-con	Superconductivity

physics (single archives) (9)

Code	Name
gr-qc	General Relativity and Quantum Cosmology
hep-ex	High Energy Physics - Experiment
hep-lat	High Energy Physics - Lattice
hep-ph	High Energy Physics - Phenomenology
hep-th	High Energy Physics - Theory
math-ph	Mathematical Physics
nucl-ex	Nuclear Experiment
nucl-th	Nuclear Theory
quant-ph	Quantum Physics

nlin (5)

Code	Name
nlin.AO	Adaptation and Self-Organizing Systems
nlin.CD	Chaotic Dynamics
nlin.CG	Cellular Automata and Lattice Gases
nlin.PS	Pattern Formation and Solitons
nlin.SI	Exactly Solvable and Integrable Systems

physics (22)

Code	Name
physics.acc-ph	Accelerator Physics
physics.ao-ph	Atmospheric and Oceanic Physics
physics.app-ph	Applied Physics
physics.atm-clus	Atomic and Molecular Clusters
physics.atom-ph	Atomic Physics
physics.bio-ph	Biological Physics
physics.chem-ph	Chemical Physics
physics.class-ph	Classical Physics
physics.comp-ph	Computational Physics
physics.data-an	Data Analysis, Statistics and Probability
physics.ed-ph	Physics Education
physics.flu-dyn	Fluid Dynamics
physics.gen-ph	General Physics
physics.geo-ph	Geophysics
physics.hist-ph	History and Philosophy of Physics
physics.ins-det	Instrumentation and Detectors
physics.med-ph	Medical Physics
physics.optics	Optics
physics.plasm-ph	Plasma Physics
physics.pop-ph	Popular Physics
physics.soc-ph	Physics and Society
physics.space-ph	Space Physics

q-bio (10)

Code	Name
q-bio.BM	Biomolecules
q-bio.CB	Cell Behavior
q-bio.GN	Genomics
q-bio.MN	Molecular Networks
q-bio.NC	Neurons and Cognition
q-bio.OT	Other Quantitative Biology
q-bio.PE	Populations and Evolution
q-bio.QM	Quantitative Methods
q-bio.SC	Subcellular Processes
q-bio.TO	Tissues and Organs

q-fin (9)

Code	Name
q-fin.CP	Computational Finance
q-fin.EC	Economics
q-fin.GN	General Finance
q-fin.MF	Mathematical Finance
q-fin.PM	Portfolio Management
q-fin.PR	Pricing of Securities
q-fin.RM	Risk Management
q-fin.ST	Statistical Finance
q-fin.TR	Trading and Market Microstructure

stat (6)

Code	Name
stat.AP	Applications
stat.CO	Computation
stat.ME	Methodology
stat.ML	Machine Learning
stat.OT	Other Statistics
stat.TH	Statistics Theory

Examples

Daily digest of new cs.AI + cs.LG papers (schedule it every weekday morning)

{ "mode": "latest", "categories": ["cs.AI", "cs.LG"], "announceTypes": ["new"], "limit": 1000, "fields": ["id", "title", "authors", "primaryCategory", "abstract", "pdfUrl"] }

Today's agent papers only (keywords filtered locally on the announcement)

{ "mode": "latest", "categories": ["cs.AI", "cs.MA", "cs.CL"], "query": "agent agents agentic", "matchMode": "any", "limit": 200 }

Weekly digest of new AI-agent papers

{ "query": "agent", "searchIn": "title", "categories": ["cs.AI", "cs.MA", "cs.CL"], "submittedFrom": "2026-09-07", "sortBy": "submittedDate", "limit": 200 }

Follow a researcher (new papers and new versions)

{ "authors": ["Vaswani_A"], "sortBy": "lastUpdatedDate", "limit": 50 }

Prior-art search before a priority date

{ "query": "speculative decoding", "searchIn": "title", "matchMode": "phrase", "submittedTo": "2023-01-31", "sortBy": "submittedDate", "sortOrder": "ascending", "limit": 100 }

Quant-finance corpus for a RAG index (lean fields)

{ "categories": ["q-fin"], "excludeCategories": ["q-fin.GN"], "submittedFrom": "2025-01-01", "limit": 2000, "fields": ["id", "title", "abstract", "published", "primaryCategory", "pdfUrl"] }

Enrich a bibliography by arXiv id

{ "idList": ["1706.03762", "arXiv:2005.14165", "https://arxiv.org/abs/2303.08774"], "includeAbstract": false }

Output

One item per paper (de-duplicated by id within a run). Example (trimmed):

{
  "id": "1706.03762",
  "idVersioned": "1706.03762v7",
  "version": 7,
  "title": "Attention Is All You Need",
  "authors": ["Ashish Vaswani", "Noam Shazeer", "Niki Parmar"],
  "affiliations": [],
  "abstract": "The dominant sequence transduction models are based on complex recurrent or convolutional neural networks…",
  "published": "2017-06-12T17:57:34Z",
  "updated": "2023-08-02T00:41:18Z",
  "primaryCategory": "cs.CL",
  "primaryCategoryName": "Computation and Language",
  "categories": ["cs.CL", "cs.LG"],
  "doi": null,
  "doiUrl": null,
  "journalRef": null,
  "comment": "15 pages, 5 figures",
  "url": "https://arxiv.org/abs/1706.03762",
  "pdfUrl": "https://arxiv.org/pdf/1706.03762v7",
  "htmlUrl": "https://arxiv.org/html/1706.03762",
  "source": "arxiv",
  "fetchedAt": "2026-09-13T08:00:00.000Z"
}

Field	Type	Meaning
id	string	Stable arXiv id without version — use as primary key.
idVersioned / version	string / int	Latest version id and number.
title, abstract	string	Whitespace-normalized. abstract is null when includeAbstract: false.
authors	string[]	In paper order.
affiliations	string[]	Only when authors supplied them (often empty).
published / updated	ISO date	v1 submission / latest version.
primaryCategory, primaryCategoryName	string	Code and human name.
categories	string[]	All categories incl. cross-lists.
doi, doiUrl, journalRef, comment	string|null	As provided by the authors.
url, pdfUrl, htmlUrl	string	Abstract page, PDF, HTML rendering (HTML exists for most papers since Dec 2023).
source, fetchedAt	string	Provenance.
retrievedVia	string	Search mode: which service returned the row — arxiv-api (normal), arxiv-search-page or openalex (fallbacks, see Limits & FAQ).

In latest mode each item also has announceType (new / cross / replace / replace-cross) and announcedDate (YYYY-MM-DD). The feed itself carries no submission timestamps, DOI or affiliations, so after picking the papers the actor looks them up in the arXiv export API by id_list (100 ids per request, 3 s apart, ≤ 90 s in total) and fills published (v1 submitted), updated (this version submitted), doi/doiUrl, affiliations and journalRef. The export API is sometimes throttled; papers it does not answer for keep published/updated = null (the log and SUMMARY.submissionDates = { filled, missing, note } say so) — announcedDate is always set (comment and journalRef are also filled when the weekend listing page is used). version/idVersioned come from the feed; for a replacement read from the listing page the version may be unknown (version: null, idVersioned = id).

Field (latest mode)	Type	Meaning
announceType	string	new first submission, cross new paper cross-listed into your categories, replace / replace-cross new version of an older paper.
announcedDate	date	The arXiv announcement (mailing) the paper appeared in.

A SUMMARY record in the key-value store holds, for search, the exact search_query sent, totalResults reported by arXiv, the number saved, source / sources (which service served the rows) and fallbackReason / fallbackWarnings (why a fallback was used and which inputs it could only approximate); for latest, the feed URL(s), source (rss or listing), announcedDate, papers in the announcement, matched and saved, and submissionDates (how many rows got published/updated from the export API).

Use it from code / agents

curl -X POST "https://api.apify.com/v2/acts/yadroo~arxiv-papers/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"query":"agentic commerce","categories":["cs.AI"],"sortBy":"submittedDate","limit":50}'

import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('yadroo/arxiv-papers').call({ authors: ['Vaswani_A'], sortBy: 'lastUpdatedDate', limit: 50 });
const { items } = await client.dataset(run.defaultDatasetId).listItems();

from apify_client import ApifyClient
client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("yadroo/arxiv-papers").call(run_input={"categories": ["q-fin.TR"], "submittedFrom": "2026-09-01", "limit": 100})
items = client.dataset(run["defaultDatasetId"]).list_items().items

MCP: add https://mcp.apify.com to Claude / Cursor / any MCP client and call the yadroo/arxiv-papers tool with the same JSON input.

Pricing

Pay per event: $0.001 per run start + $0.001 per paper. Typical runs: 25-paper lookup ≈ $0.026; 200-paper weekly digest ≈ $0.20; 2 000-paper corpus ≈ $2.

Limits & FAQ

• arXiv API throttling (search mode) → automatic fallback — the export API (export.arxiv.org) regularly runs out of capacity for minutes: requests hang or return 429 Rate exceeded. The actor keeps the 3 s spacing between pages, retries once, and then switches to arXiv's own website search (arxiv.org/search, a separate service): same papers and fields, but published/updated are day-precision dates (YYYY-MM-DD), affiliations is empty, relevance order differs slightly, and categories are matched on each result's category tags. If arXiv is down altogether, keyword/author searches without categories fall back to OpenAlex (arXiv-hosted works only; stemmed matching, no categories/versions/comments). Category-only or date-only browsing cannot use either fallback. Every row says where it came from in retrievedVia; the log and SUMMARY.fallbackReason explain the switch. The run fails only if every usable source is down.
• latest mode is not affected — it reads rss.arxiv.org (and arxiv.org/list/<cat>/new), which are separate from the API. If you want new papers in a category, use {"mode": "latest", "categories": [...]}. A category-only search (no query/authors/idList, no old date window) that hits throttling falls back to latest automatically with a warning — you then get only the current announcement, not the full history.
• Weekends & holidays — arXiv announces Sunday–Thursday evenings (US Eastern), so the RSS feed is empty on Saturdays/Sundays and holidays; latest then reads the most recent announcement from arxiv.org/list/<cat>/new (one request per category, 1 s apart), so a Sunday run returns Friday's papers. Check announcedDate to avoid processing the same announcement twice.
• Freshness — the API index lags new submissions by a few hours; new papers are announced Sun–Thu at 20:00 ET.
• Matching — whole words only, no stemming (transformer ≠ transformers). Use matchMode: "any" with both forms.
• 0 results — the run logs a hint (check category codes, author name form). Malformed raw syntax fails with arXiv's own error text.
• Depth — up to 2 000 items per run; startOffset up to 30 000. Split larger jobs by date window or category.
• Full text — PDFs are not downloaded; pdfUrl/htmlUrl are given for a follow-up step.
• Roadmap — OAI-PMH bulk mode for whole-category snapshots, Semantic Scholar citation counts.

---

Made by Yadroo. Sibling actors: openalex-works (citations & open access across all publishers), wikipedia-search, openlibrary-books, hackernews-search, rss-to-json.