arXiv Scraper: Preprints, Authors & Categories

What it does

Scrape arXiv: title, authors, abstract, categories, DOI, journal reference, and PDF links. Search by query, category, author or date. The primary source for AI/ML preprints.

Give it searchQueries and it returns one structured record per paper, 13 fields, ready to join on DOI or feed straight into a vector store. arXiv's own query syntax is passed through untouched, so anything you can express on arxiv.org's advanced search works here: cat:cs.LG AND all:transformer, au:Hinton AND abs:diffusion, ti:"retrieval augmented".

Multiple queries run in one pass, results are de-duplicated by arXiv ID across all of them, and pagination is handled for you.

Input

Field	Type	Default	Notes
searchQueries	array	[]	arXiv query syntax. Prefixes: all:, ti:, abs:, au:, cat:, jr: (journal ref), co: (comment). Combine with AND / OR / ANDNOT, group with parentheses. Each query is run and paginated separately.
categories	array	[]	arXiv categories, cs.LG, cs.CL, stat.ML, q-bio.NC… ORed together, then ANDed onto every query.
fromDate	string	-	Submission-date lower bound, YYYY-MM-DD. Applied as arXiv's submittedDate:[… TO …] range.
sortBy	string	submittedDate	One of submittedDate, lastUpdatedDate, relevance. Always descending.
maxResultsPerQuery	integer	100	Cap per query. Fetching stops as soon as the cap is met. 0 = no limit.

At least one of searchQueries or categories is required. They combine sensibly:

• Query only → searches all of arXiv.
• Query + categories → the query, restricted to those categories.
• Categories only → the categories become the search, which is the cheapest way to pull a whole subject feed.
• fromDate + sortBy: submittedDate is the combination you want for a scheduled "what's new" run.

{
 "searchQueries": ["all:\"large language model\" AND ti:agent"],
 "categories": ["cs.LG", "cs.CL"],
 "fromDate": "2026-01-01",
 "sortBy": "submittedDate",
 "maxResultsPerQuery": 25
}

Output

One dataset item per paper. Real record from the run above:

{
 "arxivId": "2608.04828v1",
 "title": "Skill-Use: Can LLMs Actually Use Skills in Agentic Harnesses?",
 "abstract": "Large language model (LLM) agents increasingly rely on skills, structured documents that specify when to act, which procedure to follow, and which tools are allowed. …",
 "authors": ["Jinyi Han", "Yuanjian Xu", "Ying Liao", "Xinyi Wang", "Zishang Jiang", "Zixiang Di", "Fanyang Lu", "Zhichao Hu", "Yanghua Xiao"],
 "primaryCategory": "cs.CL",
 "categories": ["cs.CL"],
 "published": "2026-08-05T13:29:16Z",
 "updated": "2026-08-05T13:29:16Z",
 "doi": null,
 "journalRef": null,
 "comment": null,
 "pdfUrl": "https://arxiv.org/pdf/2608.04828v1",
 "absUrl": "https://arxiv.org/abs/2608.04828v1",
 "scrapedAt": "2026-08-06T11:28:08.231Z"
}

Field	Meaning
arxivId	arXiv identifier including the version suffix, 2608.04828v1, or an old-style cond-mat/0102536v1
title	Paper title, line-wrapping removed
abstract	Full abstract as one paragraph, XML entities decoded
authors	Author names in submission order
primaryCategory	The single category the authors filed it under
categories	Every category it is cross-listed in, primary included
published / updated	ISO-8601 timestamps for v1 submission and the latest revision
doi	Publisher DOI once the paper is formally published, null while it is preprint-only
journalRef	Free-text journal citation, e.g. J. Chem. Phys. 115, 1626 (2001)
comment	Author's note, page count, figures, conference acceptance
pdfUrl / absUrl	Direct PDF link and the abstract landing page
scrapedAt	Run timestamp

doi, journalRef and comment are null for most fresh preprints and populated for published work, in a 13-paper condensed-matter run, all 13 had a DOI and 11 had a journal reference.

A RUN_SUMMARY record lands in the key-value store:

{
 "queriesRequested": 2,
 "queriesFailed": 1,
 "failures": [
 { "query": "all:not_a_real_term_zzzq AND badsyntax:(", "error": "arXiv rejected the query (400), check the search syntax" }
 ],
 "papersSaved": 13,
 "filters": {
 "searchQueries": ["cat:cond-mat.str-el AND ti:cusp", "all:not_a_real_term_zzzq AND badsyntax:("],
 "categories": [],
 "fromDate": null,
 "sortBy": "relevance",
 "maxResultsPerQuery": 210
 },
 "finishedAt": "2026-08-06T11:31:44.902Z"
}

Use cases

1. Track AI research output daily. Schedule this every morning with a one-day floor; diff on arxivId to see only what landed overnight.

{
 "searchQueries": ["cat:cs.LG OR cat:cs.CL OR cat:cs.AI"],
 "fromDate": "2026-08-05",
 "sortBy": "submittedDate",
 "maxResultsPerQuery": 500
}

2. Build a paper-recommendation feed. Pull abstracts for a topic, embed them, and rank against a user profile. abstract + categories + absUrl is everything a RAG index needs.

{
 "searchQueries": [
 "all:\"retrieval augmented generation\"",
 "all:\"vector database\" AND abs:embedding",
 "ti:\"mixture of experts\""
 ],
 "categories": ["cs.CL", "cs.IR"],
 "maxResultsPerQuery": 300
}

3. Monitor a research group's publications. One query per author, deduplicated automatically, so co-authored papers appear once.

{
 "searchQueries": ["au:\"Yoshua Bengio\"", "au:\"Yann LeCun\"", "au:\"Geoffrey Hinton\""],
 "sortBy": "submittedDate",
 "maxResultsPerQuery": 50
}

Limits and behaviour

• Atom XML, not JSON. arXiv answers application/atom+xml. The Actor reads it with a small purpose-built extractor, CDATA sections are preserved verbatim and entities are decoded in a single pass, so < correctly becomes the text < rather than a stray tag.
• 1 request per 3 seconds. arXiv asks for this and blocks clients that ignore it. The Actor spaces every request out accordingly, so a 1,000-paper run takes roughly 15 seconds of waiting on top of transfer time. Fetching 200 papers per request keeps that overhead low.
• Pagination is automatic, capped by maxResultsPerQuery, and stops early when arXiv's totalResults is exhausted.
• Deep paging is arXiv's weak spot. Very large offsets get slow and occasionally flaky. For more than ~30,000 results, slice by fromDate into date windows instead of raising the cap.
• A failing query never aborts the run. Bad syntax returns HTTP 400, is logged, and is recorded in RUN_SUMMARY.failures; other queries continue. The run only errors out if every query fails.
• Transient errors are retried. 429 and 5xx get three attempts with linear backoff. 400s and malformed feeds fail fast, because retrying them cannot help.
• De-duplication is global. A paper matched by several queries is saved once, so you are charged once.
• Public data only. No authentication, no personal data, no access-control bypass.

FAQ

Do I need a proxy? No. Proxy configuration is not required to run this Actor.

Do I need an account on arXiv? No. You supply no credentials.

What happens if a source is unavailable? It is reported in RUN_SUMMARY.failures and the run continues with the remaining queries.

Can I schedule it? Yes, it is designed for scheduled runs. Pair fromDate with sortBy: submittedDate and diff on arxivId.

Does it fetch full text? No, metadata plus the abstract. pdfUrl gives you the direct link if you need the PDF.

Why is doi null? Because the preprint has not been formally published yet, or the authors never added the DOI. arXiv only reports what the submitter provides.

Can I search by author? Yes: au:"Yoshua Bengio". Quote multi-word names.

How do I find category codes? They are on arxiv.org's category taxonomy page, cs.LG (machine learning), cs.CL (computation and language), stat.ML, q-bio.NC, math.PR, and so on.

Can I integrate it with something else? Yes, Apify API, client libraries, webhooks, scheduled runs, dataset exports (JSON/CSV/Excel) or MCP. Output is structured JSON.