arXiv Scraper & API

arXiv papers from the official arXiv API (export.arxiv.org) as flat rows: one row per paper. Search by keyword, title, abstract, author, category or submission date, look up a list of arXiv IDs, or monitor a category and receive only the papers that are new since the last run. No API key, no proxy, no scraping of web pages.

{
  "categories": ["cs.CL"],
  "submittedFrom": "2026-09-01",
  "submittedTo": "2026-09-05",
  "maxResults": 500
}

PDFs are linked, never downloaded. Each row carries the pdfUrl and absUrl links; the paper itself is not fetched, stored or redistributed. That is what arXiv's API terms require, and it keeps runs fast and cheap.

What you get

arxivId, version, arxivIdVersioned, title, abstract, authors, authorCount, affiliations, primaryCategory, categories, publishedAt (first version submitted), updatedAt (this version), doi, journalRef, comment, absUrl, pdfUrl, doiUrl, plus requestedId (for ID lookups), matchedQuery (the exact query sent to arXiv), totalMatches (how many papers arXiv says match) and checkedAt.

How full the optional columns are, counted on the 100 newest hep-ex papers and 50 newest cs.CL papers on 2026-09-12: affiliations 1 of 100 (hep-ex) and 0 of 50 (cs.CL) - arXiv rarely has them, so do not buy this for affiliations; DOI 3 of 100 and 2 of 50; journal reference 3 of 100; comment 81 of 100. Missing values are empty lists or null, never invented.

What this fixes about the raw API

Everything below was measured against the live API on 2026-09-12, not taken from the documentation.

1. Unknown IDs silently disappear, and the order is not yours

Ask id_list=…,hep-th/9901001,1706.03762 and 1706.03762 comes back first. Ask for 2609.99999 and arXiv answers HTTP 200 with zero results and no error. Here every row is matched back to the ID you asked for, the rows come back in your order, and an ID arXiv does not have gets its own not-found row (uncharged) instead of vanishing.

2. One old-style ID with a version breaks the whole batch

hep-th/9901001v1 makes arXiv answer HTTP 500 - and it takes every other ID in the same request down with it. When a batch fails, each ID is retried on its own, so only the one bad ID is reported (with a hint to drop the version) and the rest are returned.

3. A typo in a category code looks like an empty field - and six correct codes behave the same way

cat:xx.YY returns HTTP 200 and zero results, so a misspelled code arrives as "no papers" rather than as an error. Codes are checked against arXiv's official list before anything is sent, and an unknown code is refused with a row that says so.

The trap is that six codes arXiv documents as real are aliases, and searching them returns zero every time:

Each one is swapped for the real code before the query is sent, and every row says which swap was made. primaryCategoryOnly compares against the real code too, so an alias does not silently discard everything.

The bare pre-2009 archive names astro-ph (105,380 papers), cond-mat (14,215) and q-bio (1,356) are accepted as well: papers filed before the subcategories existed still carry them. Bare physics, math, cs, nlin, q-fin, stat, eess and econ, and retired archives such as cmp-lg and alg-geom, all return zero on arXiv, so they are refused rather than sold as an empty answer. Every code this Actor actually sends was queried on its own against arXiv and returns papers; the six aliases above are never sent as themselves.

4. An OR without brackets quietly drops your date range

cat:cs.CL OR cat:cs.LG AND submittedDate:[…] matched 118,884 papers; with brackets it matched 243. Several categories or several authors are always sent in brackets.

5. Unquoted words are not an AND

all:large language model matched 1,524,365 papers. Here the keywords large language model are sent as all:large AND all:language AND all:model (101,171), and "large language model" in quotes is sent as a phrase (76,560).

One thing arXiv does on its side: a quoted phrase is matched after arXiv drops short words such as is and not. "attention is all you need" in the title returned 35 papers, and one of them was Attention Isn't All You Need …. The query is sent exactly as you wrote it; this is how arXiv matches phrases.

The words AND, OR, ANDNOT and NOT are operators. Sent as a plain search word, arXiv drops the word and then rejects the broken query it is left with: all:transformer AND all:OR AND all:attention came back HTTP 400, Invalid query string: 'all:transformer AND AND all:attention' - an error with no explanation of which word caused it. Here those four words are quoted before sending (all:"OR", which returns 5,008 papers), so they are searched as words instead of failing.

Symbols inside a word are arXiv's business, not ours. all:C++ matched 176,351 papers - not the same as all:C (731,629), so the + is not simply thrown away, but it is not an exact match for the token C++ either. Quoting does not change it: all:"C++" returned the identical 176,351, and ti:C++ and ti:"C++" both returned 16,471. Searching for a symbol-heavy name, use an unambiguous word next to it (

title: cuda kernel

6. Only the first 10,000 results of a search can be read

start=9995 works; start=10000, 20000 and 30000 all return HTTP 500, whatever the documentation says. maxResults is capped at 10,000 and no request is sent past that point. To go further, split the search by submittedFrom / submittedTo.

7. arXiv is often busy

arXiv's terms ask for one request at a time, at least three seconds apart. This Actor sends one request at a time and waits 3.5 seconds after the previous answer - and even at that spacing arXiv has answered HTTP 429 and then HTTP 503 for about a minute before recovering. So 429 and 503 are handled twice over: the request is retried after 10, 20, 40 seconds (or longer if arXiv sends Retry-After), and the gap used by every later request in the run is doubled - 3.5s, then 7s, then 14s - so a run that meets a busy arXiv backs off instead of hammering it. An empty page from a search that reports matches is read again rather than taken as "no more results". If a later page still cannot be read, the rows already returned are kept and an uncharged row says the result is incomplete.

If you start several runs at the same time, they cannot space themselves out for you. The three-second rule applies to everything you send arXiv, but each run is a separate container and cannot see the others - the spacing above is kept within one run only. Two runs at once means arXiv sees requests up to twice as often as the terms allow, and the likely result is 429s for you and, if it is sustained, a block on your IP range. Run this Actor one run at a time (in a schedule, give concurrent runs their own start times), and prefer one big run to several small parallel ones.

8. The same paper has more than one name

1706.03762 and 1706.03762v7 are the same paper, and arXiv treats them as one: asking for both in a single id_list returns one entry. Asking for both here returns one paper row and one uncharged duplicate-request row naming the identifier it duplicates - you are never charged twice for one paper. (1706.03762v1 is a different version and is returned separately.)

Old-style identifiers can carry a subclass, and arXiv's own API does not accept that form: math.GT/0309136 returns zero results while math/0309136 returns the paper. The subclass is dropped before the lookup, the row keeps the identifier you typed in requestedId, and its note says what was actually looked up.

The queries were checked against arXiv itself

For cs.CL, 1-5 September 2026, the date-range query this Actor sends returned 459 papers. Reading the category newest-first and counting by hand gave the same 459, down to the IDs. cs.CL or cs.LG on one day: 243 and 243 (3 September), 105 and 105 (5 September). Author Percy Liang: 275 of 275 rows list him. The phrase "large language model": 2,000 of 2,000 rows contain it. Separately, every category code was queried on its own: the 152 codes this Actor sends all return papers, and the six alias codes return zero, which is why they are swapped before sending. These comparisons are measured against the live API, not carried over from the docs.

Input

If the input is left completely empty, the Actor lists the newest papers in cs.CL and says so in the note column of every row. If you fill in anything, only what you filled in is used.

Monitoring a category for new papers

Turn on monitoringMode and schedule the run (daily is typical). The first run returns the newest maxResults papers and remembers them; each later run returns only papers it has not returned before, and when nothing is new it returns a single uncharged no-new-since-last-run row. A paper is returned once: a new version of it (v2, v3) is not returned again. If more new papers arrive than maxResults, the oldest are returned first and the rest follow on the next run.

Two things worth knowing before you schedule it:

• The first run sets the starting line. It is the oldest paper that first run returned, and later runs never reach back past it. Raising maxResults afterwards therefore does not fetch the papers that were below the original cut - it only raises the ceiling for papers arriving from now on. Start with the maxResults you actually want, or use resetMonitoringState to start over; for a backlog, run a normal search with submittedFrom / submittedTo instead.
• A paper held in moderation can be missed. Monitoring walks back 14 days from the newest paper it has seen. A paper that appears on arXiv more than 14 days after its submission date sorts below that line and is not picked up. Papers already found but not yet delivered (because of maxResults or a charge limit) do hold the line open, so those always arrive on a later run.

Charging

One charge per paper row delivered. Rows that only explain something - bad-input, no-results, not-found, failed, not-looked-up, ignored-input, duplicate-request, no-new-since-last-run - are never charged. The same paper is delivered once per run even if two of your identifiers point to it, or if arXiv returns it on two pages of one search. If a run hits the maximum total charge you set, it stops before reading further pages and says the result is incomplete.

Source and terms

arXiv API (export.arxiv.org/api/query), used under the arXiv API Terms of Use: at most one request every three seconds over a single connection; descriptive metadata (titles, abstracts, authors, identifiers, categories) is free to use under CC0; the e-prints themselves are not stored or redistributed - rows link to arXiv.org. This Actor is not affiliated with, endorsed by or supported by arXiv. Thank you to arXiv for use of its open access interoperability.