# arXiv Scraper & API - Papers by Category, Keyword, Author, Date (`neverempty/arxiv-scraper`) Actor

arXiv papers from the official arXiv API as flat rows: ID, version, title, abstract, authors, categories, dates, DOI, journal ref and abs/PDF links. Search by keyword, title, author, category or date range, look up IDs in bulk, or monitor a category for new papers. PDFs are linked, never copied.

- **URL**: https://apify.com/neverempty/arxiv-scraper.md
- **Developed by:** [NeverEmpty](https://apify.com/neverempty) (community)
- **Categories:** Developer tools, AI, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.18 / 1,000 paper returneds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## arXiv Scraper & API

arXiv papers from the official arXiv API (`export.arxiv.org`) as flat rows: one row per paper. Search by
keyword, title, abstract, author, category or submission date, look up a list of arXiv IDs, or monitor a
category and receive only the papers that are new since the last run. No API key, no proxy, no scraping of
web pages.

```json
{
  "categories": ["cs.CL"],
  "submittedFrom": "2026-09-01",
  "submittedTo": "2026-09-05",
  "maxResults": 500
}
```

**PDFs are linked, never downloaded.** Each row carries the `pdfUrl` and `absUrl` links; the paper itself is
not fetched, stored or redistributed. That is what arXiv's API terms require, and it keeps runs fast and
cheap.

### What you get

`arxivId`, `version`, `arxivIdVersioned`, `title`, `abstract`, `authors`, `authorCount`, `affiliations`,
`primaryCategory`, `categories`, `publishedAt` (first version submitted), `updatedAt` (this version),
`doi`, `journalRef`, `comment`, `absUrl`, `pdfUrl`, `doiUrl`, plus `requestedId` (for ID lookups),
`matchedQuery` (the exact query sent to arXiv), `totalMatches` (how many papers arXiv says match) and
`checkedAt`.

How full the optional columns are, counted on the 100 newest hep-ex papers and 50 newest cs.CL papers on
2026-09-12: **affiliations 1 of 100** (hep-ex) and **0 of 50** (cs.CL) - arXiv rarely has them, so do not
buy this for affiliations; **DOI 3 of 100** and 2 of 50; **journal reference 3 of 100**; **comment 81 of
100**. Missing values are empty lists or `null`, never invented.

### What this fixes about the raw API

Everything below was measured against the live API on 2026-09-12, not taken from the documentation.

#### 1. Unknown IDs silently disappear, and the order is not yours

Ask `id_list=…,hep-th/9901001,1706.03762` and `1706.03762` comes back first. Ask for `2609.99999` and arXiv
answers HTTP 200 with zero results and no error. Here every row is matched back to the ID you asked for, the
rows come back in your order, and an ID arXiv does not have gets its own `not-found` row (uncharged)
instead of vanishing.

#### 2. One old-style ID with a version breaks the whole batch

`hep-th/9901001v1` makes arXiv answer HTTP 500 - and it takes every other ID in the same request down with
it. When a batch fails, each ID is retried on its own, so only the one bad ID is reported (with a hint to
drop the version) and the rest are returned.

#### 3. A typo in a category code looks like an empty field - and six *correct* codes behave the same way

`cat:xx.YY` returns HTTP 200 and zero results, so a misspelled code arrives as "no papers" rather than as an
error. Codes are checked against arXiv's official list before anything is sent, and an unknown code is
refused with a row that says so.

The trap is that six codes arXiv documents as real are **aliases**, and searching them returns zero every
time:

| You ask for | Actually searched | Papers there |
|---|---|---|
| `cs.SY` | `eess.SY` | 48,161 |
| `cs.NA` | `math.NA` | 51,534 |
| `math.IT` | `cs.IT` | 56,004 |
| `math.MP` | `math-ph` | 92,197 |
| `q-fin.EC` | `econ.GN` | 7,532 |
| `stat.TH` | `math.ST` | 28,939 |

Each one is swapped for the real code before the query is sent, and every row says which swap was made.
`primaryCategoryOnly` compares against the real code too, so an alias does not silently discard everything.

The bare pre-2009 archive names `astro-ph` (105,380 papers), `cond-mat` (14,215) and `q-bio` (1,356) are
accepted as well: papers filed before the subcategories existed still carry them. Bare `physics`, `math`,
`cs`, `nlin`, `q-fin`, `stat`, `eess` and `econ`, and retired archives such as `cmp-lg` and `alg-geom`, all
return zero on arXiv, so they are refused rather than sold as an empty answer. Every code this Actor actually
sends was queried on its own against arXiv and returns papers; the six aliases above are never sent as
themselves.

#### 4. An OR without brackets quietly drops your date range

`cat:cs.CL OR cat:cs.LG AND submittedDate:[…]` matched **118,884** papers; with brackets it matched **243**.
Several categories or several authors are always sent in brackets.

#### 5. Unquoted words are not an AND

`all:large language model` matched **1,524,365** papers. Here the keywords `large language model` are sent as
`all:large AND all:language AND all:model` (101,171), and `"large language model"` in quotes is sent as a
phrase (76,560).

One thing arXiv does on its side: a quoted phrase is matched after arXiv drops short words such as *is* and
*not*. `"attention is all you need"` in the title returned 35 papers, and one of them was *Attention Isn't All
You Need …*. The query is sent exactly as you wrote it; this is how arXiv matches phrases.

**The words AND, OR, ANDNOT and NOT are operators.** Sent as a plain search word, arXiv drops the word and
then rejects the broken query it is left with: `all:transformer AND all:OR AND all:attention` came back
HTTP 400, `Invalid query string: 'all:transformer AND AND all:attention'` - an error with no explanation of
which word caused it. Here those four words are quoted before sending (`all:"OR"`, which returns 5,008
papers), so they are searched as words instead of failing.

**Symbols inside a word are arXiv's business, not ours.** `all:C++` matched 176,351 papers - not the same as
`all:C` (731,629), so the `+` is not simply thrown away, but it is not an exact match for the token `C++`
either. Quoting does not change it: `all:"C++"` returned the identical 176,351, and `ti:C++` and `ti:"C++"`
both returned 16,471. Searching for a symbol-heavy name, use an unambiguous word next to it (`title: cuda
kernel`) rather than trusting the punctuation.

#### 6. Only the first 10,000 results of a search can be read

`start=9995` works; `start=10000`, `20000` and `30000` all return HTTP 500, whatever the documentation says.
`maxResults` is capped at 10,000 and no request is sent past that point. To go further, split the search
by `submittedFrom` / `submittedTo`.

#### 7. arXiv is often busy

arXiv's terms ask for one request at a time, at least three seconds apart. This Actor sends one request at a
time and waits **3.5 seconds after the previous answer** - and even at that spacing arXiv has answered
HTTP 429 and then HTTP 503 for about a minute before recovering. So 429 and 503 are handled twice over: the
request is retried after 10, 20, 40 seconds (or longer if arXiv sends `Retry-After`), **and the gap used by
every later request in the run is doubled** - 3.5s, then 7s, then 14s - so a run that meets a busy arXiv
backs off instead of hammering it. An empty page from a search that reports matches is read again rather
than taken as "no more results". If a later page still cannot be read, the rows already returned are kept
and an uncharged row says the result is incomplete.

> **If you start several runs at the same time, they cannot space themselves out for you.**
> The three-second rule applies to everything you send arXiv, but each run is a separate container and
> cannot see the others - the spacing above is kept *within one run only*. Two runs at once means arXiv sees
> requests up to twice as often as the terms allow, and the likely result is 429s for you and, if it is
> sustained, a block on your IP range. **Run this Actor one run at a time** (in a schedule, give concurrent
> runs their own start times), and prefer one big run to several small parallel ones.

#### 8. The same paper has more than one name

`1706.03762` and `1706.03762v7` are the same paper, and arXiv treats them as one: asking for both in a
single `id_list` returns **one** entry. Asking for both here returns one paper row and one uncharged
`duplicate-request` row naming the identifier it duplicates - you are never charged twice for one paper.
(`1706.03762v1` is a different version and is returned separately.)

Old-style identifiers can carry a subclass, and arXiv's own API does not accept that form:
`math.GT/0309136` returns zero results while `math/0309136` returns the paper. The subclass is dropped
before the lookup, the row keeps the identifier you typed in `requestedId`, and its `note` says what was
actually looked up.

#### The queries were checked against arXiv itself

For cs.CL, 1-5 September 2026, the date-range query this Actor sends returned **459** papers. Reading the
category newest-first and counting by hand gave the same **459**, down to the IDs. cs.CL or cs.LG on one
day: **243** and **243** (3 September), **105** and **105** (5 September). Author `Percy Liang`: 275 of 275
rows list him. The phrase `"large language model"`: 2,000 of 2,000 rows contain it. Separately, every
category code was queried on its own: the 152 codes this Actor sends all return papers, and the six alias
codes return zero, which is why they are swapped before sending. These comparisons are measured against the
live API, not carried over from the docs.

### Input

| Field | What it does |
|---|---|
| `categories` | Category codes such as `cs.CL`, `math.AG`, `hep-th`. Several mean OR. Cross-listed papers match too. The six alias codes are swapped for the real one (see 3); bare `astro-ph`, `cond-mat` and `q-bio` work for pre-2009 papers. |
| `keywords` | Words anywhere in the record, all required. Quotes keep a phrase together. |
| `title` / `abstract` | Words that must appear in the title / abstract. |
| `authors` | Author names, each searched as a whole name. Several mean OR. |
| `submittedFrom` / `submittedTo` | First-version submission dates, `YYYY-MM-DD`, GMT. The end day is included. |
| `searchQuery` | A raw arXiv query, combined with the fields above by AND. |
| `arxivIds` | Look up IDs instead of searching (`2609.11917`, `2609.11917v2`, `arXiv:…`, `hep-th/9901001`, `math.GT/0309136`, abs/pdf links). Two names for one paper are charged once. Search fields are not applied to a lookup: arXiv would use them to silently drop IDs. |
| `primaryCategoryOnly` | Keep only papers whose primary category is one you listed. On 1-5 September 2026, 159 of the 459 cs.CL papers were cross-lists. |
| `sortBy` / `sortOrder` | Submission date (default), last updated, or relevance. |
| `maxResults` | Papers to return, up to 10,000. Only this many are requested from arXiv, so a small number is a fast run. In an ID lookup, IDs beyond this number are not looked up (each gets an uncharged row). |
| `monitoringMode` | Only papers not returned by an earlier run of the same search. |
| `resetMonitoringState` | Forget what monitoring has already returned for this search. |
| `maxRetries` | Attempts per request when arXiv is busy. HTTP 400 is never retried. |

If the input is left completely empty, the Actor lists the newest papers in `cs.CL` and says so in the
`note` column of every row. If you fill in anything, only what you filled in is used.

### Monitoring a category for new papers

Turn on `monitoringMode` and schedule the run (daily is typical). The first run returns the newest
`maxResults` papers and remembers them; each later run returns only papers it has not returned before, and
when nothing is new it returns a single uncharged `no-new-since-last-run` row. A paper is returned once: a
new version of it (v2, v3) is not returned again. If more new papers arrive than `maxResults`, the oldest
are returned first and the rest follow on the next run.

Two things worth knowing before you schedule it:

- **The first run sets the starting line.** It is the oldest paper that first run returned, and later runs
  never reach back past it. Raising `maxResults` afterwards therefore does *not* fetch the papers that were
  below the original cut - it only raises the ceiling for papers arriving from now on. Start with the
  `maxResults` you actually want, or use `resetMonitoringState` to start over; for a backlog, run a normal
  search with `submittedFrom` / `submittedTo` instead.
- **A paper held in moderation can be missed.** Monitoring walks back 14 days from the newest paper it has
  seen. A paper that appears on arXiv more than 14 days after its submission date sorts below that line and
  is not picked up. Papers already found but not yet delivered (because of `maxResults` or a charge limit)
  do hold the line open, so those always arrive on a later run.

### Charging

One charge per paper row delivered. Rows that only explain something - `bad-input`, `no-results`,
`not-found`, `failed`, `not-looked-up`, `ignored-input`, `duplicate-request`, `no-new-since-last-run` - are
never charged. The same paper is delivered once per run even if two of your identifiers point to it, or if
arXiv returns it on two pages of one search. If a run hits the maximum total charge you set, it stops before
reading further pages and says the result is incomplete.

### Source and terms

arXiv API (`export.arxiv.org/api/query`), used under the
[arXiv API Terms of Use](https://info.arxiv.org/help/api/tou.html): at most one request every three seconds
over a single connection; descriptive metadata (titles, abstracts, authors, identifiers, categories) is
free to use under CC0; the e-prints themselves are not stored or redistributed - rows link to arXiv.org.
This Actor is not affiliated with, endorsed by or supported by arXiv. Thank you to arXiv for use of its
open access interoperability.

# Actor input Schema

## `categories` (type: `array`):

arXiv category codes such as cs.CL, cs.LG, math.AG, hep-th or q-bio.NC (case-sensitive; full list at arxiv.org/category\_taxonomy). Several codes mean OR. A category matches cross-listed papers too; turn on 'Primary category only' to keep only papers whose main category is one of these. Six codes are aliases that arXiv answers with zero results (cs.SY, cs.NA, math.IT, math.MP, q-fin.EC, stat.TH): they are searched under the real code instead and every row says so. The bare pre-2009 archive names astro-ph, cond-mat and q-bio work too. An unknown code is refused with a row saying so, because arXiv itself answers it with zero results.

## `keywords` (type: `string`):

Words searched across title, abstract, authors and comments. Every word must appear (AND). Put a phrase in double quotes to keep it together, e.g. "large language model".

## `title` (type: `string`):

Every word must appear in the title. Quotes keep a phrase together, e.g. "attention is all you need".

## `abstract` (type: `string`):

Every word must appear in the abstract. Quotes keep a phrase together.

## `authors` (type: `array`):

Author names, e.g. Percy Liang or Yoshua Bengio. Each name is searched as a whole; several names mean papers by any of them (OR).

## `submittedFrom` (type: `string`):

Keep papers whose first version was submitted on or after this day (GMT). Can be used on its own to list everything submitted in a date range.

## `submittedTo` (type: `string`):

Keep papers whose first version was submitted on or before this day (GMT); the whole day is included.

## `searchQuery` (type: `string`):

A raw arXiv API query such as ti:transformer AND au:"Ashish Vaswani" or (cat:cs.CV OR cat:eess.IV) AND abs:diffusion. Combined with the fields above using AND. Use parentheses around OR: without them arXiv applies the OR last and a date-limited search can turn into the whole category.

## `arxivIds` (type: `array`):

Look up papers by identifier instead of searching: 2609.11917, 2609.11917v2, arXiv:1706.03762, hep-th/9901001 or an arxiv.org/abs or /pdf link. An ID without a version returns the latest version. IDs arXiv does not have come back as not-found rows (arXiv itself just leaves them out). When IDs are given, the search fields are not applied.

## `primaryCategoryOnly` (type: `boolean`):

Skip cross-lists: keep only papers whose primary category is one of the categories above. On 1-5 September 2026, 159 of the 459 papers in cs.CL were cross-lists from other categories. Skipped papers are not charged.

## `sortBy` (type: `string`):

Newest submissions first is what most people want. Relevance is arXiv's own ranking.

## `sortOrder` (type: `string`):

Descending = newest first.

## `maxResults` (type: `integer`):

How many papers to return. You are charged only for the paper rows you receive. arXiv serves at most the first 10,000 results of one search (it answers HTTP 500 beyond that), so narrow a larger search by date range.

## `monitoringMode` (type: `boolean`):

Return only papers not delivered by an earlier run of this same search. The first run returns the newest papers; later runs return what was added since. A paper is delivered once, not again when a new version appears. Papers announced more than 14 days after their submission date can be missed.

## `resetMonitoringState` (type: `boolean`):

Forget what monitoring mode has already delivered for this search, so the next monitoring run starts over.

## `maxRetries` (type: `integer`):

arXiv often answers HTTP 429 or 503 when it is busy; the Actor waits (10s, 20s, 40s ...) and tries again up to this many times in total. After a 429 or 503 it also widens the gap between every later request in the run (3.5s, then 7s, then 14s). HTTP 400 is a permanent answer and is never retried.

## Actor input object example

```json
{
  "categories": [
    "cs.CL"
  ],
  "primaryCategoryOnly": false,
  "sortBy": "submittedDate",
  "sortOrder": "descending",
  "maxResults": 100,
  "monitoringMode": false,
  "resetMonitoringState": false,
  "maxRetries": 4
}
```

# Actor output Schema

## `results` (type: `string`):

One row per arXiv paper, plus uncharged rows that explain an input that was refused, a search that matched nothing, an identifier arXiv does not have, or a request that could not be read.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "categories": [
        "cs.CL"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("neverempty/arxiv-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "categories": ["cs.CL"] }

# Run the Actor and wait for it to finish
run = client.actor("neverempty/arxiv-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "categories": [
    "cs.CL"
  ]
}' |
apify call neverempty/arxiv-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,neverempty/arxiv-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/o6jWPObyLNILGH5MO/builds/Xrdubn27bHk8cyIw1/openapi.json