# ArXiv Papers Scraper (`automation-lab/arxiv-paper-search-export`) Actor

Search arXiv by topic, author, category, date, or paper ID and export structured paper metadata for literature monitoring and research datasets.

- **URL**: https://apify.com/automation-lab/arxiv-paper-search-export.md
- **Developed by:** [Stas Persiianenko](https://apify.com/automation-lab) (community)
- **Categories:** Education
- **Stats:** 2 total users, 1 monthly users, 80.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.48 / 1,000 paper extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## ArXiv Papers Scraper

Search and export **arxiv papers** as structured records for literature monitoring, systematic review preparation, and research datasets.

The Actor queries the official public arXiv Atom API. Search by topic, author, category, submission date, or known paper ID, then save titles, abstracts, ordered authors, categories, dates, DOI metadata, and PDF links to an Apify dataset.

No arXiv login, cookie, API key, or proxy is required.

### What can ArXiv Papers Scraper do?

- Search titles, abstracts, and author names with a topic query.
- Filter papers by one or more arXiv categories.
- Filter by author and first-submission date.
- Retrieve known papers by arXiv ID, abstract URL, or PDF URL.
- Sort by submission date, last update, or relevance.
- Export up to 500 unique papers per run.
- Normalize Atom metadata into integration-ready JSON.
- Provide canonical abstract and PDF links.
- Preserve DOI, journal reference, and submission comments when available.
- Support repeatable scheduled literature-monitoring runs.

### Who is it for?

**Researchers** build focused reading lists without copying metadata by hand.

**Librarians and research-support teams** collect bounded topic or category exports for discovery workflows.

**Data scientists** create reproducible metadata datasets for analysis, classification, or citation-pipeline inputs.

**R\&D teams** schedule a query and compare exports to identify newly submitted work.

**Developers and AI agents** retrieve clean records through the Apify API or MCP instead of parsing Atom XML.

This Actor searches public arXiv metadata. It does not submit papers, access private accounts, or bypass arXiv login.

### Why use this Actor?

The source API is public, but production workflows still need query construction, pagination, validation, rate-limit handling, metadata normalization, deduplication, storage, and integrations.

This Actor packages those steps into one reusable task:

1. validate the requested search scope;
2. build a structured arXiv query;
3. fetch bounded Atom pages politely;
4. retry temporary failures;
5. normalize and deduplicate papers;
6. charge only for saved records;
7. write typed JSON to the default dataset.

It fails clearly rather than returning fabricated or cached research records after an upstream error.

### Input parameters

| Field | Type | Description |
|---|---|---|
| `query` | string | Topic text matched across titles, abstracts, and author names. |
| `author` | string | Author-name filter, such as `Geoffrey Hinton`. |
| `categories` | string\[] | arXiv category codes; multiple values are combined with OR. |
| `paperIds` | string\[] | Known IDs or arXiv abs/PDF URLs. |
| `dateFrom` | date | Earliest first-submission date, `YYYY-MM-DD`. |
| `dateTo` | date | Latest first-submission date, `YYYY-MM-DD`. |
| `sortBy` | enum | `submittedDate`, `lastUpdatedDate`, or `relevance`. |
| `sortOrder` | enum | `descending` or `ascending`. |
| `maxItems` | integer | Maximum unique records, from 1 to 500. |
| `maxRequestRetries` | integer | Temporary-request retries, from 0 to 5. |

Provide at least one of `query`, `author`, `categories`, or `paperIds`.

Search filters are combined with AND. Multiple categories are combined with OR.

When `paperIds` are supplied, the Actor retrieves those exact records and applies the topic, author, category, and date filters locally as well.

### Quick start

1. Open the Actor in Apify Console.
2. Enter a topic such as `graph neural networks`.
3. Optionally add `cs.LG` to **arXiv categories**.
4. Choose a maximum number of papers.
5. Click **Start**.
6. Open the **Dataset** tab to preview or download JSON, CSV, Excel, XML, or RSS.

Example input:

```json
{
  "query": "graph neural networks",
  "categories": ["cs.LG"],
  "sortBy": "submittedDate",
  "sortOrder": "descending",
  "maxItems": 25
}
```

### Search by author

Use the dedicated author field instead of embedding arXiv query syntax:

```json
{
  "author": "Geoffrey Hinton",
  "sortBy": "submittedDate",
  "sortOrder": "descending",
  "maxItems": 20
}
```

The filter is sent to the official author index. The returned record keeps the complete ordered author list.

### Search arXiv math and science categories

Category codes are source-specific identifiers such as:

- `cs.AI` — Artificial Intelligence
- `cs.CL` — Computation and Language
- `cs.LG` — Machine Learning
- `math.OC` — Optimization and Control
- `quant-ph` — Quantum Physics
- `stat.ML` — Machine Learning in Statistics

Example category export:

```json
{
  "categories": ["math.OC", "stat.ML"],
  "sortBy": "submittedDate",
  "maxItems": 100
}
```

Consult arXiv's current category taxonomy when choosing codes. Invalid category shapes are rejected before a source request.

### Monitor papers in a submission-date window

Use a bounded date range for repeatable reviews:

```json
{
  "query": "retrieval augmented generation",
  "categories": ["cs.AI"],
  "dateFrom": "2025-01-01",
  "dateTo": "2025-12-31",
  "sortBy": "submittedDate",
  "sortOrder": "descending",
  "maxItems": 100
}
```

For recurring monitoring, schedule this Actor in Apify and move the date window forward. Downstream automation can compare `paperId` and `updatedAt` with an earlier dataset.

The Actor does not itself send alerts or calculate changes between runs.

### Retrieve known paper IDs

Use IDs when you already have a reading list:

```json
{
  "paperIds": [
    "1706.03762",
    "https://arxiv.org/abs/2303.08774",
    "https://arxiv.org/pdf/2302.13971.pdf"
  ],
  "maxItems": 3
}
```

Versioned identifiers such as `1706.03762v7` are supported.

Legacy-style arXiv identifiers are also accepted.

### Output fields

| Field | Meaning |
|---|---|
| `paperId` | arXiv ID, including returned version. |
| `title` | Normalized paper title. |
| `abstract` | Normalized abstract text. |
| `authors` | Ordered array of author names. |
| `categories` | All assigned arXiv category codes. |
| `primaryCategory` | Primary category code. |
| `publishedAt` | Initial submission timestamp. |
| `updatedAt` | Latest version timestamp. |
| `pdfUrl` | Direct PDF URL. |
| `absUrl` | Canonical abstract-page URL. |
| `doi` | DOI when arXiv provides one; otherwise `null`. |
| `journalReference` | Journal reference when available. |
| `comment` | Source submission comment when available. |
| `query` | Structured query used, or `null` for ID retrieval. |
| `fetchedAt` | UTC extraction timestamp. |

### Example output

A real topic-search result has this shape:

```json
{
  "paperId": "2608.27413v1",
  "title": "Scaling Graph Neural Networks for Friend Recommendation: Multi-Hash User Embeddings and Temporal Neighbor Sampling",
  "abstract": "Friend recommendation is inherently graph-structured...",
  "authors": ["Maksim Utushkin", "Andrei Ovsiannikov", "Alexander D'yakonov"],
  "categories": ["cs.IR", "cs.LG", "cs.SI"],
  "primaryCategory": "cs.IR",
  "publishedAt": "2026-08-27T17:41:33Z",
  "updatedAt": "2026-08-27T17:41:33Z",
  "pdfUrl": "https://arxiv.org/pdf/2608.27413v1",
  "absUrl": "https://arxiv.org/abs/2608.27413v1",
  "doi": null,
  "journalReference": null,
  "comment": "12 pages, 4 figures, 8 tables...",
  "query": "all:\"graph neural networks\" AND (cat:cs.LG)",
  "fetchedAt": "2026-08-28T06:36:49.711Z"
}
```

Optional metadata remains `null` when the paper's arXiv entry does not supply it.

### How much does it cost to export arXiv papers?

The Actor uses pay per event:

- **$0.001** when a run starts;
- **$0.0008 per saved paper** on the BRONZE tier.

Approximate BRONZE examples:

| Saved papers | Estimated price |
|---:|---:|
| 5 | $0.005 |
| 25 | $0.021 |
| 100 | $0.081 |
| 500 | $0.401 |

A no-result run pays only the start fee. Failed input validation happens before source extraction, while the live pricing system remains the final billing authority.

Higher Apify subscription tiers receive the lower item prices shown in Console.

### Export to spreadsheets and data pipelines

Every result is stored in the default Apify dataset.

From the Dataset tab you can download:

- JSON for applications and archives;
- CSV or Excel for analysts;
- XML for legacy systems;
- RSS for compatible readers.

Use Apify integrations to send completed datasets to Google Drive, Slack, webhooks, Zapier, Make, or your own service.

For durable pipelines, store `paperId` as the stable join key and inspect `updatedAt` for version changes.

### Schedule literature monitoring

A practical recurring workflow is:

1. create a saved task with a topic, category, and date range;
2. add a daily or weekly schedule;
3. receive a run-finished webhook;
4. fetch the dataset through the API;
5. compare paper IDs with your existing catalog;
6. route new papers to a review queue.

Keep date windows explicit for reproducible monitoring.

ArXiv publication timing and source API indexing determine freshness.

### Run with the Apify API

#### cURL

```bash
curl -X POST \
  "https://api.apify.com/v2/acts/automation-lab~arxiv-paper-search-export/runs?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"query":"graph neural networks","categories":["cs.LG"],"maxItems":25}'
```

#### JavaScript

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/arxiv-paper-search-export').call({
  query: 'graph neural networks',
  categories: ['cs.LG'],
  maxItems: 25,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);
```

#### Python

```python
from apify_client import ApifyClient

client = ApifyClient(token="YOUR_APIFY_TOKEN")
run = client.actor("automation-lab/arxiv-paper-search-export").call(run_input={
    "query": "graph neural networks",
    "categories": ["cs.LG"],
    "maxItems": 25,
})
items = client.dataset(run["defaultDatasetId"]).list_items().items
print(items)
```

Never commit an Apify token to source control.

### Use through MCP

Add this Actor to Claude Code:

```bash
claude mcp add --transport http apify \
  "https://mcp.apify.com?tools=automation-lab/arxiv-paper-search-export"
```

#### Claude Desktop, Cursor, and VS Code setup

Use this MCP configuration in Claude Desktop, Cursor, or VS Code:

```json
{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=automation-lab/arxiv-paper-search-export"
    }
  }
}
```

Example prompts:

- "Find five recent cs.LG papers about graph neural networks and summarize their metadata."
- "Export papers by Geoffrey Hinton with their arXiv PDF links."
- "Collect quant-ph papers from this date range for my literature review."

MCP invokes the same validated Actor input and returns the same dataset records.

### Reliability and source etiquette

The implementation uses the official public Atom API, bounded request pages, a 30-second request timeout, and increasing retry delays.

For multi-page runs it waits at least three seconds between source requests in line with arXiv API guidance.

The Actor does not need browser rendering, residential proxies, or account sessions.

Temporary rate limits can still occur. The Actor retries only the bounded number requested and then fails with a clear error instead of silently returning stale data.

### Limits

- `maxItems` is capped at 500 per run.
- Search is governed by arXiv's index and query semantics.
- Topic text is treated as one quoted phrase for precise matching.
- Date filters use first-submission time, not the latest revision time.
- DOI, journal, and comment fields are optional source metadata.
- This Actor exports metadata and links; it does not download PDF files.
- It does not calculate citations, affiliations, full-text entities, or semantic similarity.
- It does not send alerts or merge datasets across runs.

### Troubleshooting

#### The run says I must provide a search field

Supply at least one of `query`, `author`, `categories`, or `paperIds`. The Actor intentionally rejects unbounded blank searches.

#### My category is rejected

Use an arXiv category code such as `cs.AI`, `math.OC`, or `quant-ph`, not a descriptive category name.

#### A search returns no papers

Try a broader phrase, remove one filter, verify the category code, or widen the date range. All active filters are combined with AND.

#### The arXiv API rate-limited the run

Wait before retrying, keep retry count bounded, and avoid overlapping large scheduled runs. The Actor already applies polite pagination delays.

#### A paper has no DOI or journal reference

Those fields are optional in arXiv. `null` means the source entry did not include the value.

### Legality and responsible use

ArXiv exposes public scholarly metadata for discovery and reuse, but users remain responsible for their workflow.

- Follow arXiv's API terms and rate guidance.
- Respect paper copyrights and licenses when following PDF links.
- Do not infer sensitive personal information about authors.
- Verify important bibliographic details against the source record.
- Attribute arXiv and original authors where appropriate.

This Actor is an independent automation tool and is not affiliated with or endorsed by Cornell University or arXiv.

### Related Actor

For broader scholarly web search and citation-result metadata, use [Google Scholar Scraper](https://apify.com/automation-lab/google-scholar-scraper).

Choose this Actor when you need official arXiv metadata and category/date filtering. Choose Google Scholar Scraper when you need discovery across publishers and repositories.

### FAQ

#### Does this Actor require an arXiv login?

No. It uses public metadata and never asks for account credentials.

#### Can I search by paper title?

Yes. Put the title or a distinctive phrase in `query`. Topic search covers titles and abstracts.

#### Can I fetch a specific version?

Yes. Supply a versioned ID such as `1706.03762v7`.

#### Can I export PDFs?

The output includes a direct `pdfUrl`, but the Actor does not download or store PDF binaries.

#### Are multiple categories AND or OR?

Multiple category values are OR. Category filtering is then ANDed with topic, author, and date filters.

#### How do I detect revised papers?

Store `paperId` and `updatedAt`, then compare them across scheduled datasets. Versioned IDs may also change when arXiv returns a newer version.

#### Is the output suitable for a research dataset?

It is suitable as structured source metadata. Review source licenses, document your query and collection date, and validate records for your methodology.

#### What happens if the source is unavailable?

The Actor performs bounded retries and fails the run if it cannot obtain valid Atom data. It does not substitute fabricated or cached results.

# Actor input Schema

## `query` (type: `string`):

Text to match across arXiv paper titles, abstracts, and author names.

## `author` (type: `string`):

Author name filter, for example Geoffrey Hinton.

## `categories` (type: `array`):

One or more arXiv category codes, such as cs.LG, cs.AI, math.OC, or quant-ph.

## `paperIds` (type: `array`):

Fetch known papers by ID or abs/PDF URL. Other supplied filters are applied to these papers too.

## `dateFrom` (type: `string`):

Only include papers first submitted on or after this date (YYYY-MM-DD).

## `dateTo` (type: `string`):

Only include papers first submitted on or before this date (YYYY-MM-DD).

## `sortBy` (type: `string`):

Order search results by submission date, last update date, or relevance.

## `sortOrder` (type: `string`):

Return newest/highest-ranked or oldest/lowest-ranked results first.

## `maxItems` (type: `integer`):

Maximum number of unique paper records to save (1–500).

## `maxRequestRetries` (type: `integer`):

Retries for temporary arXiv API or network failures. Retries use increasing delays.

## Actor input object example

```json
{
  "query": "graph neural networks",
  "categories": [
    "cs.LG"
  ],
  "sortBy": "submittedDate",
  "sortOrder": "descending",
  "maxItems": 5,
  "maxRequestRetries": 2
}
```

# Actor output Schema

## `overview` (type: `string`):

Open the normalized paper records in the overview table view.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "query": "graph neural networks",
    "categories": [
        "cs.LG"
    ],
    "maxItems": 5
};

// Run the Actor and wait for it to finish
const run = await client.actor("automation-lab/arxiv-paper-search-export").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "query": "graph neural networks",
    "categories": ["cs.LG"],
    "maxItems": 5,
}

# Run the Actor and wait for it to finish
run = client.actor("automation-lab/arxiv-paper-search-export").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "query": "graph neural networks",
  "categories": [
    "cs.LG"
  ],
  "maxItems": 5
}' |
apify call automation-lab/arxiv-paper-search-export --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,automation-lab/arxiv-paper-search-export"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/fn6kvjS5DVaRfGsSq/builds/kFLpEQZgr1mB6mgK6/openapi.json