# Arxiv Papers Scraper (`zerobreak/arxiv-papers-scraper`) Actor

arXiv Papers Scraper is a Python utility designed to efficiently extract research papers, abstracts, author details, and publication metadata from arXiv, converting academic archives into structured, analysis-ready formats.

- **URL**: https://apify.com/zerobreak/arxiv-papers-scraper.md
- **Developed by:** [ZeroBreak](https://apify.com/zerobreak) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.99 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## arXiv Papers Scraper

<p align="center"><img src="https://imgur.com/cSvJ2uU.png" alt="arXiv Papers Scraper"  width="800" /></p>

arXiv Papers Scraper uses the public arXiv Atom API to search academic papers and save matching records for export as JSON, CSV, or Excel. No login or API key is required.

Search using standard arXiv query syntax, such as `all:python`, `ti:machine learning`, `au:smith`, or `cat:cs.AI`. The actor stores paper titles in `paperTitle` because the dataset reserves the field name `title`. The overview table still displays this column as **Title**.

### Use Cases

- **Literature reviews:** Collect papers on a topic before a reading group or research session.
- **Dataset building:** Export titles, authors, abstracts, and paper links for research datasets.
- **Category monitoring:** Schedule recurring searches for categories such as `cat:cs.LG`.
- **Research briefings:** Gather recent submissions in a field without manually browsing search results.
- **Paper link tracking:** Store abstract URLs alongside Atom IDs for future reference.

### Input

| Parameter | Type | Default | Description |
|---|---|---|---|
| `searchQuery` | string | `all:python` | arXiv query sent as `search_query`. Supports syntax such as `all:`, `ti:`, `au:`, and `cat:`. |
| `sortBy` | string | `submittedDate` | Sorting option: `relevance`, `lastUpdatedDate`, or `submittedDate`. Results are ordered descending. |
| `maxItems` | integer | `100` | Maximum number of papers to save, from 1 to 1,000. |
| `timeoutSecs` | integer | `300` | Maximum time in seconds for starting new API calls. |
| `requestTimeoutSecs` | integer | `30` | Timeout in seconds for each Atom API request. |

#### Example Input

```json
{
  "searchQuery": "cat:cs.AI",
  "sortBy": "submittedDate",
  "maxItems": 100,
  "timeoutSecs": 300,
  "requestTimeoutSecs": 30
}
```

### What Data Does This Actor Extract?

Each paper is saved as one dataset row. If a query is invalid or an API request fails, the actor writes an error row instead of failing the entire run.

#### Example Output

```json
{
  "paperTitle": "ePython: An implementation of Python for the many-core Epiphany coprocessor",
  "link": "https://arxiv.org/abs/2010.14827v1",
  "published": "2020-10-28T09:01:27Z",
  "author": "Nick Brown",
  "summary": "The Epiphany is a many-core, low power, low on-chip memory architecture.",
  "id": "http://arxiv.org/abs/2010.14827v1",
  "sourceUrl": "https://export.arxiv.org/api/query?search_query=all:python&start=0&max_results=3&sortBy=submittedDate&sortOrder=descending",
  "scrapedAt": "2026-10-12T00:00:00+00:00"
}
```

#### Output Fields

| Field | Type | Description |
|---|---|---|
| `paperTitle` | string | Paper title with markup removed. |
| `link` | string | URL of the paper's abstract page. |
| `published` | string | Published or updated timestamp provided by the feed. |
| `author` | string | Author names in feed order, joined by commas. |
| `summary` | string | Paper abstract with markup removed. |
| `id` | string | Atom entry ID. Duplicate IDs are discarded. |
| `sourceUrl` | string | Atom API URL that returned the paper. |
| `scrapedAt` | string | UTC timestamp when the record was saved. |
| `error` | string | Present only for error records; contains an error explanation. |
| `input` | string | Query or setting associated with an error record. |

### How It Works

1. Reads the search query, sorting option, item limit, and timeout settings.
2. Requests results from the public arXiv Atom API using pagination.
3. Parses the Atom feed and extracts the paper fields described above.
4. Skips papers whose Atom IDs have already been saved.
5. Requests subsequent pages until it reaches the configured item limit, the feed's total result count, or a short page.
6. Waits 3.1 seconds between API calls.
7. Retries HTTP 429 responses, HTTP 5xx responses, and timeouts with waits of 2, 4, and 8 seconds, or follows the `Retry-After` value when provided by arXiv.
8. Writes one error row and finishes the run if the query is empty, the sorting option is unsupported, the feed contains no results, or the response body is not a valid Atom feed.

The actor uses the public export Atom API and identifies itself through its User-Agent, which includes a contact email.

### Limits and Terms

- **Request frequency:** arXiv asks API clients to make no more than one request every three seconds over a single connection. The actor uses one connection and waits 3.1 seconds between calls.
- **Page size:** Each request asks for at most 100 results, staying within the API page limit.
- **Maximum results:** A single run can save up to 1,000 papers.
- **Metadata licensing:** arXiv metadata is released under CC0. When republishing records, link back to arXiv.
- **Paper references:** The `link` and `id` fields point to the paper's abstract page or its Atom entry identifier, respectively.

### FAQ

**Does the arXiv Atom API require an API key?**

No. The public export API does not require an API key or login.

**How many papers can one run export?**

The maximum is 1,000 papers per run. The default is 100. Large collections take longer because the actor respects the API's request-frequency guidance.

**What happens if no papers match the query?**

The actor completes the run and writes a single row containing an `error` field with a short explanation.

**Why is the title field called `paperTitle`?**

The dataset schema reserves the field name `title`. The actor uses `paperTitle` for the paper's title, while the overview table labels the column **Title**.

**Can I sort results by relevance or date?**

Yes. Set `sortBy` to `relevance`, `lastUpdatedDate`, or `submittedDate`.

**Can I schedule recurring searches?**

Yes. Recurring runs can help keep research collections and category-monitoring datasets up to date.

### Pricing

The actor is free to run. Apify platform usage may still incur charges based on the resources consumed.

### Trademark Notice

arXiv and related names, logos, and trademarks belong to their respective owners. Their use in this documentation is solely to identify the data source and does not imply affiliation, sponsorship, endorsement, or partnership.

### Disclaimer

This actor is an independent tool and is not affiliated with, endorsed by, sponsored by, or officially associated with arXiv or its owners. Data is collected from publicly available sources. Users are responsible for complying with applicable terms of use, rate limits, attribution requirements, and licensing conditions.

### Is Web Scraping Legal?

This actor collects publicly available arXiv metadata through its public API without logging in or bypassing access controls. The legality of collecting and using public data depends on the circumstances and applicable laws.

- **Personal data:** If collected records contain personal data, privacy laws such as the GDPR or CCPA may apply. Collect and retain personal data only when you have an appropriate legal basis.
- **Copyright:** Metadata and full-text papers can have different licensing conditions. Do not republish copyrighted article content without the necessary permission.
- **Terms of use:** Respect the source's terms, request-frequency guidance, and applicable attribution or licensing requirements.
- **Legal advice:** This section provides general information, not legal advice. Consult a qualified lawyer if you are unsure about your obligations.

# Actor input Schema

## `searchQuery` (type: `string`):

arXiv query string passed to the Atom API search_query parameter. Examples: all:python, cat:cs.AI, ti:transformer AND au:vaswani.

## `sortBy` (type: `string`):

Sort order sent to the Atom API. Allowed values: relevance, lastUpdatedDate, submittedDate.

## `maxItems` (type: `integer`):

Maximum number of papers to save. Values outside 1 to 1000 are clamped.

## `timeoutSecs` (type: `integer`):

Overall run time limit in seconds. The actor stops starting new requests after this many seconds.

## `requestTimeoutSecs` (type: `integer`):

Per-request timeout in seconds for each call to the arXiv Atom API.

## `proxyConfiguration` (type: `object`):

Optional proxy for Atom API requests. Leave Apify Proxy off unless export.arxiv.org blocks the default IP.

## Actor input object example

```json
{
  "searchQuery": "cat:cs.AI",
  "sortBy": "submittedDate",
  "maxItems": 100,
  "timeoutSecs": 300,
  "requestTimeoutSecs": 30,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `datasetUrl` (type: `string`):

Default dataset with one row per paper or error.

## `itemsUrl` (type: `string`):

Dataset items as JSON.

## `overviewUrl` (type: `string`):

Dataset items in the overview table view.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "searchQuery": "all:python",
    "sortBy": "submittedDate",
    "maxItems": 100,
    "timeoutSecs": 300,
    "requestTimeoutSecs": 30,
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("zerobreak/arxiv-papers-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "searchQuery": "all:python",
    "sortBy": "submittedDate",
    "maxItems": 100,
    "timeoutSecs": 300,
    "requestTimeoutSecs": 30,
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("zerobreak/arxiv-papers-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "searchQuery": "all:python",
  "sortBy": "submittedDate",
  "maxItems": 100,
  "timeoutSecs": 300,
  "requestTimeoutSecs": 30,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call zerobreak/arxiv-papers-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,zerobreak/arxiv-papers-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/QRser8I6Wc8bgMyu2/builds/FCLuEXhNsOkMtQI5W/openapi.json