# arXiv Papers Scraper: Search Papers, Abstracts & Authors (`tidyharvest/arxiv-papers-scraper`) Actor

Run one or many arXiv searches by keywords, category, author, date range or ID and get titles, abstracts, authors, categories, dates, DOIs and PDF links from the official arXiv API. No login or proxy setup. $0.90 per 1,000 papers.

- **URL**: https://apify.com/tidyharvest/arxiv-papers-scraper.md
- **Developed by:** [TidyHarvest](https://apify.com/tidyharvest) (community)
- **Categories:** AI, Education, Developer tools
- **Stats:** 2 total users, 1 monthly users, 33.3% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$0.90 / 1,000 paper scrapeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## arXiv Papers Scraper: Search Papers, Abstracts & Authors

Search **arXiv** and get clean, structured **paper metadata**: **title, abstract, authors (with affiliations when listed), arXiv ID and version, primary and cross-list categories, submission and update dates, DOI, journal reference, comments, and links to the abstract page and PDF.** Search by keywords, category (`cs.AI`, `cs.LG`, `hep-th`...), author, submission date range or a list of arXiv IDs.

It reads the **official arXiv API**, so you need no login, API key, browser or proxy setup of your own. **$0.90 per 1,000 papers.**

### What you can do with it

- **Daily new-paper feeds:** schedule a run for the latest `cs.CL` or `q-bio` submissions and send them to Slack, email, Notion or Google Sheets.
- **Literature reviews and research monitoring:** pull every paper matching a topic in a date range, with abstracts, into a spreadsheet.
- **RAG and AI datasets:** build a corpus of abstracts and metadata to embed, classify or summarize with an LLM.
- **Author and lab tracking:** follow what specific researchers publish.
- **Bibliometrics:** count papers by category, month or cross-list, or collect DOIs and journal references.

### How to use it

1. Type one or more searches, pick categories (or leave everything empty for the latest `cs.AI` papers).
2. Optionally add authors, a submission date range, a sort order and a **Max papers** cap.
3. Run it, then download the results as JSON, CSV or Excel, or connect them to any Apify integration.

### Input

| Field | What it does |
|---|---|
| `queries` | Free-text searches, one per line, each run with the filters below; every word must match, `"exact phrase"` in quotes. Rows carry the `query` that found them; a paper found twice is returned once. |
| `maxItemsPerQuery` | Cap per search when you give several (empty = only `maxItems` applies). |
| `categories` | arXiv category codes, matched with OR (`cs.AI`, `cs.LG`, `stat.ML`, `astro-ph`...). |
| `authors` | Author names, matched with OR. |
| `dateFrom`, `dateTo` | First-submission date range (YYYY-MM-DD). |
| `ids` | Specific arXiv IDs or abs/pdf links (`1706.03762`, `2005.14165v4`, `hep-th/9901001`). |
| `searchQuery` | Advanced arXiv query syntax (`ti:`, `abs:`, `au:`, `cat:`, `AND`/`OR`/`ANDNOT`). |
| `sortBy`, `sortOrder` | Relevance, submission date or last-updated date; newest or oldest first. |
| `maxItems` | Max papers for the whole run (default 50, up to 10,000). Also caps your cost. |

Filled fields are combined with AND. Example:

```json
{ "queries": ["retrieval augmented generation"], "categories": ["cs.CL", "cs.IR"], "dateFrom": "2026-01-01", "maxItems": 200 }
```

Several topics in one run, 100 papers each:

```json
{ "queries": ["mixture of experts", "speculative decoding", "\"state space model\""], "categories": ["cs.LG", "cs.CL"], "maxItemsPerQuery": 100, "maxItems": 300 }
```

### Output

One item per paper:

```json
{
  "arxivId": "1706.03762", "version": 7,
  "title": "Attention Is All You Need",
  "summary": "The dominant sequence transduction models are based on complex recurrent or convolutional neural networks ...",
  "authors": [{"name": "Ashish Vaswani", "affiliations": []}],
  "authorNames": ["Ashish Vaswani", "Noam Shazeer", "..."],
  "primaryCategory": "cs.CL", "categories": ["cs.CL", "cs.LG"],
  "published": "2017-06-12T17:57:34Z", "updated": "2023-08-02T00:41:18Z",
  "doi": null, "journalRef": null, "comment": "15 pages, 5 figures",
  "absUrl": "https://arxiv.org/abs/1706.03762v7", "pdfUrl": "https://arxiv.org/pdf/1706.03762v7",
  "source": "arXiv API (export.arxiv.org)",
  "attribution": "Thank you to arXiv for use of its open access interoperability. Metadata from arXiv.org (CC0)."
}
```

| Field | Meaning |
|---|---|
| `arxivId`, `version` | arXiv identifier without version, and the latest version number. |
| `title`, `summary` | Title and abstract (whitespace cleaned). |
| `authors`, `authorNames` | Authors with affiliations (when the authors provided them), and a plain name list. |
| `primaryCategory`, `categories` | Main category and all cross-lists. |
| `published`, `updated` | First submission (v1) and latest version timestamps, UTC. |
| `doi`, `journalRef`, `comment` | Author-supplied publication info; `null` when not given. |
| `absUrl`, `pdfUrl` | Links to the paper on arXiv.org. |
| `source`, `attribution` | Where the data came from and the arXiv acknowledgement. |

### Pricing

Pay per result: **$0.90 per 1,000 papers** (`paper-scraped`). The `maxItems` cap applies before you are charged, and the run stops at your spending limit. A search that finds nothing fails instead of returning an empty dataset.

### Limits and notes

- **Speed:** arXiv asks API users to make at most one request every 3 seconds on a single connection. This actor follows that rule, so it returns about 100 papers every 5 seconds (500 papers in about 25 seconds).
- **Proxy included:** arXiv often turns away requests from shared cloud servers, so the actor connects through Apify Proxy, keeping one IP per run. This costs you nothing beyond the per-paper price.
- arXiv's API returns at most about 30,000 results per search. Split very large pulls by date range.
- Returns **metadata only**. PDFs are linked, not downloaded: the papers' copyright belongs to their authors, and arXiv asks that readers be sent to arXiv.org for the full text.
- Affiliations, DOI and journal references are filled only when authors supplied them.
- Abstract and metadata are released by arXiv under CC0. Thank you to arXiv for use of its open access interoperability.
- This actor is not affiliated with, endorsed by or supported by arXiv or Cornell University.

# Actor input Schema

## `queries` (type: `array`):

Free-text searches across title, abstract, authors and comments; each line runs its own search with the filters below. Every word must match; put "exact phrases" in double quotes. Each paper says which search found it, and a paper found twice is returned once. Leave empty to list the latest papers in the chosen categories.

## `query` (type: `string`):

Same as one line in the list above; kept for existing integrations. Every word must match; put "exact phrases" in double quotes. Leave empty to list the latest papers in the chosen categories.

## `maxItemsPerQuery` (type: `integer`):

Cap for each line when you give several searches. Empty = no per-search cap (only Max papers applies).

## `categories` (type: `array`):

arXiv category codes, matched with OR (e.g. `cs.AI`, `cs.LG`, `cs.CL`, `stat.ML`, `q-bio.GN`, `astro-ph`). Full list: arxiv.org/category_taxonomy. With no input at all the run returns the latest `cs.AI` papers.

## `authors` (type: `array`):

Author names, matched with OR (e.g. `Yann LeCun` or a surname like `Bengio`).

## `dateFrom` (type: `string`):

Only papers first submitted on or after this date (YYYY-MM-DD).

## `dateTo` (type: `string`):

Only papers first submitted on or before this date (YYYY-MM-DD).

## `ids` (type: `array`):

Look up specific papers by ID (`1706.03762`, `2005.14165v4`, `hep-th/9901001`) or abs/pdf link. Combined with other fields, only these papers are searched.

## `searchQuery` (type: `string`):

Raw arXiv API query, ANDed with the fields above. Prefixes: `ti:` title, `abs:` abstract, `au:` author, `cat:` category, `co:` comment, `jr:` journal ref; operators AND, OR, ANDNOT. Example: `ti:transformer ANDNOT cat:cs.CV`.

## `sortBy` (type: `string`):

Auto = relevance when you give search terms, otherwise newest submissions first.

## `sortOrder` (type: `string`):

Descending = newest (or most relevant) first.

## `maxItems` (type: `integer`):

Maximum papers to return for the whole run (also caps your cost). arXiv allows about 100 papers every 3 seconds, so 1,000 papers take about 35 seconds.

## Actor input object example

```json
{
  "queries": [
    "large language model agents"
  ],
  "categories": [
    "cs.AI"
  ],
  "sortBy": "auto",
  "sortOrder": "descending",
  "maxItems": 50
}
```

# Actor output Schema

## `papers` (type: `string`):

One item per paper.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "queries": [
        "large language model agents"
    ],
    "categories": [
        "cs.AI"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("tidyharvest/arxiv-papers-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "queries": ["large language model agents"],
    "categories": ["cs.AI"],
}

# Run the Actor and wait for it to finish
run = client.actor("tidyharvest/arxiv-papers-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "queries": [
    "large language model agents"
  ],
  "categories": [
    "cs.AI"
  ]
}' |
apify call tidyharvest/arxiv-papers-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,tidyharvest/arxiv-papers-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/bSOuNSRDivMdyArca/builds/Hvld16KQ0dqc0e0Lh/openapi.json