# arXiv Research Papers Scraper — AI/ML Preprints, Abstracts & (`spotless_studios/arxiv-papers-scraper`) Actor

Scrape arXiv preprints, machine learning papers, authors, abstracts, PDF download links, categories (cs.AI, cs.LG, cs.CV), and publication dates via the official API.

- **URL**: https://apify.com/spotless_studios/arxiv-papers-scraper.md
- **Developed by:** [Spotless Studios](https://apify.com/spotless_studios) (community)
- **Categories:** AI
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 arxiv research paper extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## arXiv Research Papers & AI/ML Preprints Scraper

Extract academic preprints, machine learning research, author lists, abstracts, PDF download links, categories (`cs.AI`, `cs.LG`, `cs.CV`), and submission dates from arXiv via the official query API.

### Features

- **Scholarly Preprint Extraction**: Scrapes paper title, arXiv identifier, abstract summary, publication and update timestamps, author list, categories, journal reference, DOI, and direct PDF download link.
- **Granular Subject Targeting**: Query across computer science (`cs.AI`, `cs.LG`, `cs.CL`, `cs.CV`), statistics (`stat.ML`), physics (`quant-ph`), mathematics, and economics.
- **Publication Date Filtering**: Filter for papers submitted after a specific date.
- **Automated Research Watch**: Use `onlyNewItems` with a `stateKey` to automatically monitor arXiv daily and trigger downstream alerts when new preprints on your research topic hit the repository.
- **Fast & Direct**: Consumes clean XML Atom feeds with zero browser overhead, consuming minimal memory (128 MB).

### Input Parameters

| Parameter | Type | Default | Description |
|---|---|---|---|
| `searchQuery` | String | `"large language models reasoning"` | Research topic or keywords |
| `category` | String | `""` | arXiv subject category code (e.g. `cs.AI`, `cs.LG`) |
| `author` | String | `""` | Filter by specific author name |
| `sortBy` | String | `"submittedDate"` | `submittedDate`, `lastUpdatedDate`, or `relevance` |
| `sortOrder` | String | `"descending"` | `descending` or `ascending` |
| `publishedAfter` | String | `""` | Date cutoff (YYYY-MM-DD) |
| `maxResults` | Integer | `100` | Maximum preprints to extract |
| `onlyNewItems` | Boolean | `false` | Monitoring mode: only emit newly submitted preprints |
| `outputPreset` | String | `"full"` | `full`, `lean`, or `contacts` |

### Output Example

```json
{
  "arxivId": "2404.18920v1",
  "title": "Agentic Reasoning in Large Language Models via Dynamic Self-Reflection",
  "summary": "We introduce a novel architecture for autonomous multi-step reasoning in code generation...",
  "published": "2026-10-25T14:00:00.000Z",
  "updated": "2026-10-25T14:00:00.000Z",
  "authors": ["Alexander Chen", "Sarah Jenkins"],
  "primaryCategory": "cs.AI",
  "categories": ["cs.AI", "cs.LG", "cs.CL"],
  "arxivUrl": "https://arxiv.org/abs/2404.18920v1",
  "pdfUrl": "https://arxiv.org/pdf/2404.18920v1.pdf",
  "doi": null,
  "comment": "Accepted at NeurIPS 2026, 14 pages",
  "scrapedAt": "2026-10-27T12:00:00.000Z"
}
```

***

### More from Spotless Studios

**[All actors on Apify](https://apify.com/spotless_studios)** · Support: spotless.studios.info@gmail.com

| Actor | Best for |
|-------|----------|
| [SLED Government Bids](https://apify.com/spotless_studios/sled-procurement-scraper) | Daily US gov RFP / bid alerts |
| [Webflow Experts](https://apify.com/spotless_studios/webflow-experts-scraper) | Webflow agency & freelancer leads |
| [Stripe Partners](https://apify.com/spotless_studios/stripe-partners-scraper) | Stripe implementation partner lists |
| [UK Planning Applications](https://apify.com/spotless_studios/uk-planning-applications-scraper) | UK planning leads + agent contacts |
| [Product Hunt](https://apify.com/spotless_studios/product-hunt-scraper) | Every PH launch + makers / websites |
| [Duunitori Jobs](https://apify.com/spotless_studios/duunitori-jobs-scraper) | Finland job listings |
| [Clutch Agencies](https://apify.com/spotless_studios/clutch-agencies-scraper) | Clutch agency prospecting |
| [Skool Communities](https://apify.com/spotless_studios/skool-communities-scraper) | Skool community & creator leads |

# Actor input Schema

## `searchQuery` (type: `string`):

Search query across title, abstract, and authors (e.g. 'reasoning agents', 'multimodal transformer', 'quantum error correction').

## `searchQueries` (type: `array`):

Optional list of multiple search queries to run sequentially in a single job.

## `titleQuery` (type: `string`):

Require these keywords in the paper title (ti: qualifier).

## `abstractQuery` (type: `string`):

Require these keywords in the paper abstract (abs: qualifier).

## `category` (type: `string`):

Filter by primary arXiv subject taxonomy (e.g. 'cs.AI', 'cs.LG', 'cs.CL', 'cs.CV', 'cs.RO', 'stat.ML', 'quant-ph').

## `categories` (type: `array`):

Filter across multiple arXiv categories (e.g. \['cs.AI', 'cs.CL', 'cs.LG']).

## `author` (type: `string`):

Filter preprints by author name (e.g. 'Yann LeCun', 'Ilya Sutskever').

## `sortBy` (type: `string`):

Sort order for arXiv preprints.

## `sortOrder` (type: `string`):

Descending (newest first) or ascending.

## `publishedAfter` (type: `string`):

Only return papers published after this date (YYYY-MM-DD, ISO, or relative like '14 days' / '3 months').

## `publishedBefore` (type: `string`):

Only return papers published before this date.

## `requireCodeUrl` (type: `boolean`):

When enabled, only saves papers that include a GitHub, GitLab, or HuggingFace repository link in their abstract or author comments.

## `requireDoi` (type: `boolean`):

When enabled, only saves preprints that have a journal DOI assigned.

## `arxivIds` (type: `array`):

Scrape specific papers directly by arXiv ID or URL (e.g. '2303.08774', 'https://arxiv.org/abs/1706.03762').

## `includeKeywords` (type: `array`):

Keep only papers matching these keywords or /regex/ in title, abstract, authors, or comments.

## `excludeKeywords` (type: `array`):

Skip papers matching any of these keywords or /regex/.

## `keywordMatch` (type: `string`):

Require any or all includeKeywords to match.

## `onlyNewItems` (type: `boolean`):

Remember seen arXiv IDs across runs and emit only newly published preprints.

## `stateKey` (type: `string`):

Separate state key per saved search or schedule.

## `maxResults` (type: `integer`):

Maximum number of papers to save.

## `outputPreset` (type: `string`):

Choose full record, lean summary, or contacts & links only.

## `outputFields` (type: `array`):

Keep only these specific fields in each saved record.

## `excludeFields` (type: `array`):

Remove specific fields from each saved record.

## `includeEmptyFields` (type: `boolean`):

When disabled, strips null and empty values from each saved record.

## `requiredFields` (type: `array`):

Only save records where every listed field has a non-empty value (e.g. codeUrls, doi).

## `maxPages` (type: `integer`):

Maximum search pages per query (100 papers per page).

## `maxRequestRetries` (type: `integer`):

Retries per failed HTTP request.

## `requestTimeoutSecs` (type: `integer`):

Timeout per HTTP request.

## `proxyConfiguration` (type: `object`):

Optional proxy settings.

## `proxyEscalation` (type: `boolean`):

Automatically step up from direct to Apify datacenter/residential proxies if blocked.

## Actor input object example

```json
{
  "searchQuery": "reasoning agents",
  "sortBy": "submittedDate",
  "sortOrder": "descending",
  "requireCodeUrl": false,
  "requireDoi": false,
  "keywordMatch": "any",
  "onlyNewItems": false,
  "stateKey": "default",
  "maxResults": 50,
  "outputPreset": "full",
  "includeEmptyFields": true,
  "maxPages": 15,
  "maxRequestRetries": 3,
  "requestTimeoutSecs": 30,
  "proxyConfiguration": {
    "useApifyProxy": false
  },
  "proxyEscalation": true
}
```

# Actor output Schema

## `items` (type: `string`):

Scrape arXiv preprints, machine learning papers, authors, abstracts, PDF download links, categories (cs.AI, cs.LG, cs.CV), and publication dates via the official API.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "searchQuery": "reasoning agents",
    "maxResults": 50,
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("spotless_studios/arxiv-papers-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "searchQuery": "reasoning agents",
    "maxResults": 50,
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("spotless_studios/arxiv-papers-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "searchQuery": "reasoning agents",
  "maxResults": 50,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call spotless_studios/arxiv-papers-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,spotless_studios/arxiv-papers-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Sd3NabJYM9l0xhJEn/builds/Tb23RBIU1evHSeGsF/openapi.json