# arXiv Paper Search Scraper API + AI Summaries (`data-drip/arxiv-papers-ai`) Actor

Search arXiv research papers by keyword, category and date via the official arXiv API: title, abstract, categories, dates, PDF link, DOI and journal ref. Optional AI adds a plain-English TL;DR, contributions, methods and applications.

- **URL**: https://apify.com/data-drip/arxiv-papers-ai.md
- **Developed by:** [Ibraheem Ganayim](https://apify.com/data-drip) (community)
- **Categories:** AI, Education, Developer tools
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 paper scrapeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## arXiv Paper Search Scraper API + AI Summaries

**arXiv Paper Scraper at a glance:** data from the official arXiv API · no login, no browser, no proxies · **$1 per 1,000 papers** · optional AI with your own key · JSON, CSV, Excel export, API and MCP ready

### What does the arXiv Paper Scraper do?

Search **arXiv research papers** by keyword, category and submission date and export clean metadata: **title, abstract, categories, publish/update dates, PDF link, DOI, journal reference and comments**. Turn on the optional **AI summaries** to get a **plain-English TL;DR, key contributions, methods, practical applications and difficulty level** for every paper, perfect for literature reviews, research newsletters and AI research agents.

The Actor uses the **official arXiv API** (`export.arxiv.org/api`) and follows arXiv's polite-use rules (one request every 3 seconds). No browser, no proxies, no login.

### Why this arXiv scraper?

- **Keyword search** in all fields, title only or abstract only, or exact phrases with quotes
- **Category filters**: `cs.AI`, `cs.CL`, `cs.LG`, `stat.ML`, `q-bio`, `physics`... (OR-ed)
- **Date range** filter on submission date, sort by newest, last updated or relevance
- **Fetch specific papers** by arXiv ID or URL
- **Thousands of papers per run**, paginated automatically
- **AI summaries** (optional, your own OpenAI/OpenRouter key): TL;DR, contributions, methods, applications, field, level
- **Agent and RAG friendly**: clean JSON, one paper per row, stable `arxivId`

### Use cases

- **Daily/weekly paper digests** for a research topic (schedule the Actor)
- **Literature reviews** and related-work sections
- **AI research agents and MCP tools** that need fresh papers
- **RAG knowledge bases** of abstracts and summaries
- **Trend tracking** of methods and fields over time
- **Tech scouting** for startups, VCs and R\&D teams

### How to use the arXiv Paper Scraper

1. Click **Try for free** (or **Start**) on this page and sign in to Apify.
2. Enter keywords (or an exact phrase), and optionally arXiv categories such as `cs.AI` or `cs.CL`.
3. Optionally set a submission date range, sort order and maximum number of papers, or list specific arXiv IDs.
4. Optionally enable AI summaries and paste your OpenAI or OpenRouter key.
5. Click **Start**, then download results as JSON, CSV or Excel, or fetch them via the API.

### Input

| Field | Description |
|---|---|
| `query` | Keywords (all must match), `"exact phrase"`, or raw arXiv syntax such as `ti:transformer AND abs:vision` |
| `searchIn` | `all` (default), `ti` (title) or `abs` (abstract) |
| `categories` | Optional arXiv categories, e.g. `["cs.AI", "cs.CL"]` |
| `arxivIds` | Or fetch specific papers: `1706.03762`, `https://arxiv.org/abs/2005.14165` |
| `dateFrom` / `dateTo` | Submission date range (YYYY-MM-DD) |
| `sortBy` / `sortOrder` | `submittedDate` (default), `lastUpdatedDate` or `relevance`; descending or ascending |
| `maxResults` | Maximum papers (default 100) |
| `enableAi`, `llmProvider`, `llmApiKey`, `llmModel` | Optional AI summaries with your own key |

```json
{ "query": "retrieval augmented generation", "categories": ["cs.CL", "cs.AI"], "dateFrom": "2026-10-01", "maxResults": 200 }
```

### Output example

One row per paper, straight from the official arXiv API:

```json
{
  "arxivId": "2610.40454",
  "version": "v1",
  "title": "Glance: The Generator Is Its Own Reranker",
  "abstract": "Retrieval-augmented generation commonly serves separate models for reranking and answering, duplicating model residency and passage processing. We introduce Glance...",
  "primaryCategory": "cs.CL",
  "categories": ["cs.CL", "cs.IR"],
  "publishedAt": "2026-10-29T17:01:22Z",
  "updatedAt": "2026-10-29T17:01:22Z",
  "absUrl": "https://arxiv.org/abs/2610.40454v1",
  "pdfUrl": "https://arxiv.org/pdf/2610.40454v1",
  "doi": null,
  "journalRef": null,
  "comment": "12 pages, 2 figures, 6 tables.",
  "authorCount": 2,
  "ai": {
    "tldr": "One 9B model both reranks retrieved passages and answers, beating a separate cross-encoder reranker.",
    "contributions": ["Decision adapter + pointer head for reranking from a shared prefill", "~0.1 GiB extra memory per skill", "Better nDCG@10 than bge-reranker-v2-m3"],
    "methods": ["adapter fine-tuning", "MS MARCO", "SciFact", "HotpotQA"],
    "applications": ["cheaper RAG serving", "single-model search assistants"],
    "field": "Retrieval-augmented generation",
    "difficulty": "expert"
  },
  "scrapedAt": "2026-10-31T20:04:24.565Z"
}
```

The `ai` object only appears when AI summaries are enabled with a key (values shown are illustrative).

### How much does it cost?

This Actor uses **pay per event** pricing: you pay only for results saved to your dataset, with no monthly fee. Apify platform compute is included.

| Event | Price |
|---|---|
| Paper scraped | **$0.001** per paper ($1 per 1,000) |
| AI paper summary | **$0.003** per paper (plus your own LLM tokens) |

**Examples**

- 1,000 papers: **$1**
- 10,000 papers: **$10**
- 1,000 papers with AI: **$4** plus your LLM tokens (typically cents with `gpt-4o-mini`)

Apify's Free plan includes $5 of monthly usage, enough for about **5,000 papers** without AI. Set **Maximum cost per run** in the run options to cap spend; the Actor stops cleanly at the limit.

### Use it via API, Python, n8n, Make or AI agents

Run the Actor and get results in one call (replace `<APIFY_TOKEN>` with your token):

```bash
curl -X POST "https://api.apify.com/v2/acts/data-drip~arxiv-papers-ai/run-sync-get-dataset-items?token=<APIFY_TOKEN>" \
  -H 'Content-Type: application/json' \
  -d '{"query": "retrieval augmented generation", "categories": ["cs.CL", "cs.AI"], "dateFrom": "2026-10-01", "maxResults": 200}'
```

```python
from apify_client import ApifyClient

client = ApifyClient('<APIFY_TOKEN>')
run = client.actor('data-drip/arxiv-papers-ai').call(run_input={"query": "retrieval augmented generation", "categories": ["cs.CL", "cs.AI"], "dateFrom": "2026-10-01", "maxResults": 200})
for item in client.dataset(run['defaultDatasetId']).iterate_items():
    print(item)
```

- **Schedules**: run it hourly, daily or weekly from Apify Schedules.
- **Integrations**: send results to Google Sheets, Slack, Zapier, Make, n8n or a webhook.
- **AI agents (MCP)**: add `data-drip/arxiv-papers-ai` as a tool in the Apify MCP server (`https://mcp.apify.com`) for Claude, Cursor and other MCP clients.

### Limitations

- Returns abstracts and PDF links, not parsed full text.
- Author names are replaced by an author count (no personal data).
- arXiv asks for one request every 3 seconds, so very large runs (10,000+ papers) take a while.

### FAQ

**Is it allowed?** Yes. The Actor uses arXiv's official public API, which is meant for programmatic access, and keeps to arXiv's rate guidance of one request every 3 seconds. arXiv metadata is openly licensed. Thank you to arXiv for use of its open access interoperability.

**Why are author names not included?** We keep the output free of personal data, so you get `authorCount` instead of names. The `absUrl` links to the full paper page with all bylines.

**Can I get the full text?** The Actor returns the abstract and a direct `pdfUrl`. Full-text PDF parsing is not included.

**How fresh is the data?** Queries hit arXiv live, so new submissions appear as soon as arXiv announces them. Schedule a daily run with `dateFrom` for a digest.

**Do I need an AI key?** No. Without a key the Actor is a fast metadata scraper and you pay only for papers. To enable AI, paste your own OpenAI or OpenRouter key into **LLM API key** (stored as an encrypted secret). LLM tokens are billed by your provider, not by us.

**Can I schedule it?** Yes. Use Apify Schedules (for example daily) and connect the dataset to Google Sheets, Slack or a webhook to get only fresh papers.

**How do I get help or request a feature?** Open an issue on the Actor's **Issues** tab.

***

**Related searches:** arxiv scraper, arxiv api, research paper search, academic papers scraper, paper summaries, literature review, ai research digest, rag dataset, scientific papers, machine learning papers.

# Actor input Schema

## `query` (type: `string`):

Words to search for (all must match). Wrap in quotes for an exact phrase. Advanced: raw arXiv syntax such as ti:transformer AND abs:vision.

## `searchIn` (type: `string`):

Which field the keywords must appear in.

## `categories` (type: `array`):

Optional category filter, e.g. cs.AI, cs.CL, cs.LG, stat.ML, q-bio.GN. Several categories are OR-ed.

## `arxivIds` (type: `array`):

arXiv IDs or URLs, e.g. 1706.03762 or https://arxiv.org/abs/2005.14165.

## `dateFrom` (type: `string`):

Only papers submitted on or after this date (YYYY-MM-DD).

## `dateTo` (type: `string`):

Only papers submitted on or before this date.

## `sortBy` (type: `string`):

Order of results.

## `sortOrder` (type: `string`):

Direction of sorting.

## `maxResults` (type: `integer`):

Maximum number of papers to return. arXiv allows one page of 100 every 3 seconds.

## `enableAi` (type: `boolean`):

Add a plain-English TL;DR, key contributions, methods, practical applications, field and difficulty level for each paper. Requires your own OpenAI or OpenRouter key.

## `llmProvider` (type: `string`):

Which API your key belongs to.

## `llmApiKey` (type: `string`):

Your OpenAI or OpenRouter API key. Stored encrypted.

## `llmModel` (type: `string`):

Defaults to gpt-4o-mini (OpenAI) or openai/gpt-4o-mini (OpenRouter).

## Actor input object example

```json
{
  "query": "large language model agents",
  "searchIn": "all",
  "categories": [
    "cs.AI"
  ],
  "sortBy": "submittedDate",
  "sortOrder": "descending",
  "maxResults": 100,
  "enableAi": false,
  "llmProvider": "openai"
}
```

# Actor output Schema

## `papers` (type: `string`):

No description

## `aiSummaries` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "query": "large language model agents",
    "categories": [
        "cs.AI"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("data-drip/arxiv-papers-ai").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "query": "large language model agents",
    "categories": ["cs.AI"],
}

# Run the Actor and wait for it to finish
run = client.actor("data-drip/arxiv-papers-ai").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "query": "large language model agents",
  "categories": [
    "cs.AI"
  ]
}' |
apify call data-drip/arxiv-papers-ai --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,data-drip/arxiv-papers-ai"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/WM3u3eeZF8BBY7Ef1/builds/b3mLTnlpCEp8nd70S/openapi.json