# arXiv Scraper: Preprints, Authors & Categories (`arman-bd/arxiv-papers-scraper`) Actor

Scrape arXiv: title, authors, abstract, categories, DOI, journal reference, and PDF links. Search by query, category, author or date. The primary source for AI/ML preprints.

- **URL**: https://apify.com/arman-bd/arxiv-papers-scraper.md
- **Developed by:** [Arman Hossain](https://apify.com/arman-bd) (community)
- **Categories:** Developer tools, AI, MCP servers
- **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 paper scrapeds

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## arXiv Scraper: Preprints, Authors & Categories

![arXiv Scraper — Preprints, authors, abstracts, categories and DOIs — arXiv query syntax passed through untouched](https://api.apify.com/v2/key-value-stores/ZQOcNAOHrIgTacAmy/records/arxiv-papers-scraper.jpg)

### What it does

Scrape arXiv: title, authors, abstract, categories, DOI, journal reference, and PDF links. Search by query, category, author or date. The primary source for AI/ML preprints.

Give it `searchQueries` and it returns one structured record per paper, 13 fields, ready to join on DOI or feed straight into a vector store. arXiv's own query syntax is passed through untouched, so anything you can express on arxiv.org's advanced search works here: `cat:cs.LG AND all:transformer`, `au:Hinton AND abs:diffusion`, `ti:"retrieval augmented"`.

Multiple queries run in one pass, results are de-duplicated by arXiv ID across all of them, and pagination is handled for you.

### Input

| Field | Type | Default | Notes |
|---|---|---|---|
| `searchQueries` | array | `[]` | arXiv query syntax. Prefixes: `all:`, `ti:`, `abs:`, `au:`, `cat:`, `jr:` (journal ref), `co:` (comment). Combine with `AND` / `OR` / `ANDNOT`, group with parentheses. Each query is run and paginated separately. |
| `categories` | array | `[]` | arXiv categories, `cs.LG`, `cs.CL`, `stat.ML`, `q-bio.NC`… ORed together, then ANDed onto every query. |
| `fromDate` | string | - | Submission-date lower bound, `YYYY-MM-DD`. Applied as arXiv's `submittedDate:[… TO …]` range. |
| `sortBy` | string | `submittedDate` | One of `submittedDate`, `lastUpdatedDate`, `relevance`. Always descending. |
| `maxResultsPerQuery` | integer | `100` | Cap per query. Fetching stops as soon as the cap is met. `0` = no limit. |

**At least one of `searchQueries` or `categories` is required.** They combine sensibly:

- Query only → searches all of arXiv.
- Query + categories → the query, restricted to those categories.
- Categories only → the categories *become* the search, which is the cheapest way to pull a whole subject feed.
- `fromDate` + `sortBy: submittedDate` is the combination you want for a scheduled "what's new" run.

```json
{
 "searchQueries": ["all:\"large language model\" AND ti:agent"],
 "categories": ["cs.LG", "cs.CL"],
 "fromDate": "2026-01-01",
 "sortBy": "submittedDate",
 "maxResultsPerQuery": 25
}
```

### Output

One dataset item per paper. Real record from the run above:

```json
{
 "arxivId": "2608.04828v1",
 "title": "Skill-Use: Can LLMs Actually Use Skills in Agentic Harnesses?",
 "abstract": "Large language model (LLM) agents increasingly rely on skills, structured documents that specify when to act, which procedure to follow, and which tools are allowed. …",
 "authors": ["Jinyi Han", "Yuanjian Xu", "Ying Liao", "Xinyi Wang", "Zishang Jiang", "Zixiang Di", "Fanyang Lu", "Zhichao Hu", "Yanghua Xiao"],
 "primaryCategory": "cs.CL",
 "categories": ["cs.CL"],
 "published": "2026-08-05T13:29:16Z",
 "updated": "2026-08-05T13:29:16Z",
 "doi": null,
 "journalRef": null,
 "comment": null,
 "pdfUrl": "https://arxiv.org/pdf/2608.04828v1",
 "absUrl": "https://arxiv.org/abs/2608.04828v1",
 "scrapedAt": "2026-08-06T11:28:08.231Z"
}
```

| Field | Meaning |
|---|---|
| `arxivId` | arXiv identifier **including the version suffix**, `2608.04828v1`, or an old-style `cond-mat/0102536v1` |
| `title` | Paper title, line-wrapping removed |
| `abstract` | Full abstract as one paragraph, XML entities decoded |
| `authors` | Author names in submission order |
| `primaryCategory` | The single category the authors filed it under |
| `categories` | Every category it is cross-listed in, primary included |
| `published` / `updated` | ISO-8601 timestamps for v1 submission and the latest revision |
| `doi` | Publisher DOI once the paper is formally published, `null` while it is preprint-only |
| `journalRef` | Free-text journal citation, e.g. `J. Chem. Phys. 115, 1626 (2001)` |
| `comment` | Author's note, page count, figures, conference acceptance |
| `pdfUrl` / `absUrl` | Direct PDF link and the abstract landing page |
| `scrapedAt` | Run timestamp |

`doi`, `journalRef` and `comment` are `null` for most fresh preprints and populated for published work, in a 13-paper condensed-matter run, all 13 had a DOI and 11 had a journal reference.

A `RUN_SUMMARY` record lands in the key-value store:

```json
{
 "queriesRequested": 2,
 "queriesFailed": 1,
 "failures": [
 { "query": "all:not_a_real_term_zzzq AND badsyntax:(", "error": "arXiv rejected the query (400), check the search syntax" }
 ],
 "papersSaved": 13,
 "filters": {
 "searchQueries": ["cat:cond-mat.str-el AND ti:cusp", "all:not_a_real_term_zzzq AND badsyntax:("],
 "categories": [],
 "fromDate": null,
 "sortBy": "relevance",
 "maxResultsPerQuery": 210
 },
 "finishedAt": "2026-08-06T11:31:44.902Z"
}
```

### Use cases

**1. Track AI research output daily.** Schedule this every morning with a one-day floor; diff on `arxivId` to see only what landed overnight.

```json
{
 "searchQueries": ["cat:cs.LG OR cat:cs.CL OR cat:cs.AI"],
 "fromDate": "2026-08-05",
 "sortBy": "submittedDate",
 "maxResultsPerQuery": 500
}
```

**2. Build a paper-recommendation feed.** Pull abstracts for a topic, embed them, and rank against a user profile. `abstract` + `categories` + `absUrl` is everything a RAG index needs.

```json
{
 "searchQueries": [
 "all:\"retrieval augmented generation\"",
 "all:\"vector database\" AND abs:embedding",
 "ti:\"mixture of experts\""
 ],
 "categories": ["cs.CL", "cs.IR"],
 "maxResultsPerQuery": 300
}
```

**3. Monitor a research group's publications.** One query per author, deduplicated automatically, so co-authored papers appear once.

```json
{
 "searchQueries": ["au:\"Yoshua Bengio\"", "au:\"Yann LeCun\"", "au:\"Geoffrey Hinton\""],
 "sortBy": "submittedDate",
 "maxResultsPerQuery": 50
}
```

### Limits and behaviour

- **Atom XML, not JSON.** arXiv answers `application/atom+xml`. The Actor reads it with a small purpose-built extractor, CDATA sections are preserved verbatim and entities are decoded in a single pass, so `<` correctly becomes the text `<` rather than a stray tag.
- **1 request per 3 seconds.** arXiv asks for this and blocks clients that ignore it. The Actor spaces every request out accordingly, so a 1,000-paper run takes roughly 15 seconds of waiting on top of transfer time. Fetching 200 papers per request keeps that overhead low.
- **Pagination is automatic**, capped by `maxResultsPerQuery`, and stops early when arXiv's `totalResults` is exhausted.
- **Deep paging is arXiv's weak spot.** Very large offsets get slow and occasionally flaky. For more than ~30,000 results, slice by `fromDate` into date windows instead of raising the cap.
- **A failing query never aborts the run.** Bad syntax returns HTTP 400, is logged, and is recorded in `RUN_SUMMARY.failures`; other queries continue. The run only errors out if *every* query fails.
- **Transient errors are retried.** 429 and 5xx get three attempts with linear backoff. 400s and malformed feeds fail fast, because retrying them cannot help.
- **De-duplication is global.** A paper matched by several queries is saved once, so you are charged once.
- **Public data only.** No authentication, no personal data, no access-control bypass.

### FAQ

**Do I need a proxy?** No. Proxy configuration is not required to run this Actor.

**Do I need an account on arXiv?** No. You supply no credentials.

**What happens if a source is unavailable?** It is reported in `RUN_SUMMARY.failures` and the run continues with the remaining queries.

**Can I schedule it?** Yes, it is designed for scheduled runs. Pair `fromDate` with `sortBy: submittedDate` and diff on `arxivId`.

**Does it fetch full text?** No, metadata plus the abstract. `pdfUrl` gives you the direct link if you need the PDF.

**Why is `doi` null?** Because the preprint has not been formally published yet, or the authors never added the DOI. arXiv only reports what the submitter provides.

**Can I search by author?** Yes: `au:"Yoshua Bengio"`. Quote multi-word names.

**How do I find category codes?** They are on arxiv.org's category taxonomy page, `cs.LG` (machine learning), `cs.CL` (computation and language), `stat.ML`, `q-bio.NC`, `math.PR`, and so on.

**Can I integrate it with something else?** Yes, Apify API, client libraries, webhooks, scheduled runs, dataset exports (JSON/CSV/Excel) or MCP. Output is structured JSON.

# Actor input Schema

## `searchQueries` (type: `array`):

One or more arXiv queries in native syntax. Prefixes: all: (everything), ti: (title), abs: (abstract), au: (author), cat: (category). Combine with AND / OR / ANDNOT and group with parentheses. e.g. 'cat:cs.LG AND all:transformer' or 'au:Hinton AND abs:diffusion'. Each query is run and paginated separately.

## `categories` (type: `array`):

Restrict every query to these arXiv categories, ORed together and ANDed onto the query. e.g. cs.LG, cs.CL, cs.AI, stat.ML, math.PR, q-bio.NC. Leave empty to search all of arXiv. If you give categories but no query, the categories alone become the search.

## `fromDate` (type: `string`):

Lower bound on the submission date, as YYYY-MM-DD. Applied as arXiv's submittedDate range filter. Leave empty for no date floor.

## `sortBy` (type: `string`):

Ordering arXiv applies before pagination. Results are always returned newest/best first (descending).

## `maxResultsPerQuery` (type: `integer`):

Cap the number of papers saved per search query. Pages are 200 at a time and fetching stops as soon as the cap is met. Set 0 for no limit. be careful, some queries match hundreds of thousands of papers.

## Actor input object example

```json
{
  "searchQueries": [
    "all:\"large language model\"",
    "au:Bengio"
  ],
  "categories": [
    "cs.LG",
    "cs.CL"
  ],
  "fromDate": "2026-01-01",
  "sortBy": "submittedDate",
  "maxResultsPerQuery": 100
}
```

# Actor output Schema

## `items` (type: `string`):

Every record the run produced.

## `runsummary` (type: `string`):

The RUN\_SUMMARY record from the run's key-value store.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "searchQueries": [
        "cat:cs.LG AND all:transformer"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("arman-bd/arxiv-papers-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "searchQueries": ["cat:cs.LG AND all:transformer"] }

# Run the Actor and wait for it to finish
run = client.actor("arman-bd/arxiv-papers-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "searchQueries": [
    "cat:cs.LG AND all:transformer"
  ]
}' |
apify call arman-bd/arxiv-papers-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=arman-bd/arxiv-papers-scraper",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/t3bpD6iZqKBviFxSk/builds/0HkeGEXjG60chJOP5/openapi.json