arXiv Paper Scraper

Scrape academic papers from arXiv.org. Search by keyword, browse categories, or get latest papers. Extract titles, abstracts, authors, PDF links, and citation data via arXiv API.

Use cases

• Track new preprints in a research area on a schedule
• Build a literature-review corpus with abstracts and PDF links
• Feed a RAG pipeline with fresh academic text
• Monitor a competitor lab's or author's publication output
• Detect emerging topics by watching category submission volume

Input

Parameter	Type	Required	Default	Description
searchQuery	string	Yes	"machine learning"	Search term to find papers on arXiv (e.g., 'machine learning', 'quantum computing')
category	string	No	—	arXiv category filter (e.g., 'cs.AI', 'cs.LG', 'physics')
sortBy	string	No	"relevance"	Sort order for results Allowed: relevance, lastUpdatedDate, submittedDate.
maxResults	integer	No	20	Maximum number of papers to scrape

Example input

{
  "searchQuery": "machine learning",
  "sortBy": "relevance",
  "maxResults": 20
}

Output

The exact fields depend on the mode you run. This is real output from an actual run of this Actor:

{
  "title": "Changing Data Sources in the Age of Machine Learning for Official Statistics",
  "authors": [
    "Cedric De Boom",
    "Michael Reusens"
  ],
  "abstract": "Data science has become increasingly essential for the production of official statistics, as it enables the automated collection, processing, and anal…",
  "categories": [
    "stat.ML",
    "cs.LG"
  ],
  "primaryCategory": "stat.ML",
  "arxivId": "2306.04338",
  "url": "http://arxiv.org/abs/2306.04338v1",
  "pdfUrl": "https://arxiv.org/pdf/2306.04338v1",
  "publishedDate": "2023-06-07T11:08:12Z",
  "updatedDate": "2023-06-07T11:08:12Z",
  "commentText": "Presented at UNECE Machine Learning for Official Statistics Workshop 2023",
  "journalRef": "UNECE Machine Learning for Official Statistics Workshop 2023"
}

Field	Type
title	string
authors	array
abstract	string
categories	array
primaryCategory	string
arxivId	string
url	string
pdfUrl	string
publishedDate	string
updatedDate	string
commentText	string
journalRef	string

The dataset also ships a preset table view (Papers), so the key columns are readable straight away in Apify Console, and exportable to JSON, CSV, Excel, or XML.

How to run it

In Apify Console — open the Actor, fill in the input form, click Start, then download the results from the Dataset tab.

With the JavaScript client

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });

const run = await client.actor('cloud9_ai/arxiv-paper-scraper').call({
  "searchQuery": "machine learning",
  "sortBy": "relevance",
  "maxResults": 20
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

With the Python client

from apify_client import ApifyClient

client = ApifyClient('YOUR_APIFY_TOKEN')

run = client.actor('cloud9_ai/arxiv-paper-scraper').call(run_input={
  "searchQuery": "machine learning",
  "sortBy": "relevance",
  "maxResults": 20
})

for item in client.dataset(run['defaultDatasetId']).iterate_items():
    print(item)

With the API — POST https://api.apify.com/v2/acts/cloud9_ai~arxiv-paper-scraper/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN with the input JSON as the body.

Notes and limits

• No API key, account, or login is needed — just the input above.
• maxResults caps how much a single run collects, which is also what caps the run's cost.
• Requests are paced and failed requests are retried automatically, so runs stay inside the source's rate limits.
• Only publicly available data is collected. How you use the output is your responsibility, including the source's terms of use and any applicable data-protection law.

Support

Found a bug, or need a field that isn't in the output? Open an issue on the Issues tab of this Actor in Apify Console. Issues there are read and answered.

License

Apache-2.0