arXiv Papers Scraper

Scrape academic papers from arXiv by category, keyword, or author. Extract titles, authors, abstracts, PDF URLs, DOIs, categories, publication dates, and journal references as structured JSON. No API key required — uses the free arXiv API.

This actor is ideal for building AI/ML training datasets, literature reviews, citation analysis, and research monitoring. It uses CheerioCrawler for fast HTTP-based XML parsing, automatically paginates through arXiv results, and supports sorting and filtering so you get exactly the data you need.

Features

• Search by keyword (searchQuery), category (category), or author (author) — or combine all three
• Automatic pagination through arXiv results (up to 100 per page)
• Sort by relevance, lastUpdatedDate, or submittedDate
• Extracts: paper title, authors, abstract, published/updated dates, PDF URL, arXiv ID, primary category, all categories, DOI, comments, and journal reference
• Optional includeAbstract toggle to reduce output size
• Polite rate limiting (single concurrency) respecting arXiv API guidelines
• No API key, authentication, or login required

Input

Configure the actor through the input schema. All fields are optional with sensible defaults.

Field	Type	Default	Description
searchQuery	String	—	Free-text keyword search across titles and abstracts
category	String	cs.AI	arXiv subject category (e.g. cs.AI, cs.CL, stat.ML, cs.CV, math.CO)
author	String	—	Author name to search for
sortBy	String	relevance	Sort order: relevance, lastUpdatedDate, or submittedDate
maxItems	Integer	50	Maximum number of papers to scrape
includeAbstract	Boolean	true	Include full abstract text in results

Example input

{
    "category": "cs.AI",
    "searchQuery": "large language model",
    "sortBy": "submittedDate",
    "maxItems": 100
}

Search for papers by a specific author:

{
    "author": "Yann LeCun",
    "maxItems": 50,
    "includeAbstract": true
}

Output

Results are stored in the default dataset and can be exported as JSON, CSV, Excel, or via API.

{
    "id": "2401.12345v1",
    "title": "A Novel Approach to Large Language Model Training",
    "authors": ["Jane Smith", "John Doe"],
    "abstract": "In this paper we present...",
    "published": "2024-01-22T00:00:00Z",
    "updated": "2024-01-23T00:00:00Z",
    "pdfUrl": "https://arxiv.org/pdf/2401.12345v1",
    "url": "https://arxiv.org/abs/2401.12345v1",
    "primaryCategory": "cs.AI",
    "categories": ["cs.AI", "cs.CL", "stat.ML"],
    "doi": "10.1234/example.2024.12345",
    "comments": "15 pages, 3 figures",
    "journalRef": "Nature Machine Intelligence 2024"
}

Popular arXiv categories

Category	Field
cs.AI	Computer Science — Artificial Intelligence
cs.CL	Computer Science — Computation and Language (NLP)
cs.CV	Computer Science — Computer Vision
cs.LG	Computer Science — Machine Learning
stat.ML	Statistics — Machine Learning
math.CO	Mathematics — Combinatorics
physics	Physics (general)
q-bio	Quantitative Biology
econ	Economics

Browse all categories at arxiv.org/category_taxonomy.

Use cases

• AI/ML training datasets — collect abstracts and metadata for fine-tuning or benchmarking
• Literature reviews — systematically gather papers on a topic with full metadata
• Citation analysis — track publications by author or across a category
• Research monitoring — get the latest papers in your field via scheduled runs
• Trend analysis — identify emerging topics and active research areas
• Data pipelines — feed structured paper data into RAG systems, knowledge graphs, or databases

How to use

1. Open the actor on the Apify Store.
2. Click Try for free or Start with Free Trial.
3. Set the search criteria — category, keyword, and/or author. Adjust maxItems and sort order as needed.
4. Click Start and wait for the run to complete.
5. Export the results to JSON, CSV, or connect via the Apify API.

For scheduled runs, use Apify's scheduler to scrape arXiv daily or weekly and auto-export to your webhook, database, or Google Sheets. The arXiv API updates continuously, so frequent runs will capture newly submitted papers.

Notes

• The arXiv API is free and requires no API key.
• Results are limited to arXiv-hosted papers only (preprints).
• Rate limiting is polite (1 request at a time) to comply with arXiv's API usage guidelines.