arXiv Paper Search Scraper

Search arXiv papers by keyword, author, category, sort order, and publication date. Export clean paper metadata including titles, abstracts, authors, arXiv IDs, categories, DOI, journal reference, abstract links, and PDF links.

What does arXiv Paper Search Scraper do?

arXiv Paper Search Scraper helps you collect public research paper metadata from arXiv into an Apify dataset.

Use it to monitor new papers, build literature-review datasets, track authors, watch AI categories, or export paper links for newsletters and research workflows.

Who is it for?

• 🧑‍🔬 AI researchers tracking new papers in machine learning, NLP, robotics, or computer vision.
• 📚 Academic teams building repeatable literature search workflows.
• 📰 Newsletter writers curating recent papers for readers.
• 📈 Competitive intelligence teams monitoring research trends.
• 🧩 Data engineers feeding paper metadata into databases, dashboards, or vector indexes.

Why use it?

• Official public arXiv metadata source
• No login required
• Clean JSON output
• Keyword, category, author, and date filters
• Works with Apify integrations, API, datasets, webhooks, and MCP
• Low-cost HTTP actor designed for repeatable monitoring

What data can you extract?

Field	Description
atomId	Canonical arXiv Atom ID / abstract URL
arxivId	Clean arXiv identifier
paperTitle	Paper title
summary	Abstract text
authors	List of author names
primaryCategory	Main arXiv category
categories	All category tags
publishedAt	Initial publication timestamp
updatedAt	Latest update timestamp
comment	Optional author comment
journalRef	Optional journal reference
doi	Optional DOI
absUrl	Abstract page URL
pdfUrl	PDF URL
query	Query used in the run
position	Result position

How much does it cost to scrape arXiv papers?

The actor uses pay-per-event pricing:

• A small start event per run
• A per-paper result event for each paper saved

You control cost with maxItems. Start with 25 to test your query, then increase for larger literature searches.

Quick start

1. Enter a search query such as machine learning.
2. Optionally add arXiv categories such as cs.AI or cs.LG.
3. Optionally add an author name.
4. Choose sorting and date filters.
5. Run the actor.
6. Export the dataset as JSON, CSV, Excel, XML, or HTML.

Input options

query

Required. A keyword or phrase to search across arXiv paper metadata.

Examples:

• large language models
• graph neural networks
• diffusion models
• quantum error correction

categories

Optional list of arXiv category codes.

Common examples:

• cs.AI
• cs.LG
• cs.CL
• cs.CV
• stat.ML
• math.OC
• quant-ph

author

Optional author filter. Use names such as Yoshua Bengio, Fei-Fei Li, or Geoffrey Hinton.

sortBy

Supported values:

• relevance
• lastUpdatedDate
• submittedDate

sortOrder

Supported values:

• descending
• ascending

startDate and endDate

Optional publication-date filters in YYYY-MM-DD format.

maxItems

Maximum number of papers to save. Use a small number for tests and a larger value for monitoring jobs.

Example input

{
  "query": "large language models",
  "categories": ["cs.CL", "cs.AI"],
  "sortBy": "submittedDate",
  "sortOrder": "descending",
  "maxItems": 50
}

Example output

{
  "atomId": "http://arxiv.org/abs/2401.00001v1",
  "arxivId": "2401.00001v1",
  "paperTitle": "Example paper title",
  "summary": "Paper abstract text...",
  "authors": ["Jane Doe", "John Smith"],
  "primaryCategory": "cs.CL",
  "categories": ["cs.CL", "cs.AI"],
  "publishedAt": "2024-01-01T00:00:00Z",
  "updatedAt": "2024-01-02T00:00:00Z",
  "comment": "12 pages",
  "journalRef": null,
  "doi": null,
  "absUrl": "http://arxiv.org/abs/2401.00001v1",
  "pdfUrl": "http://arxiv.org/pdf/2401.00001v1",
  "query": "large language models",
  "position": 1
}

Tips for better results

• Use exact research terms instead of very broad words.
• Add categories when you want topic-specific output.
• Use submittedDate + descending for recent-paper monitoring.
• Use relevance for literature-discovery searches.
• Keep maxItems low until you confirm the query matches your intent.

Category examples

Topic	Categories
AI	cs.AI, cs.LG
NLP	cs.CL
Computer vision	cs.CV
Statistics	stat.ML
Optimization	math.OC
Quantum physics	quant-ph

Integrations

Use this actor with:

• Google Sheets exports for research lists
• Webhooks for new-paper alerts
• Apify datasets for BI pipelines
• Make or Zapier workflows for newsletters
• Vector databases for semantic search
• Slack or Discord bots for lab updates

API usage

Node.js

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('fetch_cat/arxiv-paper-search-scraper').call({
  query: 'large language models',
  categories: ['cs.CL'],
  maxItems: 25
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Python

from apify_client import ApifyClient

client = ApifyClient('YOUR_APIFY_TOKEN')
run = client.actor('fetch_cat/arxiv-paper-search-scraper').call(run_input={
    'query': 'large language models',
    'categories': ['cs.CL'],
    'maxItems': 25,
})
items = client.dataset(run['defaultDatasetId']).list_items().items
print(items)

cURL

curl -X POST 'https://api.apify.com/v2/acts/fetch_cat~arxiv-paper-search-scraper/runs?token=YOUR_APIFY_TOKEN' \
  -H 'Content-Type: application/json' \
  -d '{"query":"large language models","categories":["cs.CL"],"maxItems":25}'

MCP: use with AI assistants

You can run this actor from MCP-compatible clients through Apify MCP.

Claude Code

Add the Apify MCP server with this actor enabled:

$claude mcp add apify-arxiv "https://mcp.apify.com/?tools=fetch_cat/arxiv-paper-search-scraper"

Claude Desktop

Add this server entry to your Claude Desktop MCP configuration and set your Apify token in the environment:

{
  "mcpServers": {
    "apify-arxiv": {
      "command": "npx",
      "args": ["-y", "mcp-remote", "https://mcp.apify.com/?tools=fetch_cat/arxiv-paper-search-scraper"],
      "env": {
        "APIFY_TOKEN": "YOUR_APIFY_TOKEN"
      }
    }
  }
}

Example MCP prompts

Example prompts you can ask in Claude Code, Claude Desktop, or another MCP-compatible assistant after adding the Apify MCP server:

• "Using the Apify MCP tool fetch_cat/arxiv-paper-search-scraper, find 25 new arXiv papers about retrieval augmented generation and give me the dataset URL."
• "Run the arXiv Paper Search Scraper via MCP for category cs.CL, query multilingual LLM, sort by submitted date, and summarize the newest titles."
• "Use MCP to collect 100 recent papers about graph neural networks, then group the returned authors and PDF links by primary category."
• "Create a weekly literature-monitoring workflow: run this actor for agentic AI, export CSV, and tell me which papers are likely relevant to enterprise search."

Common workflows

Literature review seed list

Run a focused keyword query and export the result as CSV for spreadsheet review.

Research newsletter

Schedule a run for a topic and use the dataset in a newsletter automation.

Lab monitoring

Track author names or categories and send new records to Slack.

Trend analysis

Export results by category and analyze publication dates, terms, and author networks.

Data freshness

arXiv metadata changes as new papers and revisions are submitted. Use sorting by submitted or updated date for monitoring jobs.

Limits and reliability

The actor respects practical pagination and is designed for normal research workloads. Very broad queries may have many thousands of matches. Use category and date filters to keep results focused.

FAQ

Why did I get fewer papers than maxItems?

The source may have fewer matching papers after your filters. Date filters can also reduce the saved item count.

Why are DOI or journal fields empty?

Not every arXiv paper has DOI or journal reference metadata. Empty optional fields are returned as null.

How do I find recent AI papers?

Use a query such as artificial intelligence, categories like cs.AI and cs.LG, and set sortBy to submittedDate with descending order.

Legality and ethics

This actor collects public metadata from arXiv. Users are responsible for complying with applicable laws, arXiv terms, and fair-use expectations. Do not use exported data to spam authors or misrepresent research work.

Related actors

• GitHub Repository Search Scraper for software repository discovery.
• Google Patents Search Scraper for patent research workflows.
• Hacker News Search Scraper for technology discussion monitoring.

Changelog

0.1

Initial version with keyword, category, author, date, sorting, and clean arXiv paper metadata output.

Support

If a query behaves unexpectedly, share the run ID and input so the issue can be reproduced.

Output formats

Apify datasets can be exported as JSON, JSONL, CSV, Excel, XML, RSS, or HTML.

Scheduling

Use Apify schedules to run the same query daily, weekly, or monthly.

Webhooks

Attach a webhook to trigger downstream processing when a run succeeds.

Dataset retention

Dataset retention follows your Apify account settings and run storage configuration.

Performance notes

Small runs normally finish quickly. Larger runs take longer because the actor paginates through matching papers and saves normalized records.

Versioning

Use the latest build for production workflows unless a specific task or integration requires a fixed version.