arXiv Paper Search Scraper

Search public arXiv paper metadata by keyword, author, category, sort order, and publication date, then export titles, abstracts, authors, IDs, DOI, journal references, abstract links, and PDF links.

arXiv Paper Search Scraper is a lightweight research-data actor for literature review seed lists, paper monitoring, AI research feeds, newsletter curation, and dataset pipelines.

At a glance

• Input: required query, optional categories, author, date filters, sorting, and maxItems.
• Output: paper title, abstract, authors, arXiv ID, categories, published/updated dates, DOI, journal reference, abstract URL, and PDF URL.
• Controls: filter by arXiv categories, author name, publication date range, sort field, and sort order.
• Exports: download JSON, CSV, Excel, XML, RSS, or HTML, or read results through API/MCP.
• Best for: literature search, recent-paper monitoring, research newsletters, lab alerts, and paper metadata enrichment.

Input recipes

Use these concrete input patterns when starting a new run:

• Recent LLM Papers: query: "large language models", categories: ["cs.CL", "cs.AI"], sortBy: "submittedDate", sortOrder: "descending", maxItems: 25.
• Graph Neural Network Review: query: "graph neural networks", categories: ["cs.LG"], sortBy: "relevance", maxItems: 30.
• Author Watch: query: "representation learning", author: "Yoshua Bengio", maxItems: 20.
• Quantum Error Correction Scan: query: "quantum error correction", categories: ["quant-ph"], sortBy: "submittedDate", maxItems: 25.
• Date-Bounded AI Safety Search: query: "AI safety", categories: ["cs.AI"], startDate: "2025-01-01", endDate: "2025-12-31", maxItems: 25.

What can it do?

• Search arXiv metadata: query public arXiv records by keyword, phrase, category, author, and publication date.
• Export paper discovery rows: save titles, abstracts, authors, arXiv IDs, categories, dates, DOI, journal references, abstract links, and PDF links.
• Monitor recent research: sort by submitted or updated date and schedule the same query for recurring paper feeds.
• Build literature-review seed lists: collect focused records for research notes, lab alerts, newsletters, or dataset enrichment.
• Use as an arXiv data workflow: export CSV/Excel/JSON, run through the Apify API, or expose the Actor to AI agents through Apify MCP.

Input example

{
  "query": "large language models",
  "categories": ["cs.CL", "cs.AI"],
  "sortBy": "submittedDate",
  "sortOrder": "descending",
  "maxItems": 25
}

Output example

{
  "atomId": "http://arxiv.org/abs/2401.00001v1",
  "arxivId": "2401.00001v1",
  "paperTitle": "Example paper title",
  "summary": "Paper abstract text...",
  "authors": ["Jane Doe", "John Smith"],
  "primaryCategory": "cs.CL",
  "categories": ["cs.CL", "cs.AI"],
  "publishedAt": "2024-01-01T00:00:00Z",
  "updatedAt": "2024-01-02T00:00:00Z",
  "comment": "12 pages",
  "journalRef": null,
  "doi": null,
  "absUrl": "http://arxiv.org/abs/2401.00001v1",
  "pdfUrl": "http://arxiv.org/pdf/2401.00001v1",
  "query": "large language models",
  "position": 1
}

Input configuration

Setting	JSON key	Description
Search query	query	Required keyword or phrase searched across arXiv metadata.
arXiv categories	categories	Optional category codes such as cs.AI, cs.LG, cs.CL, stat.ML, or quant-ph.
Author name	author	Optional author filter, such as Yoshua Bengio or Fei-Fei Li.
Maximum papers	maxItems	Maximum number of paper rows to save, from 1 to 5,000.
Resume from result index	startIndex	Optional zero-based offset for continuing a time-bounded run. Copy nextStart from the PENDING_WORK record.
Sort by	sortBy	relevance, lastUpdatedDate, or submittedDate.
Sort order	sortOrder	descending or ascending.
Published after	startDate	Optional publication-date lower bound in YYYY-MM-DD format.
Published before	endDate	Optional publication-date upper bound in YYYY-MM-DD format.

Output fields

Field	Description
atomId	Canonical arXiv Atom ID or abstract URL.
arxivId	Clean arXiv identifier.
paperTitle	Paper title.
summary	Abstract text.
authors	List of author names.
primaryCategory	Main arXiv category when available.
categories	All category tags returned for the paper.
publishedAt	Initial publication timestamp.
updatedAt	Latest update timestamp.
comment	Optional author comment.
journalRef	Optional journal reference.
doi	Optional DOI.
absUrl	arXiv abstract page URL.
pdfUrl	arXiv PDF URL when available.
query	Search query used for the run.
position	Result position after filtering.

Pricing

This Actor uses Apify pay-per-event pricing. The prices below come from the current Actor pricing configuration. Apify public plans map to Store discount tiers, so the table shows both the user-facing plan context and the pricing tier name. The final price shown in Apify depends on the user account plan and any custom agreement.

Event	What is charged	Price
start	One-time fee charged when a run starts. Covers fixed startup cost (init, proxy warmup, first HTTP setup).	$0.005

Event	What is charged	Free / no discount	Starter / Bronze	Scale / Silver	Business / Gold	Custom / Platinum	Custom / Diamond
result	Per arXiv paper saved to the dataset	$0.575 / 1,000	$0.5 / 1,000	$0.39 / 1,000	$0.3 / 1,000	$0.2 / 1,000	$0.14 / 1,000

Apify may also charge platform usage for compute, storage, proxies, or data transfer outside this Actor pricing. Check the Actor run and the Apify Pricing tab for the exact cost shown to your account.

Common workflows

• Literature review seed list: search a focused topic and export titles, abstracts, authors, and PDF links to a spreadsheet.
• Recent-paper monitoring: sort by submittedDate descending and schedule the same query weekly.
• Author tracking: combine query and author to follow a researcher or lab topic.
• Category feeds: use arXiv category codes to keep AI, NLP, computer vision, statistics, or quantum searches focused.
• Research enrichment: send abstracts, authors, categories, and URLs to a database, vector index, or newsletter workflow.

Tips for best results

• Use precise terms: retrieval augmented generation is easier to review than AI.
• Add categories: category filters reduce off-topic matches for broad topics.
• Use date filters carefully: startDate and endDate are applied to publication dates after arXiv returns results.
• Respect arXiv pacing: larger runs take longer because the actor paginates through public arXiv API results.
• Check optional fields: DOI, journal reference, and comments are present only when authors provide them.

Limits and caveats

• Public metadata only: the actor exports arXiv metadata, not paywalled paper content.
• No citation graph: it does not collect citation counts or references.
• Date filtering is post-fetch: very broad queries with narrow dates may scan more rows than they save.
• Normal workload design: keep queries focused for repeatable monitoring and predictable run times.

API usage

Node.js example:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('fetch_cat/arxiv-paper-search-scraper').call({
  query: 'large language models',
  categories: ['cs.CL', 'cs.AI'],
  sortBy: 'submittedDate',
  sortOrder: 'descending',
  maxItems: 25
});
console.log(run.defaultDatasetId);

Python example:

from apify_client import ApifyClient

client = ApifyClient('YOUR_APIFY_TOKEN')
run = client.actor('fetch_cat/arxiv-paper-search-scraper').call(run_input={
    'query': 'large language models',
    'categories': ['cs.CL', 'cs.AI'],
    'sortBy': 'submittedDate',
    'sortOrder': 'descending',
    'maxItems': 25,
})
print(run['defaultDatasetId'])

cURL example:

curl -X POST 'https://api.apify.com/v2/acts/fetch_cat~arxiv-paper-search-scraper/runs?token=YOUR_APIFY_TOKEN' \
  -H 'Content-Type: application/json' \
  -d '{"query":"large language models","categories":["cs.CL","cs.AI"],"sortBy":"submittedDate","sortOrder":"descending","maxItems":25}'

MCP and AI agents

Use the official Apify MCP server for agent workflows. The default server can search and run Actors; the focused URL exposes only this Actor.

Claude CLI add command:

$claude mcp add apify-arxiv --transport http "https://mcp.apify.com?tools=fetch_cat/arxiv-paper-search-scraper"

JSON config block:

{
  "mcpServers": {
    "apify-arxiv": {
      "url": "https://mcp.apify.com?tools=fetch_cat/arxiv-paper-search-scraper"
    }
  }
}

Example prompts for MCP usage:

• "Use arXiv Paper Search Scraper to find 25 recent large language model papers in cs.CL and cs.AI, then summarize titles, authors, categories, and PDF links."
• "Run an arXiv search for graph neural networks in cs.LG and return a CSV-ready table with arXiv IDs, authors, abstracts, and PDF URLs."
• "Monitor new AI safety papers in cs.AI from 2025 and list DOI, journal reference, and PDF link when available."

FAQ

Does it require an arXiv account or API key?

No. It uses public arXiv metadata.

Can I export papers to CSV or Excel?

Yes. Apify datasets export to CSV, Excel, JSON, JSONL, XML, RSS, and HTML.

Can I search by author?

Yes. Use author along with a required query. The actor builds an arXiv search query with both filters.

Why are DOI or journal fields empty?

Not every arXiv paper has DOI or journal reference metadata. Missing optional values are saved as null.

Is this an arXiv API alternative?

It is a packaged Apify workflow around public arXiv metadata search, not an official arXiv API.

Related actors

• Google Scholar Profiles Scraper
• GitHub Repositories Search Scraper
• Hacker News Search Scraper

Support

If a run fails, returns no data, or a field looks wrong, open an issue from the Actor page.

Please include the Apify run ID or run URL, input JSON, one example public URL, query, or input item, what you expected, and what the dataset returned. Small reproducible inputs make parsing or site-layout issues much faster to fix.

Privacy and data handling

This Actor runs with Apify limited permissions and only processes data needed for the documented run. It uses the inputs you provide and the public records needed to produce the documented dataset to produce the output dataset and sends requests to arXiv public search/API endpoints; results are stored in Apify run storage for your account. FetchCat does not use your inputs or outputs for advertising, does not use them for model training, and does not retain them outside the Apify run except for transient support debugging when you explicitly share run details. You are responsible for using the Actor lawfully, respecting the target site's terms, and avoiding unnecessary personal or sensitive data in inputs.