ArXiv Papers Scraper

Search and export arxiv papers as structured records for literature monitoring, systematic review preparation, and research datasets.

The Actor tries the official public arXiv Atom API first, with bounded public HTML retrieval when the API is unavailable. Search by topic, author, category, submission date, or known paper ID, then save titles, abstracts, ordered authors, categories, dates, available DOI metadata, and PDF links to an Apify dataset. Each record identifies its metadata source and date precision.

No arXiv login, cookie, API key, or proxy is required.

What can ArXiv Papers Scraper do?

• Search titles, abstracts, and author names with a topic query.
• Filter papers by one or more arXiv categories.
• Filter by author and first-submission date.
• Retrieve known papers by arXiv ID, abstract URL, or PDF URL.
• Sort by submission date, last update, or relevance.
• Export up to 500 unique papers per run.
• Normalize Atom metadata into integration-ready JSON.
• Provide canonical abstract and PDF links.
• Preserve DOI, journal reference, and submission comments when available.
• Support repeatable scheduled literature-monitoring runs.

Who is it for?

Researchers build focused reading lists without copying metadata by hand.

Librarians and research-support teams collect bounded topic or category exports for discovery workflows.

Data scientists create reproducible metadata datasets for analysis, classification, or citation-pipeline inputs.

R&D teams schedule a query and compare exports to identify newly submitted work.

Developers and AI agents retrieve clean records through the Apify API or MCP instead of parsing Atom XML.

This Actor searches public arXiv metadata. It does not submit papers, access private accounts, or bypass arXiv login.

Why use this Actor?

The source API is public, but production workflows still need query construction, pagination, validation, rate-limit handling, metadata normalization, deduplication, storage, and integrations.

This Actor packages those steps into one reusable task:

1. validate the requested search scope;
2. build a structured arXiv query;
3. fetch bounded Atom pages politely, recovering through public HTML when needed;
4. retry temporary HTML/network failures within the requested bound;
5. normalize and deduplicate papers across retrieval routes;
6. save each useful record through the SDK's charge-aware dataset path;
7. stop when the result cap or spending allowance is reached and write an export summary.

It fails clearly rather than returning fabricated or cached research records after an upstream error.

Input parameters

Field	Type	Description
query	string	Topic text matched across titles, abstracts, and author names.
author	string	Author-name filter, such as Geoffrey Hinton.
categories	string[]	arXiv category codes; multiple values are combined with OR.
paperIds	string[]	Known IDs or arXiv abs/PDF URLs.
dateFrom	date	Earliest first-submission date, YYYY-MM-DD.
dateTo	date	Latest first-submission date, YYYY-MM-DD.
sortBy	enum	submittedDate, lastUpdatedDate, or relevance.
sortOrder	enum	descending or ascending.
maxItems	integer	Maximum unique records, from 1 to 500.
maxRequestRetries	integer	Temporary-request retries, from 0 to 5.

Provide at least one of query, author, categories, or paperIds.

Search filters are combined with AND. Multiple categories are combined with OR.

When paperIds are supplied, the Actor retrieves those exact records and applies the topic, author, category, and date filters locally as well.

Quick start

1. Open the Actor in Apify Console.
2. Enter a topic such as graph neural networks.
3. Optionally add cs.LG to arXiv categories.
4. Choose a maximum number of papers.
5. Click Start.
6. Open the Dataset tab to preview or download JSON, CSV, Excel, XML, or RSS.

Example input:

{
  "query": "graph neural networks",
  "categories": ["cs.LG"],
  "sortBy": "submittedDate",
  "sortOrder": "descending",
  "maxItems": 25
}

Search by author

Use the dedicated author field instead of embedding arXiv query syntax:

{
  "author": "Geoffrey Hinton",
  "sortBy": "submittedDate",
  "sortOrder": "descending",
  "maxItems": 20
}

The filter is sent to the official author index. The returned record keeps the complete ordered author list.

Search arXiv math and science categories

Category codes are source-specific identifiers such as:

• cs.AI — Artificial Intelligence
• cs.CL — Computation and Language
• cs.LG — Machine Learning
• math.OC — Optimization and Control
• quant-ph — Quantum Physics
• stat.ML — Machine Learning in Statistics

Example category export:

{
  "categories": ["math.OC", "stat.ML"],
  "sortBy": "submittedDate",
  "maxItems": 100
}

Consult arXiv's current category taxonomy when choosing codes. Invalid category shapes are rejected before a source request.

Monitor papers in a submission-date window

Use a bounded date range for repeatable reviews:

{
  "query": "retrieval augmented generation",
  "categories": ["cs.AI"],
  "dateFrom": "2025-01-01",
  "dateTo": "2025-12-31",
  "sortBy": "submittedDate",
  "sortOrder": "descending",
  "maxItems": 100
}

For recurring monitoring, schedule this Actor in Apify and move the date window forward. Downstream automation can join by basePaperId, then compare version and updatedAt with an earlier dataset. A null version is unresolved, not proof that no revision exists.

The Actor does not itself send alerts or calculate changes between runs.

Retrieve known paper IDs

Use IDs when you already have a reading list:

{
  "paperIds": [
    "1706.03762",
    "https://arxiv.org/abs/2303.08774",
    "https://arxiv.org/pdf/2302.13971.pdf"
  ],
  "maxItems": 3
}

Versioned identifiers such as 1706.03762v7 are supported.

Legacy-style arXiv identifiers are also accepted.

Output fields

Field	Meaning
paperId	arXiv ID, including the version when resolved by the source.
basePaperId	Stable unversioned arXiv ID for joining scheduled exports.
version	Resolved integer revision number, or null when unavailable.
metadataSource	atom, html-abs, or html-search.
datePrecision	second, day, or unavailable; HTML search days are represented at midnight UTC.
title	Normalized paper title.
abstract	Normalized abstract text.
authors	Ordered array of author names.
categories	All assigned arXiv category codes.
primaryCategory	Primary category code.
publishedAt	Initial submission time, or initial submission day for HTML search.
updatedAt	Selected/current revision time or day; empty string if unavailable, never fabricated from another revision.
pdfUrl	Direct PDF URL.
absUrl	Canonical abstract-page URL.
doi	DOI when arXiv provides one; otherwise null.
journalReference	Journal reference when available.
comment	Source submission comment when available.
query	Structured query used, or null for ID retrieval.
fetchedAt	UTC extraction timestamp.

Example output

A real topic-search result has this shape:

{
  "paperId": "2608.27413v1",
  "basePaperId": "2608.27413",
  "version": 1,
  "metadataSource": "atom",
  "datePrecision": "second",
  "title": "Scaling Graph Neural Networks for Friend Recommendation: Multi-Hash User Embeddings and Temporal Neighbor Sampling",
  "abstract": "Friend recommendation is inherently graph-structured...",
  "authors": ["Maksim Utushkin", "Andrei Ovsiannikov", "Alexander D'yakonov"],
  "categories": ["cs.IR", "cs.LG", "cs.SI"],
  "primaryCategory": "cs.IR",
  "publishedAt": "2026-08-27T17:41:33Z",
  "updatedAt": "2026-08-27T17:41:33Z",
  "pdfUrl": "https://arxiv.org/pdf/2608.27413v1",
  "absUrl": "https://arxiv.org/abs/2608.27413v1",
  "doi": null,
  "journalReference": null,
  "comment": "12 pages, 4 figures, 8 tables...",
  "query": "all:\"graph neural networks\" AND (cat:cs.LG)",
  "fetchedAt": "2026-08-28T06:36:49.711Z"
}

Optional metadata remains null when the selected source surface does not supply it. HTML search can expose less metadata than the abstract page or Atom entry. A DOI is source metadata, not evidence of peer review.

The key-value store's EXPORT-SUMMARY records saved/scanned counts, retrieval routes, stop reason (sourceExhausted, maxItems, spendingLimit, scanLimit, or aborted), and fallback limitations. A scan-limit stop is a partial bounded search, not proof that no other matching papers exist. ID retrieval restores requested order before filtering and saving, validates exact requested versions, and recovers omitted/mismatched Atom entries from abstract pages. An unrecoverable requested entry fails with its input position; HTTP 404 is reported as not found, distinct from a retrieval/parser failure. Filtered records and entries beyond the result/spending cap are not missing-source errors. Graceful abort/migration stops later emissions while preserving an already saved/charged prefix; no atomic billing guarantee is made for hard kills.

How much does it cost to export arXiv papers?

The Actor uses pay per event:

• $0.001 when a run starts;
• $0.0008 per saved paper on the BRONZE tier.

Approximate BRONZE examples:

Saved papers	Estimated price
5	about 0.005 USD
25	about 0.021 USD
100	about 0.081 USD
500	about 0.401 USD

A no-result run pays only the start fee. Runtime validation happens before the start charge and source requests. The Actor checks allowance before starting, saves and charges useful records incrementally, and stops when the remaining paper allowance is exhausted. A spending-limited run can return fewer than maxItems records. Storage failures propagate before paper charging; already saved rows remain accessible if a later source request fails. The platform's live tariff remains the final billing authority.

Higher Apify subscription tiers receive the lower item prices shown in Console.

Export to spreadsheets and data pipelines

Every result is stored in the default Apify dataset.

From the Dataset tab you can download:

• JSON for applications and archives;
• CSV or Excel for analysts;
• XML for legacy systems;
• RSS for compatible readers.

Use Apify integrations to send completed datasets to Google Drive, Slack, webhooks, Zapier, Make, or your own service.

For durable pipelines, use basePaperId as the stable join key. Inspect resolved version and updatedAt for changes, respecting datePrecision. paperId may include a revision and is not a stable cross-version join key.

Schedule literature monitoring

A practical recurring workflow is:

1. create a saved task with a topic, category, and date range;
2. add a daily or weekly schedule;
3. receive a run-finished webhook;
4. fetch the dataset through the API;
5. compare paper IDs with your existing catalog;
6. route new papers to a review queue.

Keep date windows explicit for reproducible monitoring.

ArXiv publication timing and source API indexing determine freshness.

Run with the Apify API

cURL

curl -X POST \
  "https://api.apify.com/v2/acts/automation-lab~arxiv-paper-search-export/runs?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"query":"graph neural networks","categories":["cs.LG"],"maxItems":25}'

JavaScript

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/arxiv-paper-search-export').call({
  query: 'graph neural networks',
  categories: ['cs.LG'],
  maxItems: 25,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Python

from apify_client import ApifyClient

client = ApifyClient(token="YOUR_APIFY_TOKEN")
run = client.actor("automation-lab/arxiv-paper-search-export").call(run_input={
    "query": "graph neural networks",
    "categories": ["cs.LG"],
    "maxItems": 25,
})
items = client.dataset(run["defaultDatasetId"]).list_items().items
print(items)

Never commit an Apify token to source control.

Use through MCP

Add this Actor to Claude Code:

claude mcp add --transport http apify \
  "https://mcp.apify.com?tools=automation-lab/arxiv-paper-search-export"

Claude Desktop, Cursor, and VS Code setup

Use this MCP configuration in Claude Desktop, Cursor, or VS Code:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=automation-lab/arxiv-paper-search-export"
    }
  }
}

Example prompts:

• "Find five recent cs.LG papers about graph neural networks and summarize their metadata."
• "Export papers by Geoffrey Hinton with their arXiv PDF links."
• "Collect quant-ph papers from this date range for my literature review."

MCP invokes the same validated Actor input and returns the same dataset records.

Reliability and source etiquette

The implementation uses Atom first (10-second timeout), then independently served public HTML search or abstract pages when Atom is unavailable. HTML requests have a 30-second timeout and bounded increasing delays for transient errors. It never emits cached or fabricated papers.

For multi-page runs it waits at least three seconds between source requests in line with arXiv API guidance.

The Actor does not need browser rendering, residential proxies, or account sessions.

Temporary rate limits can still occur. A blocked/timed-out Atom route switches to HTML rather than repeatedly retrying the same blocked endpoint. If both routes fail or a results page is malformed, the run fails clearly. Earlier saved papers remain available.

HTML search supports source-side topic/author and original-submission date scoping where available; category and phrase matching are checked locally. It scans at most ten pages (500 candidates). Sparse filters may yield fewer than the requested cap; EXPORT-SUMMARY discloses scan exhaustion. HTML submission ordering follows the source's most-recent-submission order and is not guaranteed to be original-submission order. HTML does not support exact lastUpdatedDate or ascending-relevance ordering: those requests fail clearly if Atom is unavailable. Known-ID mode preserves input order and does not apply search sorting.

Diagnostic privacy

Failed operations send sanitized diagnostic input, exceptions, and Actor/build/run IDs to our private GlitchTip service for repair. Secret fields and URL query strings are removed. Reports are retained for 30 days. No private paper body, credentials, or raw upstream response is included in diagnostics. This Actor uses no AI model or external enrichment service.

Public paper metadata includes author names; it does not collect author email addresses or account credentials. Queries are sent to arXiv. Output and the export summary stay in your run-owned Apify dataset/key-value store, under your Apify retention settings; delete those stores to remove them. No cross-run cache or author profile is created. Apify stores run logs under its account retention rules; diagnostic reports are removed after 30 days.

For help, open an issue on this Actor's Apify Store page with a non-sensitive example and run link. Do not post tokens or private credentials.

Limits

• maxItems is capped at 500 per run.
• Search is governed by arXiv's index and query semantics.
• Topic text is treated as one quoted phrase for precise matching.
• Date filters use first-submission time, not the latest revision time.
• DOI, journal, and comment fields are optional source-surface metadata.
• HTML search has day-precision dates and may not resolve versions; abstract pages expose precise submission history when available.
• HTML fallback scans are bounded and can be partial; inspect EXPORT-SUMMARY before interpreting an empty or short export.
• This Actor exports metadata and links; it does not download PDF files.
• It does not calculate citations, affiliations, full-text entities, or semantic similarity.
• It does not send alerts or merge datasets across runs.

Troubleshooting

The run says I must provide a search field

Supply at least one of query, author, categories, or paperIds. The Actor intentionally rejects unbounded blank searches.

My category is rejected

Use an arXiv category code such as cs.AI, math.OC, or quant-ph, not a descriptive category name.

A search returns no papers

Try a broader phrase, remove one filter, verify the category code, or widen the date range. All active filters are combined with AND.

The arXiv API rate-limited the run

Wait before retrying, keep retry count bounded, and avoid overlapping large scheduled runs. The Actor already applies polite pagination delays.

A paper has no DOI or journal reference

Those fields are optional. null means the selected source surface did not include the value; it does not prove absence on all arXiv surfaces.

Legality and responsible use

ArXiv exposes public scholarly metadata for discovery and reuse, but users remain responsible for their workflow.

• Follow arXiv's API terms and rate guidance.
• Respect paper copyrights and licenses when following PDF links.
• Do not infer sensitive personal information about authors.
• Verify important bibliographic details against the source record.
• Attribute arXiv and original authors where appropriate.

This Actor is an independent automation tool and is not affiliated with or endorsed by Cornell University or arXiv.

Related Actor

For broader scholarly web search and citation-result metadata, use Google Scholar Scraper.

Choose this Actor when you need official arXiv metadata and category/date filtering. Choose Google Scholar Scraper when you need discovery across publishers and repositories.

FAQ

Does this Actor require an arXiv login?

No. It uses public metadata and never asks for account credentials.

Can I search by paper title?

Yes. Put the title or a distinctive phrase in query. Topic search covers titles and abstracts.

Can I fetch a specific version?

Yes. Supply a versioned ID such as 1706.03762v7.

Can I export PDFs?

The output includes a direct pdfUrl, but the Actor does not download or store PDF binaries.

Are multiple categories AND or OR?

Multiple category values are OR. Category filtering is then ANDed with topic, author, and date filters.

How do I detect revised papers?

Join scheduled datasets by basePaperId, then compare version and updatedAt. Versioned paperId values change across revisions; HTML search may leave version unresolved and dates have day precision.

Is the output suitable for a research dataset?

It is suitable as structured source metadata. Review source licenses, document your query and collection date, and validate records for your methodology.

What happens if the source is unavailable?

The Actor attempts Atom then bounded HTML recovery. It fails if neither route returns valid metadata, or if fallback cannot honor a requested exact ordering. It does not substitute fabricated or cached results.