arXiv Research Paper Scraper

What does arXiv Search Scraper do?

arXiv Search Scraper searches arXiv.org through its public Atom API and returns structured research-paper metadata. It is an arXiv API alternative for scheduled research discovery, literature monitoring, and dataset collection; it does not access private papers, user accounts, or protected content.

Why use arXiv Search Scraper?

Use it to build literature datasets, monitor a subject area, find papers by author or title, or prepare inputs for research and scientific NLP workflows. The Actor provides:

• Full-text, title, author, abstract, comment, journal-reference, category, report-number, and ID searches.
• Relevance, last-updated, or submitted-date sorting in either direction.
• Pagination with deduplication and a configurable item limit.
• Author arrays, subject categories, abstract links, PDF links, DOIs, and both date-only and full Atom timestamps.
• Source URL and query provenance on every paper record.

As an Apify Actor, it can be scheduled, called through the API, connected to integrations, monitored, and used with downstream dataset workflows. The API path uses public structured responses and does not require a browser or an arXiv API key.

What data can arXiv Search Scraper extract?

Field	Type	Description
arxivId	string	Version-independent arXiv identifier.
version	integer	Public Atom entry version, when available.
title	string	Paper title.
authors	array	Author name strings.
abstract	string	Full public abstract.
primaryCategory, categories	string, array	Subject classification.
submittedDate, updatedDate	string	Date-only timestamps retained for compatibility.
publishedAt, updatedAt	string	Full public Atom timestamps.
paperUrl, pdfUrl, doi	string	Public paper, PDF, and DOI links.
journalRef, comments, reportNumber	string	Optional publication metadata.
searchQuery, sourceUrl, scrapedAt	string	Query and extraction provenance.

How to scrape arXiv

1. Open the Actor input tab and enter a query such as large language models or graph neural networks.
2. Choose the search field, sort mode, direction, and maximum number of papers.
3. Start the Actor. Results are written to the default dataset with one item per paper.
4. Download the dataset in JSON, HTML, CSV, or Excel format, or consume it through the API tab.

How much does it cost?

The Actor has no arXiv subscription or API-key cost. Apify usage is billed according to the compute and storage plan attached to your account. Small bounded searches normally use little compute because the Actor requests structured Atom data without a browser. Pagination observes a delay between requests to respect arXiv guidance, so larger searches take longer.

Input

See the input tab for full configuration options. query is required. searchType accepts all, ti, au, abs, co, jr, cat, rn, and id; sortBy accepts relevance, lastUpdatedDate, and submittedDate; sortOrder accepts ascending and descending; maxItems is limited to 1-1000.

Output

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel. The historical id, abstractUrl, and date-only fields remain available for compatibility. Optional diagnostics are stored separately in the key-value store with found: false and dataAvailable: false; they are never counted as successful paper records.

[
  {
    "arxivId": "9999.00001",
    "version": 1,
    "title": "Example study of public data systems",
    "authors": ["Example Author"],
    "primaryCategory": "cs.AI",
    "paperUrl": "https://arxiv.org/abs/9999.00001",
    "publishedAt": "2024-01-05T12:00:00.000Z"
  },
  {
    "arxivId": "9999.00002",
    "version": 2,
    "title": "Sample methods for research discovery",
    "authors": ["Sample Researcher"],
    "primaryCategory": "stat.ML",
    "paperUrl": "https://arxiv.org/abs/9999.00002",
    "publishedAt": "2024-01-06T12:00:00.000Z"
  }
]

Tips, FAQ, and support

Keep maxItems small while testing, and use a specific search field when you need predictable matching. The Actor uses the official public Atom API, applies a 30-second request timeout, retries only bounded transient network/server failures, and waits between pages. A zero-result run is recorded as a diagnostic rather than a successful paper item. If public access is blocked or rate-limited, the Actor stops without bypassing the response. For feedback, use the Issues tab and the API tab for programmatic access.

Our Actors are ethical and do not extract private user data, such as email addresses, gender, or location. They only extract what the user has chosen to share publicly. Personal data in public papers may still be protected by applicable law; use the Actor only for a legitimate purpose and consult legal counsel when unsure.