arXiv Research Paper Scraper

arXiv Research Paper Scraper finds public research papers by keyword phrase or arXiv category and turns the arXiv API feed into clean, structured dataset rows. It is built for AI research tracking, academic trend monitoring, technical due diligence, startup research, competitive intelligence, and automated paper alerts.

The Actor does not use a generic webpage scraper. It calls the public arXiv API, parses Atom XML, normalizes authors, categories, PDF links, published dates, updated dates, and abstracts, then adds source-aware relevance scoring so buyers can understand why each paper matched their query or category.

What This Actor Helps You Do

Use this Actor when you need to answer questions such as:

• What new papers are appearing around a research topic such as RAG, LLM agents, graph neural networks, robotics, or diffusion models?
• Which arXiv categories are producing fresh papers in a technical area?
• Which papers should be reviewed for a market map, due diligence memo, technical newsletter, or internal research brief?
• Which papers were newly discovered since the previous scheduled run?
• Which topics are gaining momentum across AI, machine learning, natural language processing, computer vision, quantitative finance, or engineering categories?

Key Features

• Search arXiv by one or more keyword phrases.
• Search exact arXiv categories such as cs.AI, cs.LG, cs.CL, cs.CV, stat.ML, or q-fin.
• Sort by submitted date, last updated date, or relevance.
• Capture paper title, arXiv URL, PDF URL, authors, categories, primary category, published date, updated date, and optional abstract.
• Score rows against query/category terms using relevanceScore and matchedSourceTerms.
• Balance output across multiple queries or categories with maxResultsPerSource.
• Use change detection to output newly discovered papers between scheduled runs.
• Produce a clean dataset schema for review, export, and downstream automations.

Input Overview

The simplest run uses searchQueries, such as retrieval augmented generation or robot learning. You can also add categories to monitor specific arXiv sections.

Important input fields:

• searchQueries: Research phrases to search across arXiv metadata.
• categories: Optional arXiv category IDs to monitor.
• sortBy: Submitted date, last updated date, or relevance.
• sortOrder: Descending for newest first, ascending for oldest first.
• maxResults: Maximum total paper rows to write.
• maxResultsPerSource: Optional cap per query or category.
• pagesPerSource: Number of arXiv API pages to request per source.
• includeAbstract: Include abstract text in the dataset.
• minimumRelevanceScore: Require matched source terms before writing a row.
• monitorChanges: Save a snapshot for detecting newly discovered papers.
• onlyChanges: Output only new papers when change detection is enabled.

Example Input

{
  "searchQueries": [
    "retrieval augmented generation",
    "large language model agents"
  ],
  "categories": [
    "cs.AI",
    "cs.CL"
  ],
  "sortBy": "submittedDate",
  "sortOrder": "descending",
  "maxResults": 50,
  "maxResultsPerSource": 15,
  "pagesPerSource": 1,
  "includeAbstract": true,
  "minimumRelevanceScore": 0
}

Output Fields

Each dataset row represents one arXiv paper.

Core fields include:

• paperId: Stable arXiv paper identifier.
• paperTitle: Paper title from the arXiv feed.
• url: arXiv abstract page.
• pdfUrl: Direct PDF link when available.
• authors: Author names.
• categories: All arXiv categories attached to the paper.
• primaryCategory: First category in the feed.
• publishedAt: Original publication timestamp.
• updatedAt: Last updated timestamp.
• sourceType: Whether the row came from a query or category source.
• sourceQuery and sourceCategory: The input source that found the paper.
• relevanceScore: Number of meaningful source terms matched in the paper metadata.
• matchedSourceTerms: Source terms that matched the paper.
• abstract: Optional paper abstract, capped for practical dataset size.
• scrapedAt: Actor run timestamp.

Practical Workflows

Research teams can schedule this Actor daily or weekly to monitor new papers in specific categories. Product teams can track emerging terms that may affect roadmaps or technical strategy. Investors and analysts can build topic-specific paper feeds for due diligence. Newsletter operators can turn fresh arXiv rows into editorial queues. AI teams can collect paper metadata for internal knowledge bases and review dashboards.

For monitoring, enable monitorChanges. The Actor saves a snapshot and can output only newly discovered papers on later runs when onlyChanges is enabled.

Data Quality Notes

arXiv search can return broad results for broad phrases. Use specific phrases, category filters, and maxResultsPerSource to keep output focused. Set minimumRelevanceScore above 0 when you want stricter matching. Leave it at 0 when monitoring categories where the category itself is already the main filter.

maxResults is a cap, not a guarantee. Final row count depends on arXiv API results, category size, sorting mode, duplicate paper IDs, and relevance filters.

Limitations

This Actor reads public arXiv API data only. It does not download or parse full PDF text, infer citations, classify paper quality, or access private publication databases. arXiv API availability and rate limiting may affect very large or frequent runs. The Actor is not affiliated with arXiv or Cornell University.