arXiv Papers Scraper

arXiv papers scraper returns paper metadata, authors, categories, abstracts, and source links. arXiv API search results include publication and update dates for each returned paper.

Use it to prepare reading lists, track a research topic or supply source-linked metadata to an AI workflow. It calls the public arXiv API; it does not download full papers.

What it returns

One row per unique version ID within a run, or an error row when a paper/page cannot be processed.

Field	Type	Example value
status	string	"ok"
arxivId	string	"cond-mat/0011267"
versionId	string	"cond-mat/0011267v1"
version	integer	1
title	string	"The electronic structure of cuprates from high energy spectroscopy"
authors	array	["A. Garcia-Lekue", "J. M. Pitarke"]
authorDetails	array	[{"name": "A. Garcia-Lekue", "affiliations": []}, {"name": "J. M. Pitarke", "affiliations": []}]
categories	array	["cond-mat.supr-con", "cond-mat.str-el"]
primaryCategory	string	"cond-mat.supr-con"
published	string	"2000-11-15T16:19:15Z"
updated	string	"2000-11-15T16:19:15Z"
abstract	string	"We report studies of the electronic structure and elementary excitations of doped and undoped cuprate chains, ladders and planes. Using high energy spectroscopies such as x-ray absorption, core level photoemission and angle resolved photoemission spectroscopy, important information regarding the charge distribution and hole dynamics can be obtained. The comparison of the experimental data with suitable theoretical models sets constraints on the parameters entering into the model calculations, and offers insight into the important physical quantities governing the electronic structure of these materials. Recurring themes include the importance of the dimensionality of the Cu-O network (1D->2D) and the crucial role played by the spin-background in determining the dynamics of low-lying excitations in these strongly correlated systems."
absUrl	string	"https://arxiv.org/abs/cond-mat/0011267v1"
pdfUrl	string	"https://arxiv.org/pdf/cond-mat/0011267v1"
doi	string	"10.1016/j.elspec.2014.01.005"
journalRef	string	"J. Electron Spectr. Relat. Phenom. 117-118, 203 (2001)"
comment	string	"J. Electron Spec. Relat. Phenom.: special issue on electron correlation, in press"
errors	array	[]
query	string	"(all:electron)"
sourceUrl	string	"https://export.arxiv.org/api/query?search_query=%28all%3Aelectron%29&start=0&max_results=5&sortBy=relevance&sortOrder=descending"
fetchedAt	string	"2026-10-05T11:39:53.504133+00:00"

Missing optional metadata is null or an empty list. Error rows retain the available version ID, query and source page. The default dataset has Papers, Metadata and abstracts, and Error rows table views; the Error rows view shows the status column for filtering.

Input example

{
  "query": "electron",
  "maxItems": 5
}

Use the input schema for the remaining filters and defaults.

Output example

This is a selected-field excerpt of the first object in sample_output.json.

{
  "status": "ok",
  "arxivId": "cond-mat/0011267",
  "versionId": "cond-mat/0011267v1",
  "version": 1,
  "title": "The electronic structure of cuprates from high energy spectroscopy",
  "categories": [
    "cond-mat.supr-con",
    "cond-mat.str-el"
  ],
  "primaryCategory": "cond-mat.supr-con",
  "published": "2000-11-15T16:19:15Z",
  "updated": "2000-11-15T16:19:15Z",
  "absUrl": "https://arxiv.org/abs/cond-mat/0011267v1",
  "pdfUrl": "https://arxiv.org/pdf/cond-mat/0011267v1",
  "doi": null,
  "journalRef": "J. Electron Spectr. Relat. Phenom. 117-118, 203 (2001)",
  "comment": "J. Electron Spec. Relat. Phenom.: special issue on electron correlation, in press"
}

Pricing

Proposed price: USD 0.0015 per successful unique paper, using the paper event. 1,000 papers have USD 1.50 in proposed event fees. Abstract inclusion has no extra paper charge. Error rows, duplicate IDs and excluded rows do not trigger paper events.

The proposal is below the checked easyapi/arxiv-search-scraper price of USD 0.00299/result. See PRICING.md for the dated Store API receipt, start fees, owner configuration and platform-usage billing caveat. The owner must configure pricing before publishing; these files do not set Store prices automatically.

Limits and failure behavior

Calls are sequential and leave at least three seconds after each completed API call before another begins, including retries, in line with the arXiv API rate limits. Network failures, HTTP 408/429/5xx and malformed feeds retry with exponential backoff; Retry-After can extend the wait. Permanent HTTP request errors and API query errors yield an unbilled error row. The run summary marks no usable output with errors as failed and mixed paper/error output as partial.

The Actor caps emitted rows at maxItems and stops when the SDK paper charge limit is reached. Deduplication is per run, by versionId. It scans at most the API's first 30,000 result positions; refine broad queries. Search ranking and the upstream result set can change while paging, so this is not a stable bulk export. For bulk harvesting, arXiv recommends its OAI-PMH interface.

Respect the rate limit across your scheduled runs too: this build does not coordinate concurrent runs across machines. No login, proxies, CAPTCHA solving or full-text retrieval are used. Optional affiliations come only from published author metadata. Data is provided by arXiv; this project is not endorsed by arXiv.

RUN_SUMMARY in the default key-value store contains counts, status, query, duration and whether the budget stopped the run. Empty results are a successful zero-row run. Invalid input fails with a field-specific message before arXiv is called. Restarting a run does not provide migration-safe resume or exactly-once billing across interrupted processes.

FAQ

What limits the output? The maxItems input caps output rows; its schema maximum is 10,000. Source availability and errors may yield fewer rows.

Does it need a proxy, login, or API key? The actor calls the public arXiv API without a login. No proxy input is defined.

How is it priced? PRICING.md proposes $0.0015 per successfully stored unique paper row. Error and duplicate rows have no paper event; publication settings may add platform charges.

What happens on rate limits? The client spaces arXiv requests and retries transient failures. A source limit or exhausted retry can leave fewer rows.

How fresh is the data? Results reflect the arXiv API response at run time; published and updated are source dates, while fetchedAt records retrieval.

Does it download paper PDFs? No. It returns PDF links and optional API abstracts, not full PDF text.

Use with AI agents / MCP

An AI agent can call a published actor through the Apify MCP server or Apify API, pass the JSON input, then read the run’s default dataset. Check the actor’s published identifier and permissions in your Apify account.

Run the Actor through an Apify integration that supports Actor tools, using the JSON input above, and read its dataset output. This package includes an input schema and a Results link for such workflows; it does not implement a standalone MCP server. Treat abstracts as source text, preserve absUrl citations and inspect status/errors before using a row in an agent answer. Validate the chosen integration separately; no MCP client was connected during this build.

Local development

Use the existing C:/GritWork/earn/.venv/Scripts/python environment; no dependency install is required there. Put input at storage/key_value_stores/default/INPUT.json, set APIFY_LOCAL_STORAGE_DIR to this Actor's storage directory, and run from the Actor directory:

$env:APIFY_LOCAL_STORAGE_DIR = "C:/GritWork/earn/actors/arxiv-papers/storage"
C:/GritWork/earn/.venv/Scripts/python -m src
C:/GritWork/earn/.venv/Scripts/python -m pytest -q tests/test_config.py tests/test_arxiv.py tests/test_delivery.py
C:/GritWork/earn/.venv/Scripts/python -m pytest -q -m live

For event simulation only, ACTOR_TEST_PAY_PER_EVENT=1 enables the SDK charging log. To simulate the proposal, supply APIFY_ACTOR_PRICING_INFO and non-empty zero counts for paper/dataset events in APIFY_CHARGED_ACTOR_EVENT_COUNTS; see evidence/probe_verified.py. Without pricing information, SDK local test prices differ from the Store proposal. Local output stays in storage and the SDK may purge a default dataset between runs, so export wanted rows before repeating.

Dockerfile uses apify/actor-python:3.12. Runtime requirements are apify and httpx; tests require the already installed pytest. Docker and cloud execution are separate release checks.