arXiv Research Paper Search Scraper

What does arXiv Research Paper Search Scraper do?

This Actor searches the public arXiv Atom API and returns normalized research-paper records. Search across all fields or a specific field such as title, author, abstract, category, report number, or arXiv ID; keep paper metadata, public links, search provenance, and validated response metadata in one dataset.

Why use arXiv Research Paper Search Scraper?

It is useful for literature discovery, systematic-review preparation, research monitoring, topic tracking, and building citation or paper-indexing workflows. The Actor supports bounded offset pagination, a global item limit, stable arXiv-ID deduplication, category/date filters, API sorting, configurable page size, deterministic source/page ranks, and safe direct or authorized proxy requests. Apify scheduling, API access, run history, monitoring, and integrations can be used around the Actor.

The Actor uses arXiv's structured Atom response instead of brittle browser scraping. Every response is checked for HTTP status, allowed arXiv origin, XML/Atom content type, size, block signals, and a valid feed before any paper fields are parsed. It does not log credentials, cookies, authorization headers, proxy URLs, raw responses, or private data.

The submitted query is sent as a quoted phrase in the selected field (for example, all:"large language models"). An unquoted search in arXiv's web form can return a different total and order, so compare the output using the recorded searchUrl and sort settings.

What data can it extract?

Group	Fields
Identity	id, arxivId, version
Paper content	title, titleCharacterCount, titleWordCount, abstract, abstractCharacterCount, abstractWordCount, hasAbstract, authors, authorsDetailed, authorCount, affiliations, affiliationCount, primaryCategory, categories, categoryCount, hasVersion
Dates and references	submittedDate, updatedDate, publishedAt, updatedAt, comments, journalRef, doi, reportNumber, hasDoi, hasComments, hasJournalReference, hasReportNumber
Public links	abstractUrl, paperUrl, pdfUrl, links, linkCount
Search context	query, normalizedQuery, searchType, sortBy, sortOrder, searchUrl, page, pageStart, pageSize, position, sourceRank, totalResults
Provenance	httpStatus, responseContentType, requestAttempts, extractionMethod, source, scrapedAt

How to scrape arXiv

1. Open the Actor in Apify Console and select the Input tab.
2. Enter a query and choose the search field, sort, limits, and optional category/date filters.
3. Set pageSize and maxPages to control bounded pagination, then run the Actor.
4. Inspect the dataset, OUTPUT summary, API link, or scheduled run history.

How much will it cost?

The Actor uses lightweight HTTP requests to arXiv's public Atom API and does not launch a browser. Compute usage depends on the number of pages, response size, request delays, retries, proxy routing, and total run time. Larger page limits and queries that return many filtered-out entries can require more requests before the global maxItems limit is reached.

Input

See the input tab for the complete configuration. query is required and is normalized for whitespace, with a maximum of 500 characters.

Minimal search

{ "query": "large language models" }

Filtered, sorted, paginated search

{
  "query": "machine learning",
  "searchType": "all",
  "sortBy": "submittedDate",
  "sortOrder": "descending",
  "includeCategories": ["cs.AI", "cs.LG"],
  "excludeCategories": ["cs.HC"],
  "minSubmittedDate": "2024-01-01",
  "maxSubmittedDate": "2026-12-31",
  "maxItems": 100,
  "pageSize": 50,
  "maxPages": 3
}

Developer options

{
  "query": "quantum computing",
  "pageSize": 25,
  "maxPages": 2,
  "requestDelayMs": 3000,
  "requestTimeoutSecs": 60,
  "maxRequestRetries": 2,
  "proxyConfiguration": { "useApifyProxy": false }
}

searchType accepts all, ti, au, abs, co, jr, cat, rn, or id. sortBy accepts relevance, lastUpdatedDate, or submittedDate; sortOrder is ascending or descending. includeCategories keeps a paper when at least one exact category matches, while excludeCategories removes a paper when any exact category matches. Date filters are inclusive YYYY-MM-DD values and must be in chronological order. maxItems is global (1–1000), pageSize is 1–100, and maxPages is 1–100. The default page limit covers the requested item count at the selected page size, capped at 100 pages.

Developer options are requestDelayMs (0–10000 ms, default 3000), requestTimeoutSecs (10–180 seconds, default 60), and maxRequestRetries (0–5, default 2). proxyConfiguration supports authorized useApifyProxy, proxyUrls, apifyProxyGroups, and apifyProxyCountry; custom proxy URLs and Apify Proxy cannot be enabled together. Proxy use does not bypass access controls.

The complete input field definitions, defaults, allowed values, and limits are in .actor/input_schema.json.

Output

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel.

See the samples/sample-record.json from the public Atom API. I compared its title, authors, abstract, categories, and submission date with the matching arXiv abstract page in Chrome. Fields such as affiliations remain empty when arXiv does not provide them in the Atom entry.

The dataset contract is defined in .actor/dataset_schema.json. Types and requiredness for each field:

The OUTPUT key-value record contains status, dataset count, source/filter/duplicate/malformed counts, pages completed or failed, total available results, query/sort settings, connection mode, and safe failure categories. A no-results run produces an empty dataset with status: "no-results". Any failed or blocked page makes the Apify run fail, even if earlier pages yielded valid papers; only normalized paper records enter the dataset, never diagnostic or placeholder rows.

Tips and advanced options

Use a page size close to the intended result batch to reduce requests, while keeping the recommended three-second delay for respectful public API traffic. Pagination is offset-based and bounded by maxPages; filtered pages are still followed when they contain source entries. Deduplication uses the stable arXiv ID, so multiple versions of one paper are represented once with the version reported by the retained entry.

FAQ, support, and responsible use

If a run fails after returning papers, inspect OUTPUT for the failure category and completed page count; papers from successful pages remain valid dataset records. Retry with a lower page limit or a longer timeout when appropriate. Report issues in the Actor's Issues tab with the run ID and summary. This Actor is not affiliated with arXiv; follow arXiv's terms, API guidance, rate limits, privacy expectations, and applicable law. It is intended for public scholarly metadata only and does not bypass authentication, CAPTCHAs, paywalls, or other access controls.