arXiv Paper Scraper — AI Research, Abstracts & PDF Links | from $1.50/1K

Used by AI/ML researchers building training datasets, literature review automation tools, and academic institutions monitoring emerging research.

Pricing: $1.50 per 1,000 papers (metadata + PDF links). Include full abstracts for an additional $0.50/1k (includeAbstract=true).

Search arXiv research papers by keyword, arXiv ID, or category. Returns structured metadata including title, authors, abstract, categories, PDF URL, DOI, publish dates, and parse_confidence. Uses the official arXiv Atom XML API — no proxy, no auth required. Pay per result.

---

What is this actor for?

arXiv (arxiv.org) is the world's largest open-access repository for scientific preprints, with over 2.3 million papers in physics, mathematics, computer science, quantitative biology, and more. Researchers, analysts, and ML practitioners use arXiv daily for literature discovery.

This actor provides a clean, structured interface to the official arXiv API — no scraping, no fragility, no proxy costs for buyers.

What data does it return?

Each result row has 17 fields:

Field	Description
arxiv_id	arXiv paper ID (e.g. 2501.05032v2)
title	Paper title
summary	Full abstract text (toggle off with includeAbstract=false)
authors	List of author name strings
primary_category	Primary arXiv category (e.g. cs.CL, cs.LG)
categories	All categories the paper is cross-listed in
published_at	Original submission date (ISO 8601 UTC)
updated_at	Last updated date (ISO 8601 UTC)
pdf_url	Direct PDF link
abstract_url	arXiv abstract page URL
doi	DOI if available (often present for journal-published papers)
comment	Author comment (conference/workshop note, code link, etc.)
journal_ref	Journal reference if published
query	Search query that returned this paper (null for ID lookups)
scraped_at	Timestamp of the scrape run (ISO 8601 UTC)
parse_confidence	Quality score 0.0–1.0; 1.0 = all fields parsed correctly
warnings	List of warning codes when confidence < 1.0

Input

Parameter	Type	Default	Description
searchQueries	string[]	—	Keyword queries. Supports arXiv field prefixes: ti: (title), au: (author), abs: (abstract), cat: (category), all: (all fields).
idList	string[]	—	Specific arXiv IDs to fetch (e.g. 2501.05032).
category	string	—	Filter to category (e.g. cs.LG, cs.CL). Applied as cat: prefix when no query given.
sortBy	enum	relevance	relevance / lastUpdatedDate / submittedDate
maxItems	integer	100	Total results cap. 0 = unlimited.
includeAbstract	boolean	true	Include full abstract in output.
pageSize	integer	50	Results per API page (1–200).
delaySeconds	integer	3	Delay between API calls. arXiv requires ≥3 seconds.

Usage examples

Search for recent LLM papers:

{
  "searchQueries": ["ti:\"large language models\""],
  "sortBy": "submittedDate",
  "maxItems": 50
}

Look up specific papers by ID:

{
  "idList": ["2501.05032", "2402.14679", "1706.03762"]
}

Browse a category:

{
  "category": "cs.LG",
  "sortBy": "submittedDate",
  "maxItems": 100
}

Combined query + category:

{
  "searchQueries": ["abs:\"retrieval augmented generation\""],
  "category": "cs.CL",
  "sortBy": "submittedDate",
  "maxItems": 200
}

arXiv search query syntax

arXiv supports prefix operators for targeted searches:

• ti:transformer — papers with "transformer" in the title
• au:"Yann LeCun" — papers by Yann LeCun
• abs:diffusion — papers with "diffusion" in the abstract
• cat:cs.CV — papers in Computer Vision
• all:"attention mechanism" — any field contains phrase
• Combine with AND, OR, ANDNOT

Pricing examples

Run	Items	Cost
100 papers (metadata, no abstract)	100	~$0.15
100 papers with full abstracts	100	~$0.20
1,000 papers (metadata)	1,000	~$1.50
Weekly category monitor (cs.LG, 200 papers)	800/mo	~$1.20/mo

You only pay for papers successfully pushed to the dataset.

FAQ

Do I need a proxy or API key? No. The arXiv API is fully public and requires no authentication. Zero proxy cost for buyers.

What formats can I export results in? JSON, CSV, JSONL, Excel — all via Apify dataset export. The PDF URL field lets you download source PDFs programmatically.

Can I monitor a category for new papers on a schedule? Yes. Use Apify Schedules + sortBy: submittedDate to fetch the latest papers daily or weekly. Combine with a webhook to push new papers into Slack, Notion, or a literature database.

What if the actor returns empty results? Check your searchQueries syntax — arXiv requires the correct field prefix format (e.g. ti:"attention" not title:attention). For ID lookups, verify the ID exists on arxiv.org. The OUTPUT key-value store reports any failed queries with reason codes.

Edge cases & known limits

• Rate limit: arXiv requests ≥3 seconds between API calls. The actor respects this automatically. Large runs may take time — plan accordingly.
• max_results cap: arXiv API caps single-page results at 2000. The actor paginates automatically for large queries.
• No auth required: The API is publicly accessible with no token or proxy.
• Versioned IDs: arXiv returns versioned IDs (e.g. 2501.05032v2). The arxiv_id field contains the version-qualified ID from the response.
• LaTeX in abstracts: arXiv abstracts may contain LaTeX markup (e.g. $\textbf{...}$). The summary field returns raw text — no LaTeX rendering.
• Not affiliated with arXiv: This actor uses the official public API. arXiv and Cornell University are not affiliated with this actor.

Why choose this actor?

• parse_confidence on every record: immediately see if parsing succeeded. No silent failures.
• Zero proxy cost: official API, no residential proxies needed — buyers pay nothing for egress.
• All 3 modes: search + ID lookup + category browse in a single actor.
• Full metadata: doi, comment, journal_ref, versioned PDF URLs — fields competitors skip.
• Batch-friendly: handles multiple queries in one run with automatic pagination.

Competitor comparison

	This actor	scrapestorm/arxiv	Any REST client
Official Atom XML API	Yes	Unknown	Yes (DIY)
parse_confidence	Yes	No	No
Abstract toggle (save cost)	Yes	Always-on	No
Category + keyword + ID in one actor	Yes	Partial	Separate calls
Proxy needed	No	Unknown	No
Price	$1.50/1k	$9.99/1k	Free (DIY infra)

We are 6.6× cheaper than scrapestorm for academic paper data.

AI training datasets

Pull all papers in cs.LG since 2020 — abstract + title + category = a ready-made AI research corpus. Use includeAbstract=true for full text, includeAbstract=false for fast metadata-only runs.

Use with AI agents (MCP)

Ask your AI agent "find me the 10 most cited papers on RAG from 2024" — this actor returns structured paper metadata with abstract and PDF link, ready for downstream summarization.

Point your MCP client at this tool:

{
  "mcpServers": {
    "apify": {
      "command": "npx",
      "args": [
        "mcp-remote",
        "https://mcp.apify.com/?tools=bovi/arxiv-scraper",
        "--header",
        "Authorization: Bearer <YOUR_APIFY_TOKEN>"
      ]
    }
  }
}

Minimal call:

{ "searchQueries": ["large language models"], "maxItems": 20 }

crossref-scraper cross-sell

Need journal metadata, citation counts, or DOIs for published versions? Use our crossref-scraper alongside this actor to enrich arXiv papers with citation data.

Integrations

Built for ML researchers and R&D teams building literature datasets and monitoring new papers by keyword or category — the JSON/dataset output drops into the tools you already run, no glue code:

• n8n / Make / Zapier — trigger a run or pipe every new dataset item into 500+ apps (Google Sheets, Airtable, Slack, HubSpot, your database) with no code: n8n, Make, Zapier.
• Webhooks — fire your own endpoint the moment a run finishes, to push results straight into your pipeline (docs).
• MCP server — expose this actor as a tool to Claude, Cursor, or any MCP client so an AI agent can pull this data mid-conversation (guide).
• API & SDKs — fetch the dataset as JSON, CSV, or Excel through the Apify REST API or the Python / JS SDKs.

See all Apify integrations.

Usage statistics

This Actor creates a small, content-free summary at the end of each run. It is used only to monitor reliability and improve this Actor. A copy is saved as USAGE_STATS in your own Apify key-value store, so you can see the exact record created for your run.

Set disableUsageStats to true in the input to opt out. Nothing is sent then; your USAGE_STATS record only says that statistics were disabled.

Only these fields are recorded:

• schema version, Actor name and build number;
• UTC start and finish hour (not a precise timestamp);
• run duration, number of results and time to the first result, each as a coarse range;
• whether the result was empty, the end status, and an error type from a fixed list;
• memory setting and counts of charged events;
• names of the input fields you set, never their values;
• the selected option for input fields that offer a fixed list of choices (for example a sort order).

We do not collect input text, search terms, URLs, domains, usernames, email addresses, names, proxy credentials, tokens, scraped records, output items, raw error messages, stack traces, or hashes of any of those values. Records are kept for no longer than 13 months, used only as aggregated operational statistics, and never sold or shared.

Additional fields (Phase 2)

This Actor also records your Apify user ID, whether Apify marks the account as paying, the size range of list inputs, the selected country when the input offers a fixed list of countries, and one category from a fixed Actor taxonomy. We use these fields only for aggregate reliability, repeat-use and cross-Actor analysis; reports suppress any cell with fewer than five distinct users.

The same disableUsageStats: true input flag turns these fields off too. The user ID is removed after 13 months; we do not export, sell, share, or attempt to re-identify this data.

Run-outcome signals (v2)

To learn whether a run did what it was asked to do, the record also holds a few more coarse ranges and yes/no flags. None of them contains content:

• the result limit you asked for (a range, when the input has one) and what share of it was delivered;
• results delivered per input item you listed (a range);
• output quality as ranges: how fully the result fields were filled, the share of rows that look like errors, the share of duplicate rows, and how many different fields appeared. These are counted in memory while results are saved; no result content is kept;
• how the run was started (console, API, schedule, webhook, another Actor);
• how it ended: stopped by you, timed out, reached the requested limit, stopped by the charge limit, and how many times the platform moved the run;
• if this Actor reports it: how many items to process worked or failed (ranges) and one failure reason from a fixed list;
• a short code made from the names of the input fields you set, never their values.

Repeat-run fingerprint (v2)

When your Apify user ID is recorded (see above), the record also holds an 8-character one-way code made from your input (proxy settings left out) and this Actor's name. It only lets us see that the same account ran the same input again soon after an unsatisfying run; we never see the input itself. It is stored only in the database, never published, and reports use it in aggregate with the same five-user minimum. It is the one exception to the statement above that no hashes are collected, and disableUsageStats: true turns it off.