📚 arXiv Scraper: Papers, Citations & Impact Metrics

arXiv Research Papers Scraper searches arXiv by keyword, author, category or arXiv ID and returns one row per paper: title, abstract, authors, categories, PDF link, DOI and dates. Each paper can be enriched with OpenAlex citation counts, FWCI and open-access status, plus citing-paper and reference rows. An optional only-new mode returns just the papers a scheduled search has not delivered before. Export CSV, Excel, JSON or XML. No login, no API key. Verified against live arXiv on 2026-10-04: 10 papers in 14 seconds.

The arXiv Research Papers Scraper queries arXiv's official API and joins every result with OpenAlex, the open scholarly graph, so a paper row can carry impact data instead of bare metadata. It is built for research teams, analysts, librarians and RAG pipeline builders.

🔎 What does the arXiv Scraper do?

• Search arXiv papers by keyword across all fields, title, abstract, author, category or comment.
• Fetch papers by arXiv ID (for example 1706.03762) in the same run.
• Filter by category with a dropdown of all 155 official arXiv categories (cs.AI, cs.CL, stat.ML, quant-ph and more).
• Sort by relevance, last updated date or submitted date, ascending or descending.
• Add citation metrics from OpenAlex: citation count, FWCI, open-access status, top concepts.
• Export citing papers (who cites a paper) and references (what a paper cites) as separate rows.
• Monitor new papers with an only-new mode for scheduled runs: papers already delivered are skipped and not charged.

📊 What data can you extract from arXiv?

Field	Description
recordType	paper, citation or reference
arxivId, url, pdfUrl	arXiv ID with version, abstract page and direct PDF link
title, summary	Paper title and full abstract
authors	Author list
primaryCategory, categories	Main arXiv category and all categories
published, updated	First submission and last update timestamps
doi, journalRef, comment	DOI, journal reference and author comment (Not Disclosed when arXiv has none)
citationCount, fwci, openAccessStatus, openAlexId	OpenAlex impact data (Not Available when no confident match)
topConcepts	Up to 5 OpenAlex concepts, present only when OpenAlex returns them
searchTerm	The search term that found the paper (absent for ID lookups)
sourceArxivId, sourceTitle, publicationDate, citedByCount, venue	On citation and reference rows
scrapedAt	Run timestamp (UTC)

🧪 Sample output from a real run

Input: arxivIds: ["1706.03762"]. Abstract and author list shortened here.

{
  "recordType": "paper",
  "arxivId": "1706.03762v7",
  "title": "Attention Is All You Need",
  "url": "https://arxiv.org/abs/1706.03762v7",
  "pdfUrl": "https://arxiv.org/pdf/1706.03762v7",
  "summary": "The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. ...",
  "authors": ["Ashish Vaswani", "Noam Shazeer", "Niki Parmar", "Jakob Uszkoreit"],
  "primaryCategory": "cs.CL",
  "categories": ["cs.CL", "cs.LG"],
  "published": "2017-06-12T17:57:34Z",
  "updated": "2023-08-02T00:41:18Z",
  "doi": "Not Disclosed",
  "journalRef": "Not Disclosed",
  "comment": "15 pages, 5 figures",
  "citationCount": 26739,
  "fwci": "Not Disclosed",
  "openAccessStatus": "gold",
  "openAlexId": "W2626778328",
  "topConcepts": ["Computer science", "Machine translation"],
  "scrapedAt": "2026-10-04T05:03:31.911Z"
}

💵 How much does it cost to scrape arXiv papers?

Pay per event, no subscription. Prices at the Free tier (volume tiers are cheaper on the actor page):

Event	Price	When it is charged
Paper record	$0.015	One arXiv paper saved to the dataset
Citation metrics (OpenAlex)	$0.012	Only when OpenAlex returns a confident title match
Citing paper	$0.010	One citing-paper row (off by default)
Reference	$0.010	One reference row (off by default)

Measured example: a 10-paper run for "large language models" delivered 10 papers, 9 matched in OpenAlex, so it charged 10 paper events and 9 metrics events (about $0.26). With metrics off, 1,000 papers cost about $15. With metrics on and every paper matched, 1,000 papers cost about $27 (1,000 x $0.015 + 1,000 x $0.012). You are never charged for rows that failed, for unknown arXiv IDs, for empty searches or for papers OpenAlex could not match. Set a maximum charge per run and the actor stops cleanly. Free Apify users get a 10-paper preview.

▶️ How to scrape arXiv papers in 3 steps

1. Open the actor and click Try for free.
2. Enter search terms (or arXiv IDs), optionally pick categories, and set Max Items.
3. Click Start, then download the dataset as CSV, Excel, JSON or XML.

⚙️ Input

Field	Default	What it does
searchTerms	["large language models"]	Keywords, one search per term
arxivIds	[]	Exact arXiv IDs to fetch directly
maxItems	10	Maximum papers (free users capped at 10, paid up to 1,000,000)
searchField	all	all, ti, abs, au, cat or co
categories	[]	One or more arXiv categories, combined with AND
sortBy / sortOrder	relevance / descending	Also lastUpdatedDate, submittedDate
includeMetrics	true	OpenAlex citation count, FWCI, open access, concepts
includeCitations, maxCitationsPerPaper	false, 25	Citing-paper rows per paper (max 200)
includeReferences, maxReferencesPerPaper	false, 25	Reference rows per paper (max 200)
onlyNew	false	Monitor mode: skip papers this same search already returned in an earlier run
stateKey	empty string	Optional monitor name, for independent monitors on the same search
resetSeen	false	Clear the stored list of seen papers before the run

{
  "searchTerms": ["quantum error correction"],
  "categories": ["quant-ph"],
  "sortBy": "submittedDate",
  "sortOrder": "descending",
  "maxItems": 50,
  "includeMetrics": true
}

📤 Output

The dataset has one row per paper, citing paper or reference, told apart by recordType. Optional fields (such as topConcepts) are left out when empty. If a search finds nothing, the run ends as succeeded with the message "No arXiv papers matched the input. Nothing was charged."

🔔 Only new papers (monitor mode)

Turn on onlyNew and the actor skips every paper this same search already returned in an earlier run. Skipped papers are not charged. Use stateKey to name independent monitors and resetSeen to start over.

• The seen list lives in a key-value store, arxiv-seen, in your own Apify account.
• The key is the arXiv ID without the version suffix, so a new version (v2) of a paper you already have is not delivered again.
• maxItems means "up to N new papers". Citing-paper and reference rows follow their paper. If nothing is new the run still succeeds and charges nothing.

⚖️ arXiv scraper vs alternatives

Competitor features and prices read from their public Apify Store pages on 2026-10-09.

Capability	This actor	ryanclinton	gentle_cloud
Output fields	29	68, many computed	14 plain fields
Citation count and FWCI	Yes, via OpenAlex	Not listed; BibTeX block	Not listed
Citing papers, references	Optional rows	Not listed	Not listed
Search and filters	Fields, 155 categories, IDs	Boolean prefixes, filters, modes	Keyword, author, category, IDs
Only-new mode	Yes, skipped free	Monitor mode with change deltas	Not listed
Computed scores	None	Relevance, maturity, freshness	None
Export	CSV, Excel, JSON, XML	Dataset, integrations	JSON, CSV, Excel
Price	$15 per 1,000, +$12 metrics	$2.00 per 1,000	Platform usage only

The rivals are much cheaper per paper. ryanclinton fits teams that want ready-made relevance and maturity scores and change tracking; gentle_cloud fits plain metadata at platform cost. Our price buys OpenAlex citation counts, FWCI and optional citing and reference rows in one dataset, with free skips in monitor mode. Papers OpenAlex cannot match are not charged the metrics event.

💼 Use cases

R&D monitoring with weekly onlyNew runs, literature reviews in one spreadsheet, citation graph edge lists, and feeding RAG pipelines with current paper metadata.

🔌 Run the arXiv scraper via API and integrations

curl -X POST "https://api.apify.com/v2/acts/RecordsData~arxiv-research-papers-scraper/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"searchTerms":["retrieval augmented generation"],"maxItems":20}'

Use the Apify API, schedules and webhooks, or connect Zapier, Make, n8n, Google Sheets, Slack and Airbyte.

🤖 arXiv Research Papers Scraper MCP server for AI agents

AI agents can call this Actor through the Apify MCP server at https://mcp.apify.com?tools=RecordsData/arxiv-research-papers-scraper.

Add this to your MCP client configuration (Claude Desktop, Cursor, VS Code or any client that supports remote MCP servers):

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=RecordsData/arxiv-research-papers-scraper"
    }
  }
}

Example agent prompt:

Use the arXiv Research Papers Scraper to find the 20 most recently submitted papers on "quantum error correction" in the quant-ph category, with citation metrics on, and list the three with the highest citation count.

Tool description: choose this Actor when an agent needs current arXiv papers with abstracts, PDF links and citation impact, or the citing papers and references of a known arXiv ID. It takes the same fields as the Input table above and returns one dataset row per paper.

🛡️ Is it legal to scrape arXiv?

The actor reads public paper metadata through arXiv's official API and OpenAlex's open API. It waits about 3 seconds between arXiv requests, as arXiv asks, and identifies itself with a contact address. It does not download PDFs, only links to them. Check arXiv's terms for how you reuse abstracts. Independent tool, not affiliated with arXiv or OpenAlex. Thank you to arXiv for use of its open access interoperability.

❓ Frequently asked questions

How do I search arXiv papers by keyword and export to CSV?

Open the arXiv Research Papers Scraper on Apify, enter one or more search terms, choose the field to search (all, title, abstract, author, category or comment) and set Max Items. Click Start, then open the Storage tab and download the dataset as CSV. Excel, JSON and XML downloads are available from the same dataset, with one row per arXiv paper.

How do I get citation counts for arXiv papers?

Leave Citation metrics switched on in the arXiv Research Papers Scraper, which is the default. Each arXiv paper is matched by title against OpenAlex, and when the match is confident the row gets citation count, FWCI, open-access status and top concepts. Papers without a confident match are marked Not Available and are not charged for metrics.

Why are some fields "Not Disclosed" or "Not Available" in the arXiv Scraper?

In the arXiv Research Papers Scraper, Not Disclosed means the source has no value, for example many arXiv papers have no DOI or journal reference, and OpenAlex has no FWCI for very recent papers. Not Available means OpenAlex had no confident match for that arXiv paper. In a verified 10-paper run, 9 papers matched in OpenAlex.

Can I get the papers that cite a specific arXiv paper?

Yes. In the arXiv Research Papers Scraper, add the arXiv ID, for example 1706.03762, and turn on Citing papers. Each work that cites the paper becomes its own dataset row with DOI, venue, authors and citation count, newest first. You can set how many citing papers to fetch per paper, up to 200. Each citing paper costs one citing-paper event.

What happens if I enter a wrong arXiv ID?

If you enter an arXiv ID that does not exist, arXiv rejects it, and the arXiv Research Papers Scraper logs a warning and skips it. Nothing is charged for that ID. Valid arXiv IDs in the same list still resolve normally, so one typo does not stop the rest of the run or waste your budget.

Why did my arXiv search return 0 results?

Either the search matched nothing on arXiv, so try fewer words or search all fields, or the category filter is too narrow. The arXiv Research Papers Scraper then ends the run as succeeded with zero charge. If arXiv itself is unreachable and no paper was delivered, the run fails with a clear message and also charges nothing.

What happens if I send an empty or invalid input to the arXiv Scraper?

The Apify platform rejects invalid values, such as an unknown searchField or category, with a 400 error when you start the run, so no run is created. If you send neither search terms nor arXiv IDs, the arXiv Research Papers Scraper ends as succeeded with one free error row and charges nothing. An empty input uses the defaults: 10 papers for "large language models".

How fast is the arXiv Research Papers Scraper?

A verified 10-paper run with citation metrics on finished in 14 seconds. The arXiv Research Papers Scraper waits about 3 seconds between arXiv requests, as arXiv asks, so large runs are paced by that limit. Plan for longer runtimes when you ask for hundreds or thousands of papers, and use the maximum charge setting to cap cost.

How do I get only new arXiv papers since my last run?

Set onlyNew to true in the arXiv Research Papers Scraper, sort by submittedDate descending and schedule the actor. Each run returns only papers the same search has not delivered before, and skipped papers are free. The seen list is stored in a key-value store named arxiv-seen in your own Apify account. Use stateKey for separate monitors and resetSeen to start from scratch.

Does a new version of a paper count as new in monitor mode?

No. The arXiv Research Papers Scraper tracks each paper by its arXiv ID without the version suffix, so 2303.08774v2 and 2303.08774v3 are the same paper. Once a paper has been delivered, a later revision is skipped and not charged. If you want a revised paper again, turn on resetSeen for one run or use a different stateKey.

Why do citation numbers differ from Google Scholar?

The arXiv Research Papers Scraper takes citation counts from OpenAlex, which is open and reproducible, while Google Scholar uses its own closed index, so the numbers differ. The OpenAlex ID is included in every arXiv paper row, so you can look up the same work on OpenAlex and check any number yourself.

🔗 Want related data? Other RecordsData scrapers

• PubMed Scraper for biomedical literature
• SEC EDGAR Filings Scraper for official company filings
• Grants.gov Opportunities Scraper for federal funding to pair with the literature

🆘 Support

Questions or a field you need? Open the Issues tab on the actor page or write to contact.punkrecordsdata@gmail.com.

Last updated: 2026-10-11