arXiv Scraper - Papers, Authors & Abstracts

Scrape arXiv preprints in bulk. Extract paper title, abstract, authors, categories, publication and update dates, abstract page and PDF links from any search query to CSV/JSON. No API key, no login.

---

What does the arXiv Scraper do?

This Actor turns any arXiv search into a structured dataset. You give it by category or by topic or author, it walks the result pages one after another, and it writes one clean row per paper into your dataset — ready to export as JSON, CSV or Excel, or to pull straight from the Apify API.

arXiv publishes an official, keyless Atom API and the Actor queries it directly. A single broad query reports hundreds of thousands of matches, and the offset parameter walks them in blocks, so depth is bounded by real data rather than by guesswork. Pagination is followed automatically until it runs out of results, hits your page limit or hits your Max items cap, whichever comes first. Every row is de-duplicated across the whole run, so you are never billed twice for the same paper.

There is no API key, no login and no browser involved. That keeps runs fast and cheap, and it means you can schedule the Actor without worrying about credentials expiring.

Who is it for?

• Research and R&D teams tracking publications in a field.
• AI teams building corpora of abstracts and metadata.
• Competitive-intelligence analysts watching what rivals publish.
• Librarians and bibliometricians assembling citation-adjacent datasets.
• Journalists and analysts monitoring a scientific topic.

Use cases

• Export every paper matching a topic and analyse how it evolved by year.
• Track which authors publish most in a category.
• Build an abstract corpus for embedding or classification.
• Monitor a category weekly for newly posted preprints.
• Collect PDF links for a bulk download pipeline.

Why use this arXiv Scraper?

• 🔑 Keyless — no account, no API token, no cookies to paste.
• 📦 13 fields per paper — everything the result page exposes, already typed.
• 📄 Real pagination — it walks page after page instead of returning the first screen.
• 🎯 Precise caps — Max items stops the run exactly where you want it, so the bill is predictable.
• 📊 Export anywhere — JSON, CSV, Excel or HTML, plus the Apify API and integrations.
• 💸 Pay per result — you pay for rows you actually receive, with no platform fees to calculate.

What data can you extract?

Every run produces one row per paper, with these fields:

Field	Type	Description
paperId	string	arXiv identifier including version
title	string	Paper title
abstract	string	Full abstract text
authors	string	Authors, separated by a pipe
primaryCategory	string	Primary arXiv category
categories	string	All categories, separated by a pipe
publishedAt	string	ISO timestamp the first version was posted
updatedAt	string	ISO timestamp of the latest revision
absUrl	string	Abstract page URL
pdfUrl	string	Direct PDF URL
query	string	Search query this row came from
page	number	Result block the paper appeared in
scrapedAt	string	ISO timestamp of extraction

Output example

{
  "absUrl": "https://arxiv.org/abs/2609.19144v1",
  "abstract": "Direct preference alignment methods are widely used to align large language models (LLMs) with human preferences because of their computational and\u2026",
  "authors": "Peter Chen | Xi Chen | Wotao Yin | Tianyi Lin",
  "categories": "cs.CL | cs.AI | cs.LG",
  "page": 1,
  "paperId": "2609.19144v1",
  "pdfUrl": "https://arxiv.org/pdf/2609.19144v1",
  "primaryCategory": "cs.CL",
  "publishedAt": "2026-09-16T17:59:35Z",
  "query": "cat:cs.LG",
  "scrapedAt": "2026-09-17T08:41:04.130Z",
  "title": "A Zeroth-Order Paradigm for LLM Preference Alignment",
  "updatedAt": "2026-09-16T17:59:35Z"
}

How to use

Option A — by category

{
  "maxItems": 1000,
  "maxPagesPerSearch": 10,
  "searchQueries": [
    "cat:cs.LG"
  ]
}

1. Open the Actor and fill in the category field.
2. Set Max pages per search and Max items to bound the run.
3. Click Start, then export from the Output tab.

Option B — by topic or author

{
  "maxItems": 500,
  "maxPagesPerSearch": 5,
  "searchQueries": [
    "all:transformer",
    "au:bengio"
  ]
}

Paste one or more arXiv URLs into Start URLs and the Actor paginates each of them independently.

Input parameters

Parameter	Type	Default	Description
searchQueries	array	–	arXiv query syntax, e.
maxPagesPerSearch	integer	3	How many result pages to walk for each search.
maxItems	integer	100	Stop after this many papers.
maxConcurrency	integer	2	Parallel requests.
proxyConfiguration	object	{"useApifyProxy": true}	Proxy used to fetch pages.

Tips for best results

• Use arXiv's own query syntax: all:transformer, cat:cs.LG, au:hinton all work.
• The API returns up to 100 entries per request, so large exports need few calls.
• Restrict by category (cat:cs.CL) to keep a run focused and fast.
• updatedAt differs from publishedAt when a paper has been revised — useful for tracking activity.
• pdfUrl is a direct link, ready for a bulk download step.
• Schedule a weekly run on the same query to build a publication timeline.
• Authors are pipe-separated so the column survives a CSV round trip.
• arXiv asks for courteous use; keep concurrency low and the run will not be throttled.
• Abstracts are full text, so exports can get large — cap results if you only need metadata.
• Pair with the Crossref or OpenAlex scrapers to add citation data.

Integrations

Send results straight into the tools you already use: Google Sheets, Slack, Zapier, Make, Airtable or any Webhook. You can also schedule the Actor to run hourly, daily or weekly and have each run append to the same dataset, which is how you build a price or availability history rather than a one-off snapshot.

API usage

Run the Actor and collect results from any language. Replace <YOUR_TOKEN> with your Apify API token.

cURL

curl -X POST "https://api.apify.com/v2/acts/logiover~arxiv-paper-scraper/run-sync-get-dataset-items?token=<YOUR_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"maxItems": 1000, "maxPagesPerSearch": 10, "searchQueries": ["cat:cs.LG"]}'

Node.js

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_TOKEN>' });
const run = await client.actor('logiover/arxiv-paper-scraper').call({"maxItems": 1000, "maxPagesPerSearch": 10, "searchQueries": ["cat:cs.LG"]});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Python

from apify_client import ApifyClient

client = ApifyClient('<YOUR_TOKEN>')
run = client.actor('logiover/arxiv-paper-scraper').call(run_input={"maxItems": 1000, "maxPagesPerSearch": 10, "searchQueries": ["cat:cs.LG"]})
for item in client.dataset(run['defaultDatasetId']).iterate_items():
    print(item)

Use with AI agents (MCP)

This Actor is available through the Apify MCP server, so an AI agent can call it as a tool. Point your agent at https://mcp.apify.com and it can run the arXiv Scraper on demand — for example: "Pull the first 500 papers from arXiv and summarise the price distribution." The agent receives the same structured rows you would get from the UI.

FAQ

Do I need a arXiv account or API key?

No. The Actor reads publicly available pages only. There is nothing to authenticate and no credentials to rotate.

How many papers can I get in one run?

As many as the search exposes. Raise Max pages per search and Max items together; the run stops at whichever limit it reaches first.

Why did I get fewer rows than I asked for?

The search ran out of papers. That is normal for narrow queries — broaden the search or add more searches to one run.

Are results de-duplicated?

Yes. Each paper is emitted once per run, even when it appears on several pages, so you are never billed twice for the same record.

Why are some fields empty?

arXiv does not publish every attribute for every paper. Empty means the source did not show it, not that extraction failed.

What export formats are supported?

JSON, CSV, Excel, HTML and RSS from the Output tab, plus the Apify API and any integration you connect.

How fast is it?

It is pure HTTP with no browser, so a page of results typically takes a second or two. Raise Max concurrency carefully — the source rate-limits aggressive crawling.

Can I schedule it?

Yes. Use the Apify scheduler to run it on any interval and append each run to the same dataset for time-series analysis.

Does it work behind a proxy?

It uses Apify Proxy automatically. You can switch groups or supply your own proxies in Proxy configuration.

How often does the data change?

arXiv updates continuously. Re-run whenever you need current data; the Actor always reads the live pages, never a cache.

Is the output schema stable?

Yes. Field names and types are fixed, so downstream pipelines will not break between runs.

What if the site changes its layout?

Open an issue on the Issues tab and it gets fixed. The Actor is actively maintained.

Is it legal?

This Actor reads only publicly available pages on arXiv — the same content any visitor sees without logging in. It does not bypass authentication, does not collect private data and does not attempt to defeat access controls. You are responsible for how you use the output: respect the source's terms of service, applicable copyright, and data-protection law such as GDPR where personal data is involved. Scraping public data is generally lawful in the EU and the US, but the responsibility for the downstream use of that data sits with you.

Related scrapers

• Crossref Scraper — DOI and citation metadata
• OpenAlex Scraper — Works, authors and institutions
• PubMed Scraper — Biomedical literature
• DataCite Scraper — Research data DOIs