arXiv Papers Scraper — Abstracts, Authors, Categories & PDF Links

Search and extract papers from arXiv.org as clean, structured JSON, CSV, Excel or XML: title, full abstract, authors, subject categories, submission/update dates, DOI, journal reference, and direct PDF links. Search by keyword, filter by category, sort by newest or relevance, or fetch exact papers by arXiv ID. No account, no API key — fast and reliable.

What does the arXiv Papers Scraper do?

The arXiv Papers Scraper searches the full arXiv corpus and extracts complete, structured metadata for every matching paper. Use plain keywords or structured field queries, narrow by subject category, and export thousands of papers with abstracts and PDF links in a single run — ideal for literature reviews, research monitoring, and building AI/ML datasets.

Features

• 🔎 Keyword & field search — plain terms or structured queries (ti:, au:, abs:, cat:) across the full arXiv corpus
• 🗂️ Category filters — narrow to fields like cs.CL, cs.AI, math.PR, stat.ML, and combine them
• 🆔 Fetch by ID — pass a list of arXiv IDs to pull specific papers directly
• 📅 Sort your way — by relevance, newest submitted, or most recently updated
• 📄 Full abstracts + PDF links — the complete summary text plus direct abs and pdf URLs
• 👥 Structured authors — clean author list, count, and a display string

What you get

Each paper is one dataset record:

{
  "query": "large language models",
  "arxivId": "2403.09676v1",
  "arxivIdBase": "2403.09676",
  "title": "Unmasking the Shadows of AI: Investigating Deceptive Capabilities in Large Language Models",
  "summary": "This research critically navigates the intricate landscape of AI deception...",
  "authors": ["Linge Guo"],
  "authorsList": "Linge Guo",
  "authorCount": 1,
  "primaryCategory": "cs.CL",
  "categories": ["cs.CL", "cs.AI"],
  "published": "2024-02-07T00:21:46Z",
  "updated": "2024-02-07T00:21:46Z",
  "doi": null,
  "journalRef": null,
  "comment": "AI deception, Large Language Models, ChatGPT",
  "absUrl": "https://arxiv.org/abs/2403.09676v1",
  "pdfUrl": "https://arxiv.org/pdf/2403.09676v1"
}

Output schema

Field	Type	Description
query	string	The search term this paper matched (echoed for traceability)
arxivId	string	Full arXiv identifier including version (e.g. 2403.09676v1)
arxivIdBase	string	arXiv identifier without the version suffix
title	string	Paper title
summary	string	Full abstract text
authors	array	List of author names
authorsList	string	Authors joined as a single display string
authorCount	integer	Number of authors
primaryCategory	string	Primary arXiv subject category (e.g. cs.CL)
categories	array	All subject categories the paper is tagged with
published	string (ISO date)	Original submission date/time
updated	string (ISO date)	Most recent revision date/time
doi	string	DOI, if the paper has one (else null)
journalRef	string	Journal reference, if published (else null)
comment	string	Author-provided comment / note
absUrl	string (URL)	Link to the arXiv abstract page
pdfUrl	string (URL)	Direct link to the paper PDF

Input

{
  "searchTerms": ["large language models", "cat:cs.CL"],
  "categories": ["cs.AI"],
  "sortBy": "newest",
  "maxResultsPerQuery": 200,
  "includeAbstract": true
}

Field	Type	Description
searchTerms	array	Keywords or structured field:value queries. Multi-word phrases are matched as phrases.
categories	array	arXiv subject categories to filter by (e.g. cs.CL, stat.ML). Combine with terms or use alone.
idList	array	Fetch specific papers by arXiv ID (skips search when provided).
sortBy	string	relevance (default), newest, or lastUpdated.
maxResultsPerQuery	integer	Max papers per query (1–2000).
includeAbstract	boolean	Include the full abstract text (default true).

Use cases

• Literature reviews — pull every recent paper on a topic with abstracts in one run.
• Research monitoring — track new submissions in your categories, sorted newest-first.
• AI/ML datasets — build corpora of titles + abstracts for classification, search, or RAG.
• Citation & metadata — grab DOIs, journal refs, and PDF links at scale.
• Trend analysis — measure output by category, author or time to spot emerging topics.

How to scrape arXiv papers

1. Add the arXiv Papers Scraper to your Apify account.
2. Enter one or more search terms (keywords or field:value queries), or an idList for exact papers.
3. Optionally set categories, sortBy and maxResultsPerQuery.
4. Click Run.
5. Export the dataset as JSON, CSV, Excel or XML, or pull it via the Apify API.

Need more research data?

• Crossref Scraper — DOIs, citation counts and metadata from 150M+ registered works.
• OpenLibrary Scraper — book metadata, authors, subjects and editions.
• ClinicalTrials Scraper — study records from the clinical trials registry.

FAQ

Do I need an API key or login? No. There's no account, login or API key required — just add your search terms and run.

How many papers can I scrape per run? Up to 2,000 papers per query via maxResultsPerQuery, and you can pass multiple search terms in one run, so a single run can return many thousands of papers.

How does billing work? You pay only per paper you get, so small filtered runs stay cheap. See the Pricing tab for current rates.

What format is the output? Clean structured JSON — one record per paper — exportable as JSON, CSV, Excel or XML. Each record includes the title, full abstract, authors, categories, dates, DOI and direct PDF links.

Can I filter by category or fetch specific papers? Yes. Use categories to narrow to fields like cs.CL, cs.AI or stat.ML, combine them with keywords, or pass an idList of arXiv IDs to pull exact papers directly.

Is this only public data? Yes. It returns only openly available arXiv metadata and doesn't touch anything behind a login.

Can I connect this to other tools? The arXiv Papers Scraper can be connected with almost any cloud service or web app thanks to integrations on the Apify platform. It works with Make, Zapier, Slack, Airbyte, GitHub, Google Drive and many more, plus the Apify API, JavaScript/Python clients and MCP. Or use webhooks to trigger an action whenever a run finishes.

Related Actors

• Crossref Scraper
• OpenLibrary Scraper
• ClinicalTrials Scraper
• Wikipedia Scraper

Notes

Original clean-room implementation. Returns only public arXiv metadata; you are responsible for compliance with arXiv's terms. arXiv is a trademark of Cornell University; this Actor is independent and not affiliated with or endorsed by arXiv.