arXiv Papers Scraper — Abstracts, Authors, Categories & PDF Links

Search and extract papers from arXiv.org as clean, structured JSON: title, full abstract, authors, subject categories, submission/update dates, DOI, journal reference, and direct PDF links. Search by keyword, filter by category, sort by newest or relevance, or fetch exact papers by arXiv ID. No account, no API key.

Built on arXiv's official export API, so results are complete and reliable — ideal for literature reviews, research monitoring, and building AI/ML datasets.

Features

• 🔎 Keyword & field search — plain terms or structured queries (ti:, au:, abs:, cat:) across the full arXiv corpus
• 🗂️ Category filters — narrow to fields like cs.CL, cs.AI, math.PR, stat.ML, and combine them
• 🆔 Fetch by ID — pass a list of arXiv IDs to pull specific papers directly
• 📅 Sort your way — by relevance, newest submitted, or most recently updated
• 📄 Full abstracts + PDF links — the complete summary text plus direct abs and pdf URLs
• 👥 Structured authors — clean author list, count, and a display string
• ⚡ Fast & polite — pure JSON, no browser; respects arXiv rate limits with automatic retries

What you get

Each paper is one dataset record:

{
  "query": "large language models",
  "arxivId": "2403.09676v1",
  "arxivIdBase": "2403.09676",
  "title": "Unmasking the Shadows of AI: Investigating Deceptive Capabilities in Large Language Models",
  "summary": "This research critically navigates the intricate landscape of AI deception...",
  "authors": ["Linge Guo"],
  "authorsList": "Linge Guo",
  "authorCount": 1,
  "primaryCategory": "cs.CL",
  "categories": ["cs.CL", "cs.AI"],
  "published": "2024-02-07T00:21:46Z",
  "updated": "2024-02-07T00:21:46Z",
  "doi": null,
  "journalRef": null,
  "comment": "AI deception, Large Language Models, ChatGPT",
  "absUrl": "https://arxiv.org/abs/2403.09676v1",
  "pdfUrl": "https://arxiv.org/pdf/2403.09676v1"
}

Input

{
  "searchTerms": ["large language models", "cat:cs.CL"],
  "categories": ["cs.AI"],
  "sortBy": "newest",
  "maxResultsPerQuery": 200,
  "includeAbstract": true
}

Field	Description
searchTerms	Keywords or structured field:value queries. Multi-word phrases are matched as phrases.
categories	arXiv subject categories to filter by (e.g. cs.CL, stat.ML). Combine with terms or use alone.
idList	Fetch specific papers by arXiv ID (skips search when provided).
sortBy	relevance (default), newest, or lastUpdated.
maxResultsPerQuery	Max papers per query (1–2000).
includeAbstract	Include the full abstract text (default true).

Use cases

• Literature reviews — pull every recent paper on a topic with abstracts in one run
• Research monitoring — track new submissions in your categories, sorted newest-first
• AI/ML datasets — build corpora of titles + abstracts for classification, search, or RAG
• Citation & metadata — grab DOIs, journal refs, and PDF links at scale

Pricing

Pay-per-event: you're billed a small amount per run start and per paper scraped — you only pay for the papers you actually get. See the Pricing tab for current rates.

FAQ

Do I need an API key? No. This Actor uses arXiv's public export API, so there's no account, login or API key required — just add your search terms and run.

How many papers can I scrape per run? Up to 2,000 papers per query via maxResultsPerQuery, and you can pass multiple search terms in one run, so a single run can return many thousands of papers.

Is scraping arXiv legal? Yes. It reads only openly available metadata through arXiv's official export API and follows its usage guidelines (paced requests, an identified client). It returns arXiv's own metadata as-is and doesn't touch anything behind a login.

What format is the output? Clean structured JSON — one dataset record per paper — which you can export as JSON, CSV, Excel or feed straight into an API. Each record includes the title, full abstract, authors, categories, dates, DOI and direct PDF links.

Can I filter by category or fetch specific papers? Yes. Use categories to narrow to fields like cs.CL, cs.AI or stat.ML, combine them with keywords, or pass an idList of arXiv IDs to pull exact papers directly.

Related Actors

Building a research or literature dataset? These other hipersoft scrapers pair well with this one:

• OpenAlex Scraper — 250M+ scholarly works with citations, authors and abstracts
• Semantic Scholar Scraper — papers with citation and influential-citation metrics across all fields
• Crossref Scraper — DOIs, citation counts and metadata from 150M+ registered works
• PubMed Scraper — biomedical papers, abstracts and MeSH terms from the NLM index

Notes

Uses arXiv's public export API and complies with its usage guidelines (paced requests, identified client). Results reflect arXiv's own metadata. arXiv is a trademark of Cornell University; this Actor is an independent tool and is not affiliated with or endorsed by arXiv.