arXiv Scraper API – AI Research Papers & PDF

arXiv scraper for AI research papers via the official arXiv API: search by keyword or category (cs.AI, cs.CL, …) and get title, abstract, DOI, version and PDF link. CC0 metadata. RAG/LLM-ready JSON, MCP-friendly.

What does it do?

• Query arXiv Atom API by free text and/or category codes.
• Return arXiv id, version, title, abstract, DOI, PDF URL and categories when present.
• Sort by newest or relevance; cap with maxItems.
• Metadata is CC0; abstracts are author-supplied.

Use cases

• AI research monitoring: watch new cs.AI / cs.CL papers daily.
• RAG corpora: ingest titles + abstracts + PDF links into vector stores.
• Literature reviews: export JSON/CSV of matching preprints.
• AI agents: fetch latest arXiv papers through Apify MCP.

Input

Field	Description
query	Free-text query. Example: retrieval augmented generation
categories	Optional arXiv category codes. Example: cs.AI, cs.CL
sortBy	newest or relevance. Default newest.
maxItems	Max papers. Example: 25

Example input:

{
  "query": "large language model",
  "categories": [
    "cs.AI",
    "cs.CL"
  ],
  "sortBy": "newest",
  "maxItems": 10
}

Output

Real output (sample from a prior run on 27 Oct 2026, Europe/Madrid):

[
  {
    "arxivId": "2610.32755",
    "version": "v1",
    "title": "Semantic Representation for Agentic Self-Play in Automated Laboratories",
    "abstract": "Navigating complex, high-dimensional biological search spaces necessitates scalable physical and computational intelligence to explore beyond local optima through iterative self-play across ideation, experimentation, and automation. Translating experimental designs into reliable physical execution remains bottlenecked by static laboratory automation and unmodeled physical variability. To address this, we introduce VERified Infrastructure for Discovery Laboratories, or VERID, a foundational infrastructure designed to bridge the gap between the high-level reasoning of artificial intelligence (AI) agents and complex, error-prone physical realities of wet labs. VERID is a persistent, high-fidelity semantic digital twin of the laboratory workcell that provides four core services: queryable introspection for agentic reasoning, time-compressed simulation for protocol validation, comprehensive trace persistence for a complete scientific record, and a framework for safe execution. We present three case studies using aqueous dye and deionized water mixtures to validate the optical and fluidic foundations of VERID. These studies demonstrate how VERID enables pre-emptive detection of critical errors in experimental protocols, stateful mid-run error recovery and post-hoc identification of subtle physical failures. The simulation functions as a discrete accounting engine, allowing agents to safely and inexpensively explore experimental designs in silico and refine experimental strategies through iterative self-play. VERID also captures structured multimodal experimental data for downstream analysis, knowledge mining and agent training. We argue that this semantic layer is an essential prerequisite for deploying autonomous agents in the biochemical and biomedical sciences, ensuring data integrity, operational safety, and paving the way for tackling previously intractable grand challenges in biology.",
    "published": "2026-10-23T15:00:16Z",
    "updated": "2026-10-23T15:00:16Z",
    "primaryCategory": "cs.RO",
    "categories": [
      "cs.RO",
      "cs.AI"
    ],
    "authorsCount": 15,
    "doi": null,
    "journalRef": null,
    "comment": "16 pages, 4 figures",
    "url": "https://arxiv.org/abs/2610.32755",
    "pdfUrl": "https://arxiv.org/pdf/2610.32755v1"
  },
  {
    "arxivId": "2610.32738",
    "version": "v1",
    "title": "Agentic Hazard Mapper: Emulating Expert Map-Making with AI Agents for Risk Communication",
    "abstract": "Hazard maps are essential tools for public risk communication, preparedness, and evacuation planning, yet effective cartographic design is not always ensured due to limited access to cartographic expertise across local governments and communities. We present Agentic Hazard Mapper, a multi-agent system that automates hazard-map production by emulating professional cartographic workflows and iterative design trade-offs while adapting design decisions to different data, locations, and hazard contexts. We conducted a formative study with map experts to inform the system design, followed by a user study and interviews with map experts and hazard practitioners. The generated maps improved participants' spatial understanding and evacuation-planning performance, reduced perceived workload and task completion time, and were generally preferred over corresponding municipal maps. We further discuss human oversight, adaptation across users and communication contexts, and opportunities to extend agentic support across broader stages of hazard-map production and risk communication.",
    "published": "2026-10-23T14:53:40Z",
    "updated": "2026-10-23T14:53:40Z",
    "primaryCategory": "cs.AI",
    "categories": [
      "cs.AI"
    ],
    "authorsCount": 5,
    "doi": null,
    "journalRef": null,
    "comment": null,
    "url": "https://arxiv.org/abs/2610.32738",
    "pdfUrl": "https://arxiv.org/pdf/2610.32738v1"
  }
]

Export as JSON, CSV or Excel, read through the API, or send to Google Sheets, Make, Zapier, n8n or webhooks. You can also run this Actor from AI agents through the Apify MCP server.

Pricing

Pay per event: $2.20 per 1,000 papers plus $0.00005 per run start. There is no extra platform usage charge beyond Apify’s normal platform fees if any apply to your plan.

FAQ

Is a login required? No. Uses the public arXiv API.

License? arXiv metadata is CC0; paper PDFs keep their original licenses — check each paper.

Can AI agents use it? Yes. JSON with abstracts and PDF links works well for RAG and MCP clients.

Data source and license

Source: arXiv API. Metadata CC0. Not affiliated with arXiv or Cornell.

This Actor is not affiliated with or endorsed by the data source. You are responsible for using the data in line with the source's license and terms.