arXiv Papers Scraper - New Papers Watch

Get arXiv Papers Scraper - New Papers Watch results as clean JSON, CSV or Excel, from the Apify API or on a schedule. No login, about $2.00 for 1,000 results.

What does arXiv Papers Scraper - New Papers Watch do?

arXiv Papers Scraper - New Papers Watch extracts structured data from oaipmh.arxiv.org. Collect new and updated arXiv papers by category (e.g. cs.AI, cs.LG, stat.ML, quant-ph) from arXiv's official OAI-PMH metadata interface: title, authors, abstract, categories, dates, comments, journal reference, DOI and links, filtered by keywords, and watched for new papers and new versions. Metadata and links only. It works as an API for oaipmh.arxiv.org data: run it from Apify Console, on a schedule, or from your own code, and get clean, typed JSON with numbers as numbers and dates in ISO 8601.

What you get

Data	18 fields per item: arxivId, title, authors, abstract, categories, primaryCategory, ...
Formats	JSON, CSV, Excel, HTML, or the Apify API
Price	$2.00 per 1,000 results, pay per result
Access	Public oaipmh.arxiv.org data only: no login, no cookies, robots.txt respected
Licence	arXiv metadata is CC0 1.0 (public domain); papers keep their own licences and are linked, not downloaded. Not endorsed by arXiv.

Why use arXiv Papers Scraper - New Papers Watch?

• Analysts and researchers: export the data to a spreadsheet or database.
• Developers: call it from your own code through the Apify API.
• Monitoring: run it on a schedule and compare results over time.
• AI agents and RAG: give an LLM live structured data.

Main features:

• Follows pagination up to maxPages pages per start URL and stops at maxItems results.
• Filters: categories (categories), keywords (keywords), updatedAfter (new or updated after), onlyNewPapers (only new papers), onlyChanges (only changes since the last run), so you only get (and pay for) the results you need.
• Polite by default: respects robots.txt, at most maxConcurrency parallel requests and a delay between requests.
• Checks every result against field validators, so layout changes show up as clear data-quality warnings.
• Runs on the Apify platform: scheduling, API access, integrations, monitoring and datasets you can export.

What data can arXiv Papers Scraper - New Papers Watch extract?

Field	Type	Description
arxivId	string	arXiv identifier, e.g. 2610.01234
title	string	Title
authors	array	Author names, in order
abstract	string	Abstract (arXiv metadata, CC0)
categories	array	arXiv categories
primaryCategory	string	First listed category
submittedDate	ISO 8601 date	First version (YYYY-MM-DD)
updatedDate	ISO 8601 date	Latest version (YYYY-MM-DD)
isNewVersion	boolean	An update of an older paper
comments	string	Author comments (pages, figures, venue)
journalRef	string	Journal reference, when published
doi	string	DOI, when published
license	string	Licence URL of the paper
url	string	Abstract page
pdfUrl	string	PDF link (not downloaded)
source	string	Always 'arXiv' (metadata via OAI-PMH)
changeType	string	new, changed, unchanged since the last run with the same input
changes	object	What changed since the last run: {field: {from, to}}

How to scrape oaipmh.arxiv.org

1. Open arXiv Papers Scraper - New Papers Watch in Apify Console and go to the Input tab.
2. Enter what to scrape (see the Input section below), for example the start URLs.
3. Set Max items to the number of results you need.
4. Click Start and wait for the run to finish.
5. Download the results from the Output tab, or fetch them with the API.

How much will it cost to scrape oaipmh.arxiv.org?

This Actor is priced per result: $2.00 per 1,000 results, with no extra charge for platform usage. That is about $2.00 for 1,000 results: 100 results cost $0.20 and 10,000 results cost $20.00. Set a maximum cost per run and the Actor stops when it is reached.

Input

See the Input tab for full configuration options.

Field	Type	Required	Description
categories	array	no	arXiv categories, e.g. cs.AI, cs.LG, stat.ML, quant-ph (a whole archive like 'cs' works too).
keywords	array	no	Keep papers whose title or abstract contains any of these words.
updatedAfter	string	no	YYYY-MM-DD. Default: the last 3 days.
onlyNewPapers	boolean	no	Skip new versions of older papers (keep papers first submitted in the date range).
onlyChanges	boolean	no	For scheduled runs: return (and pay for) only new and changed records, compared with the last run with the same input. The snapshot is kept in a named key-value store; delete it to start over.
maxItems	integer	no	Maximum number of items to return (0 = unlimited).
maxPages	integer	no	Maximum listing pages to follow per start URL (pagination).
maxConcurrency	integer	no	arXiv asks for a single connection at a time: keep 1.
requestDelayMs	integer	no	Minimum delay between requests, in milliseconds (at least 250). arXiv asks for at most one request every 3 seconds: keep at least 3000.
proxyType	string	no	none (direct connection), datacenter (Apify Proxy, cheapest) or residential (opt-in, billed per GB, fewer blocks). The actor never switches by itself.
proxyCountry	string	no	Two-letter country code for the proxy IP (optional).

Example input:

{
  "categories": [
    "cs.AI"
  ],
  "maxItems": 2000,
  "maxPages": 2,
  "maxConcurrency": 1,
  "requestDelayMs": 3500,
  "proxyType": "none"
}

Output

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel. Example results from a real run:

[
  {
    "arxivId": "1709.09051",
    "title": "A polynomial time algebraic solution to exact marginal inference in Markov Random Field models",
    "authors": [
      "Ikhlef Bechar"
    ],
    "abstract": "This paper develops on algebraic grounds a polynomial time exact linear solution to the hard combinatorial problem of marginal inference in Markov random field (MRF) models under general assumptions. To prove our claim, we first implicitly remodel a MRF joint distribution as the unique solution of s…",
    "categories": [
      "math.OC",
      "cs.AI",
      "math.CO"
    ],
    "primaryCategory": "math.OC",
    "submittedDate": "2026-09-29",
    "updatedDate": "2026-10-01",
    "isNewVersion": true,
    "comments": "Paper still awaits a peer-review by the academic community",
    "journalRef": null,
    "doi": null,
    "license": "http://creativecommons.org/licenses/by-nc-sa/4.0/",
    "url": "https://arxiv.org/abs/1709.09051",
    "pdfUrl": "https://arxiv.org/pdf/1709.09051",
    "source": "arXiv",
    "changeType": "new",
    "changes": {}
  },
  {
    "arxivId": "2306.07197",
    "title": "AROID: Improving Adversarial Robustness Through Online Instance-Wise Data Augmentation",
    "authors": [
      "Lin Li",
      "Jianing Qiu",
      "Michael Spratling"
    ],
    "abstract": "Deep neural networks are vulnerable to adversarial examples. Adversarial training (AT) is an effective defense against adversarial examples. However, AT is prone to overfitting which degrades robustness substantially. Recently, data augmentation (DA) was shown to be effective in mitigating robust ov…",
    "categories": [
      "cs.CV",
      "cs.AI",
      "cs.LG"
    ],
    "primaryCategory": "cs.CV",
    "submittedDate": "2024-08-14",
    "updatedDate": "2026-10-01",
    "isNewVersion": true,
    "comments": "published at the IJCV in press",
    "journalRef": "International Journal of Computer Vision (IJCV), 133:929-50, 2024",
    "doi": "10.1007/s11263-024-02206-4",
    "license": "http://creativecommons.org/licenses/by/4.0/",
    "url": "https://arxiv.org/abs/2306.07197",
    "pdfUrl": "https://arxiv.org/pdf/2306.07197",
    "source": "arXiv",
    "changeType": "new",
    "changes": {}
  },
  {
    "arxivId": "2404.00424",
    "title": "Quantformer: from attention to profit with a quantitative transformer trading strategy",
    "authors": [
      "Zhaofeng Zhang",
      "Banghao Chen",
      "Shengxin Zhu",
      "Nicolas Langrené"
    ],
    "abstract": "In traditional quantitative trading practice, navigating the complicated and dynamic financial market presents a persistent challenge. Fully capturing various market variables, including long-term information, as well as essential signals that may lead to profit remains a difficult task for learning…",
    "categories": [
      "q-fin.MF",
      "cs.AI",
      "cs.CE"
    ],
    "primaryCategory": "q-fin.MF",
    "submittedDate": "2025-08-18",
    "updatedDate": "2026-10-01",
    "isNewVersion": true,
    "comments": "The implementation details and code is available on https://github.com/zhangmordred/QuantFormer",
    "journalRef": "Expert Systems with Applications 313 131567 (2026)",
    "doi": "10.1016/j.eswa.2026.131567",
    "license": "http://creativecommons.org/licenses/by/4.0/",
    "url": "https://arxiv.org/abs/2404.00424",
    "pdfUrl": "https://arxiv.org/pdf/2404.00424",
    "source": "arXiv",
    "changeType": "new",
    "changes": {}
  }
]

Integrations and API

• Apify API: start a run and get the results in one HTTP request:

curl -X POST "https://api.apify.com/v2/acts/compass_lab~arxiv-papers-scraper/run-sync-get-dataset-items?token=<YOUR_APIFY_TOKEN>" \
  -H "Content-Type: application/json" -d '{"categories": ["cs.AI"], "maxItems": 2000}'

• Python (pip install apify-client):

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("compass_lab/arxiv-papers-scraper").call(run_input={"categories": ["cs.AI"], "maxItems": 2000})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

• JavaScript (npm install apify-client):

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_APIFY_TOKEN>' });
const run = await client.actor('compass_lab/arxiv-papers-scraper').call({"categories": ["cs.AI"], "maxItems": 2000});
const { items } = await client.dataset(run.defaultDatasetId).listItems();

• Make, Zapier, n8n, Google Sheets, webhooks: use the Apify integrations (Integrations tab) to send each run's results where you need them, or to start a run from your workflow.
• Schedules: run it hourly, daily or weekly from Apify Console (Schedules) and always have fresh results.

Tips and advanced options

• Keep Max items and Max pages as low as you need: fewer pages means a faster, cheaper run.
• Raise Delay between requests if the site responds slowly; keep Max concurrency low to stay polite.
• Missing values are null. Fields that often come back empty are listed in the run log as data-quality warnings.
• Watch for changes: every record has changeType (new, changed or unchanged compared with the last run with the same input) and changes ({"field": {"from": ..., "to": ...}}). Schedule the Actor with Only changes since the last run on to get, and pay for, only what moved. new means new to this input: not returned by an earlier run with the same filters. The last snapshot is kept in a named key-value store called <actor-name>-watch-<id> (the run log prints its name); delete it, or change the input, to start over.

FAQ, disclaimers and support

Is it legal to scrape oaipmh.arxiv.org?

D91 (Mouad, 2026-10-04) confirmed the scholarly line, covering the missing robots.txt on oaipmh.arxiv.org; export.arxiv.org (the legacy API) is Disallow: / and is never used (D94). arXiv terms: metadata CC0; 'no more than one request every three seconds', a single connection; notification services are allowed; no PDFs stored or served, no claim of endorsement.

The data is published under the arXiv metadata is CC0 1.0 (public domain); papers keep their own licences and are linked, not downloaded. Not endorsed by arXiv.: you may reuse it, including commercially, provided you credit the source as the licence requires.

Our Actors are ethical and do not extract any private user data, such as email addresses, gender, or location. They only extract what the user has chosen to share publicly. We therefore believe that our Actors, when used for ethical purposes by Apify users, are safe. However, you should be aware that your results could contain personal data. Personal data is protected by the GDPR in the European Union and by other regulations around the world. You should not scrape personal data unless you have a legitimate reason to do so. If you're unsure whether your reason is legitimate, consult your lawyers.

How many results can I get?

Up to maxItems per run (0 means no limit), as many as the source lists. Each result is one dataset item, and you are only charged for items that are saved.

Can I run it on a schedule or from my own code?

Yes. Schedule it in Apify Console (Schedules), or call it with the run-sync-get-dataset-items endpoint or the Python/JavaScript clients shown in Integrations and API above.

What are the limitations?

• None identified so far.

Where can I get help?

Report problems or ideas on the Issues tab. To call this Actor from your own code, see the API tab.

Related actors

the same clean, typed output across sources, so you can combine them in one dataset.

Actor	What it scrapes	Price
Breezy HR Jobs Scraper	Job listings from Breezy HR	$1.00 / 1,000
Open Canada Dataset Catalogue Scraper	Dataset records from Open Canada	$1.00 / 1,000
Canada Job Bank Jobs Scraper	Job listings from Canada Job Bank	$1.00 / 1,000
data.gouv.fr Dataset Catalogue Scraper	Dataset records from data.gouv.fr	$1.00 / 1,000
Django Weblog Scraper	Posts from Django Weblog	$1.00 / 1,000
Greenhouse Jobs Scraper	Job listings from Greenhouse	$1.60 / 1,000
Lever Jobs Scraper	Job listings from Lever	$1.60 / 1,000
Pinpoint Jobs Scraper	Job listings from Pinpoint	$1.00 / 1,000
Python Jobs Scraper (python.org)	Job listings from Python Jobs Scraper (python.org)	$2.00 / 1,000
Recruitee Jobs Scraper	Job listings from Recruitee	$1.00 / 1,000
We Work Remotely Jobs Scraper	Job listings from We Work Remotely	$2.50 / 1,000
Working Nomads Jobs Scraper	Job listings from Working Nomads	$2.50 / 1,000