arXiv Scraper

A paper goes up on arXiv before peer review and then keeps being revised in place, so it has two dates worth having: when it was first submitted, and when it was last touched. Both are in every row, next to the full abstract, every author, the category list, DOI when there is one, and direct links to the abstract page and the PDF. It all comes from arXiv's official API, so there's no key and no scraping of the HTML site.

What you get back

{
  "ok": true,
  "arxivId": "1706.03762",
  "title": "Attention Is All You Need",
  "abstract": "The dominant sequence transduction models are based on complex recurrent or convolutional neural networks...",
  "authors": ["Ashish Vaswani", "Noam Shazeer", "Niki Parmar"],
  "primaryCategory": "cs.CL",
  "categories": ["cs.CL", "cs.LG"],
  "publishedAt": "2017-06-12T17:57:34Z",
  "updatedAt": "2023-08-02T00:41:18Z",
  "doi": null,
  "absUrl": "http://arxiv.org/abs/1706.03762v7",
  "pdfUrl": "http://arxiv.org/pdf/1706.03762v7"
}

abstract is the whole thing, whitespace collapsed to single spaces — not a truncated preview. arxivId has the version suffix stripped so it's stable across revisions, while absUrl and pdfUrl point at the specific version arXiv served. publishedAt is the first submission, updatedAt the latest revision, so updatedAt > publishedAt tells you a paper has been revised.

doi is null for most papers. arXiv only has one when the authors registered a journal DOI themselves, and plenty never do.

Input

The query field takes arXiv's own search syntax, not plain keywords. Field prefixes are all:, ti: (title), au: (author), abs: (abstract) and cat: (category), combined with AND, OR and ANDNOT.

{
  "query": "cat:cs.CL AND abs:retrieval augmented",
  "sortBy": "submittedDate",
  "maxItems": 300
}

More examples that work:

• all:large language models — everything, anywhere in the record
• cat:cs.AI — a whole category
• ti:attention AND au:vaswani — title and author together
• abs:diffusion ANDNOT cat:eess.IV — subtract a category

Field	Default	Notes
query	empty	Required in practice. An empty query returns a BAD_INPUT row, not results. The Console prefills all:large language models; an API call has to send its own.
sortBy	relevance	Or submittedDate / lastUpdatedDate, newest first.
maxItems	50	1 to 30,000. See the timing note below.
proxyConfiguration	off	The API is open. No proxy needed.

Limits

• Big runs are slow on purpose. arXiv serves at most 100 results per request and asks callers to wait about three seconds between them. The actor does. So 1,000 papers is roughly ten requests and half a minute of deliberate sleeping; 10,000 papers is a five-minute run that is mostly waiting. That's arXiv's rule, not a throttle we added.
• arXiv caps how deep any single query can be paged at around 30,000 results. Past that you have to narrow the query. Add a category, or split it into date ranges.
• Duplicates are dropped on arXiv ID within the run.
• A query that matches nothing gets one NO_RESULTS row, and network or rate-limit failures get a coded diagnostic row. Neither is billed.

Billing

$2.00 per 1,000 papers, so $0.002 each, with no run-start fee. Charged per paper returned. Diagnostics and empty results are free.

Delivery

Papers land in the dataset and export as JSON, CSV, Excel or XML. Set notionConnector and notionParentId and each paper is also written as a Notion page after the run, which is one way to build a literature-review database. Authorize the connector once under Settings → API & Integrations → MCP connectors; leave the fields empty to skip.

Why people run it

Literature triage. One run gives you a few hundred rows with the whole abstract sitting in each one, so you can read down a spreadsheet and decide what's worth opening rather than clicking through arXiv's search page.

People also use it to watch a single category. cat:cs.LG with sortBy: submittedDate, run daily and diffed against yesterday's dataset, is a new-papers feed.

abstract comes back as clean single-spaced text, so it chunks into an embedding index with no cleanup step in between.