arXiv Papers Extractor

Search arXiv and get papers back as clean, typed records — title, full abstract, authors, categories, DOI, and a direct PDF link. Uses arXiv's official public API, so there is no scraping, no API key and no rate-limit contract to negotiate.

What it does

arXiv hosts millions of preprints across physics, mathematics, computer science, quantitative biology, finance and statistics. Its official API returns Atom XML, which is awkward to work with. This Actor queries that API for you and emits flat JSON records you can load into a notebook, a vector database or a spreadsheet without writing a parser.

You can search with plain language (diffusion models), use arXiv's fielded query syntax for precision, filter by category, or skip search entirely and fetch exact papers by ID.

Features

• Official arXiv API — no HTML scraping, so results don't break when a page template changes.
• Fielded query syntax supported: ti: (title), au: (author), abs: (abstract), cat: (category), all:, combined with AND / OR / ANDNOT.
• Category filter — narrow to cs.LG, cs.CL, stat.ML, q-fin.TR and any other arXiv category.
• Fetch by ID — pass a list of arXiv IDs in idList to retrieve exact papers.
• Sort control — by submittedDate, lastUpdatedDate or relevance, ascending or descending.
• Full abstracts included, not truncated snippets — ready for embedding or summarisation.
• Direct PDF URLs on every record, so downstream download steps are trivial.

What people use it for

• Literature reviews — pull everything published on a topic in a date window.
• RAG and embeddings — abstracts are clean, well-formed text, ideal for vector search.
• Research monitoring — schedule a daily run on a category and get new preprints automatically.
• Competitive/technical intelligence — track which labs and authors publish in your field.
• Dataset building — assemble a labelled corpus of abstracts by category for ML experiments.

Query syntax cheat-sheet

Plain text works fine, but arXiv's fielded syntax gives you much sharper results:

Goal	Query
Phrase in the title	ti:"reinforcement learning"
Specific author	au:hinton
Phrase in the abstract	abs:"chain of thought"
Everything by an author in a field	au:lecun AND cat:cs.LG
Two topics together	all:transformer AND all:distillation
Exclude a topic	all:llm ANDNOT all:survey

Useful categories

cs.LG machine learning · cs.CL computation and language (NLP) · cs.CV computer vision · cs.AI artificial intelligence · cs.CR cryptography and security · stat.ML statistics/ML · q-fin.TR trading and market microstructure · math.OC optimisation. The full list is on arXiv's category taxonomy page.

Input

Everything is optional. query plus category covers most searches; idList overrides searching and fetches specific papers instead.

Field	Type	Default	What it does
query	string	transformer	Free text (e.g. 'diffusion models') or fielded arXiv syntax (ti:, au:, abs:, all:, cat:). Example: ti:"reinforcement learning" AND au:silver
category	string	—	arXiv category filter, e.g. cs.LG, cs.CL, stat.ML, q-fin.TR. ANDed with the query.
idList	array	[]	Fetch specific papers by ID, e.g. 2301.00001. Overrides paging behavior.
maxResults	integer	50	Cap on papers returned. Range 1–2000.
sortBy	string	submittedDate	Result ordering. Options: relevance, lastUpdatedDate, submittedDate.
sortOrder	string	descending	Ascending or descending. Options: ascending, descending.

Input example

{
  "query": "machine learning",
  "category": "cs.LG",
  "maxResults": 10,
  "sortBy": "submittedDate",
  "sortOrder": "descending"
}

Output

One record per paper, including the arXiv ID and version, the full abstract, the author list, category tags, and links to both the abstract page and the PDF.

Every dataset record contains: arxivId, version, title, abstract, authors, primaryCategory, categories, published, updated, doi, journalRef, comment, absUrl, pdfUrl, fetchedAt.

Output example

A real record from a run of this Actor:

{
  "arxivId": "2608.19181",
  "version": "v1",
  "title": "Beyond Teacher Likelihood: Group-Calibrated On-Policy Distillation for Long-Context Reasoning",
  "abstract": "On-policy distillation (OPD) trains a student on its own responses using dense token-level guidance from a stronger teacher. In long-context tasks, however, token-level teacher support can favor locally plausible respons …",
  "authors": [
    "Zhu Zhang",
    "Jixun Wang",
    "Xiaoang Xu",
    "… +6 more"
  ],
  "primaryCategory": "cs.LG",
  "categories": [
    "cs.LG",
    "cs.AI",
    "cs.CL"
  ],
  "published": "2026-08-19T17:54:58Z",
  "updated": "2026-08-19T17:54:58Z",
  "doi": null,
  "journalRef": null,
  "comment": "20 pages, 5 figures",
  "absUrl": "https://arxiv.org/abs/2608.19181",
  "pdfUrl": "https://arxiv.org/pdf/2608.19181v1",
  "fetchedAt": "2026-08-20T02:12:58.385Z"
}

Export the dataset as JSON, CSV, Excel, XML or JSONL from the Console, or pull it programmatically through the Apify API and any of the official clients.

How to use it

1. Click Try for free (or Start if you already have an Apify account).
2. Fill in the input fields described above — the defaults already produce a working run.
3. Press Start and watch the log; results stream into the dataset as they are found.
4. When the run finishes, open the Output/Storage tab and export as JSON, CSV or Excel.

Runs can be scheduled (hourly, daily, weekly) and wired into Slack, Google Sheets, Zapier, Make, webhooks or your own backend through Apify integrations. Everything the Console does is also available over the Apify API.

Pricing

This Actor is billed on Apify's pay-per-event model: a small charge when a run starts, plus a charge for each result written to the dataset. You only pay for records you actually receive — a run that finds nothing costs only the start event. Current rates are always shown on the Pricing tab of this page, and the run log prints your usage as it goes.

Free-plan credits from Apify cover a large amount of light usage, so you can evaluate the Actor before committing to anything.

FAQ

Do I need an arXiv API key?

No. arXiv's API is public and free. The Actor paces its requests to stay inside arXiv's published rate guidance.

Can I get the full paper text, not just the abstract?

Each record includes a direct pdfUrl. Fetch and parse that PDF in a downstream step if you need the body text — this Actor deliberately returns metadata plus abstract, which is what most pipelines index.

How do I fetch new papers only?

Sort by submittedDate descending and schedule the run. Compare arxivId values against what you already stored, or push the dataset into your own database with a unique index on arxivId.

Why does a paper show a version suffix?

arXiv preprints are revised. version (v1, v2, …) tells you which revision the record describes — worth storing if you track changes.

Is there a result limit?

maxResults caps each run. arXiv's API paginates large result sets, so for very large harvests run several narrower queries (by category or date range) rather than one enormous one.

Other Actors by cynix_dev

Actor	What it does
CoinGecko Markets — Crypto Data API	Live cryptocurrency market data from CoinGecko as clean typed JSON: price, market cap, volume, 24h change, ATH/ATL, …
FX Rates & History	Latest and historical foreign exchange rates (ECB reference data) as clean, typed dataset records.
USGS Earthquakes — GeoJSON Extractor	Pull live and historical earthquakes from the USGS FDSN event service as clean typed JSON: magnitude, place, time, lat/lon/depth, …
Launch Library 2 — Rocket Launch Tracker	Upcoming, previous, and specific rocket launches from The Space Devs' Launch Library 2 API.
Open Food Facts Extractor	Search and extract food-product data from Open Food Facts as clean typed JSON: name, brand, ingredients, allergens, nutrition …
Website to RAG Chunks	Crawl any website and turn its pages into clean, chunked, metadata-rich Markdown records ready for RAG pipelines, vector stores, …

Legal and responsible use

This Actor collects only publicly available information. You are responsible for how you use the data, including compliance with the target site's Terms of Service, robots directives, copyright, and data protection law such as GDPR and CCPA. Do not use it to gather personal data without a lawful basis.

Support and feedback

Found a bug, hit a site change, or need an extra field? Open a ticket on the Issues tab of this Actor — issues are read and fixed. Feature requests and custom-scraper enquiries are welcome through the same channel.