arXiv Paper Search — Preprint Metadata API

The arXiv preprint server hosts 2M+ scholarly papers across physics, mathematics, computer science, quantitative biology, and quantitative finance. This actor searches the arXiv public API and returns each preprint as a structured JSON record — title, authors, abstract, primary category, PDF link, arXiv ID, submission date — without auth, proxy rotation, or captchas.

Backed by http://export.arxiv.org/api/query. Polite-pool only. No rate-limit gate to burn.

What it returns

One dataset record per paper. Real sample from a 2026-09-01 smoke run on input {"searchQuery":"cat:cs.AI AND ti:transformer","maxResults":2,"sortBy":"submittedDate"} (run id c790fidNndrvf14cG, dataset d43iqmpKu2EV9O8Vp):

{
  "id": "http://arxiv.org/abs/2608.31167v1",
  "arxivId": "2608.31167v1",
  "title": "SUN: Persistent Programs For Language-Grounded Control-to-Learning-to-Real Policies",
  "authors": [
    "Weiqi Wang", "Zhi Li", "Yudong Lei", "David Martinez", "Xiaofeng Gao",
    "Yuxin Jiang", "Chenfanfu Jiang", "Yingnian Wu", "Demetri Terzopoulos", "Ran Gong"
  ],
  "summary": "Bridging model-based control and learned policies in long-horizon manipulation has harbored a silent disagreement: control executes specified objectives, learning amortizes that behavior into a reactive policy, yet existing protocols discard task semantics, leaving rewards hand-crafted and behavior drifting from what control verified. We introduce Semantically UNified (SUN) Programs, typed executables where geometric and contact relations are defined once and compiled into aligned Model Predictive Control (MPC) costs, satisfaction predicates, RL rewards, transition guards, and diagnostics. Our system, Kuafu, driven by large vision language systems, automatically synthesizes SUN Programs from language and scene semantics, screens feasibility via MPC, and retains semantics while training stage-conditioned policies. Across nine tasks, Kuafu achieves 82.03% macro-success, outperforming sparse-reward (35.67%) and Stage-BC (24.75%) baselines...",
  "published": "2026-08-31T17:59:16Z",
  "updated": "2026-08-31T17:59:16Z",
  "primaryCategory": "cs.RO",
  "categories": ["cs.RO", "cs.AI"],
  "pdfUrl": "http://arxiv.org/pdf/2608.31167v1",
  "absUrl": "http://arxiv.org/abs/2608.31167v1",
  "searchQuery": "cat:cs.AI AND ti:transformer",
  "fetchedAt": "2026-09-01T23:47:11.000Z"
}

Input

Matches the actor's INPUT_SCHEMA.json exactly:

Field	Required	Default	Description
searchQuery	yes	cat:cs.AI	arXiv query syntax (cat:cs.AI, au:lecun, all:transformer AND cat:cs.LG, ti:agent). Full reference: https://arxiv.org/help/api/user-manual#query_details
maxResults	no	25	Cap on returned records, 1–200. arXiv recommends 25–100 per run; deep paging is rate-limited.
sortBy	no	submittedDate	relevance | lastUpdatedDate | submittedDate.
start	no	0	0-indexed pagination offset, 0–10000.

Example input:

{
  "searchQuery": "cat:cs.CL AND all:agent AND au:hinton",
  "maxResults": 10,
  "sortBy": "relevance"
}

How it works

The actor translates searchQuery into the arXiv search_query parameter, dispatches one GET /api/query per page, and walks start until maxResults records are collected. Each Atom XML entry is parsed into a flat record and pushed to the dataset. No pagination state lives outside a single run.

Run

make install
make run        # local smoke against INPUT_SCHEMA.json
make push       # build + push to Apify
make run-prod   # cloud smoke on the published build

Limits and gotchas

• No auth, no key, no proxy. arXiv caps unauthenticated traffic at ~1 request per 3 seconds; the actor's single-call-per-run shape stays well below that.
• Atom XML parsing only. arXiv does not expose JSON. The actor ignores everything except the standard <entry> schema.
• Field shape varies. Comment, DOI, journal-ref, and primaryCategory are present only when arXiv provides them; absent fields render as null.
• Sort by submittedDate is the default because relevance is undefined for category-only queries and silently falls back to date order.

License

The arXiv public API is hosted by Cornell University. Metadata is in the public domain. See https://arxiv.org/help/license for per-paper licensing.