# arXiv Scraper - Scientific Papers, Abstracts & PDFs (`benthepythondev/arxiv-scraper`) Actor arXiv Scraper for the official arXiv API. Search 2M+ scientific papers in CS, physics, math and biology by keyword, title, author, abstract or category. Extract title, authors, abstract, categories, DOI, dates and PDF links. For AI/ML research, literature reviews and RAG datasets. - **URL**: https://apify.com/benthepythondev/arxiv-scraper.md - **Developed by:** [ben](https://apify.com/benthepythondev) (community) - **Categories:** AI, Automation - **Stats:** 3 total users, 1 monthly users, 95.5% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing Pay per usage This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## arXiv Scraper — Scientific Papers, Abstracts & PDFs **Search [arXiv.org](https://arxiv.org) — 2M+ open-access scientific papers in physics, CS, math, biology, economics and more — via the official arXiv API.** Built for **AI/ML research, literature reviews, RAG datasets, and research analytics**. Keyless, fast and reliable — no proxy or browser needed. ### What you get Per paper: - **title**, **arxiv_id** - **authors**, **author_count** - **abstract** (full text) - **primary_category**, **categories** - **published**, **updated** - **doi**, **journal_ref**, **comment** - **pdf_url**, **abstract_url** - **scraped_at** ### Why this Actor | | arXiv Scraper | Manual search | Raw arXiv API | |---|---|---|---| | Clean flat JSON output | Yes | — | Atom XML to parse | | Search + filters + paging | Yes | Slow | DIY | | PDF + abstract links | Yes | Manual | Yes | | Pay per result | Yes | — | — | ### Input Use the simple fields, or a raw `searchQuery` for full arXiv syntax. | Field | Type | Description | |---|---|---| | `allFields` | string | Keyword across title/abstract/authors | | `title` | string | Title contains | | `author` | string | Author name | | `abstract` | string | Abstract contains | | `category` | string | arXiv category (e.g. `cs.LG`, `cs.CL`, `cs.AI`) | | `searchQuery` | string | Advanced raw query (overrides the above) | | `sortBy` | string | Relevance / Newest / Recently updated | | `maxResults` | integer | Max papers to return | #### Example: newest LLM papers ```json { "allFields": "large language models", "sortBy": "newest", "maxResults": 100 } ```` #### Example: a category, advanced syntax ```json { "searchQuery": "cat:cs.CL AND abs:retrieval augmented", "sortBy": "newest", "maxResults": 200 } ``` ### Sample output ```json { "arxiv_id": "2605.30351v1", "title": "VideoMLA: Low-Rank Latent KV Cache for Minute-Scale Video", "authors": ["Hidir Yesiltepe", "Jiazhen Hu"], "primary_category": "cs.CV", "categories": ["cs.CV", "cs.AI"], "published": "2026-05-28T17:59:57Z", "abstract": "Long-rollout causal video diffusion...", "pdf_url": "https://arxiv.org/pdf/2605.30351v1", "abstract_url": "https://arxiv.org/abs/2605.30351v1" } ``` ### Use cases - **AI/ML research** — track the latest papers in a field or category - **RAG / LLM datasets** — build corpora of abstracts + PDF links by topic - **Literature reviews** — gather and rank relevant papers fast - **Research analytics** — analyse output by category, author and time ### Pricing Pay-per-result. You are charged only for the papers returned — empty runs cost nothing. ### Notes & legal - Uses the official arXiv API. Please respect arXiv's API terms and rate limits (the Actor waits between requests). - Use data only for lawful purposes. ### Related actors More scrapers from the same author: - [OpenAlex Scraper](https://apify.com/benthepythondev/openalex-scraper) — academic papers & citations - [PubMed Scraper](https://apify.com/benthepythondev/pubmed-scraper) — biomedical literature & citations - [Reddit Archive Scraper](https://apify.com/benthepythondev/reddit-archive-scraper) — years of historical posts & comments # Actor input Schema ## `searchQuery` (type: `string`): Raw arXiv query, e.g. 'all:transformer AND cat:cs.LG'. If set, it overrides the simple fields below. See arXiv API docs for syntax. ## `allFields` (type: `string`): Search across title, abstract, authors and more. ## `title` (type: `string`): Only papers whose title contains this term. ## `author` (type: `string`): Filter by author name. ## `abstract` (type: `string`): Only papers whose abstract contains this term. ## `category` (type: `string`): arXiv category code, e.g. 'cs.LG' (Machine Learning), 'cs.CL' (NLP), 'cs.AI', 'physics.optics', 'math.PR', 'q-bio'. ## `sortBy` (type: `string`): How to sort the results. ## `maxResults` (type: `integer`): Maximum number of papers to return. ## Actor input object example ```json { "searchQuery": "", "allFields": "large language models", "title": "", "author": "", "abstract": "", "category": "", "sortBy": "relevance", "maxResults": 25 } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "allFields": "large language models", "maxResults": 25 }; // Run the Actor and wait for it to finish const run = await client.actor("benthepythondev/arxiv-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "allFields": "large language models", "maxResults": 25, } # Run the Actor and wait for it to finish run = client.actor("benthepythondev/arxiv-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "allFields": "large language models", "maxResults": 25 }' | apify call benthepythondev/arxiv-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=benthepythondev/arxiv-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "arXiv Scraper - Scientific Papers, Abstracts & PDFs", "description": "arXiv Scraper for the official arXiv API. Search 2M+ scientific papers in CS, physics, math and biology by keyword, title, author, abstract or category. Extract title, authors, abstract, categories, DOI, dates and PDF links. For AI/ML research, literature reviews and RAG datasets.", "version": "1.0", "x-build-id": "E5hJ9g3kbcUDIhVzt" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/benthepythondev~arxiv-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-benthepythondev-arxiv-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/benthepythondev~arxiv-scraper/runs": { "post": { "operationId": "runs-sync-benthepythondev-arxiv-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/benthepythondev~arxiv-scraper/run-sync": { "post": { "operationId": "run-sync-benthepythondev-arxiv-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "searchQuery": { "title": "Search Query (advanced, optional)", "type": "string", "description": "Raw arXiv query, e.g. 'all:transformer AND cat:cs.LG'. If set, it overrides the simple fields below. See arXiv API docs for syntax.", "default": "" }, "allFields": { "title": "Keyword (all fields)", "type": "string", "description": "Search across title, abstract, authors and more." }, "title": { "title": "Title contains", "type": "string", "description": "Only papers whose title contains this term.", "default": "" }, "author": { "title": "Author", "type": "string", "description": "Filter by author name.", "default": "" }, "abstract": { "title": "Abstract contains", "type": "string", "description": "Only papers whose abstract contains this term.", "default": "" }, "category": { "title": "Category", "type": "string", "description": "arXiv category code, e.g. 'cs.LG' (Machine Learning), 'cs.CL' (NLP), 'cs.AI', 'physics.optics', 'math.PR', 'q-bio'.", "default": "" }, "sortBy": { "title": "Sort By", "enum": [ "relevance", "newest", "updated" ], "type": "string", "description": "How to sort the results.", "default": "relevance" }, "maxResults": { "title": "Max Results", "minimum": 1, "maximum": 2000, "type": "integer", "description": "Maximum number of papers to return.", "default": 50 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```