# arXiv Paper Scraper — AI Research, Abstracts & PDF Links (`bovi/arxiv-scraper`) Actor Search arXiv papers by keyword, ID list, or category. Returns title, authors, abstract, categories, PDF URL, DOI, publish dates, and parse\_confidence. Official Atom XML API — no proxy, no auth. Pay per result. - **URL**: https://apify.com/bovi/arxiv-scraper.md - **Developed by:** [Vitalii Bondarev](https://apify.com/bovi) (community) - **Categories:** Developer tools, MCP servers, AI - **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $1.84 / 1,000 arxiv papers This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## arXiv Paper Scraper — AI Research, Abstracts & PDF Links | from $1.50/1K Used by AI/ML researchers building training datasets, literature review automation tools, and academic institutions monitoring emerging research. **Pricing:** $1.50 per 1,000 papers (metadata + PDF links). Include full abstracts for an additional $0.50/1k (`includeAbstract=true`). **Search arXiv research papers by keyword, arXiv ID, or category. Returns structured metadata including title, authors, abstract, categories, PDF URL, DOI, publish dates, and parse_confidence. Uses the official arXiv Atom XML API — no proxy, no auth required. Pay per result.** --- ### What is this actor for? arXiv (arxiv.org) is the world's largest open-access repository for scientific preprints, with over 2.3 million papers in physics, mathematics, computer science, quantitative biology, and more. Researchers, analysts, and ML practitioners use arXiv daily for literature discovery. This actor provides a clean, structured interface to the official arXiv API — no scraping, no fragility, no proxy costs for buyers. ### What data does it return? Each result row has 17 fields: | Field | Description | |---|---| | `arxiv_id` | arXiv paper ID (e.g. `2501.05032v2`) | | `title` | Paper title | | `summary` | Full abstract text (toggle off with `includeAbstract=false`) | | `authors` | List of author name strings | | `primary_category` | Primary arXiv category (e.g. `cs.CL`, `cs.LG`) | | `categories` | All categories the paper is cross-listed in | | `published_at` | Original submission date (ISO 8601 UTC) | | `updated_at` | Last updated date (ISO 8601 UTC) | | `pdf_url` | Direct PDF link | | `abstract_url` | arXiv abstract page URL | | `doi` | DOI if available (often present for journal-published papers) | | `comment` | Author comment (conference/workshop note, code link, etc.) | | `journal_ref` | Journal reference if published | | `query` | Search query that returned this paper (null for ID lookups) | | `scraped_at` | Timestamp of the scrape run (ISO 8601 UTC) | | `parse_confidence` | Quality score 0.0–1.0; 1.0 = all fields parsed correctly | | `warnings` | List of warning codes when confidence < 1.0 | ### Input | Parameter | Type | Default | Description | |---|---|---|---| | `searchQueries` | string[] | — | Keyword queries. Supports arXiv field prefixes: `ti:` (title), `au:` (author), `abs:` (abstract), `cat:` (category), `all:` (all fields). | | `idList` | string[] | — | Specific arXiv IDs to fetch (e.g. `2501.05032`). | | `category` | string | — | Filter to category (e.g. `cs.LG`, `cs.CL`). Applied as `cat:` prefix when no query given. | | `sortBy` | enum | `relevance` | `relevance` / `lastUpdatedDate` / `submittedDate` | | `maxItems` | integer | `100` | Total results cap. 0 = unlimited. | | `includeAbstract` | boolean | `true` | Include full abstract in output. | | `pageSize` | integer | `50` | Results per API page (1–200). | | `delaySeconds` | integer | `3` | Delay between API calls. arXiv requires ≥3 seconds. | ### Usage examples **Search for recent LLM papers:** ```json { "searchQueries": ["ti:\"large language models\""], "sortBy": "submittedDate", "maxItems": 50 } ```` **Look up specific papers by ID:** ```json { "idList": ["2501.05032", "2402.14679", "1706.03762"] } ``` **Browse a category:** ```json { "category": "cs.LG", "sortBy": "submittedDate", "maxItems": 100 } ``` **Combined query + category:** ```json { "searchQueries": ["abs:\"retrieval augmented generation\""], "category": "cs.CL", "sortBy": "submittedDate", "maxItems": 200 } ``` ### arXiv search query syntax arXiv supports prefix operators for targeted searches: - `ti:transformer` — papers with "transformer" in the title - `au:"Yann LeCun"` — papers by Yann LeCun - `abs:diffusion` — papers with "diffusion" in the abstract - `cat:cs.CV` — papers in Computer Vision - `all:"attention mechanism"` — any field contains phrase - Combine with `AND`, `OR`, `ANDNOT` ### Pricing examples | Run | Items | Cost | |---|---|---| | 100 papers (metadata, no abstract) | 100 | ~$0.15 | | 100 papers with full abstracts | 100 | ~$0.20 | | 1,000 papers (metadata) | 1,000 | ~$1.50 | | Weekly category monitor (cs.LG, 200 papers) | 800/mo | ~$1.20/mo | You only pay for papers successfully pushed to the dataset. ### FAQ **Do I need a proxy or API key?** No. The arXiv API is fully public and requires no authentication. Zero proxy cost for buyers. **What formats can I export results in?** JSON, CSV, JSONL, Excel — all via Apify dataset export. The PDF URL field lets you download source PDFs programmatically. **Can I monitor a category for new papers on a schedule?** Yes. Use Apify Schedules + `sortBy: submittedDate` to fetch the latest papers daily or weekly. Combine with a webhook to push new papers into Slack, Notion, or a literature database. **What if the actor returns empty results?** Check your `searchQueries` syntax — arXiv requires the correct field prefix format (e.g. `ti:"attention"` not `title:attention`). For ID lookups, verify the ID exists on arxiv.org. The `OUTPUT` key-value store reports any failed queries with reason codes. ### Edge cases & known limits - **Rate limit**: arXiv requests ≥3 seconds between API calls. The actor respects this automatically. Large runs may take time — plan accordingly. - **max\_results cap**: arXiv API caps single-page results at 2000. The actor paginates automatically for large queries. - **No auth required**: The API is publicly accessible with no token or proxy. - **Versioned IDs**: arXiv returns versioned IDs (e.g. `2501.05032v2`). The `arxiv_id` field contains the version-qualified ID from the response. - **LaTeX in abstracts**: arXiv abstracts may contain LaTeX markup (e.g. `$\textbf{...}$`). The `summary` field returns raw text — no LaTeX rendering. - **Not affiliated with arXiv**: This actor uses the official public API. arXiv and Cornell University are not affiliated with this actor. ### Why choose this actor? - **parse\_confidence on every record**: immediately see if parsing succeeded. No silent failures. - **Zero proxy cost**: official API, no residential proxies needed — buyers pay nothing for egress. - **All 3 modes**: search + ID lookup + category browse in a single actor. - **Full metadata**: doi, comment, journal\_ref, versioned PDF URLs — fields competitors skip. - **Batch-friendly**: handles multiple queries in one run with automatic pagination. ### Competitor comparison | | This actor | scrapestorm/arxiv | Any REST client | |---|---|---|---| | Official Atom XML API | Yes | Unknown | Yes (DIY) | | `parse_confidence` | Yes | No | No | | Abstract toggle (save cost) | Yes | Always-on | No | | Category + keyword + ID in one actor | Yes | Partial | Separate calls | | Proxy needed | No | Unknown | No | | Price | $1.50/1k | $9.99/1k | Free (DIY infra) | We are 6.6× cheaper than scrapestorm for academic paper data. ### AI training datasets Pull all papers in `cs.LG` since 2020 — abstract + title + category = a ready-made AI research corpus. Use `includeAbstract=true` for full text, `includeAbstract=false` for fast metadata-only runs. ### Use with AI agents (MCP) Ask your AI agent "find me the 10 most cited papers on RAG from 2024" — this actor returns structured paper metadata with abstract and PDF link, ready for downstream summarization. Point your MCP client at this tool: ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=bovi/arxiv-scraper", "--header", "Authorization: Bearer " ] } } } ``` Minimal call: ```json { "searchQueries": ["large language models"], "maxItems": 20 } ``` ### crossref-scraper cross-sell Need journal metadata, citation counts, or DOIs for published versions? Use our [crossref-scraper](https://apify.com/bovi/crossref-scraper) alongside this actor to enrich arXiv papers with citation data. ### Integrations Built for ML researchers and R\&D teams building literature datasets and monitoring new papers by keyword or category — the JSON/dataset output drops into the tools you already run, no glue code: - **n8n / Make / Zapier** — trigger a run or pipe every new dataset item into 500+ apps (Google Sheets, Airtable, Slack, HubSpot, your database) with no code: [n8n](https://docs.apify.com/platform/integrations/n8n), [Make](https://docs.apify.com/platform/integrations/make), [Zapier](https://docs.apify.com/platform/integrations/zapier). - **Webhooks** — fire your own endpoint the moment a run finishes, to push results straight into your pipeline ([docs](https://docs.apify.com/platform/integrations/webhooks)). - **MCP server** — expose this actor as a tool to Claude, Cursor, or any [MCP client](https://mcp.apify.com) so an AI agent can pull this data mid-conversation ([guide](https://blog.apify.com/how-to-use-mcp/)). - **API & SDKs** — fetch the dataset as JSON, CSV, or Excel through the Apify REST API or the Python / JS SDKs. See all [Apify integrations](https://apify.com/integrations). # Actor input Schema ## `searchQueries` (type: `array`): Keyword search queries. Supports arXiv field prefixes: ti: (title), au: (author), abs: (abstract), cat: (category), all: (all fields). Example: ti:"transformer attention" OR au:"Yann LeCun". ## `idList` (type: `array`): Fetch specific papers by arXiv ID. Accepts short IDs (e.g. 2501.05032) or full abs URLs. Version suffixes optional (e.g. 2501.05032v2). Example: \["1706.03762", "2501.05032"]. ## `category` (type: `string`): Filter results to an arXiv category. Examples: cs.LG (Machine Learning), cs.CL (Computation and Language), stat.ML, physics.hep-th, math.CO. Applied as cat: prefix when no query given. ## `sortBy` (type: `string`): Sort order for search results. Allowed: relevance, lastUpdatedDate, submittedDate. ## `maxItems` (type: `integer`): Maximum total papers to return across all queries and ID lookups. 0 = no limit (up to API page limit per query). ## `includeAbstract` (type: `boolean`): Include the full abstract text in the output. Disable to reduce output size when you only need metadata. ## `pageSize` (type: `integer`): Number of results fetched per arXiv API call (1–200). Larger values mean fewer API calls but more memory. Respect arXiv 3-second delay between pages. ## `delaySeconds` (type: `integer`): Seconds to wait between consecutive arXiv API requests. arXiv requests ≥3 seconds. Do not set below 3. ## Actor input object example ```json { "searchQueries": [ "ti:\"large language models\"", "abs:\"retrieval augmented generation\"" ], "idList": [ "1706.03762" ], "sortBy": "relevance", "maxItems": 100, "includeAbstract": true, "pageSize": 50, "delaySeconds": 3 } ``` # Actor output Schema ## `results` (type: `string`): Dataset containing Arxiv Scraper records (arxiv\_id, title, authors, primary\_category, published\_at, updated\_at, pdf\_url, doi, parse\_confidence, warnings). # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "searchQueries": [ "ti:\"large language models\"", "abs:\"retrieval augmented generation\"" ], "idList": [ "1706.03762" ], "sortBy": "relevance", "maxItems": 100, "includeAbstract": true, "pageSize": 50, "delaySeconds": 3 }; // Run the Actor and wait for it to finish const run = await client.actor("bovi/arxiv-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "searchQueries": [ "ti:\"large language models\"", "abs:\"retrieval augmented generation\"", ], "idList": ["1706.03762"], "sortBy": "relevance", "maxItems": 100, "includeAbstract": True, "pageSize": 50, "delaySeconds": 3, } # Run the Actor and wait for it to finish run = client.actor("bovi/arxiv-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "searchQueries": [ "ti:\\"large language models\\"", "abs:\\"retrieval augmented generation\\"" ], "idList": [ "1706.03762" ], "sortBy": "relevance", "maxItems": 100, "includeAbstract": true, "pageSize": 50, "delaySeconds": 3 }' | apify call bovi/arxiv-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=bovi/arxiv-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "arXiv Paper Scraper — AI Research, Abstracts & PDF Links", "description": "Search arXiv papers by keyword, ID list, or category. Returns title, authors, abstract, categories, PDF URL, DOI, publish dates, and parse_confidence. Official Atom XML API — no proxy, no auth. Pay per result.", "version": "0.1", "x-build-id": "7U17FS9fmdj5G15fh" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/bovi~arxiv-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-bovi-arxiv-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/bovi~arxiv-scraper/runs": { "post": { "operationId": "runs-sync-bovi-arxiv-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/bovi~arxiv-scraper/run-sync": { "post": { "operationId": "run-sync-bovi-arxiv-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "searchQueries": { "title": "Search queries", "type": "array", "description": "Keyword search queries. Supports arXiv field prefixes: ti: (title), au: (author), abs: (abstract), cat: (category), all: (all fields). Example: ti:\"transformer attention\" OR au:\"Yann LeCun\".", "items": { "type": "string" } }, "idList": { "title": "arXiv IDs", "type": "array", "description": "Fetch specific papers by arXiv ID. Accepts short IDs (e.g. 2501.05032) or full abs URLs. Version suffixes optional (e.g. 2501.05032v2). Example: [\"1706.03762\", \"2501.05032\"].", "items": { "type": "string" } }, "category": { "title": "Category filter", "type": "string", "description": "Filter results to an arXiv category. Examples: cs.LG (Machine Learning), cs.CL (Computation and Language), stat.ML, physics.hep-th, math.CO. Applied as cat: prefix when no query given." }, "sortBy": { "title": "Sort by", "enum": [ "relevance", "lastUpdatedDate", "submittedDate" ], "type": "string", "description": "Sort order for search results. Allowed: relevance, lastUpdatedDate, submittedDate." }, "maxItems": { "title": "Max items total", "minimum": 0, "type": "integer", "description": "Maximum total papers to return across all queries and ID lookups. 0 = no limit (up to API page limit per query)." }, "includeAbstract": { "title": "Include abstract / summary", "type": "boolean", "description": "Include the full abstract text in the output. Disable to reduce output size when you only need metadata." }, "pageSize": { "title": "Results per API page", "minimum": 1, "maximum": 200, "type": "integer", "description": "Number of results fetched per arXiv API call (1–200). Larger values mean fewer API calls but more memory. Respect arXiv 3-second delay between pages." }, "delaySeconds": { "title": "Delay between API calls (seconds)", "minimum": 3, "maximum": 60, "type": "integer", "description": "Seconds to wait between consecutive arXiv API requests. arXiv requests ≥3 seconds. Do not set below 3." } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```