# arXiv Papers Scraper (`maximedupre/arxiv-papers-scraper`) Actor Search public arXiv paper records by query, category, author, title, abstract, date range, or advanced syntax. Export paper metadata, URLs, DOI values, and monitoring-ready results. - **URL**: https://apify.com/maximedupre/arxiv-papers-scraper.md - **Developed by:** [Maxime Duprรฉ](https://apify.com/maximedupre) (community) - **Categories:** Education, Developer tools, Automation - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing $1.80 / 1,000 paper results This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ### ๐Ÿ”Ž arXiv paper search for research feeds arXiv Papers Scraper searches public arXiv paper records and saves one dataset row per matching paper. Use it to collect paper metadata for literature monitoring, category feeds, author tracking, date-limited searches, or advanced arXiv queries without user credentials or an API key. - Run an [arXiv papers scraper](https://apify.com/maximedupre/arxiv-papers-scraper/examples/arxiv-papers-scraper) to collect matching research records for a dataset or dashboard. - Use an [arXiv paper monitor](https://apify.com/maximedupre/arxiv-papers-scraper/examples/arxiv-paper-monitor) to watch recurring searches and return newly delivered papers. - Build an [arXiv category feed](https://apify.com/maximedupre/arxiv-papers-scraper/examples/arxiv-category-feed) for selected category codes such as `cs.AI` or `stat.ML`. - Run [arXiv author search](https://apify.com/maximedupre/arxiv-papers-scraper/examples/arxiv-author-search) workflows to track papers from specific researchers. - Create an [arXiv metadata export](https://apify.com/maximedupre/arxiv-papers-scraper/examples/arxiv-metadata-export) with IDs, titles, authors, categories, dates, DOI values, and URLs. - Use [advanced arXiv search](https://apify.com/maximedupre/arxiv-papers-scraper/examples/advanced-arxiv-search) when simple keywords are not precise enough for your query. - Collect [arXiv PDF URLs](https://apify.com/maximedupre/arxiv-papers-scraper/examples/arxiv-pdf-urls) for direct source links without downloading or hosting files. #### ๐Ÿ“ฆ What data you get Each saved result represents one arXiv paper record. Depending on source metadata and your input settings, rows can include: | Field | Description | | --- | --- | | `arxivId` | Source-native arXiv identifier. | | `title` | Paper title as published by arXiv. | | `authors` and `authorCount` | Authors in source order and the number of listed authors. | | `abstract` | Paper abstract when requested and available. | | `primaryCategory` and `categories` | Primary arXiv category and all listed categories, including cross-listings. | | `isCrossListed` | Whether the paper appears in more than one arXiv category. | | `submittedDate` and `updatedDate` | Initial submission date and most recent update date. | | `abstractUrl` and `pdfUrl` | Canonical arXiv abstract page URL and direct source PDF URL. | | `doi`, `comment`, `journalRef` | Optional DOI, author comment, and journal reference when arXiv provides them. | | `abstractReadMinutes` | Estimated abstract reading time when the abstract is included. | | `metadataCompleteness` | Ratio from 0 to 1 for useful metadata completeness in the emitted row. | | `extractionTimestamp` | ISO timestamp when the Actor extracted the record. | #### โ–ถ๏ธ How to run it 1. Enter a free-text search query, choose arXiv categories, add author, title, or abstract filters, or use the advanced arXiv query field. 2. Optionally filter by submitted-date range and choose sorting by relevance, newest submitted papers, or recently updated papers. 3. Set the maximum number of papers and decide whether to include abstracts. 4. Turn on **Only new papers** for scheduled monitoring when you want recurring runs to focus on newly delivered papers. 5. Start the Actor and export the dataset from Apify in your preferred format. #### โš™๏ธ Input options | Input | What it does | | --- | --- | | Search query | Searches public arXiv records by general free text. | | arXiv categories | Limits results to selected arXiv category codes. | | Advanced arXiv query | Uses arXiv advanced query syntax for precise searches. | | Author filter | Filters by paper authors. | | Title filter | Filters by terms in the paper title. | | Abstract filter | Filters by terms in the abstract. | | Submitted from / Submitted to | Filters by submitted-date range. | | Sort results by | Orders by relevance, newest submitted papers, or recently updated papers. | | Include abstracts | Adds abstracts and abstract read-minute estimates when available. | | Only new papers | Returns papers not delivered by earlier runs with the same input. | | Maximum papers | Caps the number of paper records returned. | #### ๐Ÿงพ Output example ```json { "arxivId": "2401.12345", "title": "Example Paper Title", "authors": ["Jane Doe", "John Smith"], "authorCount": 2, "abstract": "This paper studies an example research problem.", "primaryCategory": "cs.LG", "categories": ["cs.LG", "stat.ML"], "isCrossListed": true, "submittedDate": "2024-01-23T00:00:00.000Z", "updatedDate": "2024-02-01T00:00:00.000Z", "abstractUrl": "https://arxiv.org/abs/2401.12345", "pdfUrl": "https://arxiv.org/pdf/2401.12345", "doi": "10.0000/example.doi", "comment": "12 pages, 3 figures", "journalRef": "Example Journal 1 (2024)", "abstractReadMinutes": 1, "metadataCompleteness": 1, "extractionTimestamp": "2026-07-08T12:00:00.000Z" } ```` Optional fields such as `abstract`, `doi`, `comment`, and `journalRef` appear only when requested or available from arXiv. #### ๐Ÿ’ณ Pricing This actor uses pay-per-event pricing. You are charged only when the Actor saves an accepted arXiv paper record to the dataset. Empty runs, setup checks, failed runs, and searches with no matching papers are not charged as paper results. #### ๐Ÿ”Œ Integrations https://www.youtube.com/watch?v=bNACk1\_S\_6w\&list=PLObrtcm1Kw6MUrlLNDbK9QRg8VDJg0gOW\&index=4 Use this actor with Apify integrations, API clients, webhooks, scheduled runs, and dataset exports. Send results to spreadsheets, BI tools, databases, or your own app after each run. #### โ“ FAQ ##### ๐Ÿ” Can I use this as an arXiv paper monitor? Yes. Use categories, queries, authors, title or abstract filters, or date ranges with scheduled Apify runs. The **Only new papers** option is designed for recurring monitoring so feeds can focus on newly delivered papers. ##### ๐Ÿ” Does it support advanced arXiv search syntax? Yes. You can use the advanced arXiv query input when simple search terms, category filters, author filters, title terms, or abstract terms are not specific enough. ##### ๐Ÿ“„ Does it download PDF files? No. The Actor returns the direct source `pdfUrl` from arXiv. It does not download, mirror, host, or transform PDF files. ##### ๐Ÿง  Does it summarize or rank papers? No. It returns source metadata and derived mechanical fields such as author count, cross-listing status, abstract read minutes, and metadata completeness. It does not create LLM summaries, recommendations, rankings, or quality judgments. ##### ๐Ÿ” Do I need an arXiv account or API key? No. The Actor searches public arXiv paper records without requiring user credentials or an API key. ##### ๐Ÿ“ญ What happens if no papers match my query? A no-result search is treated as a successful empty run. The pricing model charges only for accepted paper records saved to the dataset. ##### ๐Ÿงฉ Is this an alternative to the arXiv API? Use it when you want arXiv paper search and monitoring inside Apify with scheduling, webhooks, API access, and dataset exports. If you only need to call arXiv directly from your own code, the source API may be enough. ### ๐Ÿ“ Changelog - 0.1.0: Initial release. ### ๐Ÿ†˜ Support For issues, questions, or feature requests, [file a ticket](https://console.apify.com/actors/maximedupre~arxiv-papers-scraper/issues) and I'll fix or implement it in less than 24h ๐Ÿซก ### ๐Ÿ”— Other actors - [URL to BibTeX Converter โ†—](https://apify.com/maximedupre/url-to-bibtex-converter) - Use it for related arxiv and papers workflows. - [ORCID Researcher Profile Search โ†—](https://apify.com/maximedupre/orcid-researcher-profile-search) - Use it for related syntax and doi workflows. - [World Bank Projects Scraper โ†—](https://apify.com/maximedupre/world-bank-projects-scraper) - Use it for related abstract and education workflows. - [CMS Open Payments Scraper โ†—](https://apify.com/maximedupre/cms-open-payments-scraper) - Use it for related education workflows. - [GitHub Security Advisories Scraper โ†—](https://apify.com/maximedupre/github-security-advisories-scraper) - Use it for related feeds workflows. **Made with โค๏ธ by Maxime Duprรฉ** # Actor input Schema ## `searchQuery` (type: `string`): Search arXiv paper records by general free text. Leave empty when using only field filters, categories, dates, or advanced query syntax. ## `categories` (type: `array`): Limit results to one or more arXiv category codes. Use source category IDs such as cs.AI, stat.ML, math.NT, or physics.optics. ## `advancedQuery` (type: `string`): Use arXiv query syntax for precise searches. Combine with the other filters only when you want a narrower result set. ## `author` (type: `string`): Return papers matching this author name. Use when author is a required part of the search. ## `title` (type: `string`): Return papers with matching title text. Use specific terms to keep results focused. ## `abstract` (type: `string`): Return papers with matching abstract text. Useful for concepts that may not appear in the title. ## `submittedFrom` (type: `string`): Include papers submitted on or after this date. ## `submittedTo` (type: `string`): Include papers submitted on or before this date. ## `sortBy` (type: `string`): Choose how arXiv matches are ordered before the result limit is applied. ## `includeAbstract` (type: `boolean`): Save each paper abstract in the output. Turn off for smaller records when abstracts are not needed. ## `onlyNewPapers` (type: `boolean`): For recurring runs, return papers not delivered by earlier runs with the same input. Leave off to collect the current matching set each run. ## `maxResults` (type: `integer`): Maximum number of paper records to return for this run. Use a lower value for broad searches and monitoring feeds. ## Actor input object example ```json { "searchQuery": "graph neural networks", "sortBy": "relevance", "includeAbstract": true, "onlyNewPapers": false, "maxResults": 100 } ``` # Actor output Schema ## `results` (type: `string`): View arXiv paper metadata rows in the default dataset. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "searchQuery": "graph neural networks" }; // Run the Actor and wait for it to finish const run = await client.actor("maximedupre/arxiv-papers-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`๐Ÿ’พ Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // ๐Ÿ“š Want to learn more ๐Ÿ“–? Go to โ†’ https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "searchQuery": "graph neural networks" } # Run the Actor and wait for it to finish run = client.actor("maximedupre/arxiv-papers-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("๐Ÿ’พ Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # ๐Ÿ“š Want to learn more ๐Ÿ“–? Go to โ†’ https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "searchQuery": "graph neural networks" }' | apify call maximedupre/arxiv-papers-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=maximedupre/arxiv-papers-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "arXiv Papers Scraper", "description": "Search public arXiv paper records by query, category, author, title, abstract, date range, or advanced syntax. Export paper metadata, URLs, DOI values, and monitoring-ready results.", "version": "0.1", "x-build-id": "eZGdrziReuhXHWwR9" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/maximedupre~arxiv-papers-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-maximedupre-arxiv-papers-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/maximedupre~arxiv-papers-scraper/runs": { "post": { "operationId": "runs-sync-maximedupre-arxiv-papers-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/maximedupre~arxiv-papers-scraper/run-sync": { "post": { "operationId": "run-sync-maximedupre-arxiv-papers-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "searchQuery": { "title": "Search query", "type": "string", "description": "Search arXiv paper records by general free text. Leave empty when using only field filters, categories, dates, or advanced query syntax." }, "categories": { "title": "arXiv categories", "type": "array", "description": "Limit results to one or more arXiv category codes. Use source category IDs such as cs.AI, stat.ML, math.NT, or physics.optics.", "items": { "type": "string" } }, "advancedQuery": { "title": "Advanced arXiv query", "type": "string", "description": "Use arXiv query syntax for precise searches. Combine with the other filters only when you want a narrower result set." }, "author": { "title": "Author filter", "type": "string", "description": "Return papers matching this author name. Use when author is a required part of the search." }, "title": { "title": "Title filter", "type": "string", "description": "Return papers with matching title text. Use specific terms to keep results focused." }, "abstract": { "title": "Abstract filter", "type": "string", "description": "Return papers with matching abstract text. Useful for concepts that may not appear in the title." }, "submittedFrom": { "title": "Submitted from", "type": "string", "description": "Include papers submitted on or after this date." }, "submittedTo": { "title": "Submitted to", "type": "string", "description": "Include papers submitted on or before this date." }, "sortBy": { "title": "Sort results by", "enum": [ "relevance", "submittedDate", "lastUpdatedDate" ], "type": "string", "description": "Choose how arXiv matches are ordered before the result limit is applied.", "default": "relevance" }, "includeAbstract": { "title": "Include abstracts", "type": "boolean", "description": "Save each paper abstract in the output. Turn off for smaller records when abstracts are not needed.", "default": true }, "onlyNewPapers": { "title": "Only new papers", "type": "boolean", "description": "For recurring runs, return papers not delivered by earlier runs with the same input. Leave off to collect the current matching set each run.", "default": false }, "maxResults": { "title": "Maximum papers", "minimum": 1, "maximum": 1000, "type": "integer", "description": "Maximum number of paper records to return for this run. Use a lower value for broad searches and monitoring feeds.", "default": 100 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```