# arXiv Papers Scraper (`hipersoft/arxiv-scraper`) Actor Search and extract arXiv papers: title, abstract, authors, categories, published/updated dates, DOI, journal reference, and direct PDF links. Filter by category and sort by newest, recently updated or relevance. Fetch by arXiv ID too. For research, lit reviews and AI/ML monitoring. No key. - **URL**: https://apify.com/hipersoft/arxiv-scraper.md - **Developed by:** [hiper soft](https://apify.com/hipersoft) (community) - **Categories:** News - **Stats:** 2 total users, 0 monthly users, 76.9% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.001 / paper scraped This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## arXiv Papers Scraper — Abstracts, Authors, Categories & PDF Links Search and extract papers from **arXiv.org** as clean, structured JSON: title, full abstract, authors, subject categories, submission/update dates, DOI, journal reference, and direct **PDF links**. Search by keyword, filter by category, sort by newest or relevance, or fetch exact papers by arXiv ID. No account, no API key. Built on arXiv's official export API, so results are complete and reliable — ideal for literature reviews, research monitoring, and building AI/ML datasets. ### Features - 🔎 **Keyword & field search** — plain terms or structured queries (`ti:`, `au:`, `abs:`, `cat:`) across the full arXiv corpus - 🗂️ **Category filters** — narrow to fields like `cs.CL`, `cs.AI`, `math.PR`, `stat.ML`, and combine them - 🆔 **Fetch by ID** — pass a list of arXiv IDs to pull specific papers directly - 📅 **Sort your way** — by relevance, newest submitted, or most recently updated - 📄 **Full abstracts + PDF links** — the complete summary text plus direct `abs` and `pdf` URLs - 👥 **Structured authors** — clean author list, count, and a display string - ⚡ **Fast & polite** — pure JSON, no browser; respects arXiv rate limits with automatic retries ### What you get Each paper is one dataset record: ```json { "query": "large language models", "arxivId": "2403.09676v1", "arxivIdBase": "2403.09676", "title": "Unmasking the Shadows of AI: Investigating Deceptive Capabilities in Large Language Models", "summary": "This research critically navigates the intricate landscape of AI deception...", "authors": ["Linge Guo"], "authorsList": "Linge Guo", "authorCount": 1, "primaryCategory": "cs.CL", "categories": ["cs.CL", "cs.AI"], "published": "2024-02-07T00:21:46Z", "updated": "2024-02-07T00:21:46Z", "doi": null, "journalRef": null, "comment": "AI deception, Large Language Models, ChatGPT", "absUrl": "https://arxiv.org/abs/2403.09676v1", "pdfUrl": "https://arxiv.org/pdf/2403.09676v1" } ```` ### Input ```json { "searchTerms": ["large language models", "cat:cs.CL"], "categories": ["cs.AI"], "sortBy": "newest", "maxResultsPerQuery": 200, "includeAbstract": true } ``` | Field | Description | |-------|-------------| | `searchTerms` | Keywords or structured `field:value` queries. Multi-word phrases are matched as phrases. | | `categories` | arXiv subject categories to filter by (e.g. `cs.CL`, `stat.ML`). Combine with terms or use alone. | | `idList` | Fetch specific papers by arXiv ID (skips search when provided). | | `sortBy` | `relevance` (default), `newest`, or `lastUpdated`. | | `maxResultsPerQuery` | Max papers per query (1–2000). | | `includeAbstract` | Include the full abstract text (default `true`). | ### Use cases - **Literature reviews** — pull every recent paper on a topic with abstracts in one run - **Research monitoring** — track new submissions in your categories, sorted newest-first - **AI/ML datasets** — build corpora of titles + abstracts for classification, search, or RAG - **Citation & metadata** — grab DOIs, journal refs, and PDF links at scale ### Pricing Pay-per-event: you're billed a small amount per run start and per paper scraped — you only pay for the papers you actually get. See the **Pricing** tab for current rates. ### FAQ **Do I need an API key?** No. This Actor uses [arXiv](https://arxiv.org)'s public export API, so there's no account, login or API key required — just add your search terms and run. **How many papers can I scrape per run?** Up to 2,000 papers per query via `maxResultsPerQuery`, and you can pass multiple search terms in one run, so a single run can return many thousands of papers. **Is scraping arXiv legal?** Yes. It reads only openly available metadata through arXiv's official export API and follows its usage guidelines (paced requests, an identified client). It returns arXiv's own metadata as-is and doesn't touch anything behind a login. **What format is the output?** Clean structured JSON — one dataset record per paper — which you can export as JSON, CSV, Excel or feed straight into an API. Each record includes the title, full abstract, authors, categories, dates, DOI and direct PDF links. **Can I filter by category or fetch specific papers?** Yes. Use `categories` to narrow to fields like `cs.CL`, `cs.AI` or `stat.ML`, combine them with keywords, or pass an `idList` of arXiv IDs to pull exact papers directly. ### Related Actors Building a research or literature dataset? These other hipersoft scrapers pair well with this one: - [OpenAlex Scraper](https://apify.com/hipersoft/openalex-scraper) — 250M+ scholarly works with citations, authors and abstracts - [Semantic Scholar Scraper](https://apify.com/hipersoft/semantic-scholar-scraper) — papers with citation and influential-citation metrics across all fields - [Crossref Scraper](https://apify.com/hipersoft/crossref-scraper) — DOIs, citation counts and metadata from 150M+ registered works - [PubMed Scraper](https://apify.com/hipersoft/pubmed-scraper) — biomedical papers, abstracts and MeSH terms from the NLM index ### Notes Uses arXiv's public export API and complies with its usage guidelines (paced requests, identified client). Results reflect arXiv's own metadata. arXiv is a trademark of Cornell University; this Actor is an independent tool and is not affiliated with or endorsed by arXiv. # Actor input Schema ## `searchTerms` (type: `array`): Keywords or phrases to search (title + abstract + full text). You may also pass a raw arXiv query using field prefixes, e.g. ti:"attention is all you need" or au:hinton. ## `categories` (type: `array`): Optional arXiv category codes to restrict to, e.g. cs.CL, cs.LG, stat.ML, math.PR, hep-th. Combined with the search terms. ## `idList` (type: `array`): Optional. Fetch these specific papers by arXiv ID (e.g. 2201.00978 or 1706.03762v5). When set, search terms are ignored. ## `sortBy` (type: `string`): Ordering of search results. ## `maxResultsPerQuery` (type: `integer`): Maximum papers to collect per search term (or in total for an ID list). ## `includeAbstract` (type: `boolean`): Include the paper abstract in the output. ## Actor input object example ```json { "searchTerms": [ "diffusion models", "au:bengio" ], "categories": [ "cs.CL", "cs.LG" ], "idList": [ "1706.03762", "2005.14165" ], "sortBy": "relevance", "maxResultsPerQuery": 50, "includeAbstract": true } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "searchTerms": [ "large language models" ], "categories": [], "idList": [] }; // Run the Actor and wait for it to finish const run = await client.actor("hipersoft/arxiv-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "searchTerms": ["large language models"], "categories": [], "idList": [], } # Run the Actor and wait for it to finish run = client.actor("hipersoft/arxiv-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "searchTerms": [ "large language models" ], "categories": [], "idList": [] }' | apify call hipersoft/arxiv-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=hipersoft/arxiv-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "arXiv Papers Scraper", "description": "Search and extract arXiv papers: title, abstract, authors, categories, published/updated dates, DOI, journal reference, and direct PDF links. Filter by category and sort by newest, recently updated or relevance. Fetch by arXiv ID too. For research, lit reviews and AI/ML monitoring. No key.", "version": "1.0", "x-build-id": "Yqa6TNAPl4C3JNUne" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/hipersoft~arxiv-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-hipersoft-arxiv-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/hipersoft~arxiv-scraper/runs": { "post": { "operationId": "runs-sync-hipersoft-arxiv-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/hipersoft~arxiv-scraper/run-sync": { "post": { "operationId": "run-sync-hipersoft-arxiv-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "searchTerms": { "title": "Search terms", "type": "array", "description": "Keywords or phrases to search (title + abstract + full text). You may also pass a raw arXiv query using field prefixes, e.g. ti:\"attention is all you need\" or au:hinton.", "items": { "type": "string" } }, "categories": { "title": "Categories", "type": "array", "description": "Optional arXiv category codes to restrict to, e.g. cs.CL, cs.LG, stat.ML, math.PR, hep-th. Combined with the search terms.", "items": { "type": "string" } }, "idList": { "title": "arXiv IDs", "type": "array", "description": "Optional. Fetch these specific papers by arXiv ID (e.g. 2201.00978 or 1706.03762v5). When set, search terms are ignored.", "items": { "type": "string" } }, "sortBy": { "title": "Sort by", "enum": [ "relevance", "newest", "lastUpdated" ], "type": "string", "description": "Ordering of search results.", "default": "relevance" }, "maxResultsPerQuery": { "title": "Max results per search term", "minimum": 1, "maximum": 2000, "type": "integer", "description": "Maximum papers to collect per search term (or in total for an ID list).", "default": 50 }, "includeAbstract": { "title": "Include abstract", "type": "boolean", "description": "Include the paper abstract in the output.", "default": true } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```