# arXiv Papers Scraper (`grit-77/arxiv-papers-scraper`) Actor arXiv papers scraper for research reading lists. Export paper titles, authors, abstracts, categories and dates as JSON with source links; full papers are not downloaded. - **URL**: https://apify.com/grit-77/arxiv-papers-scraper.md - **Developed by:** [Grit](https://apify.com/grit-77) (community) - **Categories:** AI, Education - **Stats:** 2 total users, 1 monthly users, 92.9% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing $1.50 / 1,000 arxiv papers This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## arXiv Papers Scraper arXiv papers scraper returns paper metadata, authors, categories, abstracts, and source links. arXiv API search results include publication and update dates for each returned paper. Use it to prepare reading lists, track a research topic or supply source-linked metadata to an AI workflow. It calls the public [arXiv API](https://info.arxiv.org/help/api/user-manual.html); it does not download full papers. ### What it returns One row per unique version ID within a run, or an error row when a paper/page cannot be processed. | Field | Type | Example value | |---|---|---| | `status` | string | `"ok"` | | `arxivId` | string | `"cond-mat/0011267"` | | `versionId` | string | `"cond-mat/0011267v1"` | | `version` | integer | `1` | | `title` | string | `"The electronic structure of cuprates from high energy spectroscopy"` | | `authors` | array | `["A. Garcia-Lekue", "J. M. Pitarke"]` | | `authorDetails` | array | `[{"name": "A. Garcia-Lekue", "affiliations": []}, {"name": "J. M. Pitarke", "affiliations": []}]` | | `categories` | array | `["cond-mat.supr-con", "cond-mat.str-el"]` | | `primaryCategory` | string | `"cond-mat.supr-con"` | | `published` | string | `"2000-11-15T16:19:15Z"` | | `updated` | string | `"2000-11-15T16:19:15Z"` | | `abstract` | string | `"We report studies of the electronic structure and elementary excitations of doped and undoped cuprate chains, ladders and planes. Using high energy spectroscopies such as x-ray absorption, core level photoemission and angle resolved photoemission spectroscopy, important information regarding the charge distribution and hole dynamics can be obtained. The comparison of the experimental data with suitable theoretical models sets constraints on the parameters entering into the model calculations, and offers insight into the important physical quantities governing the electronic structure of these materials. Recurring themes include the importance of the dimensionality of the Cu-O network (1D->2D) and the crucial role played by the spin-background in determining the dynamics of low-lying excitations in these strongly correlated systems."` | | `absUrl` | string | `"https://arxiv.org/abs/cond-mat/0011267v1"` | | `pdfUrl` | string | `"https://arxiv.org/pdf/cond-mat/0011267v1"` | | `doi` | string | `"10.1016/j.elspec.2014.01.005"` | | `journalRef` | string | `"J. Electron Spectr. Relat. Phenom. 117-118, 203 (2001)"` | | `comment` | string | `"J. Electron Spec. Relat. Phenom.: special issue on electron correlation, in press"` | | `errors` | array | `[]` | | `query` | string | `"(all:electron)"` | | `sourceUrl` | string | `"https://export.arxiv.org/api/query?search_query=%28all%3Aelectron%29&start=0&max_results=5&sortBy=relevance&sortOrder=descending"` | | `fetchedAt` | string | `"2026-10-05T11:39:53.504133+00:00"` | Missing optional metadata is null or an empty list. Error rows retain the available version ID, query and source page. The default dataset has Papers, Metadata and abstracts, and Error rows table views; the Error rows view shows the status column for filtering. ### Input example ```json { "query": "electron", "maxItems": 5 } ``` Use the input schema for the remaining filters and defaults. ### Output example This is a selected-field excerpt of the first object in [sample_output.json](sample_output.json). ```json { "status": "ok", "arxivId": "cond-mat/0011267", "versionId": "cond-mat/0011267v1", "version": 1, "title": "The electronic structure of cuprates from high energy spectroscopy", "categories": [ "cond-mat.supr-con", "cond-mat.str-el" ], "primaryCategory": "cond-mat.supr-con", "published": "2000-11-15T16:19:15Z", "updated": "2000-11-15T16:19:15Z", "absUrl": "https://arxiv.org/abs/cond-mat/0011267v1", "pdfUrl": "https://arxiv.org/pdf/cond-mat/0011267v1", "doi": null, "journalRef": "J. Electron Spectr. Relat. Phenom. 117-118, 203 (2001)", "comment": "J. Electron Spec. Relat. Phenom.: special issue on electron correlation, in press" } ``` ### Pricing Proposed price: USD 0.0015 per successful unique paper, using the paper event. 1,000 papers have USD 1.50 in proposed event fees. Abstract inclusion has no extra paper charge. Error rows, duplicate IDs and excluded rows do not trigger paper events. The proposal is below the checked easyapi/arxiv-search-scraper price of USD 0.00299/result. See PRICING.md for the dated Store API receipt, start fees, owner configuration and platform-usage billing caveat. The owner must configure pricing before publishing; these files do not set Store prices automatically. ### Limits and failure behavior Calls are sequential and leave at least three seconds after each completed API call before another begins, including retries, in line with the [arXiv API rate limits](https://info.arxiv.org/help/api/tou.html). Network failures, HTTP 408/429/5xx and malformed feeds retry with exponential backoff; Retry-After can extend the wait. Permanent HTTP request errors and API query errors yield an unbilled error row. The run summary marks no usable output with errors as failed and mixed paper/error output as partial. The Actor caps emitted rows at maxItems and stops when the SDK paper charge limit is reached. Deduplication is per run, by versionId. It scans at most the API's first 30,000 result positions; refine broad queries. Search ranking and the upstream result set can change while paging, so this is not a stable bulk export. For bulk harvesting, arXiv recommends its OAI-PMH interface. Respect the rate limit across your scheduled runs too: this build does not coordinate concurrent runs across machines. No login, proxies, CAPTCHA solving or full-text retrieval are used. Optional affiliations come only from published author metadata. Data is provided by arXiv; this project is not endorsed by arXiv. RUN_SUMMARY in the default key-value store contains counts, status, query, duration and whether the budget stopped the run. Empty results are a successful zero-row run. Invalid input fails with a field-specific message before arXiv is called. Restarting a run does not provide migration-safe resume or exactly-once billing across interrupted processes. ### FAQ **What limits the output?** The maxItems input caps output rows; its schema maximum is 10,000. Source availability and errors may yield fewer rows. **Does it need a proxy, login, or API key?** The actor calls the public arXiv API without a login. No proxy input is defined. **How is it priced?** PRICING.md proposes $0.0015 per successfully stored unique paper row. Error and duplicate rows have no paper event; publication settings may add platform charges. **What happens on rate limits?** The client spaces arXiv requests and retries transient failures. A source limit or exhausted retry can leave fewer rows. **How fresh is the data?** Results reflect the arXiv API response at run time; published and updated are source dates, while fetchedAt records retrieval. **Does it download paper PDFs?** No. It returns PDF links and optional API abstracts, not full PDF text. ### Use with AI agents / MCP An AI agent can call a published actor through the Apify MCP server or Apify API, pass the JSON input, then read the run’s default dataset. Check the actor’s published identifier and permissions in your Apify account. Run the Actor through an Apify integration that supports Actor tools, using the JSON input above, and read its dataset output. This package includes an input schema and a Results link for such workflows; it does not implement a standalone MCP server. Treat abstracts as source text, preserve absUrl citations and inspect status/errors before using a row in an agent answer. Validate the chosen integration separately; no MCP client was connected during this build. ### Local development Use the existing C:/GritWork/earn/.venv/Scripts/python environment; no dependency install is required there. Put input at storage/key_value_stores/default/INPUT.json, set APIFY_LOCAL_STORAGE_DIR to this Actor's storage directory, and run from the Actor directory: ```powershell $env:APIFY_LOCAL_STORAGE_DIR = "C:/GritWork/earn/actors/arxiv-papers/storage" C:/GritWork/earn/.venv/Scripts/python -m src C:/GritWork/earn/.venv/Scripts/python -m pytest -q tests/test_config.py tests/test_arxiv.py tests/test_delivery.py C:/GritWork/earn/.venv/Scripts/python -m pytest -q -m live ``` For event simulation only, ACTOR_TEST_PAY_PER_EVENT=1 enables the SDK charging log. To simulate the proposal, supply APIFY_ACTOR_PRICING_INFO and non-empty zero counts for paper/dataset events in APIFY_CHARGED_ACTOR_EVENT_COUNTS; see evidence/probe_verified.py. Without pricing information, SDK local test prices differ from the Store proposal. Local output stays in storage and the SDK may purge a default dataset between runs, so export wanted rows before repeating. Dockerfile uses apify/actor-python:3.12. Runtime requirements are apify and httpx; tests require the already installed pytest. Docker and cloud execution are separate release checks. # Actor input Schema ## `query` (type: `string`): arXiv syntax such as ti:electron OR abs:photon. Plain text becomes an all-fields phrase search. Supply query or categories. ## `categories` (type: `array`): Match any listed category, for example cs.AI, cs.LG, hep-th or cs.\*. Combined with query using AND. ## `dateFrom` (type: `string`): Inclusive YYYY-MM-DD. Filters original submission date, not the latest revision. ## `dateTo` (type: `string`): Inclusive YYYY-MM-DD. Either date bound may be omitted. ## `maxItems` (type: `integer`): Hard cap including paper and error rows. Only successful unique paper rows trigger paper events. ## `pageSize` (type: `integer`): Smaller batches can use more calls. Calls are sequential and wait at least three seconds after the previous call. ## `includeAbstract` (type: `boolean`): Include the API abstract. False returns abstract=null. No full-text or PDF download occurs. ## `sortBy` (type: `string`): Ordering applied by the official API. ## `sortOrder` (type: `string`): Ascending or descending ordering. ## `concurrency` (type: `integer`): Fixed at one to respect the arXiv API policy. ## `maxRetries` (type: `integer`): Retry network errors, HTTP 408/429/5xx and malformed feeds with exponential backoff. Permanent request errors are not retried. ## `timeoutSecs` (type: `integer`): Timeout for each HTTP operation. ## Actor input object example ```json { "query": "all:electron", "categories": [], "maxItems": 100, "pageSize": 100, "includeAbstract": true, "sortBy": "relevance", "sortOrder": "descending", "concurrency": 1, "maxRetries": 3, "timeoutSecs": 30 } ``` # Actor output Schema ## `dataset` (type: `string`): Download paper metadata and error rows from this run. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "query": "all:electron" }; // Run the Actor and wait for it to finish const run = await client.actor("grit-77/arxiv-papers-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`πŸ’Ύ Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // πŸ“š Want to learn more πŸ“–? Go to β†’ https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "query": "all:electron" } # Run the Actor and wait for it to finish run = client.actor("grit-77/arxiv-papers-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"πŸ’Ύ Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # πŸ“š Want to learn more πŸ“–? Go to β†’ https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "query": "all:electron" }' | apify call grit-77/arxiv-papers-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,grit-77/arxiv-papers-scraper" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/elfcXeYvIsXMBdwx9/builds/ly65O7kjLSBzoCOGw/openapi.json