# Academic Papers Scraper – arXiv, PubMed, OpenAlex & Crossref (`openharvest/academic-papers-scraper`) Actor

Search academic papers in arXiv, PubMed, OpenAlex, Crossref and Semantic Scholar with one query - a Google Scholar alternative with no captchas. Get titles, authors, abstracts, DOIs, citation counts, venues and open-access PDF links. Filter by year, sort by citations, look up DOI lists.

- **URL**: https://apify.com/openharvest/academic-papers-scraper.md
- **Developed by:** [Enes Bulanık](https://apify.com/openharvest) (community)
- **Categories:** AI, Education, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$1.00 / 1,000 papers

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Academic Papers Scraper – arXiv, PubMed, OpenAlex & Crossref

Search academic papers in arXiv, PubMed, OpenAlex, Crossref and Semantic Scholar with one query - a Google Scholar alternative with no captchas. Get titles, authors, abstracts, DOIs, citation counts, venues and open-access PDF links. Filter by year, sort by citations, look up DOI lists.

**Price: $1 per 1,000 papers.** Failed sources or unknown DOIs are never charged.

### What data do you get?

One row per paper, with the same fields no matter which database it came from:

- **Title, authors, year, publication date, type** (article, preprint, review, conference paper, book chapter, clinical trial…)
- **Venue** (journal / conference / repository) and **publisher**
- **Abstract** (rebuilt from OpenAlex's inverted index, cleaned from Crossref JATS XML, structured PubMed abstracts)
- **Citation count** (OpenAlex, Crossref, Semantic Scholar)
- **Open access flag and direct PDF link** (`pdfUrl`) when a free full text exists
- **DOI, landing-page URL, arXiv ID, PubMed ID (PMID) and PubMed Central ID (PMCID)**
- **Topics** – OpenAlex topics & keywords, MeSH terms, arXiv categories or fields of study
- **`sources`** – every database where the paper was found (duplicates are merged into one row, filling missing fields from the other sources)

### Why use this instead of a Google Scholar scraper?

- **No captchas, no blocked runs** – Google Scholar has no API and blocks scrapers. This Actor uses the official, free APIs of five large open scholarly databases, so a default run takes about 5 seconds.
- **Richer data** – abstracts, open-access PDF links, PMIDs, topics and publication types that Scholar result pages don't show.
- **Low, predictable cost** – $1 per 1,000 papers, no start fee.
- **One query, five databases, no duplicates** – the same paper found in OpenAlex, Crossref and Semantic Scholar becomes one row.

| Source | Best for | Size |
|---|---|---|
| **OpenAlex** | All fields, citations, open access, topics | 250M+ works |
| **arXiv** | Preprints in CS/AI, physics, maths, statistics | 2.5M+ preprints |
| **Crossref** | Every DOI from every publisher | 170M+ records |
| **PubMed** | Medicine and life sciences | 38M+ citations |
| **Semantic Scholar** | AI-indexed literature, citation counts | 200M+ papers |

### Use cases

- **Literature reviews & systematic reviews** – collect every paper on a topic with abstracts, then screen them in a spreadsheet.
- **AI agents and RAG** – give your LLM agent a reliable "search papers" tool with abstracts and PDF links.
- **Citation & bibliometric analysis** – find the most cited papers on a topic in a year range.
- **Research monitoring** – schedule a daily run sorted by date to catch new preprints and publications.
- **DOI enrichment** – turn a list of DOIs into full metadata (authors, abstract, citations, open-access PDF).
- **Field mapping** – see which authors, venues and publishers are active on a topic.

### How to use it

1. Enter a **Search query** (e.g. `crispr gene editing`).
2. Choose the **Sources** (all five by default), **Max papers per source** and **Sort by** (relevance, date or citations).
3. Optionally set **Published from/until year** and **Open access only**.
4. Click **Start** and export to JSON, CSV, Excel, or use the API.

**DOI lookup mode:** paste DOIs into **DOIs to look up** instead of a query. Each DOI is looked up in all selected sources at once and returned as one merged record.

### Input

| Field | Type | Description |
|---|---|---|
| `query` | string | Keywords to search. Default `large language models`. arXiv syntax (`ti:`, `au:`, `cat:`) is passed through to arXiv. |
| `sources` | array | Any of `openalex`, `arxiv`, `crossref`, `pubmed`, `semanticScholar`. Default: all. |
| `maxItems` | integer | Max papers **per source** (1–1000). Default `10`. |
| `sortBy` | string | `relevance` (default), `date` (newest first) or `citations` (most cited first; arXiv and PubMed fall back to relevance). |
| `yearFrom` / `yearTo` | integer | Publication year range. |
| `openAccessOnly` | boolean | Only free-to-read papers. Default `false`. |
| `dois` | array | DOI lookup mode: `10.1038/nature14539` or `https://doi.org/...`. |
| `dedupe` | boolean | Merge the same paper from several sources. Default `true`. |
| `contactEmail` | string | Optional. Sent as `mailto` to OpenAlex/Crossref/PubMed for their faster "polite pool". Omitted if empty. |
| `semanticScholarApiKey` | string (secret) | Optional free key for faster, relevance-ranked Semantic Scholar results. |

Example input:

```json
{
    "query": "crispr gene editing",
    "sources": ["openalex", "pubmed", "semanticScholar"],
    "maxItems": 50,
    "sortBy": "citations",
    "yearFrom": 2019,
    "yearTo": 2024,
    "openAccessOnly": true
}
```

DOI lookup:

```json
{
    "dois": ["10.1038/nature14539", "https://doi.org/10.1056/NEJMoa2032183"]
}
```

### Output

Real sample (DOI lookup of `10.1038/nature14539`, shortened):

```json
{
    "itemType": "paper",
    "source": "openalex",
    "sources": ["openalex", "crossref", "pubmed"],
    "id": "W2919115771",
    "doi": "10.1038/nature14539",
    "title": "Deep learning",
    "authors": ["Yann LeCun", "Yoshua Bengio", "Geoffrey E. Hinton"],
    "year": 2015,
    "publicationDate": "2015-05-26",
    "venue": "Nature",
    "publisher": "Nature Portfolio",
    "abstract": "Deep learning allows computational models that are composed of multiple processing layers to learn representations of data with multiple levels of abstraction. ...",
    "citationCount": 85418,
    "openAccess": true,
    "pdfUrl": null,
    "url": "https://doi.org/10.1038/nature14539",
    "topics": ["Neural Networks and Applications", "deep learning", "backpropagation", "convolutional neural networks"],
    "type": "article",
    "arxivId": null,
    "pmid": "26017442",
    "pmcid": null,
    "language": "en",
    "query": "10.1038/nature14539",
    "scrapedAt": "2026-10-07T19:08:34.711Z"
}
```

An arXiv preprint looks the same, with `"source": "arxiv"`, `"type": "preprint"`, `"openAccess": true`, `"pdfUrl": "https://arxiv.org/pdf/2402.14679v2"` and `"citationCount": null` (arXiv has no citation data – enable OpenAlex or Semantic Scholar to fill it in when the paper is found there too).

**Free error / status rows** (`itemType: "error"` or `"status"`) tell you when a source was rate-limited (`RATE_LIMITED`), failed (`SOURCE_FAILED`), a DOI was invalid (`INVALID_DOI`) or not found (`NOT_FOUND`), or a search had no results (`NO_RESULTS`). They are never charged.

The dataset has ready-made views: **Overview**, **Abstracts & topics**, **Identifiers** and **Errors & status**.

### Pricing

Pay-per-event – you only pay for papers saved. Platform usage is included.

| Event | Price | When it is charged |
|---|---|---|
| `paper` | **$0.001** per paper ($1 / 1,000) | Each paper saved to the dataset (after duplicates are merged) |
| Errors / status rows | **Free** | Failed sources, invalid or unknown DOIs, empty searches |

| Job | Cost |
|---|---|
| Default run (10 papers × 5 sources, minus duplicates) | ≈ **$0.05** |
| 100 papers from each of 5 sources | ≤ **$0.50** |
| Look up 1,000 DOIs | ≤ **$1.00** |

Set **Max cost per run** in the run options – the Actor stops exactly at your budget.

### Use with AI agents (MCP)

This Actor works as a tool for AI agents through the [Apify MCP server](https://mcp.apify.com) (Claude, ChatGPT, Cursor, VS Code and other MCP clients). Add it with `https://mcp.apify.com?tools=openharvest/academic-papers-scraper`, or let the agent find it with `search-actors` ("academic papers", "Google Scholar", "arXiv", "PubMed").

Example `call-actor` input an agent can send:

```json
{
    "query": "retrieval augmented generation evaluation",
    "sources": ["openalex", "arxiv", "semanticScholar"],
    "maxItems": 10,
    "sortBy": "citations",
    "yearFrom": 2023
}
```

The agent gets flat JSON with `title`, `authors`, `year`, `abstract`, `citationCount`, `doi`, `pdfUrl` and `url` – ready to summarise or cite. Default runs finish in seconds.

### FAQ

**Is this a Google Scholar scraper?**
No – it does not touch Google Scholar (which has no API and blocks bots). It searches the open databases most of Scholar's content comes from, through their official APIs, so it never hits captchas.

**Why are some citation counts `null`?**
arXiv and PubMed don't publish citation counts. With deduplication on, a paper also found in OpenAlex or Semantic Scholar gets their citation count.

**Why do some open-access papers have no `pdfUrl`?**
Some free papers are only available as a web page (e.g. PubMed Central). Use `url`, or the `pmcid` (`https://pmc.ncbi.nlm.nih.gov/articles/<pmcid>/`).

**Semantic Scholar shows a `RATE_LIMITED` row – what now?**
Semantic Scholar's public API is shared by everyone and is often busy. The Actor retries, then falls back to its bulk search (most cited first). If it is still busy you get a free error row and the other sources are unaffected. A free API key (input `semanticScholarApiKey`) solves it.

**How does deduplication work?**
Papers with the same DOI, or the same normalized title, are merged. The first source's record is kept and empty fields are filled from the others (abstract, PDF link, highest citation count). `sources` lists every database where the paper appeared.

**How many results can I get?**
Up to 1,000 per source per run (arXiv and PubMed allow more via several runs with year ranges).

**Should I set the contact email?**
It's optional. OpenAlex, Crossref and NCBI ask API users to identify themselves with an email for their faster "polite pool". It's only sent to those APIs.

### Is it legal?

The Actor uses the **official public APIs** of OpenAlex (CC0 data), arXiv, Crossref, NCBI PubMed and Semantic Scholar, within their documented rate limits, and collects **bibliographic metadata only** – no paywalled full texts and no personal contact data. Abstracts and texts remain the copyright of their authors/publishers; respect each source's terms (the arXiv API terms of use, the Semantic Scholar API license and the NCBI usage policies) when you reuse the data. If unsure, consult a lawyer.

### Related actors

- **Google Trends Scraper – Interest Over Time & Related Queries**: see how interest in a research topic changes
- **Hacker News Scraper – Stories, Comments & Who Is Hiring Jobs**: discussion of new papers and tools
- **Bluesky Scraper – Posts, Search, Profiles & Followers**: posts by researchers and labs

### Support & feedback

Missing a field or a source? Open an issue in the **Issues** tab – feature requests are welcome.

# Actor input Schema

## `query` (type: `string`):

Keywords to search for, like in Google Scholar. Example: <code>crispr gene editing</code>. Ignored when you fill in <b>DOIs to look up</b>. If both are empty, the example query <code>large language models</code> is used.

## `sources` (type: `array`):

Which databases to search. OpenAlex (250M+ works, all fields), arXiv (preprints: CS, physics, math), Crossref (DOI registry, all publishers), PubMed (biomedicine) and Semantic Scholar (200M+ papers, citation counts).

## `maxItems` (type: `integer`):

Maximum number of papers to take from each selected source (1-1000). You pay per paper saved, so this also caps your cost. Duplicates found in several sources are merged into one paper.

## `sortBy` (type: `string`):

Order of results. <b>Citations</b> works in OpenAlex, Crossref and Semantic Scholar (arXiv and PubMed fall back to relevance). <b>Date</b> = newest first.

## `yearFrom` (type: `integer`):

Optional. Only papers published in this year or later. Example: <code>2020</code>.

## `yearTo` (type: `integer`):

Optional. Only papers published in this year or earlier. Example: <code>2024</code>.

## `openAccessOnly` (type: `boolean`):

Only return papers that are free to read (open access). Most results then include a <code>pdfUrl</code>.

## `dois` (type: `array`):

Optional DOI lookup mode: enter DOIs instead of a query to get full metadata for each (abstract, authors, citations, open-access PDF). Accepts <code>10.1038/nature14539</code> or <code>https://doi.org/10.1038/nature14539</code>. Data from all selected sources is merged into one record per DOI.

## `dedupe` (type: `boolean`):

Merge the same paper found in several sources (same DOI or same title) into one record, combining their data. Recommended.

## `contactEmail` (type: `string`):

Optional. Your email is sent to OpenAlex, Crossref and PubMed as the polite <code>mailto</code>/<code>email</code> parameter, which gives faster and more reliable responses (their 'polite pool'). Leave empty to omit it.

## `semanticScholarApiKey` (type: `string`):

Optional. Semantic Scholar's public API is shared by everyone and often rate-limited (HTTP 429). A free key from semanticscholar.org/product/api makes it fast and reliable. Without it, the Actor retries with backoff and reports a free error row if Semantic Scholar stays busy.

## Actor input object example

```json
{
  "query": "crispr gene editing",
  "sources": [
    "openalex",
    "arxiv",
    "pubmed"
  ],
  "maxItems": 50,
  "sortBy": "citations",
  "yearFrom": 2020,
  "yearTo": 2024,
  "openAccessOnly": true,
  "dois": [
    "10.1038/nature14539",
    "https://doi.org/10.1056/NEJMoa2032183"
  ],
  "dedupe": true,
  "contactEmail": "you@university.edu"
}
```

# Actor output Schema

## `results` (type: `string`):

All scraped items in the default dataset (JSON, CSV, Excel and more via the dataset export).

## `files` (type: `string`):

Records saved in the default key-value store, e.g. the run summary.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "query": "large language models",
    "sources": [
        "openalex",
        "arxiv",
        "crossref",
        "pubmed",
        "semanticScholar"
    ],
    "maxItems": 10
};

// Run the Actor and wait for it to finish
const run = await client.actor("openharvest/academic-papers-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "query": "large language models",
    "sources": [
        "openalex",
        "arxiv",
        "crossref",
        "pubmed",
        "semanticScholar",
    ],
    "maxItems": 10,
}

# Run the Actor and wait for it to finish
run = client.actor("openharvest/academic-papers-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "query": "large language models",
  "sources": [
    "openalex",
    "arxiv",
    "crossref",
    "pubmed",
    "semanticScholar"
  ],
  "maxItems": 10
}' |
apify call openharvest/academic-papers-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,openharvest/academic-papers-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/gxflINo7W8YjHB5cm/builds/x6gV4W3hChd71KkuV/openapi.json