# Academic Papers Scraper: OpenAlex, Crossref, arXiv & PubMed (`latamdatatools/academic-papers-search`) Actor

Search scientific papers across OpenAlex, Crossref, arXiv and PubMed in one run: de-duplicated by DOI, with authors, institutions, citations, abstracts and open-access PDF links. DOI lookup in bulk and alerts for new papers on a topic.

- **URL**: https://apify.com/latamdatatools/academic-papers-search.md
- **Developed by:** [Julian Genuario](https://apify.com/latamdatatools) (community)
- **Categories:** AI, Developer tools, Education
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.50 / 1,000 papers

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Academic Papers Scraper: OpenAlex, Crossref, arXiv & PubMed

Search **scientific papers in four open databases at once** and get **one clean, de-duplicated list**: title, DOI, authors with ORCID and institutions, venue, publisher, publication date, **citation count**, **abstract**, **open-access PDF link**, topics, MeSH terms, arXiv and PubMed IDs.

The same paper found in OpenAlex, Crossref, arXiv and PubMed appears **once**, with the best data of each source merged (`sources` tells you where it was found). You can also paste a list of **DOIs** (or point to a dataset / CSV) to enrich them in bulk, and schedule **alerts for new papers** on your topics.

Only official, open APIs. No Google Scholar scraping, no login, no proxies.

### What does Academic Papers Scraper do?

Academic Papers Scraper searches scientific papers across OpenAlex, Crossref, arXiv and PubMed in one run and returns research papers de-duplicated by DOI, with authors, institutions, citation counts, abstracts and open-access PDF links. It also does bulk DOI lookup and new-paper alerts for a topic, useful for literature reviews and research monitoring.

### How to use Academic Papers Scraper

1. Click **Try for free** on this page (a free Apify account is enough to start).
2. Fill in the input (see the examples below) or keep the prefilled example.
3. Click **Start** and wait for the run to finish.
4. Download the results from the **Output** tab as JSON, CSV, Excel or HTML, or read them with the Apify API.
5. Optional: create a **Schedule** to run it daily or weekly, and connect a webhook or an integration (Make, Zapier, n8n, Slack).

### Who uses it

- **Researchers and R\&D teams**: literature reviews, state of the art, follow a topic every week.
- **Pharma, biotech and medtech**: monitor PubMed and preprints for a molecule, disease or method.
- **AI agents and RAG pipelines**: fetch abstracts and open PDFs for a question (works as a tool through the Apify MCP server).
- **Libraries, publishers and analysts**: enrich DOI lists with citations, open-access status and affiliations.

### Sources

| Source | What it adds | Limit respected |
|---|---|---|
| **OpenAlex** | 250M+ works, citation counts, institutions, topics, open-access status | Small shared budget without a key; add your **free OpenAlex API key** for your own daily budget |
| **Crossref** | The DOI registry of almost every publisher: venue, publisher, references count, licenses | Public pool 1 request/s; with your contact email, the faster "polite pool" |
| **arXiv** (via OpenAlex) | Preprints in physics, CS, math, quantitative biology, finance... with PDF. Searched through OpenAlex's arXiv source, so the arXiv servers are never hit | Same OpenAlex budget as above |
| **PubMed** (via Europe PMC) | Biomedical literature (PubMed/MEDLINE records), abstracts, MeSH terms, PMC full text. Searched through the Europe PMC REST API, which mirrors PubMed | 1 request every 10 s (up to 1,000 records each), as Europe PMC's robots.txt asks |

The arXiv and NCBI APIs are not called directly: their robots.txt disallows automated access, and this Actor respects robots.txt. Results are the same papers with the same arXiv IDs and PubMed IDs; small differences in ranking and freshness versus the arXiv / PubMed websites are possible.

### Features

- Several queries per run; each one searched in every selected source.
- **Dedupe and merge** by DOI, arXiv ID, PubMed ID and title (a preprint and its published version are merged when they share a DOI or the exact title; two different DOIs are never merged).
- Filters: publication date range, open access only. Sort by relevance, newest or most cited.
- **Bulk DOI lookup** from a list, an Apify dataset or a CSV / Google Sheet.
- **Only new** mode with memory per alert name: schedule it and get only papers you have not seen.
- Crossref results are checked so every query term really appears in the title, abstract or venue (its search engine is very fuzzy).

### Input example

```json
{
  "queries": ["CRISPR base editing", "large language models protein design"],
  "sources": ["openalex", "crossref", "arxiv", "pubmed"],
  "fromDate": "2025-01-01",
  "sort": "newest",
  "maxResultsPerSource": 50,
  "onlyNew": true,
  "alertName": "weekly-crispr",
  "openAlexApiKey": "optional-free-key",
  "contactEmail": "you@example.com"
}
```

DOI enrichment only:

```json
{ "dois": ["10.1038/nature14539", "https://doi.org/10.1126/science.1225829"] }
```

### Output

```json
{
  "title": "A Programmable Dual-RNA–Guided DNA Endonuclease in Adaptive Bacterial Immunity",
  "doi": "10.1126/science.1225829",
  "doiUrl": "https://doi.org/10.1126/science.1225829",
  "publicationDate": "2012-06-29",
  "year": 2012,
  "authorNames": "Martin Jínek; Krzysztof Chylinski; Ines Fonfara; M. Hauer; Jennifer A. Doudna; Emmanuelle Charpentier",
  "firstAuthor": "Martin Jínek",
  "authorCount": 6,
  "institutions": ["Howard Hughes Medical Institute", "University of California, Berkeley", "University of Vienna", "Umeå University"],
  "venue": "Science",
  "publisher": "American Association for the Advancement of Science",
  "type": "article",
  "citationCount": 17104,
  "isOpenAccess": true,
  "openAccessStatus": "green",
  "pdfUrl": null,
  "landingUrl": "https://doi.org/10.1126/science.1225829",
  "abstract": "Clustered regularly interspaced short palindromic repeats (CRISPR)/CRISPR-associated (Cas)...",
  "topics": ["CRISPR and Genetic Engineering", "RNA regulation and disease", "Cas9"],
  "keywords": ["CRISPR/Cas9", "sequence-specific DNA cleavage", "DNA double-strand breaks"],
  "meshTerms": [],
  "arxivId": null,
  "pmid": "22745249",
  "pmcid": null,
  "openAlexId": "W2045435533",
  "issn": "0036-8075",
  "language": "en",
  "authors": [{ "name": "Martin Jínek", "orcid": "0000-0002-7601-210X", "affiliations": ["Howard Hughes Medical Institute", "University of California, Berkeley"] }],
  "matchedQueries": [],
  "sources": ["openalex"],
  "fetchedAt": "2026-10-27T10:00:00.000Z"
}
```

Fields that a source does not have are `null` (for example citation counts only come from OpenAlex and Crossref). DOIs that no source knows are reported in a free row with an `error` field.

A run summary (requests, results per source, merged duplicates, notes) is saved as `RUN_SUMMARY` in the key-value store.

### Pricing

Pay per event:

- **USD 1.50 per 1,000 papers** returned (USD 0.0015 each). Duplicates merged across sources count once.
- **USD 0.01 per run** (start).
- Free: DOIs not found, papers skipped in "only new" mode, and anything beyond your "max charge per run".

### Tips and limits

- Without an OpenAlex key, OpenAlex shares one small daily budget per IP address; on busy cloud IPs it can be used up, and then the run continues with the other sources (see `notes` in the run summary). A free key fixes it.
- arXiv results depend on OpenAlex: if the OpenAlex budget runs out, arXiv is skipped too (a free OpenAlex key fixes it). Very recent arXiv preprints can take a few days to appear in OpenAlex.
- Europe PMC is queried once every 10 s, so many queries with PubMed selected take ~10 s each.
- Crossref and PubMed sometimes carry future "issue" dates (a 2027 print issue already online).
- Abstracts come from the sources; publishers may keep copyright on some of them. Use them according to your purpose (research, indexing) or turn them off.
- Citation counts are a snapshot from OpenAlex/Crossref and differ from Google Scholar.

### Data and credits

Metadata from [OpenAlex](https://openalex.org) (CC0), [Crossref](https://www.crossref.org) (metadata openly available), arXiv records through OpenAlex ("Thank you to arXiv for use of its open access interoperability") and PubMed records through [Europe PMC](https://europepmc.org). This Actor is not affiliated with or endorsed by any of them.

### Cost examples

Prices at the current pay-per-event rate: USD 0.01 per run and USD 1.50 per 1,000 papers (papers merged across sources count once; DOIs not found and papers skipped in only-new mode are free). You can set a maximum charge per run in the run options.

| Job | Approximate cost |
|---|---|
| Test run: 1 query, 4 sources, 5 results each (measured: 19 papers after merging, 5 s) | about $0.04 |
| Literature review: 3 queries, 200 papers | about $0.31 |
| Enrich 1,000 DOIs from a CSV | about $1.51 |
| Weekly alert in only-new mode (~30 new papers) | about $0.055 per run |

### Use it from your code, an integration or an AI agent

**Python** (`pip install apify-client`):

```python
from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("latamdatatools/academic-papers-search").call(run_input={
  "queries": [
    "large language models protein design"
  ],
  "sources": [
    "openalex",
    "crossref",
    "arxiv",
    "pubmed"
  ],
  "maxResultsPerSource": 25,
  "sort": "newest"
})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["year"], item["title"], item["doi"], item["citationCount"])
```

**JavaScript / Node.js** (`npm install apify-client`):

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_APIFY_TOKEN>' });
const run = await client.actor('latamdatatools/academic-papers-search').call({
  queries: [
    'large language models protein design'
  ],
  sources: [
    'openalex',
    'crossref',
    'arxiv',
    'pubmed'
  ],
  maxResultsPerSource: 25,
  sort: 'newest'
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items.map((i) => [i.year, i.title, i.doi]));
```

**No code:**

- **Schedules:** save your input as a Task and run it weekly from Apify Console (Schedules).
- **Integrations:** send the results to Google Sheets, Slack, email, Make, Zapier, n8n or Airbyte from the Integrations tab, or call your own webhook when a run finishes.
- **AI agents:** add it as a tool in Claude, Cursor, VS Code and other MCP clients through the Apify MCP server (`https://mcp.apify.com`), so the agent can search the scientific literature and cite real papers with DOIs.

### FAQ

**Can I call it from my own code or from an AI agent?** Yes. Start runs and fetch results with the Apify API or the JavaScript / Python clients, or use it as a tool from Claude, Cursor and other MCP clients through the Apify MCP server (`mcp.apify.com`).

**Can I run it automatically?** Yes. Create a Schedule in Apify Console (for example every morning) and add a webhook or an integration to get the results where you need them.

**Which export formats are available?** JSON, JSONL, CSV, Excel (XLSX), XML, RSS and HTML table, from the Output tab or the API.

**What does it cost?** See the Pricing section above. You only pay for the events listed there.

# Actor input Schema

## `queries` (type: `array`):

Topics, keywords or titles. One per line. Each query is searched in every selected source. Use quotes for exact phrases ("base editing").

## `sources` (type: `array`):

Results from several sources are merged into one record per paper (same DOI, arXiv ID, PubMed ID or title). arXiv preprints are searched through OpenAlex (its arXiv source) and PubMed records through the Europe PMC API, which mirrors PubMed.

## `fromDate` (type: `string`):

Optional. Only papers published on or after this date.

## `toDate` (type: `string`):

Optional. Only papers published on or before this date.

## `sort` (type: `string`):

Relevance (as ranked by each source), newest first, or most cited first.

## `openAccessOnly` (type: `boolean`):

Keep only papers with free full text (OpenAlex OA filter, Europe PMC open access for PubMed, OpenAlex OA filter for arXiv, Crossref with a Creative Commons license).

## `includeAbstracts` (type: `boolean`):

Abstracts are provided by the sources; some publishers keep copyright on them. Turn off for smaller rows.

## `dois` (type: `array`):

Bulk DOI enrichment: one DOI (or doi.org link) per line. Each one returns the full record (OpenAlex first, Crossref as fallback). Can be combined with queries.

## `datasetId` (type: `string`):

ID of an Apify dataset (for example from another Actor) with a DOI field.

## `csvUrl` (type: `string`):

Public CSV link or a Google Sheets link shared as "Anyone with the link".

## `doiField` (type: `string`):

Name of the field or CSV column that holds the DOI.

## `includeNotFound` (type: `boolean`):

Adds a row with an error for DOIs that no source knows. Those rows are free.

## `onlyNew` (type: `boolean`):

Return only papers not delivered in earlier runs with the same alert name. Schedule daily or weekly to get new papers on your topics.

## `alertName` (type: `string`):

Separate memory per alert.

## `maxResultsPerSource` (type: `integer`):

How many results to read from each source for each query (before merging duplicates).

## `maxResults` (type: `integer`):

Total cap of unique papers returned (and charged).

## `openAlexApiKey` (type: `string`):

OpenAlex now gives a small shared daily budget per IP address without a key, which can run out on cloud IPs. A free personal key (openalex.org/settings/api) gives you your own daily budget. Without it, OpenAlex is skipped when the shared budget is used up and the other sources still work.

## `ncbiApiKey` (type: `string`):

No longer used: PubMed records now come from the Europe PMC API, which needs no key. Kept only so older inputs keep working.

## `contactEmail` (type: `string`):

Sent only to Crossref ("polite pool", faster and more reliable) and Europe PMC, as both APIs ask. Not stored anywhere else.

## Actor input object example

```json
{
  "queries": [
    "large language models protein design"
  ],
  "sources": [
    "openalex",
    "crossref",
    "arxiv",
    "pubmed"
  ],
  "sort": "relevance",
  "openAccessOnly": false,
  "includeAbstracts": true,
  "doiField": "doi",
  "includeNotFound": true,
  "onlyNew": false,
  "alertName": "default",
  "maxResultsPerSource": 50,
  "maxResults": 1000
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `runSummary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "queries": [
        "large language models protein design"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("latamdatatools/academic-papers-search").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "queries": ["large language models protein design"] }

# Run the Actor and wait for it to finish
run = client.actor("latamdatatools/academic-papers-search").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "queries": [
    "large language models protein design"
  ]
}' |
apify call latamdatatools/academic-papers-search --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,latamdatatools/academic-papers-search"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Z0H8PwwHoBMUOdynv/builds/xMj56i8hB6R9UeW6U/openapi.json