# Research Paper Scraper (`datalayer/research-paper-intelligence`) Actor

Scholar scrapers fail because Scholar blocks them. OpenAlex is open, keyless and unblockable. Three fields Scholar will not give you: the abstract rebuilt from the inverted index it ships instead of prose, field-weighted impact so citations compare across fields, and 134,038 retraction flags.

- **URL**: https://apify.com/datalayer/research-paper-intelligence.md
- **Developed by:** [Datalayer](https://apify.com/datalayer) (community)
- **Categories:** Education, AI
- **Stats:** 2 total users, 1 monthly users, 95.2% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.70 / 1,000 papers

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Research Paper Scraper — Citations & Retractions

Scrape **academic papers** — titles, abstracts, authors, institutions, citations
— from OpenAlex, an open catalogue of over 250 million works.

The Google Scholar scrapers on this store are rated around 2.5 stars, and not
because they are badly written. Scholar blocks automated traffic, so they fail.
OpenAlex has a public API with no key, no bot wall and no CAPTCHA. Same
literature, a source that cannot lock you out. That is the whole competitive
argument, and it is worth more than any feature.

### Three fields Scholar will never give you

**1. The abstract, rebuilt.** OpenAlex is not permitted to redistribute
abstracts as prose, so it ships an *inverted index* — `{"gene": [3], "the": [0,
12]}`. Handed through raw it is unreadable. This actor reconstructs the sentence
by placing each word at each of its positions. Measured on a live 200-paper
CRISPR query: **151 abstracts recovered**.

**2. Field-weighted citation impact (`fwci`).** Forty citations is outstanding
in one field and invisible in another, so a raw count cannot be compared across
disciplines. FWCI normalises against the work's own field and year: 1.0 is
exactly average, 2.0 is twice the field average. **196 of 200** papers in that
same live query carried one. Also included is `citationPercentileMin`/`Max` —
the percentile band within field and year.

**3. Retraction flags.** OpenAlex marks **134,038 retracted works** (measured
2026-08-14). The live CRISPR query surfaced 2 in its top 200. A literature
review or an AI pipeline that cites a retracted paper has a real problem, and
Scholar will not warn you.

### What else you get

- **Collaboration structure.** Author institutions with ROR ids, distinct
  countries, and `isInternationalCollaboration`.
- **Open access.** `isOpenAccess`, `openAccessStatus` (gold, green, hybrid,
  bronze) and a direct link to the legal full text.
- **An analysis row** over the whole query: output by year, median FWCI, how many
  works beat their field average, top-1% count, uncited share, retraction rate,
  open-access share, and the top institutions, authors, journals and countries.

A live run on `CRISPR`, 2024 onward, 200 most-cited papers: median FWCI **19.9**,
196 above field average, 122 in the top 1% of their field, 75.5% open access,
44% international. Top institutions Harvard (13), UC Berkeley (13), Broad
Institute (12); top journals Nature (18), Nature Communications (14), Cell (12).

### Who uses this

- **Biotech and pharma competitive intelligence** — who publishes in a field,
  where, and with what real impact.
- **Research offices and funders** — benchmarking an institution's output and
  citation performance.
- **Systematic reviewers** — bulk retrieval with retraction screening built in.
- **AI and RAG pipelines** — abstracts and metadata from a source that will not
  rate-limit a crawl into failure.

### Input

Give at least one narrowing filter — `searchTerm`, `institutionRor`,
`countryCode`, `authorId`, or `retractedOnly`. An unfiltered walk across 250
million works is refused rather than started.

| Field | Type | Default | Notes |
|---|---|---|---|
| `searchTerm` | string | `CRISPR` | Matched against title and abstract |
| `institutionRor` | string | `""` | ROR id, bare or full URL, e.g. `00f54p054` |
| `countryCode` | string | `""` | Two-letter country of author institutions |
| `authorId` | string | `""` | OpenAlex author id, e.g. `A5023888391` |
| `fromDate` / `toDate` | string | `2024-01-01` / `""` | ISO publication dates |
| `workTypes` | array | `[]` | article, review, preprint, dataset… |
| `openAccessOnly` | boolean | `false` | Legally readable full text only |
| `retractedOnly` | boolean | `false` | Audit a field, journal or institution for retractions |
| `minCitations` | integer | `0` | |
| `excludeParatext` | boolean | `true` | Drops indexes, covers and editorial furniture |
| `sortBy` | string | `cited_by_count:desc` | Or newest, oldest, most relevant |
| `maxWorks` | integer | `200` | 200 per page, cursor paged |
| `includeAbstract` | boolean | `true` | Reconstruct from the inverted index |

### Output

**`work`** — `title`, `doi`, `publicationDate`, `workType`, `citedByCount`,
`fwci`, `citationPercentileMin/Max`, `isRetracted`, `isOpenAccess`,
`openAccessUrl`, `sourceName`, `publisher`, `primaryTopic/Field/Domain`,
`authors`, `firstAuthor`, `correspondingAuthors`, `institutions`,
`institutionRors`, `countries`, `isInternationalCollaboration`, `abstract`.

**`analysis`** — `worksByYear`, `medianCitations`, `medianFwci`,
`worksAboveFieldAverage`, `top1PercentWorks`, `uncitedSharePercent`,
`retractedWorks`, `openAccessSharePercent`,
`internationalCollaborationSharePercent`, `topInstitutions`, `topAuthors`,
`topSources`, `topTopics`, `topCountries`.

### Notes on accuracy

- `top1PercentWorks` reads the **lower** bound of the percentile band. A work in
  the 94–100 band is not counted, because an overlapping band is not evidence.
  Overstating impact is the failure mode that matters in this dataset.
- Not every work has an FWCI — very new and very obscure works may not.
- Abstracts are missing where OpenAlex holds no inverted index for the work.
- The actor identifies itself to OpenAlex's "polite pool", which is faster and
  more reliable than anonymous access.

### Pricing

Pay per event. $0.001 per paper, $0.05 per analysis row.

# Actor input Schema

## `searchTerm` (type: `string`):

Matched against title and abstract, e.g. CRISPR base editing, solid state battery. Give at least one of this, an institution, a country or an author.

## `authorId` (type: `string`):

Restrict to one author, e.g. A5023888391.

## `institutionRor` (type: `string`):

Restrict to one institution by ROR id, e.g. 00f54p054 for Stanford. Accepts the bare id or the full https://ror.org/... URL.

## `countryCode` (type: `string`):

Two-letter country of the author institutions, e.g. au, gb, us.

## `fromDate` (type: `string`):

ISO date, e.g. 2024-01-01. Leave empty for no lower bound.

## `toDate` (type: `string`):

ISO date. Leave empty for no upper bound.

## `workTypes` (type: `array`):

Leave empty for all types.

## `openAccessOnly` (type: `boolean`):

Only papers with a legally readable full text.

## `retractedOnly` (type: `boolean`):

OpenAlex flags over 134,000 retracted works. Turn this on to audit a field, a journal or an institution for retractions.

## `minCitations` (type: `integer`):

0 means no minimum.

## `excludeParatext` (type: `boolean`):

Drop indexes, covers and editorial furniture, which OpenAlex marks as paratext.

## `includeWorks` (type: `boolean`):

Return one row per paper. Turn off to get only the analysis row.

## `includeAnalysis` (type: `boolean`):

Output by year, median field-weighted impact, retraction and open-access share, top institutions, authors, journals and countries.

## `includeAbstract` (type: `boolean`):

OpenAlex ships abstracts as an inverted index rather than prose. Turn this on to get readable abstract text back.

## `sortBy` (type: `string`):

Most cited first is the useful default for a survey; newest first is what you want for monitoring a field.

## `maxWorks` (type: `integer`):

OpenAlex serves 200 per page with cursor paging.

## `proxyConfiguration` (type: `object`):

Optional. OpenAlex is an open API and does not require a proxy.

## Actor input object example

```json
{
  "searchTerm": "CRISPR",
  "authorId": "",
  "institutionRor": "",
  "countryCode": "",
  "fromDate": "2024-01-01",
  "toDate": "",
  "workTypes": [],
  "openAccessOnly": false,
  "retractedOnly": false,
  "minCitations": 0,
  "excludeParatext": true,
  "includeWorks": true,
  "includeAnalysis": true,
  "includeAbstract": true,
  "sortBy": "cited_by_count:desc",
  "maxWorks": 200,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `works` (type: `string`):

Every paper with its reconstructed abstract, field-weighted impact, open access status and retraction flag.

## `analysis` (type: `string`):

One row for the whole query: output by year, median field-weighted impact, retraction and open-access share, top institutions and authors.

## `all` (type: `string`):

Papers and analysis together.

## `csv` (type: `string`):

Everything as a CSV download.

## `runSummary` (type: `string`):

Counts, how many abstracts were reconstructed, and how many works are retracted.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "searchTerm": "CRISPR",
    "maxWorks": 200
};

// Run the Actor and wait for it to finish
const run = await client.actor("datalayer/research-paper-intelligence").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "searchTerm": "CRISPR",
    "maxWorks": 200,
}

# Run the Actor and wait for it to finish
run = client.actor("datalayer/research-paper-intelligence").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "searchTerm": "CRISPR",
  "maxWorks": 200
}' |
apify call datalayer/research-paper-intelligence --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,datalayer/research-paper-intelligence"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/99CDoiQm02SOxRuAU/builds/UAkvR9tQhwSPgWks3/openapi.json