# PubMed Articles Scraper (`scrapyx/pubmed-articles-scraper`) Actor

Full PubMed records - abstracts, authors with affiliations, MeSH terms, grants, DOI and free-full-text links. PubMed serves at most 9,999 records per query and the history server does not open that window, so larger queries are cut into Entrez-date slices automatically.

- **URL**: https://apify.com/scrapyx/pubmed-articles-scraper.md
- **Developed by:** [Ibnu Adzim](https://apify.com/scrapyx) (community)
- **Categories:** AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.10 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## PubMed Articles Scraper

Full biomedical literature records from **PubMed**, via NCBI's own E-utilities
API — abstracts, authors with affiliations and ORCIDs, MeSH terms, grant
funding, DOIs and free-full-text links.

No API key needed. No login. Nothing to bypass.

### What makes this different

**PubMed serves at most 9,999 records per query, and the workaround everyone
cites does not work.** Every E-utilities tutorial says to pass `usehistory=y`
and page through `WebEnv` for large result sets. Measured against `cancer`
(5,687,758 matches), the history server returns the *same* refusal:
`'retstart' cannot be larger than 9998`.

Slicing the query by publication date does not rescue it either: a record
published "2020 Jan" is indexed as 2020/01/01, so **75% of a month lands on
its first day** — 40,541 records for `cancer`, four times the window, at the
finest granularity the API has.

This actor slices on **Entrez date** instead, which always carries a real day
and distributes evenly, bisecting until every slice fits. Verified: 10,050
unique records from a query that stops at 9,999 any other way, zero
duplicates, 24 requests.

### Modes

| Mode | What you give it | What you get |
| --- | --- | --- |
| **Search PubMed** | a query — full PubMed syntax works | matching articles |
| **Specific PMIDs** | PMIDs or `pubmed.ncbi.nlm.nih.gov` URLs | those articles |
| **Articles citing a PMID** | seed PMIDs | every article citing them, each tagged with the seed it cites |

### Output

One `SEARCH_SUMMARY` row per run, one `ARTICLE` row per record, and an `ERROR`
row for any input that failed — so every input maps to at least one row.

Each article carries, among ~50 fields:

- `pmid`, `doi`, `pmcId`, `articleUrl`, `doiUrl`, `pmcUrl`, `isFreeFullText`
- `title`, `abstract`, `abstractSections` (labelled sections preserved when the
  abstract is structured — 41% of clinical trials are), `hasAbstract`
- `journal`, `journalIsoAbbrev`, `issn`, `volume`, `issue`, `pages`
- `pubDate`, `pubDatePrecision`, `pubDateEarliest`, `pubDateRaw`, `entrezDate`,
  `receivedDate`, `acceptedDate`
- `authors` (last/fore name, initials, ORCID, affiliations), `authorCount`,
  `firstAuthor`, `lastAuthor`, `affiliationCount`
- `meshTerms` (descriptor, UI, major-topic flag, qualifiers), `keywords`
- `publicationTypes`, `languages`, `grants` (id, agency, country),
  `journalCountry`, `citationStatus`
- `citedByCount` when you ask for it; `citesPmid` in citedby mode

#### Dates are never invented

`pubDate` is an ISO 8601 **reduced-precision** string — `2026-09-21`,
`2026-09` or `2026`, whichever the record actually supports — and
`pubDatePrecision` says which. MEDLINE publishes seasons (`Nov-Dec`) and date
ranges (`1985 Dec 18-31`) that are not dates; filling the gaps with `01` would
produce a specific, wrong, entirely real-looking day. `pubDateEarliest` gives
you a full date when you need one, and its name says it is a floor.

### Honesty flags on the summary row

Because almost every PubMed failure arrives as HTTP 200 with the problem buried
in the body, the summary row publishes what actually happened:

- `queryTranslation` — **the query PubMed really ran.** A plain phrase is
  silently expanded through MeSH: `heart attack` runs as a five-clause boolean.
- `upstreamCount` vs `articlesReturned` — matched vs reachable
- `resultWindow`, `resultWindowHit`, `dateSlicesUsed`, `unsliceableDays`
- `maxDateDefaulted` — PubMed **ignores a from-date with no to-date entirely**,
  returning the unfiltered set under a 200. The actor completes the range and
  tells you it did.
- `pmidsNotFound`, `duplicatePmidsDropped`, `bookRecordsSkipped`
- `rateLimitHits`, `rateLimitPerSecond`, `ncbiApiKeyUsed`

An unknown `sort` is **refused**, not passed through: PubMed ignores one
without error and silently returns a different ordering.

### Speed and the rate limit

NCBI allows **3 requests/second per IP**, enforced with a real HTTP 429
(12 concurrent calls → 9 of them 429). A free NCBI API key raises it to 10/s;
supply it as `ncbiApiKey` and the pace tightens automatically.

`minRequestInterval`, not `maxConcurrency`, is the honest speed control —
concurrency buys nothing once the cap binds.

**The proxy is off by default, deliberately.** The limit is keyed on your IP, so
routing through a rotating residential pool would not fix a 429; it would evade
a published limit by making one client look like many.

### Known limits

- **9,999 records per single query.** Enabling the default date slicing goes
  past it; a *single Entrez day* holding more than 9,999 records cannot be cut
  finer, and those days are listed in `unsliceableDays` rather than silently
  truncated.
- `esummary` is deliberately not used — its JSON is a subset of the XML this
  actor already fetches.
- Book chapters (`PubmedBookArticle`) are counted but not parsed.

# Actor input Schema

## `mode` (type: `string`):

search = find articles by query. articles = fetch the PMIDs you name. citedby = fetch every article that cites the PMIDs you name.

## `searchTerm` (type: `string`):

For mode='search'. Full PubMed query syntax works: crispr AND "gene therapy"\[MeSH Terms] AND 2020:2024\[dp]. Note that PubMed silently expands a plain phrase through MeSH — 'heart attack' runs as a five-clause boolean — and the summary row publishes the query it actually ran as queryTranslation.

## `pmids` (type: `array`):

For mode='articles' and mode='citedby'. A numeric PMID like 38786024, or a pubmed.ncbi.nlm.nih.gov URL. A PMID that does not exist comes back as an honest zero, listed under pmidsNotFound.

## `maxResults` (type: `integer`):

Set 0 for unlimited. PubMed serves at most 9,999 records per query; above that this actor cuts the query into Entrez-date slices automatically (see Break the 9,999 window).

## `sort` (type: `string`):

PubMed IGNORES an unknown sort without error and returns its own default order instead, so only the four it honours are offered here. 'none' omits the parameter and takes upstream's default.

## `minDate` (type: `string`):

YYYY/MM/DD. WARNING about PubMed itself: a from-date with no to-date is ignored ENTIRELY — HTTP 200, no error, and the unfiltered result set. This actor fills the missing to-date with today and reports maxDateDefaulted so you can see it happened.

## `maxDate` (type: `string`):

YYYY/MM/DD.

## `dateType` (type: `string`):

edat = the date the record entered PubMed. pdat = publication date. mdat = last MEDLINE revision. edat is the default because it is the only one that distributes evenly: a record published '2020 Jan' is indexed as 2020/01/01, so pdat puts ~75% of a month onto its first day.

## `breakResultWindow` (type: `boolean`):

PubMed serves at most 9,999 records per query and usehistory/WebEnv does NOT open that window — the history server returns the same refusal. With this on, a larger query is cut into Entrez-date slices until every record is reachable. Turn it off to take the first 9,999 and stop.

## `includeCitedByCount` (type: `boolean`):

Adds citedByCount to every article, from NCBI's own link database. Costs one extra request per 50 articles.

## `ncbiApiKey` (type: `string`):

NCBI allows 3 requests/second per IP anonymously and 10/s with a free key from your NCBI account. Supplying one raises the default pace automatically. Leave blank to run anonymously.

## `contactEmail` (type: `string`):

NCBI's usage policy asks automated clients to identify a contact so they can reach you before blocking anything. Sent as the `email` parameter. Optional — requests without it also succeed.

## `efetchBatchSize` (type: `integer`):

How many articles to pull per efetch call, 1-1000. Fetches are POSTed because a GET dies at ~500 ids with HTTP 414 and an empty body. 1000 ids is a ~20 MB response for no gain once the rate cap binds.

## `maxConcurrency` (type: `integer`):

Concurrency buys nothing once the 3-per-second rate cap binds; minRequestInterval is the honest speed control.

## `minRequestInterval` (type: `integer`):

Politeness pacing shared across all workers. 0 uses the built-in default, which is the published NCBI limit itself: ~0.34s (3/s) anonymously, ~0.11s (10/s) when an API key is supplied. Raise it if you still see rate-limit retries.

## `proxyConfiguration` (type: `object`):

OFF by default and deliberately so. E-utilities has no WAF, and NCBI's rate limit is keyed on your IP — routing through a rotating residential pool would not fix a 429, it would evade a published limit by making one client look like many.

## Actor input object example

```json
{
  "mode": "search",
  "searchTerm": "crispr AND gene therapy",
  "pmids": [
    "38786024",
    "https://pubmed.ncbi.nlm.nih.gov/32811740/"
  ],
  "maxResults": 50,
  "sort": "none",
  "minDate": "2020/01/01",
  "maxDate": "2024/12/31",
  "dateType": "edat",
  "breakResultWindow": true,
  "includeCitedByCount": false,
  "contactEmail": "you@example.com",
  "efetchBatchSize": 200,
  "maxConcurrency": 3,
  "minRequestInterval": 0,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `items` (type: `string`):

One row per scraped record. See the dataset's default view for field definitions.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("scrapyx/pubmed-articles-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("scrapyx/pubmed-articles-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call scrapyx/pubmed-articles-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,scrapyx/pubmed-articles-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/tB7WnLfmj5wohVzhL/builds/mMMeYB423HTPVeznH/openapi.json