# PubMed Scraper - Articles, Abstracts, Authors & MeSH (`neverempty/pubmed-scraper`) Actor For literature reviews, RAG pipelines and LLM training sets: PubMed articles with the full abstract, authors with affiliations and ORCID, DOI, PMC ID and MeSH terms. Goes past PubMed's 9,999-result cap by slicing dates, so a 70,772-hit search comes back whole. Export as JSON, CSV or Excel. - **URL**: https://apify.com/neverempty/pubmed-scraper.md - **Developed by:** [NeverEmpty](https://apify.com/neverempty) (community) - **Categories:** AI, Developer tools, Other - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $5.11 / 1,000 article returneds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. - **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use). - **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`. - **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`. - **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`. - **Any other language** — the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## PubMed Scraper **For systematic reviews and literature monitoring, pharma and biotech competitive intelligence, academic research and RAG pipelines over medical literature**: PubMed articles as flat rows, straight from NCBI's official E-utilities API — title, the full abstract with its section labels, every author with affiliations and ORCID, journal, volume, issue, pages, DOI, PMC ID, MeSH terms, keywords, article types and a retraction flag. Search the way you search PubMed, or look up a list of PMIDs. **One PubMed search stops at 9,999 results. This goes past it — without counting anything twice.** E-utilities refuses to page beyond the 9,999th record (`'retstart' cannot be larger than 9998`), and the obvious workaround double-counts: an article with separate online and print dates falls into both slices. Measured on 2026-09-12, yearly publication-date slices of `crispr` added up to **77,497** against a true total of **70,772** — about **6,700** phantom articles. This Actor splits by *date added* instead, newest first, and never returns an article twice. **And a typo in a PubMed query becomes a bill for thousands of articles, silently.** All measured on 2026-09-12, all answered HTTP 200 with a normal-looking result: `crispr zzqqxxnotaword` **drops the unknown word** and returns all **70,772** `crispr` articles; `crispr[zzfield]` ignores the unknown field tag and returns **70,772** again; `"crispr gene editing zzqq"` finds no such phrase, **drops the quotation marks** and returns **25,351** articles matching the words separately; and `"Doudna JA"[Author] OR "Zzqqxx QQ"[Author]` returns Doudna's **436** articles and says nothing at all about the second author. **Dates are only ever what PubMed states.** They arrive as a full date, a year and month, a year and season (`1976 Fall`) or free text (`1976 Dec 23-30`), so `publicationDate` holds only the precision actually published (`2026-08-31`, `2024-03` or `1976`) with PubMed's own wording kept alongside. Of 300 records from 1976, **167** are in one of those looser forms. The DOI and PMC ID come only from the article's own ID list, and the PMID only from its own field — never inferred from a neighbouring record. No API key, no browser, no proxy. Export as JSON, CSV or Excel. ```json { "query": "semaglutide", "publicationTypes": ["Randomized Controlled Trial"], "dateFrom": "2024", "maxResults": 500 } ``` Look up PMIDs directly (numbers, `PMID: 33087898` or PubMed links all work): ```json { "pmids": ["33087898", "23287718", "https://pubmed.ncbi.nlm.nih.gov/22745249/"] } ``` ### What this fixes about the raw API Everything below was measured against the live API on 2026-09-12, not read from documentation. #### 1. One PubMed search stops at 9,999 results - this goes past it E-utilities refuses to page beyond the 9,999th record of a search (`'retstart' cannot be larger than 9998`). Above that the only way through is to cut the search into date ranges. The obvious cut, publication date, **double-counts**: an article with separate online and print dates falls into both. Measured on 2026-09-12, the yearly publication-date slices of `crispr` added up to 77,497 against a true total of 70,772 - about 6,700 articles counted twice. The date an article was added to PubMed gives each article exactly one date, and those slices added up to the total exactly. (PubMed grows every day, so these counts are larger by the time you read this; what stays true is which cut double-counts.) So when you ask for more than 9,999 articles, this Actor splits the search by date added, newest first, and never returns an article twice. PubMed sometimes adds more than 9,999 matching records on a single day (loading days in the 1990s did this); only the first 9,999 of such a day can be read, and a row tells you how many could not be reached instead of pretending the result is complete. #### 2. PubMed quietly widens a search it does not understand Counts measured on 2026-09-12: | what is sent | what PubMed does | |---|---| | `crispr zzqqxxnotaword` | drops the unknown word and returns all **70,772** `crispr` articles | | `crispr[zzfield]` | ignores the unknown field tag, again 70,772 | | `"crispr` (unclosed quote) | "Unmatched double quote ignored." and returns everything | | `"crispr gene editing zzqq"` | finds no such phrase, **drops the quotation marks** and returns **25,351** articles matching the words separately | In each case the answer is HTTP 200 with a normal-looking result, so a typo turns into a bill for thousands of articles you did not ask for. Here nothing is fetched or charged when PubMed says it changed the search; a row states what it would have changed and how many articles the wider search matched. Ordinary words PubMed leaves out on purpose (`of`, `on`, `the`) do not stop a run, and a quoted phrase that does match is searched as the phrase, exactly as asked. #### 3. An unknown author in a list disappears without a word `"Doudna JA"[Author] OR "Zzqqxx QQ"[Author]` returns Doudna's 436 articles and says nothing about the second name. When you give two or more authors or journals, each one is also counted on its own, and a name with no articles at all comes back as its own row - so a misspelt name in a list of twenty is visible. #### 4. Each record carries its references' IDs A PubMed record includes the reference list, with the PMIDs and DOIs of the papers it cites (46 records held 1,464 DOI entries). The DOI and PMC ID here come only from the article's own ID list, and the PMID only from its own citation - never from a reference or from a retraction notice. #### 5. Dates are not made up Publication dates come as a full date, a year and month, a year and season (`1976 Fall`) or free text (`1976 Dec 23-30`). `publicationDate` holds only what PubMed actually states (`2026-08-31`, `2024-03` or `1976`), `publicationDatePrecision` says which (`day`, `month`, `year`), and `publicationDateText` keeps PubMed's own wording, written the way PubMed writes it (`2023 Feb 1`, not `2023 Feb 01`). Of 300 records from 1976, 167 have no abstract; those rows have `abstract: null` and `hasAbstract: false`, not an empty string. `dateAddedToPubmed` is the same day PubMed's own `[edat]` index uses - checked record by record against the `EDAT` field of PubMed's MEDLINE output on 2026-09-12 - so the newest value in a run can be fed straight back in as `dateFrom` with `dateType: added` without re-paying for the boundary day. #### 6. Missing PMIDs are reported, not dropped `efetch` silently leaves out a PMID it has no record for. In a lookup, every PMID you gave comes back: as an article, or as a row saying it was not found, not read, or not looked up because the run hit its charge limit or time limit. Only article rows are charged. ### What you get `pmid`, `url`, `title`, `abstract`, `hasAbstract`, `authors` (as `Last, First`), `firstAuthor`, `lastAuthor`, `authorCount`, `affiliations`, `authorDetails` (last name, fore name, initials, collective name, ORCID and affiliations for each author), `editors` and `editorCount` (the editors of a book, who are not its authors), `journal`, `journalAbbreviation`, `issn`, `volume`, `issue`, `pages`, `publicationDate`, `publicationDatePrecision`, `publicationDateText`, `publicationYear`, `electronicPublicationDate`, `dateAddedToPubmed`, `doi`, `pmcId`, `pmcUrl`, `meshTerms`, `majorMeshTerms`, `keywords`, `publicationTypes`, `languages`, `isRetracted`, `publicationStatus`, `citationStatus`, `articleKind` (`journal-article` or `book`), `bookTitle`, `publisher`, `copyrightInformation`, `query`, `checkedAt`, plus `source`, `status` and `note` on every row. ### Input | Field | What it does | |---|---| | `query` | Anything you would type into PubMed, with its syntax: `AND`/`OR`/`NOT`, `"exact phrases"`, `[Title/Abstract]`, `[MeSH Terms]`, `[Affiliation]`. | | `authors` | Author names as PubMed writes them (`Doudna JA` or `Doudna, Jennifer A`), one per line. Several mean any of them. A name is matched exactly as written: `Smith J` matched 8,016 articles, while PubMed's unquoted `Smith J[au]` also takes every Smith JA, JB and so on (34,950) and splits `O'Brien S` into `o brien s`. For that wider form, write it in `query` instead. | | `journals` | Journal titles or abbreviations (`N Engl J Med`), one per line. Several mean any of them. | | `publicationTypes` | Article types such as Review, Systematic Review, Meta-Analysis, Randomized Controlled Trial, Case Reports, Preprint. | | `dateFrom` / `dateTo` | `2024`, `2024-03` or `2024-03-15`. A date that does not exist is refused, because PubMed answers a malformed date with zero results. | | `dateType` | `publication` (publication date) or `added` (date added to PubMed - use this to collect new articles since a previous run). | | `pmids` | PMIDs to fetch directly, up to 10,000 per run. Can be combined with a search; an article found both ways is returned and charged once. | | `sort` | `relevance` (Best Match), `pub_date` (newest first), `Author`, `JournalName`. | | `maxResults` | How many search results to return, up to 100,000. You are charged for the articles you receive. Measured rate is about 19 articles a second, so the default one-hour run timeout reaches roughly 69,000; for more than that, raise the run's timeout. A run that runs out of time stops and says so in a row rather than presenting a part as the whole. | | `maxRetries` | Attempts per request when PubMed does not answer or says too many requests arrived. | Fields are combined with AND. If you give no input at all, the Actor runs the example search `CRISPR gene editing` and every row says so in its `query` column; if you give anything, only what you gave is searched. ### Speed and limits NCBI allows three requests per second without an API key, and this Actor keeps to that (a fourth request in the same second is answered with HTTP 429; the Actor waits and retries). Articles are fetched 200 at a time, so a search of 1,000 articles takes about six requests. That limit is **per IP address, not per run**: two runs of this Actor going at the same time from the same machine share the same three requests a second and can exceed it together. Each run then spends time waiting out 429s and retrying, so it gets slower rather than failing - but if you want full speed, run them one after another. Measured throughput on 2026-09-11: **14,753 articles in 12 minutes 50 seconds** (about 19 a second), on a search large enough to be split by date added. The default run timeout of one hour therefore reaches roughly 69,000 articles; ask for more only with a longer timeout. ### Source, attribution and disclaimer Data: **PubMed, courtesy of the U.S. National Library of Medicine**, retrieved live through the NCBI E-utilities (`eutils.ncbi.nlm.nih.gov`) at the time of each run - the `checkedAt` column records when. Records in PubMed are corrected and updated after publication, so a row reflects PubMed on that date, not necessarily today. This Actor identifies itself to NCBI with its own `tool` and `email` values on every request, as the E-utilities usage guidelines require. This Actor is not affiliated with, endorsed by or sponsored by the National Library of Medicine, NCBI or the U.S. government. NCBI disclaimer: the U.S. Government does not warrant or assume any legal liability or responsibility for the accuracy, completeness or usefulness of any information, apparatus, product or process disclosed. NCBI does not endorse or recommend any commercial products, processes or services. See https://www.ncbi.nlm.nih.gov/home/about/policies/. Copyright: NLM does not claim copyright on PubMed abstracts, but journal publishers or authors may. Anyone reproducing, redistributing or making commercial use of abstracts is expected to follow the terms set by the copyright holder; the `copyrightInformation` column carries the publisher's notice where PubMed has one. PubMed data is not medical advice. # Actor input Schema ## `query` (type: `string`): Anything you would type into the PubMed search box, including its syntax: AND / OR / NOT, "exact phrases" and field tags such as \[Title/Abstract], \[MeSH Terms] or \[Affiliation]. If PubMed cannot find a word, cannot find a quoted phrase, or would ignore part of the query, it normally drops or unquotes that part and quietly returns a wider search; here nothing is fetched or charged in that case and a row says what PubMed would have changed. A quoted phrase that PubMed does match is searched as a phrase, as asked. ## `authors` (type: `array`): Author names as PubMed writes them, one per line: Doudna JA or Doudna, Jennifer A. A name is matched exactly as written, so Smith J does not also return Smith JA (for that wider form, write Smith J\[au] in the search query). Several authors mean any of them (OR). When you give two or more, each is also counted on its own and a name with no articles at all comes back as its own row, because PubMed drops an unknown author from an OR list without saying so. ## `journals` (type: `array`): Journal titles or their abbreviations, one per line: Nature, The New England journal of medicine or N Engl J Med. Several journals mean any of them (OR). ## `publicationTypes` (type: `array`): Keep only these PubMed publication types. Several types mean any of them (OR). ## `dateFrom` (type: `string`): Earliest date, written 2024, 2024-03 or 2024-03-15. Leave empty for no lower limit. A date that does not exist (2025-02-30) is refused rather than sent, because PubMed answers a malformed date with zero results. ## `dateTo` (type: `string`): Latest date, written the same way. 2024 means up to 31 December 2024. Leave empty for no upper limit. ## `dateType` (type: `string`): publication = the article's publication date (an article with separate online and print dates matches either). added = the day the record was added to PubMed, which is the one to use for new articles since your last run. ## `pmids` (type: `array`): PubMed IDs to fetch directly, one per line: 33087898, PMID: 33087898 or a pubmed.ncbi.nlm.nih.gov link. Up to 10,000 per run. A PMID PubMed has no record for comes back as its own row saying so (not charged). This can be combined with a search; an article found both ways is returned and charged once. ## `sort` (type: `string`): relevance = PubMed's Best Match. pub\_date = newest publication first. Past 9,999 results the search is split by date added, newest first, and this order applies within each part. ## `maxResults` (type: `integer`): How many articles the search returns (PMID lookups are not counted against this). You are charged for the articles you actually receive. PubMed lets one search read only 9,999 results; above that this Actor splits the search by the date each article was added to PubMed, so up to 100,000 can be returned. Measured rate is about 19 articles a second, so the default one-hour run timeout reaches roughly 69,000 - for more than that, give the run a longer timeout. A run that runs out of time stops and adds a row saying so, instead of presenting a part as the whole. ## `maxRetries` (type: `integer`): How many times to try a request in total when PubMed does not answer or answers that too many requests arrived (it allows three per second). A rejected request is never retried. ## Actor input object example ```json { "query": "CRISPR gene editing", "dateType": "publication", "sort": "relevance", "maxResults": 100, "maxRetries": 4 } ``` # Actor output Schema ## `results` (type: `string`): One row per PubMed article, plus a row whenever a search matched nothing, was changed by PubMed, a PMID was not found, or part of the run could not be read. Only article rows are charged. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "query": "CRISPR gene editing" }; // Run the Actor and wait for it to finish const run = await client.actor("neverempty/pubmed-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "query": "CRISPR gene editing" } # Run the Actor and wait for it to finish run = client.actor("neverempty/pubmed-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "query": "CRISPR gene editing" }' | apify call neverempty/pubmed-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,neverempty/pubmed-scraper" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/2Aoj1gPA463eLA3G1/builds/OiHw3CeqNxYSqR4wd/openapi.json