PubMed Scraper — Abstracts, Authors, MeSH Terms & DOI Export

Extract biomedical literature from PubMed at scale via NCBI E-utilities — no API key, no login, thousands of articles per run.

---

What does PubMed Scraper do?

PubMed Scraper connects to the NCBI E-utilities public API (eutils.ncbi.nlm.nih.gov) to search and retrieve full bibliographic records from the world's largest biomedical literature database, which indexes over 35 million citations. The scraper uses a two-step pipeline: first, esearch.fcgi converts your keyword query into a paginated list of PubMed IDs (PMIDs), then efetch.fcgi retrieves full XML records for those PMIDs in batches of up to 100.

All parsing happens server-side in the actor — you get clean, structured JSON rows with title, abstract, authors, journal, publication date, DOI, PMC ID, author keywords, MeSH medical subject headings, and article type. The scraper respects NCBI's 3 request/second rate limit for keyless access, uses built-in retry logic for transient errors, and can paginate through thousands of results in a single run. It also supports direct PMID lookup for known articles and date-range filtering for targeted literature reviews.

---

Who is it for?

• Academic researchers who need large-scale literature data for meta-analyses, systematic reviews, or bibliometric studies.
• Biotech & pharma teams monitoring the competitive research landscape for specific drug targets, diseases, or mechanisms.
• Data scientists & ML engineers building NLP models, topic classifiers, or citation graphs on biomedical text.
• Healthcare analysts extracting clinical trial publications, systematic reviews, or guidelines on specific conditions.
• Librarians & knowledge managers compiling literature databases for institutional repositories or continuing education programs.

---

Use cases

• Systematic literature review: Pull 500–5,000 abstracts on "immunotherapy lung cancer 2022–2024" for PRISMA-compliant review pipelines.
• Competitive intelligence: Monitor all publications from a competitor research group or institution by author or affiliation query.
• NLP training data: Download thousands of titled, abstracted, MeSH-tagged articles for training biomedical language models.
• Drug target surveillance: Query "(KRAS OR EGFR) AND inhibitor[TI]" to track emerging research on your target compound.
• Grant writing support: Quickly extract the 50 most recent high-impact papers for a specific MeSH heading to inform background sections.

---

Why use PubMed Scraper?

• Completely keyless — uses the NCBI public E-utilities API with no registration, no API key, and no login required.
• Rich data — extracts 12 fields per article: PMID, title, abstract, authors, journal, DOI, PMC ID, date, keywords, MeSH terms, article type, and URL.
• Bulk results — one run can retrieve thousands of articles through automatic pagination; set maxResults up to 10,000.
• Full PubMed syntax — supports advanced queries like (cancer[MeSH]) AND (2023[PDAT]) AND Clinical Trial[PT] exactly as the PubMed search engine.
• Export to CSV, JSON, Excel — Apify dataset viewer lets you download in any format with one click.
• Pay per result — you only pay for Apify compute time proportional to articles retrieved; lightweight enough to run thousands of searches cheaply.

---

What data can you extract?

The actor outputs one structured row per article with the following fields:

Field	Type	Description
pmid	string	PubMed unique identifier
title	string	Full article title
abstract	string	Full abstract text (may include structured sections)
authors	string	Semicolon-separated author names (LastName FirstName format)
journal	string	Full journal title
pubDate	string	Publication date (YYYY-MM or YYYY-MM-DD)
doi	string	Digital Object Identifier
pmcId	string	PubMed Central ID (if open access)
keywords	string	Author-provided keywords, semicolon-separated
meshTerms	string	MeSH medical subject headings, semicolon-separated
articleType	string	Publication type (Journal Article, Review, Clinical Trial, etc.)
url	string	Direct URL to article on PubMed

Example output record

{
  "pmid": "38912345",
  "title": "CRISPR-Cas9-mediated gene editing of BCL11A enhancer reduces HbS and increases fetal hemoglobin in sickle cell disease",
  "abstract": "Background: Sickle cell disease (SCD) affects millions worldwide. METHODS: We designed sgRNAs targeting the BCL11A erythroid enhancer and delivered Cas9 via lentiviral vector to CD34+ hematopoietic stem cells. RESULTS: Editing efficiency exceeded 90% in bulk populations. HbF increased 3.2-fold at 12 weeks post-transplant. CONCLUSIONS: CRISPR-mediated BCL11A disruption is a viable therapeutic strategy for SCD.",
  "authors": "Zhang L; Patel R; Okonkwo C; Kim J; Weatherall D",
  "journal": "Blood",
  "pubDate": "2024-03-15",
  "doi": "10.1182/blood.2024012345",
  "pmcId": "PMC10987654",
  "keywords": "CRISPR; sickle cell disease; fetal hemoglobin; BCL11A; gene therapy",
  "meshTerms": "Anemia, Sickle Cell; CRISPR-Cas Systems; Fetal Hemoglobin; Gene Editing; Hematopoietic Stem Cells",
  "articleType": "Journal Article; Clinical Trial",
  "url": "https://pubmed.ncbi.nlm.nih.gov/38912345/"
}

---

How to use

Option A — Search by keyword query

Enter any PubMed search query and a result limit. The actor runs esearch to find matching PMIDs, then fetches full records.

Example input (JSON):

{
  "query": "crispr gene editing cancer",
  "maxResults": 500,
  "dateFrom": "2022/01/01",
  "dateTo": "2024/12/31",
  "fetchAbstracts": true
}

Steps:

1. Go to the actor's Input tab.
2. Set Search Query to your PubMed search term (supports full PubMed query syntax).
3. Set Max Results (1–10,000).
4. Optionally set Date From and Date To (format YYYY/MM/DD).
5. Click Start.

Option B — Fetch specific PMIDs directly

If you already have a list of PMIDs (e.g. from a saved PubMed search or DOI resolver), pass them directly. This skips esearch entirely.

Example input (JSON):

{
  "pmids": ["38912345", "38876543", "37654321", "36543210"],
  "fetchAbstracts": true
}

Steps:

1. Open the Input tab.
2. In the PubMed IDs (PMIDs) field, enter your PMID array as JSON.
3. Click Start.

---

Input parameters

Parameter	Type	Default	Description
query	string	—	PubMed search query. Supports full syntax: field tags [MeSH], [Author], [TI], Boolean operators, etc.
pmids	array	—	Array of PMID strings for direct lookup. Overrides query when provided.
maxResults	integer	200	Maximum articles to return. Paginated automatically. Range: 1–10,000.
dateFrom	string	—	Filter: published on or after this date. Format YYYY/MM/DD or YYYY.
dateTo	string	—	Filter: published on or before this date. Format YYYY/MM/DD or YYYY.
fetchAbstracts	boolean	true	Fetch full abstracts via efetch. Set false for metadata-only (faster, no abstract).
proxyConfiguration	object	Datacenter	Apify proxy settings. Datacenter proxy is sufficient for NCBI.

Full input JSON:

{
  "query": "mRNA vaccine immunogenicity",
  "maxResults": 1000,
  "dateFrom": "2021/01/01",
  "dateTo": "2024/06/30",
  "fetchAbstracts": true,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["DATACENTER"]
  }
}

---

Output example

{
  "pmid": "39154321",
  "title": "Comparative immunogenicity of BNT162b2 and mRNA-1273 after booster vaccination in healthcare workers",
  "abstract": "We compared neutralizing antibody titers and T-cell responses in 312 healthcare workers who received a third dose of either BNT162b2 or mRNA-1273. Both vaccines produced robust humoral and cellular responses, with mRNA-1273 booster recipients showing 1.4-fold higher geometric mean titers at 28 days. No significant difference was observed in T-cell response magnitude.",
  "authors": "Mendez A; Huang Y; Park S; Osei-Twum C; Hakim F",
  "journal": "Nature Medicine",
  "pubDate": "2024-01-09",
  "doi": "10.1038/s41591-023-02789-1",
  "pmcId": "PMC10876543",
  "keywords": "mRNA vaccine; booster; immunogenicity; COVID-19; healthcare workers",
  "meshTerms": "COVID-19; COVID-19 Vaccines; Antibodies, Neutralizing; Immunogenicity, Vaccine; Vaccination",
  "articleType": "Journal Article; Comparative Study",
  "url": "https://pubmed.ncbi.nlm.nih.gov/39154321/"
}

---

Tips for best results

• Use PubMed query syntax: Add field tags for precision — KRAS[Gene Name] AND inhibitor[TI] AND 2023[PDAT] retrieves papers mentioning KRAS in gene name and inhibitor in title published in 2023.
• Use date filters for recency: Set dateFrom/dateTo to narrow large result sets. PubMed uses YYYY/MM/DD format.
• Boolean operators work: Combine terms with AND, OR, NOT in uppercase — (diabetes OR obesity) AND metformin AND clinical trial[PT].
• MeSH terms improve recall: Querying cancer[MeSH] captures papers indexed under any cancer MeSH subheading (broader than a keyword search).
• Start with 200, scale up: Test your query with 200 results first to validate relevance before running 5,000.
• Use pmids for known lists: If you exported PMIDs from PubMed's website, paste them directly — avoids re-running the search.
• Abstract-free mode for bulk: Set fetchAbstracts: false for quick metadata sweeps when you only need title/authors/journal.
• Parallel runs for large reviews: Run multiple actors concurrently with different dateFrom/dateTo ranges for maximum throughput.
• Export to Excel: Download the dataset as XLSX from Apify's dataset viewer for seamless spreadsheet analysis.
• Schedule monthly: Use Apify's Scheduler to monitor new publications on a topic each month automatically.

---

Integrations

Connect PubMed Scraper to your existing workflows:

• Google Sheets: Use the Apify → Google Sheets integration to automatically append new article data to a spreadsheet. Ideal for ongoing literature monitoring.
• Slack: Trigger a Slack notification when new papers on your topic are published — use Apify webhooks + Slack Incoming Webhooks.
• Zapier: Connect via Zapier's Apify integration. When a run finishes, push results to Airtable, Notion, or any Zapier-connected app.
• Make (Integromat): Use Make's Apify module to feed PubMed results into citation managers, databases, or email summaries.
• Webhooks: Configure an Apify webhook to POST to your endpoint whenever a run succeeds — pipe data into your research pipeline.
• Schedule: Use Apify's built-in Scheduler to run weekly/monthly literature sweeps on any saved query. Never miss a new paper.

---

API usage

You can trigger runs and retrieve data programmatically via the Apify API.

cURL:

curl -X POST \
  "https://api.apify.com/v2/acts/logiover~pubmed-scraper/runs?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"query": "crispr gene editing", "maxResults": 200}'

Node.js (Apify client):

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_TOKEN' });

const run = await client.actor('logiover/pubmed-scraper').call({
  query: 'mRNA vaccine immunogenicity',
  maxResults: 500,
  dateFrom: '2022/01/01',
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(`Retrieved ${items.length} articles`);
items.slice(0, 3).forEach(a => console.log(a.title));

Python:

from apify_client import ApifyClient

client = ApifyClient("YOUR_TOKEN")

run = client.actor("logiover/pubmed-scraper").call(run_input={
    "query": "BRCA1 breast cancer mutation",
    "maxResults": 1000,
    "fetchAbstracts": True,
})

dataset = client.dataset(run["defaultDatasetId"])
articles = list(dataset.iterate_items())
print(f"Retrieved {len(articles)} articles")
for a in articles[:3]:
    print(a["pmid"], a["title"][:80])

---

Use with AI agents (MCP)

PubMed Scraper is available as an MCP-compatible tool, making it usable directly from Claude, Cursor, or any MCP-enabled AI agent. Connect via the Apify MCP server and prompt your agent: "Search PubMed for recent papers on GLP-1 receptor agonists and weight loss published after 2022, get 300 results, and summarize the key findings." The agent will invoke the actor, retrieve the structured dataset, and synthesize the literature for you — no manual searching or copy-pasting.

---

FAQ

Does this actor require a PubMed API key?

No. It uses the NCBI E-utilities public endpoint, which is freely accessible without registration. For more than 3 requests/second (high-volume use), NCBI recommends registering for a free API key — but the actor is designed to stay within the keyless rate limit automatically.

What is the maximum number of results per run?

Up to 10,000 results per run via the maxResults parameter. For larger needs, run multiple actors with different date ranges or queries.

Does it cover all PubMed articles?

It searches the same index as PubMed.gov — over 35 million biomedical citations from MEDLINE and other life science journals. Coverage and indexing depend on NCBI's policies.

Why are some abstract fields empty?

Not all PubMed articles have abstracts indexed. Conference abstracts, editorials, letters, and older articles often lack full text. The abstract field will be an empty string in those cases.

Why are MeSH terms missing for some articles?

MeSH indexing is done manually by NLM staff and can lag 6–12 months for newly published articles. Recent articles may have no MeSH terms yet.

Can I search by author, journal, or institution?

Yes — use PubMed field tags: Smith J[Author], Nature[Journal], Harvard[Affiliation]. Full query syntax is supported.

How long does a run take?

For 200 articles: ~1–2 minutes. For 1,000 articles: ~5–8 minutes. For 5,000 articles: ~20–30 minutes. Speed is limited by NCBI's rate limit.

Can I filter by article type (e.g. only clinical trials)?

Yes — add Clinical Trial[PT], Review[PT], or Meta-Analysis[PT] to your query. These are PubMed Publication Type tags.

How do I export results to Excel?

Go to the run's dataset in Apify Console, click Export, and choose XLSX. Or use the Apify API with format=xlsx parameter.

How often should I re-run for monitoring?

Monthly for fast-moving topics. Use Apify Scheduler to automate this. Combine with Slack/email webhooks for alerts on new papers.

Is the DOI always present?

DOI is populated when NCBI has it indexed (most articles from 2000+). Older articles and some non-English journals may lack DOIs.

What does PMC ID mean?

PMC ID (PubMed Central ID) is present when a free full-text version is available at PubMedCentral.gov. Use it to construct links like https://www.ncbi.nlm.nih.gov/pmc/articles/{pmcId}/.

---

Is it legal?

This actor uses the NCBI E-utilities API, a publicly documented service provided by the National Library of Medicine for programmatic access to PubMed data. NCBI explicitly designed E-utilities for automated querying. Bibliographic metadata (titles, authors, abstracts) is factual data not subject to copyright in most jurisdictions. The actor operates within NCBI's rate limits and does not scrape the PubMed website HTML. Use responsibly and in accordance with NCBI's API terms and your institution's data policies.

---

Related scrapers

• OpenAlex Academic Papers Scraper — 250M+ open access academic papers across all disciplines.
• Semantic Scholar Research Scraper — AI-focused academic search with citation counts and influence scores.
• arXiv Paper Scraper — Preprints in physics, math, CS, and biology before peer review.
• ClinicalTrials.gov Scraper — Extract clinical trial registrations, phases, sponsors, and results.