PubMed Scraper: Biomedical Literature & Abstracts

PubMed Scraper searches the 35M+ citations in PubMed and returns one flat record per article, title, full abstract with its section labels intact, authors and their affiliations, journal placement, MeSH terms, keywords, DOI and PMC ID.

It reads the data the NIH publishes specifically for programmatic access: no browser, no proxies, no login, no scraping of HTML pages. An optional free NCBI key triples throughput; without one the Actor paces itself automatically.

Agent skill: SKILL.md

https://api.apify.com/v2/key-value-stores/t7YoTxpZEJOWvw4Ug/records/pubmed-articles-scraper.md

What you get

Output field	Meaning
pmid, doi, pmcId, url	PubMed ID, publisher DOI, PubMed Central ID and the PubMed page
title	Article title, inline <i>/<sup> markup stripped
abstract	Full abstract. Structured abstracts keep their headings, BACKGROUND: … METHODS: …
authors	Author names in order; consortium authors appear under their collective name
affiliations	Every distinct affiliation string on the record
journal, journalAbbrev, issn	Full journal title, its ISO abbreviation and ISSN
volume, issue, pages	Issue placement
publicationDate, publicationYear	Most precise date the record supports, 2020-12-31, 2026-12 or 1929
meshTerms	NLM Medical Subject Headings assigned by indexers
keywords	Author-supplied keywords
publicationTypes	Journal Article, Review, Randomized Controlled Trial, Published Erratum…
language, languages, country	Primary publication language, every language the record lists, and the journal's country of origin
scrapedAt	Run timestamp

A RUN_SUMMARY record in the key-value store holds per-run counts, the filters used, every query that failed or matched nothing, and the PMIDs that were unusable or resolved to no record.

Common use cases

• Systematic reviews. One query per PICO arm, filtered to Review and Meta-Analysis, exported to CSV for screening.
• Biomedical RAG corpora. title + abstract + meshTerms is a chunk with its own controlled vocabulary attached.
• Institutional output tracking. Search "Karolinska Institutet"[Affiliation] and count by year.
• Pharmacovigilance and horizon scanning. Schedule a daily run sorted by publication date and diff on pmid.
• DOI reconciliation. Feed in PMIDs and get DOIs and PMC IDs back for joining onto other sources.

Quick start

One topic, the 100 best matches:

{
 "searchQueries": ["CRISPR gene editing"]
}

Recent English-language reviews and meta-analyses:

{
 "searchQueries": ["long covid[MeSH Terms]", "post-acute sequelae SARS-CoV-2"],
 "articleTypes": ["Review", "Meta-Analysis"],
 "language": "eng",
 "fromDate": "2024-01-01",
 "sortBy": "pub_date",
 "maxResults": 500,
 "email": "you@example.com"
}

Enrich a list of PMIDs you already have, metadata only:

{
 "pmids": ["33301246", "32109013"],
 "includeAbstract": false
}

Input

Field	Type	Default	Notes
searchQueries	array	[]	PubMed query syntax. Field tags work: [Title], [Author], [Journal], [MeSH Terms], [Affiliation]. Combine with AND / OR / NOT.
pmids	array	[]	Specific articles. Bare PMID, PMID: 33301246, or a pubmed.ncbi.nlm.nih.gov URL.
fromDate / toDate	string	-	Publication date bounds as YYYY-MM-DD, YYYY-MM or YYYY. Applied as PubMed's own pdat filter. A partial toDate closes on the real last day of the period, so 2024-02 ends on the 29th in 2024 and the 28th in 2023.
articleTypes	array	[]	Publication types, ORed together and ANDed onto every query.
language	string	-	Three-letter code, eng, fre, ger, spa.
sortBy	string	relevance	relevance (Best Match), pub_date, Author or JournalName.
includeAbstract	boolean	true	Include the full abstract. Off makes the dataset several times smaller.
maxResults	integer	100	Articles saved per source — each search query is a source, and the whole pmids list is one more. Hard ceiling 9,999, see Limits.
email	string	-	Passed to NCBI so it can contact you rather than block you. Recommended.
apiKey	string	-	Free NCBI key. Raises the rate limit from 3 to 10 requests/second.

At least one of searchQueries or pmids is required, PubMed has no "return everything" mode. Both can be used in the same run; results are de-duplicated by pmid.

Output example

A real record from the run above:

{
 "pmid": "42522380",
 "doi": "10.1080/21691401.2026.2702869",
 "pmcId": null,
 "title": "Recent advances of CRISPR-based gene editing technologies and delivery strategies.",
 "abstract": "CRISPR technology is a powerful tool for gene editing, in which the efficient delivery of living target cells allows it to show great clinical potential. …",
 "authors": ["Fanyu Xue", "Zixuan Xin", "Gaojie Wang", "Jiayin Xing", "Hengxin Han", "Xiangxiang Zhao", "Ningning Song"],
 "affiliations": ["College of Life Science and Technology, Shandong Second Medical University, Weifang, China."],
 "journal": "Artificial cells, nanomedicine, and biotechnology",
 "journalAbbrev": "Artif Cells Nanomed Biotechnol",
 "issn": "2169-141X",
 "volume": "54",
 "issue": "1",
 "pages": "415-431",
 "publicationDate": "2026-12",
 "publicationYear": 2026,
 "meshTerms": ["Humans", "Gene Editing", "Animals", "CRISPR-Cas Systems", "Gene Transfer Techniques"],
 "keywords": ["Adeno-associated viral vectors", "CRISPR-Cas", "LNP delivery systems", "delivery system"],
 "publicationTypes": ["Journal Article", "Review"],
 "language": "eng",
 "languages": ["eng"],
 "country": "England",
 "url": "https://pubmed.ncbi.nlm.nih.gov/42522380/",
 "scrapedAt": "2026-08-06T12:15:43.691Z"
}

meshTerms is empty for very recent records, NLM indexers assign headings weeks to months after publication, so a paper from last week has keywords but no MeSH. pmcId is only present when a free full text exists in PubMed Central.

Writing a PubMed query

The searchQueries field is passed to PubMed untouched, so anything that works in the search box works here:

Goal	Query
Word in the title	CRISPR[Title]
Specific author	Doudna JA[Author]
Specific journal	Nature[Journal]
Controlled vocabulary	"gene editing"[MeSH Terms]
One institution	"Karolinska Institutet"[Affiliation]
Combined	(CRISPR[Title] OR Cas9[Title]) AND therapy NOT review[Publication Type]

Plain keywords work too, PubMed's term mapping expands cancer into a large synonym set on your behalf. RUN_SUMMARY.filters records exactly what was sent.

API example

curl -X POST "https://api.apify.com/v2/acts/arman-bd~pubmed-articles-scraper/run-sync-get-dataset-items?token=YOUR_TOKEN" \
 -H "Content-Type: application/json" \
 -d '{
 "searchQueries": ["CRISPR gene editing"],
 "articleTypes": ["Review"],
 "fromDate": "2025-01-01",
 "maxResults": 50
 }'

Limits and behaviour

• Two steps per query. esearch returns PMIDs, then efetch pulls the records 200 at a time. A 1,000-article run is one esearch plus five efetch calls, so cost per article falls sharply as runs get bigger.
• 9,999 results per query is PubMed's ceiling. esearch refuses retstart above 9998 and clamps retmax to 9999. Slice a broader sweep with fromDate / toDate rather than raising the cap.
• Rate limits are respected. 3 requests/second without a key, 10 with one. The Actor paces itself to stay under whichever applies, and NCBI's Retry-After is honoured on the rare 429.
• XML is parsed without a dependency. efetch answers XML, read here by a small purpose-built extractor. Presentational markup is stripped before entities are decoded, so α arrives as α and an escaped <i> stays visible text.
• Books are handled too. PubMed indexes NCBI Bookshelf entries such as GeneReviews as PubmedBookArticle records with a different structure. They are parsed rather than dropped, but carry fewer fields, no volume, issue or MeSH.
• Unknown PMIDs are named, not swallowed. efetch returns an empty set rather than an error for an ID that does not exist, so the run diffs what came back against what you asked for: entries that hold no ID at all land in RUN_SUMMARY.invalidPmids, IDs that resolve to no record in RUN_SUMMARY.unresolvedPmids.
• A query that matches nothing is a recorded outcome. A zero-result search is not an error on the wire, so the run reads PubMed's own warning list and writes the reason into RUN_SUMMARY.failures with kind: "empty" instead of finishing quietly with no rows.
• A failing query never aborts the run. It is logged and recorded in RUN_SUMMARY.failures with kind: "error"; the rest continue. The run only errors out if every source fails outright.
• Public data only. PubMed citations are public records. No authentication beyond the optional rate-limit key, no personal data, no access-control bypass.

FAQ

Do I need a proxy? No. Proxy configuration is not required to run this Actor.

Do I need an NCBI account? Not to run it. The API key is optional and only affects speed.

Why should I fill in email? NCBI asks automated clients to identify themselves and contacts the address before blocking anything that looks abusive. It is never used by this Actor for anything else.

What happens if a query fails? It is reported in RUN_SUMMARY.failures and the run continues with the rest.

Can I schedule it? Yes. Pair fromDate with sortBy: pub_date and diff on pmid for a "what's new this week" feed.

Does it fetch full text? No, the abstract only. When pmcId is present the full text is free at ncbi.nlm.nih.gov/pmc/articles/{pmcId}/.

Why is abstract null? Some records have none, editorials, errata, letters and many older papers. Nothing is invented.

Can I search MEDLINE only, or exclude preprints? Yes, with normal PubMed syntax: append NOT preprint[Publication Type] or AND medline[Subset] to your query.

Can I integrate it with something else? Yes, Apify API, client libraries, webhooks, scheduled runs, dataset exports (JSON/CSV/Excel) or MCP. Output is structured JSON.