PubMed Scraper · Papers, Authors, Journals & MeSH Terms

The PubMed Scraper extracts scientific research papers, authors, journal metadata, Medical Subject Headings (MeSH terms), and article abstracts directly from the public NCBI PubMed database (Entrez E-utilities API). Designed for biomedical researchers, clinical bioinformaticians, epidemiological analysts, and AI application developers, this scraper provides structured academic publication data with built-in HTTP backoff resilience and pay-per-event pricing.

Maintained by reapX. Every row cites the Apify run that produced it — nothing is inferred, modelled or filled in, and a field absent from the source is absent from the row. The extracted archive for this source is browsable at reapx.dev/data/pubmed-scraper/ and mirrored as an open dataset on Hugging Face and Kaggle. Questions: reapxdev@proton.me

Maintained by reapx (reapxdev@proton.me). Dedicated to public, addressable web data.

---

⬇️ Input

The actor accepts structured search queries, specialized MeSH descriptors, journal filters, author names, publication types, and publication date range limits. All fields carry working defaults or prefilled values for immediate execution.

Field	Type	Description	Default	Example
term	String	Search query string using PubMed Entrez syntax.	"cardiology"	"cancer research"[MeSH Terms]
journal	String	Filter publications by journal title or abbreviation.	Empty	"N Engl J Med"
meshTerm	String	Filter by NLM Medical Subject Headings (MeSH) descriptor.	Empty	"Cardiovascular Diseases"
author	String	Filter publications by author name.	Empty	"Fauci AS"
publicationType	String	Filter by study design or article type.	Empty	"Clinical Trial"
publicationYear	Integer	Filter publications by single calendar year.	Empty	2025
startYear	Integer	Lower bound publication year for date range search.	Empty	2020
endYear	Integer	Upper bound publication year for date range search.	Empty	2026
maxPapers	Integer	Maximum paper records to extract (1 to 10,000).	50	100

Input Example

{
  "term": "cardiology",
  "meshTerm": "Cardiovascular Diseases",
  "publicationType": "Clinical Trial",
  "startYear": 2020,
  "endYear": 2026,
  "maxPapers": 100
}

---

⬆️ Output

Every extracted PubMed publication is saved to the run's default dataset. Results include canonical PMIDs (paperId), DOIs, complete abstracts, authors, MeSH headings, journal citations, and open-access PMC identifiers.

Field Name	Type	Description	Example
paperId	String	Unique PubMed Identifier (PMID) assigned by NCBI.	"38123456"
doi	String / Null	Digital Object Identifier (DOI) assigned to the paper.	"10.1038/s41586-024-07123-x"
title	String	Full scientific title of the research paper.	"Structure and function of human cardiac receptors"
abstract	String / Null	Full text abstract of the publication.	"We report cryo-EM structures of human cardiac..."
journal	String / Null	Abbreviated publishing journal name.	"N Engl J Med"
fullJournalName	String / Null	Full untruncated name of the journal.	"The New England journal of medicine"
pubDate	String / Null	Publication date string as formatted in PubMed.	"2024 Mar 15"
pubYear	Integer / Null	Four-digit calendar publication year.	2024
authors	Array[String]	Array of indexed author names.	["Smith JA", "Doe RC", "Johnson MB"]
firstAuthor	String / Null	Primary or first author listed.	"Smith JA"
lastAuthor	String / Null	Senior or last author listed.	"Johnson MB"
meshTerms	Array[String]	NLM Medical Subject Headings (MeSH) descriptors.	["Cardiovascular Diseases", "Heart Failure"]
articleTypes	Array[String]	Article types assigned by PubMed indexers.	["Journal Article", "Clinical Trial"]
volume	String / Null	Publishing journal volume number.	"390"
issue	String / Null	Publishing journal issue number.	"11"
pages	String / Null	Page range or article locator number.	"1024-1035"
pmcId	String / Null	PubMed Central open-access repository ID.	"PMC10891234"
issn	String / Null	International Standard Serial Number (ISSN).	"0028-4793"
language	String / Null	Language code of the publication.	"eng"
pubmedUrl	String	Direct canonical URL to the paper on PubMed.	"https://pubmed.ncbi.nlm.nih.gov/38123456/"

Sample Output Record

{
  "paperId": "38123456",
  "doi": "10.1038/s41586-024-07123-x",
  "title": "Structure and function of human cardiac receptors in heart failure",
  "abstract": "We report cryo-EM structures of human cardiac receptors in complex with native ligands...",
  "journal": "Nature",
  "fullJournalName": "Nature publishing group",
  "pubDate": "2024 Mar 15",
  "pubYear": 2024,
  "authors": ["Smith JA", "Doe RC", "Johnson MB"],
  "firstAuthor": "Smith JA",
  "lastAuthor": "Johnson MB",
  "meshTerms": ["Cardiovascular Diseases", "Receptors, Cell Surface", "Heart Failure"],
  "articleTypes": ["Journal Article", "Research Support, N.I.H."],
  "volume": "627",
  "issue": "8002",
  "pages": "180-188",
  "pmcId": "PMC10891234",
  "issn": "0028-0836",
  "language": "eng",
  "pubmedUrl": "https://pubmed.ncbi.nlm.nih.gov/38123456/"
}

---

How it works

The PubMed Scraper queries official NCBI Entrez E-utilities endpoints (esearch.fcgi, esummary.fcgi, and efetch.fcgi) over standard HTTPS.

1. Search Indexing (esearch.fcgi): Constructs structured query strings combining terms, MeSH headings, journal titles, author names, publication types, and date ranges. Retrieves PMIDs matching the search criteria.
2. Summary Metadata (esummary.fcgi): Fetches detailed JSON metadata records including titles, journal names, publication dates, volume/issue numbers, and author lists in batches.
3. XML Abstracts & MeSH Parsing (efetch.fcgi): Fetches NLM XML payloads for each PMID batch to extract complete abstracts and Medical Subject Headings (MeSH descriptors).
4. Push-As-You-Go Dataset Writing: Records are pushed directly to the default dataset as they are parsed, ensuring zero data loss if a run is cancelled or hits budget caps.
5. Rate Limit Resilience: Enforces NCBI's rate limit of 3 requests per second and automatically executes exponential backoff on HTTP 429 or 5xx status codes.

---

❓ FAQ

1. How does Pay-Per-Event (PPE) pricing work?

You are charged a flat rate of $0.002 per complete paper record written to the dataset. Tiered volume discounts (BRONZE through DIAMOND) apply automatically based on your Apify plan. Searches that return zero results, fail due to invalid syntax, or encounter rate limits incur $0.00 zero charge.

2. Can I filter by multiple conditions simultaneously?

Yes! You can combine search terms, MeSH terms, journal names, author cohorts, publication types, and year ranges in a single run. The scraper merges these into an AND query string.

3. How do I extract full text articles?

PubMed stores abstracts, citations, and metadata. When an article is open-access on PubMed Central, the pmcId field (e.g. PMC10891234) is populated so you can locate the full text on PMC.

4. What happens if PubMed rate limits the run?

The scraper detects HTTP 429 rate limit responses and automatically pauses with exponential backoff before retrying, ensuring high reliability across long extraction runs.

---

💬 Your feedback

Have a feature request, bug report, or need custom research extraction pipelines? Contact the maintainers at reapxdev@proton.me.

---

Unofficial - not affiliated with NCBI or PubMed. Collects public data only. reapx. Contact reapxdev@proton.me.

🧪 Example input

A real, runnable configuration — this is an actual input this Actor has run with.

{
  "meshTerm": "Neoplasms",
  "maxPapers": 50
}

📄 Sample output

One real row from a real run of this Actor, unedited.

{
  "paperId": "42545075",
  "doi": "10.1128/aac.00137-26",
  "title": "Penicillin-resistant Streptococcus dysgalactiae subsp. equisimilis and the association with penicillin-binding protein mutations: a population genotypic and phenotypic study",
  "abstract": "Streptococcus dysgalactiae subspecies equisimilis (SDSE) is increasingly recognized as a cause of human disease. Despite decades of penicillin use, SDSE is near universally penicillin-susceptible. Here we report and characterize by whole genome sequencing (WGS) three penicillin-resistant SDSE isolates (minimum inhibitory concentration [MIC] ≥0.256 mg/L) isolated from blood and skin in an individual with recurrent cellulitis on penicillin prophylaxis. The isolates carried 11 penicillin-binding protein (PBP) PBP2x, five PBP1a, and one PBP2b mutations that were distinct from the only previously reported WGS-characterized penicillin-resistant SDSE isolates (n = 4). We then took a genotype-led approach to test the MICs of penicillin and amoxicillin for 63 isolates with PBP2x, PBP1a, and PBP2b variants selected from a well-characterized database of invasive SDSE isolates from Australia (2011-2023, n = 436). PBPs were highly conserved in Australian invasive SDSE genomes. Only a single isolate carrying a PBP2x variant with five mutations around the penicillin-binding cleft found in 1/436 invasive isolates demonstrated a four-fold rise in penicillin MIC, although remaining below clinical breakpoints. Strains with PBP2b and PBP1a variants alone were not associated with raised MICs. To provide broader context for PBP variants, we then examined a global database of SDSE genomes (n = 1,160). PBP2x mutations in isolates with raised and/or resistant penicillin MICs were found in only nine global SDSE database genomes. SDSE penicillin resistance is rare and associated with multiple PBP mutations, particularly PBP2x. WGS with phenotypic testing of novel PBP variants could be used to monitor emerging mutations associated with raised penicillin MICs.",
  "journal": "Antimicrob Agents Chemother",
  "fullJournalName": "Antimicrobial agents and chemotherapy",
  "pubDate": "2026 Aug 3",
  "pubYear": 2026,
  "authors": [
    "Xie O",
    "Tan A",
    "Nguyen TB",
    "Hayes AJ",
    "Dotel R",
    "Carter G",
    "O'Sullivan MVN",
    "Bond K",
    "McDevitt CA",
    "Korman TM",
    "Currie BJ",
    "Ascher DB",
    "Tong SYC",
    "van Hal SJ",
    "Davies MR"
  ],
  "firstAuthor": "Xie O",
  "lastAuthor": "Davies MR",
  "meshTerms": [],
  "articleTypes": [
    "Journal Article"
  ],
  "volume": null,
  "issue": null,
  "pages": "e0013726",
  "pmcId": null,
  "issn": "0066-4804",
  "language": "eng",
  "pubmedUrl": "https://pubmed.ncbi.nlm.nih.gov/42545075/"
}

⚠️ Run outcomes and error handling

This Actor reports what happened in the run's status message, and it always keeps whatever it collected. These are the outcomes you can get and what each one means.

Outcome	What it means
Success	Rows were returned and you were charged record-returned at $0.002 per row.
No matches	The source returned nothing for your filters. Nothing is charged. Widen the date window or drop a filter.

What is guaranteed either way

• Every row is pushed as it is built, not buffered to the end of the run. Anything that buffers output loses everything to a timeout, a block or a migration; this does not.
• A field absent from the source is absent from the row. Nothing is inferred, modelled or filled in to make a row look complete.