# PubMed Scraper · Papers, Authors, Journals & MeSH Terms (`reapx/pubmed-scraper`) Actor Scrape academic research papers, authors, journals, MeSH terms, abstracts, and open-access metadata from NCBI PubMed API. Features HTTP backoff resilience and pay-per-event pricing. - **URL**: https://apify.com/reapx/pubmed-scraper.md - **Developed by:** [Tarek Etman](https://apify.com/reapx) (community) - **Categories:** Developer tools, Business, Other - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $1.30 / 1,000 record returneds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. - **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use). - **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`. - **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`. - **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`. - **Any other language** — the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ![reapX — public sources in, addressable records out](https://reapx.dev/reapx.gif) ## PubMed Scraper · Papers, Authors, Journals & MeSH Terms The **PubMed Scraper** extracts scientific research papers, authors, journal metadata, Medical Subject Headings (MeSH terms), and article abstracts directly from the public NCBI PubMed database (Entrez E-utilities API). Designed for biomedical researchers, clinical bioinformaticians, epidemiological analysts, and AI application developers, this scraper provides structured academic publication data with built-in HTTP backoff resilience and pay-per-event pricing. > Maintained by **reapX**. Every row cites the Apify run that produced it — nothing is > inferred, modelled or filled in, and a field absent from the source is absent from the row. > The extracted archive for this source is browsable at > [reapx.dev/data/pubmed-scraper/](https://reapx.dev/data/pubmed-scraper/) and mirrored as an open dataset on > [Hugging Face](https://huggingface.co/datasets/reapxdev/pubmed-scraper) and > [Kaggle](https://www.kaggle.com/datasets/reapxdev/pubmed-scraper). Questions: reapxdev@proton.me Maintained by reapx (`reapxdev@proton.me`). Dedicated to public, addressable web data. *** ### ⬇️ Input The actor accepts structured search queries, specialized MeSH descriptors, journal filters, author names, publication types, and publication date range limits. All fields carry working defaults or prefilled values for immediate execution. | Field | Type | Description | Default | Example | | :--- | :--- | :--- | :--- | :--- | | `term` | String | Search query string using PubMed Entrez syntax. | `"cardiology"` | `"cancer research"[MeSH Terms]` | | `journal` | String | Filter publications by journal title or abbreviation. | *Empty* | `"N Engl J Med"` | | `meshTerm` | String | Filter by NLM Medical Subject Headings (MeSH) descriptor. | *Empty* | `"Cardiovascular Diseases"` | | `author` | String | Filter publications by author name. | *Empty* | `"Fauci AS"` | | `publicationType` | String | Filter by study design or article type. | *Empty* | `"Clinical Trial"` | | `publicationYear` | Integer | Filter publications by single calendar year. | *Empty* | `2025` | | `startYear` | Integer | Lower bound publication year for date range search. | *Empty* | `2020` | | `endYear` | Integer | Upper bound publication year for date range search. | *Empty* | `2026` | | `maxPapers` | Integer | Maximum paper records to extract (1 to 10,000). | `50` | `100` | #### Input Example ```json { "term": "cardiology", "meshTerm": "Cardiovascular Diseases", "publicationType": "Clinical Trial", "startYear": 2020, "endYear": 2026, "maxPapers": 100 } ``` *** ### ⬆️ Output Every extracted PubMed publication is saved to the run's default dataset. Results include canonical PMIDs (`paperId`), DOIs, complete abstracts, authors, MeSH headings, journal citations, and open-access PMC identifiers. | Field Name | Type | Description | Example | | :--- | :--- | :--- | :--- | | `paperId` | String | Unique PubMed Identifier (PMID) assigned by NCBI. | `"38123456"` | | `doi` | String / Null | Digital Object Identifier (DOI) assigned to the paper. | `"10.1038/s41586-024-07123-x"` | | `title` | String | Full scientific title of the research paper. | `"Structure and function of human cardiac receptors"` | | `abstract` | String / Null | Full text abstract of the publication. | `"We report cryo-EM structures of human cardiac..."` | | `journal` | String / Null | Abbreviated publishing journal name. | `"N Engl J Med"` | | `fullJournalName` | String / Null | Full untruncated name of the journal. | `"The New England journal of medicine"` | | `pubDate` | String / Null | Publication date string as formatted in PubMed. | `"2024 Mar 15"` | | `pubYear` | Integer / Null | Four-digit calendar publication year. | `2024` | | `authors` | Array\[String] | Array of indexed author names. | `["Smith JA", "Doe RC", "Johnson MB"]` | | `firstAuthor` | String / Null | Primary or first author listed. | `"Smith JA"` | | `lastAuthor` | String / Null | Senior or last author listed. | `"Johnson MB"` | | `meshTerms` | Array\[String] | NLM Medical Subject Headings (MeSH) descriptors. | `["Cardiovascular Diseases", "Heart Failure"]` | | `articleTypes` | Array\[String] | Article types assigned by PubMed indexers. | `["Journal Article", "Clinical Trial"]` | | `volume` | String / Null | Publishing journal volume number. | `"390"` | | `issue` | String / Null | Publishing journal issue number. | `"11"` | | `pages` | String / Null | Page range or article locator number. | `"1024-1035"` | | `pmcId` | String / Null | PubMed Central open-access repository ID. | `"PMC10891234"` | | `issn` | String / Null | International Standard Serial Number (ISSN). | `"0028-4793"` | | `language` | String / Null | Language code of the publication. | `"eng"` | | `pubmedUrl` | String | Direct canonical URL to the paper on PubMed. | `"https://pubmed.ncbi.nlm.nih.gov/38123456/"` | #### Sample Output Record ```json { "paperId": "38123456", "doi": "10.1038/s41586-024-07123-x", "title": "Structure and function of human cardiac receptors in heart failure", "abstract": "We report cryo-EM structures of human cardiac receptors in complex with native ligands...", "journal": "Nature", "fullJournalName": "Nature publishing group", "pubDate": "2024 Mar 15", "pubYear": 2024, "authors": ["Smith JA", "Doe RC", "Johnson MB"], "firstAuthor": "Smith JA", "lastAuthor": "Johnson MB", "meshTerms": ["Cardiovascular Diseases", "Receptors, Cell Surface", "Heart Failure"], "articleTypes": ["Journal Article", "Research Support, N.I.H."], "volume": "627", "issue": "8002", "pages": "180-188", "pmcId": "PMC10891234", "issn": "0028-0836", "language": "eng", "pubmedUrl": "https://pubmed.ncbi.nlm.nih.gov/38123456/" } ``` *** ### How it works The **PubMed Scraper** queries official NCBI Entrez E-utilities endpoints (`esearch.fcgi`, `esummary.fcgi`, and `efetch.fcgi`) over standard HTTPS. 1. **Search Indexing (`esearch.fcgi`)**: Constructs structured query strings combining terms, MeSH headings, journal titles, author names, publication types, and date ranges. Retrieves PMIDs matching the search criteria. 2. **Summary Metadata (`esummary.fcgi`)**: Fetches detailed JSON metadata records including titles, journal names, publication dates, volume/issue numbers, and author lists in batches. 3. **XML Abstracts & MeSH Parsing (`efetch.fcgi`)**: Fetches NLM XML payloads for each PMID batch to extract complete abstracts and Medical Subject Headings (MeSH descriptors). 4. **Push-As-You-Go Dataset Writing**: Records are pushed directly to the default dataset as they are parsed, ensuring zero data loss if a run is cancelled or hits budget caps. 5. **Rate Limit Resilience**: Enforces NCBI's rate limit of 3 requests per second and automatically executes exponential backoff on HTTP 429 or 5xx status codes. *** ### ❓ FAQ #### 1. How does Pay-Per-Event (PPE) pricing work? You are charged a flat rate of $0.002 per complete paper record written to the dataset. Tiered volume discounts (BRONZE through DIAMOND) apply automatically based on your Apify plan. Searches that return zero results, fail due to invalid syntax, or encounter rate limits incur **$0.00 zero charge**. #### 2. Can I filter by multiple conditions simultaneously? Yes! You can combine search terms, MeSH terms, journal names, author cohorts, publication types, and year ranges in a single run. The scraper merges these into an `AND` query string. #### 3. How do I extract full text articles? PubMed stores abstracts, citations, and metadata. When an article is open-access on PubMed Central, the `pmcId` field (e.g. `PMC10891234`) is populated so you can locate the full text on PMC. #### 4. What happens if PubMed rate limits the run? The scraper detects HTTP 429 rate limit responses and automatically pauses with exponential backoff before retrying, ensuring high reliability across long extraction runs. *** ### 💬 Your feedback Have a feature request, bug report, or need custom research extraction pipelines? Contact the maintainers at `reapxdev@proton.me`. *** *Unofficial - not affiliated with NCBI or PubMed. Collects public data only. reapx. Contact reapxdev@proton.me.* ### 🧪 Example input A real, runnable configuration — this is an actual input this Actor has run with. ```json { "meshTerm": "Neoplasms", "maxPapers": 50 } ``` ### 📄 Sample output One real row from a real run of this Actor, unedited. ```json { "paperId": "42545075", "doi": "10.1128/aac.00137-26", "title": "Penicillin-resistant Streptococcus dysgalactiae subsp. equisimilis and the association with penicillin-binding protein mutations: a population genotypic and phenotypic study", "abstract": "Streptococcus dysgalactiae subspecies equisimilis (SDSE) is increasingly recognized as a cause of human disease. Despite decades of penicillin use, SDSE is near universally penicillin-susceptible. Here we report and characterize by whole genome sequencing (WGS) three penicillin-resistant SDSE isolates (minimum inhibitory concentration [MIC] ≥0.256 mg/L) isolated from blood and skin in an individual with recurrent cellulitis on penicillin prophylaxis. The isolates carried 11 penicillin-binding protein (PBP) PBP2x, five PBP1a, and one PBP2b mutations that were distinct from the only previously reported WGS-characterized penicillin-resistant SDSE isolates (n = 4). We then took a genotype-led approach to test the MICs of penicillin and amoxicillin for 63 isolates with PBP2x, PBP1a, and PBP2b variants selected from a well-characterized database of invasive SDSE isolates from Australia (2011-2023, n = 436). PBPs were highly conserved in Australian invasive SDSE genomes. Only a single isolate carrying a PBP2x variant with five mutations around the penicillin-binding cleft found in 1/436 invasive isolates demonstrated a four-fold rise in penicillin MIC, although remaining below clinical breakpoints. Strains with PBP2b and PBP1a variants alone were not associated with raised MICs. To provide broader context for PBP variants, we then examined a global database of SDSE genomes (n = 1,160). PBP2x mutations in isolates with raised and/or resistant penicillin MICs were found in only nine global SDSE database genomes. SDSE penicillin resistance is rare and associated with multiple PBP mutations, particularly PBP2x. WGS with phenotypic testing of novel PBP variants could be used to monitor emerging mutations associated with raised penicillin MICs.", "journal": "Antimicrob Agents Chemother", "fullJournalName": "Antimicrobial agents and chemotherapy", "pubDate": "2026 Aug 3", "pubYear": 2026, "authors": [ "Xie O", "Tan A", "Nguyen TB", "Hayes AJ", "Dotel R", "Carter G", "O'Sullivan MVN", "Bond K", "McDevitt CA", "Korman TM", "Currie BJ", "Ascher DB", "Tong SYC", "van Hal SJ", "Davies MR" ], "firstAuthor": "Xie O", "lastAuthor": "Davies MR", "meshTerms": [], "articleTypes": [ "Journal Article" ], "volume": null, "issue": null, "pages": "e0013726", "pmcId": null, "issn": "0066-4804", "language": "eng", "pubmedUrl": "https://pubmed.ncbi.nlm.nih.gov/42545075/" } ``` ### ⚠️ Run outcomes and error handling This Actor reports what happened in the run's **status message**, and it always keeps whatever it collected. These are the outcomes you can get and what each one means. | Outcome | What it means | |---|---| | **Success** | Rows were returned and you were charged `record-returned` at $0.002 per row. | | **No matches** | The source returned nothing for your filters. **Nothing is charged.** Widen the date window or drop a filter. | #### What is guaranteed either way - **Every row is pushed as it is built**, not buffered to the end of the run. Anything that buffers output loses everything to a timeout, a block or a migration; this does not. - **A field absent from the source is absent from the row.** Nothing is inferred, modelled or filled in to make a row look complete. # Actor input Schema ## `term` (type: `string`): Search query string for PubMed using standard Entrez search syntax (e.g., cardiology, "cancer research"\[MeSH Terms], or Smith J\[Author]). Leave empty to query without term restriction.

The more specific the query term, the faster the extraction finishes and the lower the overall run cost. ## `journal` (type: `string`): Filter papers by publication journal name (e.g. The Lancet or N Engl J Med). Leave this field empty to search across all journals in PubMed.

Narrowing by journal focuses the scraper on specific academic titles. ## `meshTerm` (type: `string`): Filter PubMed literature by Medical Subject Headings (MeSH) descriptor (e.g. Cardiovascular Diseases or Neoplasms). Leave empty to disable MeSH term filtering.

MeSH terms provide standardized biomedical indexing across PubMed papers. ## `author` (type: `string`): Filter PubMed literature by author name (e.g. Fauci AS or Venter JC). Leave this field empty to extract papers from all authors.

Filtering by author restricts extracted publications to a specific researcher cohort. ## `publicationType` (type: `string`): Filter publications by article type (e.g., Clinical Trial, Review, Systematic Review, or Meta-Analysis). Leave empty to include all article types.

Selecting a specific publication type refines results to relevant study designs. ## `publicationYear` (type: `integer`): Filter publications by four-digit publication year (e.g. 2024 or 2025). Leave empty to search across all available publication years.

Specifying a publication year limits results to a single calendar year window. ## `startYear` (type: `integer`): Start year for publication date range filtering (e.g. 2020). Leave empty for no lower bound on publication date.

Setting a start year restricts papers to those published during or after that calendar year. ## `endYear` (type: `integer`): End year for publication date range filtering (e.g. 2026). Leave empty for no upper bound on publication date.

Setting an end year restricts papers to those published during or before that calendar year. ## `maxPapers` (type: `integer`): Maximum number of PubMed paper records to extract and save into the default dataset (1 to 10000). Prefilled with 50.

Higher limits result in longer execution times and higher total event charges.

The higher the number, the longer the run takes and the more you are charged, because you pay per row returned. ## Actor input object example ```json { "term": "cardiology", "maxPapers": 50 } ``` # Actor output Schema ## `results` (type: `string`): Every PubMed paper record extracted by this run, one row per paper, in the default dataset. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "term": "cardiology", "maxPapers": 50 }; // Run the Actor and wait for it to finish const run = await client.actor("reapx/pubmed-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "term": "cardiology", "maxPapers": 50, } # Run the Actor and wait for it to finish run = client.actor("reapx/pubmed-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "term": "cardiology", "maxPapers": 50 }' | apify call reapx/pubmed-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,reapx/pubmed-scraper" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/4Ca5K7br7KsjM2ihD/builds/aoqrJ56F0OCDIlw2W/openapi.json