# PubMed Biomedical Paper Scraper (`brilliant_gum/pubmed-scraper`) Actor Scrapes PubMed biomedical papers using the official NCBI Entrez API. Extracts full metadata including abstracts, MeSH terms, authors with affiliations, citations, grants, and more. Includes smart analytics for author networks, topic trends, and geographic distribution. - **URL**: https://apify.com/brilliant\_gum/pubmed-scraper.md - **Developed by:** [Yuliia Kulakova](https://apify.com/brilliant_gum) (community) - **Categories:** Other - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, NaN bookmarks - **User rating**: No ratings yet ## Pricing from $30.00 / 1,000 results This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## PubMed Scraper — Biomedical Research, Abstracts & Citations [![PubMed Scraper](https://i.imgur.com/SpD0gOw.png)](https://apify.com/store) Search and extract biomedical research papers from PubMed — the world's largest database of life sciences and biomedical literature. Get full metadata, abstracts, MeSH terms, citations, author networks, and smart analytics. Powered by the official NCBI Entrez API. ### What can this scraper do? - **Search by any query** — use simple keywords or advanced PubMed syntax with MeSH terms and field tags - **Full metadata extraction** — authors with affiliations and ORCID, journal info, DOI, publication types, grants, chemicals - **Abstracts** — complete structured abstracts for every article - **MeSH terms** — Medical Subject Headings with major topic indicators and qualifiers - **Citation data** — cited-by counts and reference PMIDs - **Smart analytics** — author networks, topic trends by year, top journals, funding agencies, geographic distribution - **Multiple databases** — PubMed, PubMed Central (PMC), Gene, Protein - **No API key required** — works out of the box, optional API key for faster rate limits ### Why use PubMed Scraper? | Use case | How it helps | |----------|-------------| | Literature review | Collect hundreds of papers on any topic with full abstracts and metadata | | Competitive intelligence | Track research output from specific institutions or authors | | Pharma & biotech research | Monitor clinical trials, drug targets, and therapeutic areas | | Grant landscape analysis | Identify top funding agencies and grant patterns in your field | | Bibliometric analysis | Analyze publication trends, co-authorship networks, and journal distributions | | Academic research | Build structured datasets for systematic reviews and meta-analyses | | Trend monitoring | Track emerging topics through MeSH term trends over time | ### Output data #### Article data Each article includes all available metadata: | Field | Description | |-------|-------------| | PMID & PMCID | PubMed and PubMed Central identifiers | | Title | Article title | | Authors | Full list with last name, first name, affiliations, and ORCID | | Abstract | Complete abstract text | | Journal | Name, abbreviation, ISSN, volume, issue, pages | | Publication date | ISO format date | | DOI | Digital Object Identifier | | Publication types | Journal Article, Review, Clinical Trial, Meta-Analysis, etc. | | MeSH terms | Descriptors with qualifiers and major topic flags | | Keywords | Author-provided keywords | | Grant info | Grant IDs, agencies, and countries | | Chemicals | Registry numbers and substance names | | Full text URL | PMC link for open access articles | | Cited-by count | Number of citing articles (when enabled) | | References | First 10 reference PMIDs (when enabled) | | Language & country | Publication language and journal country | | Clinical trial ID | Associated trial registry number | | Retraction status | Whether the article has been retracted | | Conflict of interest | Author disclosures | #### Smart analytics Every run can include an automatic analytics summary: | Analytics | What you get | |-----------|-------------| | Top authors | Most prolific authors with collaborator counts | | Author networks | Co-authorship connections and top collaborators | | Topic trends | MeSH major topics by year | | Journal distribution | Top journals by article count | | Funding agencies | Top grant sources | | Geographic distribution | Articles by country | | Publication types | Breakdown by article type | | Open access rate | Percentage of open access articles | | Publications by year | Yearly publication volume | ### Filters & options | Filter | Options | |--------|---------| | Date range | From/to dates (YYYY/MM/DD format) | | Sort by | Relevance, Publication date, Most cited | | Species | All, Human only, Animal only | | Language | Filter by publication language (eng, fre, de, etc.) | | Publication type | Journal Article, Review, Clinical Trial, Meta-Analysis, etc. | | Database | PubMed, PMC, Gene, Protein | | Include abstract | Full abstract text (on by default) | | Include MeSH | Medical Subject Headings (on by default) | | Include citations | Cited-by counts and references | | Include full text links | PMC open access URLs | | Include analytics | Smart analytics summary | ### How much does it cost? PubMed Scraper uses a **pay-per-result** model at **$0.03 per result**. No monthly fees — you only pay for the data you collect. | Results | Cost | |---------|------| | 100 articles | $3.00 | | 500 articles | $15.00 | | 1,000 articles | $30.00 | | 10,000 articles | $300.00 | Platform costs are included in the per-result price. ### Input examples #### Basic search ```json { "queries": ["breast cancer immunotherapy"], "maxResults": 100, "includeAbstract": true, "includeMeshTerms": true, "includeAnalytics": true } ```` #### Advanced PubMed syntax with date filter ```json { "queries": ["COVID-19[MeSH] AND vaccine[tiab]"], "maxResults": 200, "dateFrom": "2024/01/01", "dateTo": "2026/05/18", "sortBy": "pub_date", "species": "human", "language": ["eng"] } ``` #### Multi-query research with citations ```json { "queries": ["CRISPR gene therapy", "base editing clinical trials"], "maxResults": 50, "includeCitations": true, "includeFullTextLinks": true, "includeAnalytics": true } ``` #### Systematic review — specific publication types ```json { "queries": ["diabetes type 2 metformin"], "maxResults": 500, "pubTypes": ["Clinical Trial", "Randomized Controlled Trial"], "species": "human", "sortBy": "cited_by_count", "dateFrom": "2020/01/01" } ``` #### PubMed Central full text search ```json { "queries": ["machine learning drug discovery"], "database": "pmc", "maxResults": 100, "includeFullTextLinks": true, "includeAnalytics": true } ``` ### Tips for best results 1. **Start small** — test with 10-20 results to verify your query before scaling up 2. **Use PubMed syntax** — field tags like `[MeSH]`, `[tiab]`, `[au]` give precise control over search results 3. **API key** — get a free NCBI API key to increase rate limits from 3 to 10 requests/second 4. **Citations wisely** — enable `includeCitations` only when needed, it requires extra API calls 5. **Species filter** — use `human` for clinical research, `animal` for preclinical studies 6. **Sort by citations** — `cited_by_count` helps find the most impactful papers in your field 7. **Combine queries** — use multiple queries to cover different aspects of your research topic ### Limitations - PubMed API returns up to 9,999 results per query (NCBI hard cap). For larger datasets, split into narrower date windows - Citation counts are approximate and may not reflect the latest data - Full text is only available for open access articles in PMC - Rate limits: 3 requests/second without API key, 10 requests/second with key - Analytics summary is included as the last record in the dataset (field `type: "analytics"`) # Actor input Schema ## `queries` (type: `array`): Array of PubMed search terms. Supports full PubMed query syntax (e.g., 'cancer immunotherapy', 'COVID-19\[MeSH] AND vaccine\[tiab]') ## `ncbiApiKey` (type: `string`): Optional NCBI API key. Without key: 3 requests/sec limit. With key: 10 requests/sec limit. Get yours at: https://www.ncbi.nlm.nih.gov/account/ ## `database` (type: `string`): Which NCBI database to search ## `maxResults` (type: `integer`): Maximum number of papers to retrieve per query ## `dateFrom` (type: `string`): Filter papers published from this date (format: YYYY/MM/DD) ## `dateTo` (type: `string`): Filter papers published up to this date (format: YYYY/MM/DD) ## `sortBy` (type: `string`): How to sort search results ## `pubTypes` (type: `array`): Filter by publication type. Leave empty to include all types. ## `species` (type: `string`): Filter by study species ## `language` (type: `array`): Filter by publication language (ISO 639-1 codes, e.g. \['eng', 'fre']). Leave empty for all languages. ## `includeAbstract` (type: `boolean`): Include the full abstract text in results (structured if available) ## `includeMeshTerms` (type: `boolean`): Include MeSH (Medical Subject Headings) terms with major topic indicators ## `includeFullTextLinks` (type: `boolean`): Include PMC full text URL when available (open access articles) ## `includeCitations` (type: `boolean`): Include approximate cited-by count and first 10 reference PMIDs (uses additional API calls) ## `includeAnalytics` (type: `boolean`): Generate analytics summary: author networks, topic trends, journal distribution, funding analysis, geographic distribution ## `batchSize` (type: `integer`): Number of records to fetch per API call (max 200 for efetch). Higher values are more efficient. ## `requestDelayMs` (type: `integer`): Delay between API requests in milliseconds. Auto-configured based on API key presence (333ms without key, 100ms with key). Override here if needed. ## Actor input object example ```json { "queries": [ "breast cancer immunotherapy", "CRISPR gene editing" ], "database": "pubmed", "maxResults": 100, "dateFrom": "2020/01/01", "dateTo": "2024/12/31", "sortBy": "relevance", "pubTypes": [ "Journal Article", "Review" ], "species": "all", "language": [ "eng" ], "includeAbstract": true, "includeMeshTerms": true, "includeFullTextLinks": true, "includeCitations": false, "includeAnalytics": true, "batchSize": 200 } ``` # Actor output Schema ## `results` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "queries": [ "breast cancer immunotherapy" ] }; // Run the Actor and wait for it to finish const run = await client.actor("brilliant_gum/pubmed-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "queries": ["breast cancer immunotherapy"] } # Run the Actor and wait for it to finish run = client.actor("brilliant_gum/pubmed-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "queries": [ "breast cancer immunotherapy" ] }' | apify call brilliant_gum/pubmed-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=brilliant_gum/pubmed-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "PubMed Biomedical Paper Scraper", "description": "Scrapes PubMed biomedical papers using the official NCBI Entrez API. Extracts full metadata including abstracts, MeSH terms, authors with affiliations, citations, grants, and more. Includes smart analytics for author networks, topic trends, and geographic distribution.", "version": "1.0", "x-build-id": "aGx4YmP0SOG4QXYmX" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/brilliant_gum~pubmed-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-brilliant_gum-pubmed-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/brilliant_gum~pubmed-scraper/runs": { "post": { "operationId": "runs-sync-brilliant_gum-pubmed-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/brilliant_gum~pubmed-scraper/run-sync": { "post": { "operationId": "run-sync-brilliant_gum-pubmed-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "queries" ], "properties": { "queries": { "title": "Search Queries", "minItems": 1, "type": "array", "description": "Array of PubMed search terms. Supports full PubMed query syntax (e.g., 'cancer immunotherapy', 'COVID-19[MeSH] AND vaccine[tiab]')", "items": { "type": "string" } }, "ncbiApiKey": { "title": "NCBI API Key", "type": "string", "description": "Optional NCBI API key. Without key: 3 requests/sec limit. With key: 10 requests/sec limit. Get yours at: https://www.ncbi.nlm.nih.gov/account/" }, "database": { "title": "NCBI Database", "enum": [ "pubmed", "pmc", "gene", "protein" ], "type": "string", "description": "Which NCBI database to search", "default": "pubmed" }, "maxResults": { "title": "Maximum Results", "minimum": 1, "maximum": 10000, "type": "integer", "description": "Maximum number of papers to retrieve per query", "default": 100 }, "dateFrom": { "title": "Date From", "pattern": "^\\d{4}/\\d{2}/\\d{2}$", "type": "string", "description": "Filter papers published from this date (format: YYYY/MM/DD)" }, "dateTo": { "title": "Date To", "pattern": "^\\d{4}/\\d{2}/\\d{2}$", "type": "string", "description": "Filter papers published up to this date (format: YYYY/MM/DD)" }, "sortBy": { "title": "Sort Order", "enum": [ "relevance", "pub_date", "cited_by_count" ], "type": "string", "description": "How to sort search results", "default": "relevance" }, "pubTypes": { "title": "Publication Types Filter", "type": "array", "description": "Filter by publication type. Leave empty to include all types.", "items": { "type": "string" } }, "species": { "title": "Species Filter", "enum": [ "all", "human", "animal" ], "type": "string", "description": "Filter by study species", "default": "all" }, "language": { "title": "Language Filter", "type": "array", "description": "Filter by publication language (ISO 639-1 codes, e.g. ['eng', 'fre']). Leave empty for all languages.", "items": { "type": "string" } }, "includeAbstract": { "title": "Include Abstract", "type": "boolean", "description": "Include the full abstract text in results (structured if available)", "default": true }, "includeMeshTerms": { "title": "Include MeSH Terms", "type": "boolean", "description": "Include MeSH (Medical Subject Headings) terms with major topic indicators", "default": true }, "includeFullTextLinks": { "title": "Include Full Text Links", "type": "boolean", "description": "Include PMC full text URL when available (open access articles)", "default": true }, "includeCitations": { "title": "Include Citations", "type": "boolean", "description": "Include approximate cited-by count and first 10 reference PMIDs (uses additional API calls)", "default": false }, "includeAnalytics": { "title": "Include Smart Analytics", "type": "boolean", "description": "Generate analytics summary: author networks, topic trends, journal distribution, funding analysis, geographic distribution", "default": true }, "batchSize": { "title": "Batch Size", "minimum": 10, "maximum": 200, "type": "integer", "description": "Number of records to fetch per API call (max 200 for efetch). Higher values are more efficient.", "default": 200 }, "requestDelayMs": { "title": "Request Delay (ms)", "minimum": 50, "type": "integer", "description": "Delay between API requests in milliseconds. Auto-configured based on API key presence (333ms without key, 100ms with key). Override here if needed." } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```