PubMed Search Scraper

What does PubMed Search Scraper do?

PubMed Search Scraper collects biomedical literature metadata from NCBI PubMed using only public endpoints.
You can search by keyword, constrain by date range, apply publication-type filters, and download a structured dataset with article titles, authors, journal information, abstracts, DOI, keywords, and MeSH terms.

Only normalized articles are written to the dataset. Empty searches and bounded request failures are recorded in the fixed run-summary key-value record, so API consumers never receive diagnostic placeholders as article rows.

This Actor is built for API-first, deterministic extraction:

• it uses PubMed E-utilities (esearch, esummary, efetch)
• runs with bounded retries and backoff
• validates HTTP status, content type, payload size, and JSON/XML shape before parsing
• filters and normalizes outputs
• deduplicates article IDs
• returns stable fields with real URLs, IDs, and timestamps

Why use PubMed Search Scraper?

It is useful for:

• literature review pipelines
• research intelligence and grant preparation
• data collection for bibliographic analysis
• tracking publication trends for a query
• rapid extraction of titles + abstracts for downstream NLP

How to use PubMed Search Scraper

1. Open the actor and paste input in the Input tab.
2. Set a query, for example machine learning.
3. Set limits (maxResults, pageSize, maxPages) to control runtime.
4. Optionally set dateFrom, dateTo, and articleType.
5. Enable/disable optional enrichment fields.
6. Run the actor locally with apify run --purge or on cloud.

Example input:

{
  "query": "machine learning",
  "maxResults": 20,
  "pageSize": 25,
  "maxPages": 4,
  "sortBy": "pub_date",
  "dateFrom": "2025-01-01",
  "dateTo": "2026-12-31",
  "articleType": "Journal Article, Randomized Controlled Trial",
  "includeAbstract": true,
  "includeAuthors": true,
  "includeMeSH": true,
  "includeKeywords": true,
  "requestDelayMs": 250,
  "maxConcurrent": 3,
  "maxRequestRetries": 2,
  "requestTimeoutSecs": 25
}

Output

Each output item is a normalized article object with a stable identity.

{
  "recordType": "article",
  "found": true,
  "dataAvailable": true,
  "pmid": "12345678",
  "doi": "10.1000/example-doi",
  "title": "Example biomedical article title",
  "journal": "Example Journal of Medicine",
  "publicationDate": "2026-01-15T00:00:00.000Z",
  "authors": [
    { "name": "Alex Example", "initials": "A.E." }
  ],
  "abstract": "This example abstract summarizes a biomedical research finding.",
  "meshTerms": ["Machine Learning", "Biomedical Research"],
  "keywords": ["machine learning"],
  "sourceUrl": "https://pubmed.ncbi.nlm.nih.gov/12345678/",
  "scrapedAt": "2026-01-15T12:00:00.000Z"
}

If the public endpoint returns no matches, the dataset is empty and the separate summary looks like this:

{
  "recordType": "pubmed-run-summary",
  "recordsStored": 0,
  "noResults": true,
  "runStatus": "NO_RESULTS",
  "errorCode": "no_matches"
}

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel directly from Apify. The fixed run-summary key records the requested query, generated PubMed search term, sort/date/type filters, pages fetched, exact record count, and final status.

Data table

Field	Description
pmid	PubMed Identifier used as a stable identity
doi	DOI when available
title	Article title
journal	Journal title
publicationDate	Parsed publication date
authors	Normalized author objects
abstract	Short text excerpt of abstract
meshTerms	MeSH descriptors
keywords	Author keywords
sourceUrl	PubMed canonical URL
scrapedAt	Run timestamp

Pricing / cost estimation

This actor uses public NCBI endpoints and is ideal for small-to-medium literature sweeps. If your run is large, reduce maxResults / increase delays to reduce request pressure. For high-volume use, NCBI API keys can increase request limits and reduce throttling risk.

Tips and advanced options

• Use a narrow query and short date window to keep results manageable.
• Keep maxConcurrent lower if you encounter transient API throttling.
• Use articleType to narrow to specific publication types.
• Keep requestDelayMs above 100 ms when running frequently.
• For reproducibility, log each input used and persist the resulting run-summary record.

FAQ, disclaimers, and support

• PubMed content and availability can change over time; re-run for fresh data.
• publicationDate and abstracts are best-effort normalized from API payloads.
• This actor uses publicly documented NCBI endpoints and avoids bypassing paywalls or private controls.
• For custom feature requests, use the actor issue/reporting channel.