PubMed Search Scraper

What does PubMed Search Scraper do?

PubMed Search Scraper collects biomedical literature metadata from NCBI PubMed using only public endpoints.
You can search by keyword, constrain by date range, apply publication-type filters, and download a structured dataset with article titles, authors, journal information, abstracts, DOI, keywords, and MeSH terms.

Only normalized articles are written to the dataset. Empty searches and bounded request failures are represented by the terminal status in the fixed run-summary key-value record; detailed failure signals remain in run logs rather than dataset or summary rows.

This Actor is built for API-first, deterministic extraction:

• it uses PubMed E-utilities (esearch, esummary, efetch)
• runs with bounded retries and backoff
• validates HTTP status, content type, payload size, and JSON/XML shape before parsing
• filters and normalizes outputs
• deduplicates article IDs
• returns stable fields with real URLs, IDs, and timestamps

Live page comparison

The public Chrome search page at https://pubmed.ncbi.nlm.nih.gov/?term=machine%20learning was checked on 2026-09-23. It showed 281,808 results, page 1 of 28,181, ten result cards per page, and Best match sorting. The sort menu offered Most recent, Publication date, First author, and Journal.

Visible result cards included the title and PMID link, author line, journal citation, DOI when present, electronic-publication date on some records, article type, Free article / Free PMC article labels, and an abstract excerpt. The Actor's existing ESearch/ESummary/EFetch flow already retrieves these bibliographic details and additional public record metadata. Its dataset now preserves ESummary availability labels and EFetch detail such as abstract sections, language, journal ISSNs, author affiliations and identifiers, grants, database accessions, comments/corrections, and PubMed indexing dates when NCBI supplies them.

No Actor or proxy run was made for this comparison. The page observation describes the visible site; it is not an Actor-produced dataset row. Optional properties are omitted when the NCBI payload has no value. The free-article booleans report PubMed's ESummary labels and do not imply that full text is available for every reader or at every URL.

Why use PubMed Search Scraper?

It is useful for:

• literature review pipelines
• research intelligence and grant preparation
• data collection for bibliographic analysis
• tracking publication trends for a query
• rapid extraction of titles + abstracts for downstream NLP

How to use PubMed Search Scraper

1. Open the actor and paste input in the Input tab.
2. Set a query, for example machine learning.
3. Set limits (maxResults, pageSize, maxPages) to control runtime.
4. Optionally set dateFrom, dateTo, and articleType.
5. Enable/disable optional enrichment fields.
6. Run the actor locally with apify run --purge or on cloud.

Example input:

{
  "query": "machine learning",
  "maxResults": 20,
  "pageSize": 25,
  "maxPages": 4,
  "sortBy": "pub_date",
  "dateFrom": "2025-01-01",
  "dateTo": "2026-12-31",
  "articleType": "Journal Article, Randomized Controlled Trial",
  "includeAbstract": true,
  "includeAuthors": true,
  "includeMeSH": true,
  "includeKeywords": true,
  "requestDelayMs": 250,
  "maxConcurrent": 3,
  "maxRequestRetries": 2,
  "requestTimeoutSecs": 25
}

The complete input contract, including API options, defaults, and bounds, is in .actor/input_schema.json.

Output

Each output item is a normalized article object with a stable identity. The dataset schema declares 42 top-level article fields; sparse fields are omitted when NCBI has no corresponding value.

The dataset and run-summary schemas are .actor/dataset_schema.json, .actor/key_value_store_schema.json, and .actor/output_schema.json.

{
  "recordType": "article",
  "pmid": "12345678",
  "doi": "10.1000/example-doi",
  "title": "Example biomedical article title",
  "journal": "Example Journal of Medicine",
  "publicationDate": "2026-01-15T00:00:00.000Z",
  "publicationDateText": "2026 Jan 15",
  "electronicPublicationDate": "2026-01-14T00:00:00.000Z",
  "electronicPublicationDateText": "2026 Jan 14",
  "publicationStatus": "ppublish",
  "journalAbbreviation": "Example J",
  "issns": [
    { "type": "Electronic", "value": "1234-5678" }
  ],
  "languages": ["eng"],
  "authors": [
    {
      "name": "Alex Example",
      "initials": "A.E.",
      "foreName": "Alex",
      "lastName": "Example",
      "affiliations": ["Example University"]
    }
  ],
  "abstract": "This example abstract summarizes a biomedical research finding.",
  "abstractSections": [
    { "label": "BACKGROUND", "category": "BACKGROUND", "text": "Example abstract section." }
  ],
  "meshTerms": ["Machine Learning", "Biomedical Research"],
  "keywords": ["machine learning"],
  "articleTypes": ["Journal Article"],
  "hasAbstract": true,
  "availabilityLabels": ["Has Abstract", "Free PMC Article"],
  "hasFreeArticleLabel": false,
  "hasFreePmcArticleLabel": true,
  "pmcId": "PMC1234567",
  "additionalArticleIds": [
    { "type": "pii", "value": "S123456789" }
  ],
  "sourceUrl": "https://pubmed.ncbi.nlm.nih.gov/12345678/",
  "citation": "Example A. Example J 2026 Jan 15;8(2):15-24. doi: 10.1000/example-doi. Epub 2026 Jan 14. PMID: 12345678.",
  "scrapedAt": "2026-01-15T12:00:00.000Z"
}

The JSON above illustrates the output shape; it is not a record from an Actor run. Funding, linked data accessions, comments/corrections, author identifiers, author-list completeness, and PubMed creation/completion/revision dates are included when present.

If the public endpoint returns no matches, the dataset is empty; the summary's status fields are:

{
  "recordType": "pubmed-run-summary",
  "recordsStored": 0,
  "runStatus": "NO_RESULTS"
}

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel directly from Apify. The fixed run-summary key records the requested query, generated PubMed search term, sort/date/type filters, exact record count, and final status. Request diagnostics remain in the run logs.

Data table

Field	Description
recordType	Stable article record kind
pmid, sourceUrl	PubMed identifier and canonical article page
title, doi, citation	Article title, DOI when supplied, and formatted citation
journal, journalAbbreviation, issns	Full and abbreviated journal names and print/electronic ISSNs
publicationDate, publicationDateText	Parsed journal publication date and source display text
electronicPublicationDate, electronicPublicationDateText	Parsed electronic date and source display text when available
publicationStatus, medlineStatus	PubMed publication status and MEDLINE citation status
volume, issue, journalIssue, pages	Journal issue and pagination details
authors	Author names, initials, affiliations, identifiers, and collective names when supplied
authorListComplete	PubMed's author-list completeness indicator when supplied
languages	Article language codes
abstract, abstractSections, hasAbstract	Combined abstract, labeled sections, and abstract availability indicator
articleTypes	PubMed publication types, such as Review or Clinical Trial
availabilityLabels, hasFreeArticleLabel, hasFreePmcArticleLabel	ESummary labels and flags for the Free article / Free PMC article labels
pmcId, additionalArticleIds	PubMed Central ID and other external article identifiers
keywords, meshTerms	Author keywords and MeSH descriptors/qualifiers
grants	Funding agency, grant identifier, acronym, and country when present
dataBankAccessions	Article-linked database names and accession numbers
commentsCorrections	Linked comments or corrections with reference type, PMID, source, and note
dateCreated, dateCompleted, dateRevised	PubMed record creation, completion, and revision dates when supplied
scrapedAt	Actor record-normalization timestamp

Pricing / cost estimation

This actor uses public NCBI endpoints and is ideal for small-to-medium literature sweeps. If your run is large, reduce maxResults / increase delays to reduce request pressure. For high-volume use, NCBI API keys can increase request limits and reduce throttling risk.

Tips and advanced options

• Use a narrow query and short date window to keep results manageable.
• Keep maxConcurrent lower if you encounter transient API throttling.
• Use articleType to narrow to specific publication types.
• Keep requestDelayMs above 100 ms when running frequently.
• For reproducibility, log each input used and retain the resulting run-summary record.

FAQ, disclaimers, and support

• PubMed content and availability can change over time; re-run for fresh data.
• publicationDate and abstracts are best-effort normalized from API payloads.
• This actor uses publicly documented NCBI endpoints and avoids bypassing paywalls or private controls.
• For custom feature requests, use the actor issue/reporting channel.