NCBI E-utilities Scraper (PubMed, Gene, Protein, Nucleotide)

Search NCBI's E-utilities (Entrez) across PubMed, Gene, Protein and Nucleotide and get clean, structured JSON in one run. The actor runs the standard Entrez workflow for you — esearch (find matching UIDs) → esummary (fetch records) — and maps every database's quirky per-record shape into flat, predictable fields.

No API key required. NCBI allows 3 requests/second without a key and 10/second with a free key, which you can supply via the optional apiKey input.

Cost: roughly ~$0.5 per 1,000 records on typical runs (a search + summary pair per batch of 50 records). Public NCBI data, no proxies needed.

Use cases

• Literature mining — pull thousands of PubMed articles (title, authors, journal, DOI) for a topic.
• Biomedical NLP — build corpora of titles and article metadata for training and entity extraction.
• Gene research — resolve gene symbols to organism, chromosome, map location and aliases.
• Systematic reviews — reproducible, query-driven exports of the exact article set behind a search.

Modes (databases)

Pick one mode (the db driver). Each mode has its own db-specific output mapping:

mode	What it searches	Key output fields
pubmed	Biomedical articles	pmid, title, authors, journal, doi, pub_date
gene	NCBI Gene	gene_id, symbol, organism, chromosome, map_location, aliases, summary
protein	Protein sequences	uid, accession, title, length, organism
nucleotide	Nucleotide sequences	uid, accession, title, length, organism

Input

{
    "mode": "pubmed",
    "queries": ["CRISPR gene editing", "BRCA1"],
    "apiKey": "",
    "maxItemsPerQuery": 50,
    "maxItems": 200
}

Field	Type	Default	Description
mode	select	pubmed	Database / code path: pubmed, gene, protein, nucleotide.
queries	array	["CRISPR gene editing","BRCA1"]	Entrez search terms. Full Entrez syntax works, e.g. BRCA1[gene] AND human[orgn].
apiKey	string	—	Optional NCBI key. Raises the rate limit from 3 → 10 requests/second.
maxItemsPerQuery	integer	50	Max records per query.
maxItems	integer	200 (max 5000)	Total record cap across all queries.

Output examples

PubMed article:

{
    "type": "pubmed_article",
    "pmid": "42574251",
    "title": "Mucolipidosis II: novel variants, clinical variation...",
    "authors": ["Eissa NR", "Wilson MP"],
    "author_count": 7,
    "journal": "Human molecular genetics",
    "pub_date": "2026-08-10",
    "pub_date_raw": "2026 Aug 10",
    "doi": "10.1093/hmg/ddag068",
    "pmc_ref_count": null,
    "url": "https://pubmed.ncbi.nlm.nih.gov/42574251/",
    "source_api": "ncbi-eutils"
}

Gene:

{
    "type": "gene",
    "gene_id": "672",
    "symbol": "BRCA1",
    "organism": "Homo sapiens",
    "tax_id": 9606,
    "chromosome": "17",
    "map_location": "17q21.31",
    "aliases": ["BRCAI", "BRCC1", "BROVCA1", "PNCA4"],
    "url": "https://www.ncbi.nlm.nih.gov/gene/672",
    "source_api": "ncbi-eutils"
}

Rate limits & the free API key

NCBI E-utilities usage policy:

• Without a key: up to 3 requests per second per IP.
• With a free key: up to 10 requests per second.

The actor paces itself automatically to stay under whichever limit applies, and on an HTTP 429 it stops with a distinct error message telling you to slow down or add a key. Get a free key from your NCBI account: Settings → API Key Management (https://www.ncbi.nlm.nih.gov/account/). E-utilities also asks callers to identify themselves with tool and email parameters — the actor sends these automatically.

How this differs from our other research scrapers

This is our only NCBI E-utilities / Entrez actor. It is not a substitute for these:

• Semantic Scholar Scraper — cross-publisher academic graph (citations, references, influential-citation counts, TLDRs) across all fields of science. NCBI E-utilities is PubMed/NIH-centric (biomedicine + the linked Gene/Protein/Nucleotide sequence databases) and additionally returns biological entities, not just papers.
• Unpaywall Scraper — finds legal open-access full-text PDFs for a DOI. Use E-utilities to discover PubMed articles + their DOIs, then Unpaywall to fetch the OA copy. NCBI does not host third-party PDFs.
• UniProt Scraper — curated protein function / annotation knowledgebase. NCBI's Protein db here is sequence-record centric (accession, length, organism), complementary to UniProt's functional annotation.
• Clinical Trials Scraper — ClinicalTrials.gov study registry, a different corpus entirely.

Related actors: Semantic Scholar · Unpaywall · UniProt · Clinical Trials.

Data source

All data comes from the public NCBI Entrez Programming Utilities (https://eutils.ncbi.nlm.nih.gov/entrez/eutils). Please respect NCBI's usage policies.