NCBI dbSNP Variant Scraper

NCBI dbSNP Variant Scraper

Scrape variant summaries from NCBI dbSNP by gene, rsID, or any search term, up to a million per run. Each variant comes with its clinical significance, allele frequencies, gene associations, and mapped citations. No API key required. Export to CSV, JSON, Excel, or XML.

NCBI's dbSNP is the central public archive for human genetic variation, but querying it programmatically through Entrez E-utilities means handling XML parsing, rate limits, and pagination yourself. This Actor reads the Esummary endpoint directly, accepts a gene symbol, rs identifier, or free-text search, and returns each matching variant as a clean, flat row. It is built for researchers and bioinformaticians who need variant metadata in bulk without writing a single line of NCBI API code.

Who uses it	What they scrape NCBI dbSNP for
Clinical geneticists	Pull the clinical significance and review status for a list of rsIDs before a variant interpretation session.
Bioinformatics analysts	Gather allele frequencies and gene mappings for every variant in a candidate gene panel.
Population geneticists	Collect global minor allele frequencies across dbSNP populations for a set of ancestry-informative markers.
Pharma R&D teams	Extract variant-to-disease associations and PubMed citations for a target gene to prioritize lead SNPs.

What it does

This Actor collects variant summary records from NCBI dbSNP for a given search term and returns each variant as one flat row.

• 🔍 Gene or rsID search: pass a gene symbol like BRCA1 or an rs number like rs328 and get every matching variant summary.
• 📊 Flat row output: each variant is one row with allele frequencies, clinical significance, gene names, and citation IDs, ready for pandas or Excel.
• ⚙️ Configurable volume: set a hard cap from 1 to 1,000,000 variants per run so you control the size of the output dataset.
• 📎 Multi-format export: download your results as CSV, JSON, Excel, or XML directly from the Apify dataset.

Results export to CSV, JSON, Excel, or XML, or straight from the API.

What you can do with NCBI dbSNP data

🧬 Build a gene-level variant catalog.

A clinical lab analyst searches for BRCA1, collects all variant summaries, and filters by clinical significance to identify pathogenic SNPs for a test report.

📈 Compare allele frequencies across populations.

A population geneticist scrapes a set of rsIDs, extracts global minor allele frequencies, and loads them into R for a PCA of ancestry components.

📚 Gather evidence for variant interpretation.

A variant scientist pulls dbSNP summaries for a list of candidate SNPs, maps the linked PubMed IDs, and reviews the publications supporting each clinical assertion.

🔄 Sync a local variant database.

A bioinformatics engineer runs the Actor weekly with a gene list, exports the JSON, and upserts new or updated variant records into an internal knowledge base.

Why choose this scraper

	What you get
No NCBI API key	The Actor calls the public Esummary endpoint so you never register an NCBI account or manage an API key.
Fixed schema	Every variant returns the same fields, making it safe to merge runs from different genes or searches.
Bulk retrieval	Collect up to a million variant summaries in one run instead of clicking through pages of the dbSNP web interface.
Citation mapping	Each record includes linked PubMed IDs so you can trace the evidence behind a variant annotation.

How it compares

No other Store actor targets NCBI dbSNP the same way, so the honest comparison is with the alternatives teams actually weigh.

	NCBI dbSNP Variant Scraper	Build it in-house	By hand
Setup	Run it now, zero config	Days of engineering	None, but hours per pull
When NCBI dbSNP changes	Maintained for you	You fix it	You re-learn the page
Proxies, retries, anti-bot	Built in	Your problem	Browser only
Output	Fixed JSON schema, CSV/Excel export	Whatever you build	Copy-paste
Cost	Pay per result	Engineering time	Analyst hours

Configure the run

Drive the Actor with a single search term, a gene symbol, or an rs identifier, and set a maximum number of variants to cap the run. The Input tab lists every parameter.

A first run with the defaults:

{
  "maxItems": 10,
  "searchTerm": "BRCA1"
}

A larger pull:

{
  "maxItems": 200,
  "searchTerm": "BRCA1"
}

Pricing

Pay-per-result: $0.0085 per result collected. You pay only for the results written to your dataset.

Results collected	Approximate cost
100 results	$0.85
1,000 results	$8.50
10,000 results	$85.00

New Apify accounts start with $5 in free credit.

Free users

Free-plan runs return up to 10 results as a preview. Upgrade your Apify plan to collect up to 1,000,000 results per run.

Run it

1. Create a free Apify account with $5 in credit.
2. Open the NCBI dbSNP Variant Scraper.
3. Set your inputs and any filters, then click Start.
4. Export the results as CSV, Excel, JSON, or XML from the Dataset tab.

Run it programmatically through the Apify API (run-sync-get-dataset-items) or the ApifyClient for JavaScript and Python.

Use with AI agents (MCP)

Give an AI agent live access to NCBI dbSNP through the Model Context Protocol. Add the Actor to Claude, Cursor, or any MCP client:

$claude mcp add --transport http apify "https://mcp.apify.com?tools=parseforge/dbsnp-esummary-scraper"

Then prompt it in plain language to run the scraper and read back the results.

Troubleshooting

Why am I getting no results?

Your search term may not match any variant in dbSNP. Test the exact same term on the NCBI dbSNP website. Also check that you are using a valid gene symbol or rsID. If the term is valid but still returns nothing, try a broader search.

The run stopped before collecting all variants I expected.

Check the 'Maximum variants' setting. The Actor stops when it reaches that number, even if more variants match your search. Increase the maximum and rerun.

I got an error about rate limiting.

NCBI imposes rate limits on the Esummary endpoint. If you are running many large queries in quick succession, add a delay between runs. For sustained high-volume use, consider adding your own NCBI API key to the Actor code to raise the limit.

Some fields are empty in my output.

Not every variant has data for every field. For example, a variant may lack a clinical significance assertion or population frequency data. Empty fields reflect missing annotations in dbSNP, not a scraping error.

The search term with a space is not working.

URL-encode your search term or use the dbSNP search syntax. For multi-word terms, try enclosing them in quotes or using a plus sign between words, as you would in the NCBI web search.

FAQ

Question	Answer
What is dbSNP?	dbSNP is NCBI's database of single nucleotide polymorphisms and other small-scale genetic variations. It includes rsIDs, allele frequencies, clinical significance, and gene mappings for variants across many species, with a focus on human data.
Do I need an NCBI API key to use this Actor?	No. The Actor uses the public Esummary endpoint which does not require an API key. If you have very high volume needs, you may want to add your own NCBI API key in the code to raise rate limits, but it is not required to start scraping.
What search terms can I use?	You can use any term that dbSNP's search accepts: a gene symbol like BRCA1, an rs identifier like rs328, a chromosomal band like 17q21, or a free-text disease name. The term is passed directly to the Esummary search, so the same syntax that works on the NCBI website works here.
How many variants can I scrape in one run?	You set the maximum with the 'Maximum variants' input field, from 1 up to 1,000,000. The Actor will stop after collecting that many variant summaries, even if the search matches more records.
What fields does each variant row contain?	Each row includes the rsID, gene name, clinical significance, review status, allele frequencies, chromosome position, and linked PubMed IDs, among other summary fields. The exact field list is shown in the sample output on the Actor's page.
Can I scrape variants for non-human organisms?	Yes. dbSNP contains variants for many species. Your search term determines which organism's variants are returned. If you search for a mouse gene symbol, you will get mouse variants.
Is the output suitable for loading into a database?	Yes. Every run returns a flat, consistent schema. You can export to CSV or JSON and load the data directly into PostgreSQL, MySQL, or any analytics tool that accepts structured text.
How does this differ from using the NCBI Entrez API directly?	The Entrez API requires you to handle XML parsing, E-utility calls, rate limiting, and pagination logic. This Actor wraps all of that into a single run that returns clean rows. You provide a search term and a maximum count, and you get a dataset back.
Can I filter by clinical significance inside the Actor?	The Actor does not apply its own clinical significance filter. It returns all variants matching your search term. You can filter the resulting dataset by any field after export using Excel, Python, or the Apify dataset filtering.
What happens if my search term matches no variants?	The run will complete with an empty dataset. Check that your search term is spelled correctly and that it matches the format dbSNP expects. Try the same term on the NCBI dbSNP website to verify it returns results.

Related actors

Browse the full ParseForge collection for more scrapers.

🆘 Need help? Email parseforge@protonmail.com with your run ID, your input, and what you expected.

⚠️ Disclaimer. This Actor is unofficial and is not affiliated with, endorsed by, or sponsored by National Center for Biotechnology Information, U.S. National Library of Medicine. It collects only publicly available data. You are responsible for using the collected data in compliance with the source's terms of service and applicable data-protection laws, including GDPR, CCPA, and PIPL. Do not use it to collect personal data unlawfully.