NCI GDC Cancer Genomics Scraper

NCI GDC Cancer Genomics Scraper

Scrape cancer genomics data from the NCI Genomic Data Commons, up to a million records per run. Every record includes project, case, file, or annotation details with filters for primary site and program. No API key required. Export to CSV, JSON, Excel, or XML.

The NCI Genomic Data Commons API requires pagination handling and returns deeply nested JSON. This actor reads the public endpoints directly, filters by primary site or program, and returns each record in one flat schema. It covers projects, cases, files, and annotations.

Who uses it	What they scrape NCI Genomic Data Commons for
Bioinformaticians	Pull case and file metadata for a specific cancer program to build analysis cohorts.
Cancer researchers	Gather project summaries and sample counts by primary site for grant proposals.
Data engineers	Ingest GDC metadata into a data warehouse for downstream processing.
Clinical trial coordinators	Check available cases and annotations for a given program before patient enrollment.

What it does

This actor collects records from the NCI GDC public endpoints for projects, cases, files, or annotations, and returns each one as a flat row.

• 📊 Flat output: every record is returned as a single row with consistent fields, no nested JSON.
• 🔍 Site and program filters: narrow results to a specific primary site like Breast or a program like TCGA.
• 📦 Bulk export: collect up to 1,000,000 records per run and export to CSV, JSON, Excel, or XML.

Results export to CSV, JSON, Excel, or XML, or straight from the API.

What you can do with NCI Genomic Data Commons data

🧬 Build a cohort for analysis.

A bioinformatician selects the Cases entity, filters by program TCGA and primary site Breast, and exports the case IDs and clinical metadata to CSV for downstream variant calling.

📈 Compare project scope.

A researcher scrapes all Projects and uses the primary site filter to compare the number of projects per cancer type, informing a meta-analysis proposal.

🗂️ Ingest metadata into a warehouse.

A data engineer runs the actor daily on the Files entity, filters by program, and loads the flat rows into a PostgreSQL table for internal dashboards.

📋 Check annotation availability.

A clinical trial coordinator scrapes Annotations for a specific program to see which cases have curated annotations before selecting patients for a study.

Why choose this scraper

	What you get
No API key	Uses the public GDC API without authentication.
Flat schema	Nested JSON is flattened into one row per record.
Large volume	Collect up to a million records per run.
Flexible filters	Filter by primary site and program to target specific cohorts.

How it compares

No other Store actor targets NCI Genomic Data Commons the same way, so the honest comparison is with the alternatives teams actually weigh.

	NCI GDC Cancer Genomics Scraper	Build it in-house	By hand
Setup	Run it now, zero config	Days of engineering	None, but hours per pull
When NCI Genomic Data Commons changes	Maintained for you	You fix it	You re-learn the page
Proxies, retries, anti-bot	Built in	Your problem	Browser only
Output	Fixed JSON schema, CSV/Excel export	Whatever you build	Copy-paste
Cost	Pay per result	Engineering time	Analyst hours

Configure the run

Drive the actor by selecting an entity type, then optionally filter by primary site and program. Filters run as each record is read so only matches reach your dataset. The Input tab lists every parameter.

A first run with the defaults:

{
 "entity": "projects",
 "maxItems": 10
}

A larger pull:

{
 "entity": "projects",
 "maxItems": 200
}

Pricing

Pay-per-result: $0.0395 per result collected. You pay only for the results written to your dataset.

Results collected	Approximate cost
100 results	$3.95
1,000 results	$39.50
10,000 results	$395.00

New Apify accounts start with $5 in free credit.

Free users

Free-plan runs return up to 10 results as a preview. Upgrade your Apify plan to collect up to 1,000,000 results per run.

Run it

1. Create a free Apify account with $5 in credit.
2. Open the NCI GDC Cancer Genomics Scraper.
3. Set your inputs and any filters, then click Start.
4. Export the results as CSV, Excel, JSON, or XML from the Dataset tab.

Run it programmatically through the Apify API (run-sync-get-dataset-items) or the ApifyClient for JavaScript and Python.

Use with AI agents (MCP)

Give an AI agent live access to NCI Genomic Data Commons through the Model Context Protocol. Add the Actor to Claude, Cursor, or any MCP client:

$claude mcp add --transport http apify "https://mcp.apify.com?tools=parseforge/nci-gdc-cancer-scraper"

Then prompt it in plain language to run the scraper and read back the results.

Troubleshooting

Why am I getting no results?

Check your filters. If you set a Primary Site or Program that does not exist for the selected entity, no records will match. Try clearing the filters and running again.

The run is taking too long.

Reduce the Maximum records or add filters to narrow the query. The GDC API can be slow for large unfiltered requests.

I get an error about invalid entity.

Make sure you selected one of the four valid entities: projects, cases, files, or annotations.

Some fields are missing in the output.

Not all fields are present for every record. The actor returns only the fields that exist in the GDC response for that entity.

FAQ

Question	Answer
Do I need an API key or authentication?	No. The actor uses the public NCI GDC API endpoints, which do not require authentication.
What entities can I scrape?	You can scrape projects, cases, files, or annotations by selecting the entity in the input.
How do I filter by cancer type?	Use the Primary Site filter to select a specific tissue, such as Breast or Lung. Leave it empty for all sites.
Can I filter by a specific program like TCGA?	Yes, use the Program filter to select from programs such as TCGA, TARGET, or CPTAC.
What is the maximum number of records I can get?	You can set Maximum records up to 1,000,000 per run.
What output formats are supported?	The actor can export to CSV, JSON, Excel, or XML, depending on your Apify dataset settings.
Does the actor handle pagination automatically?	Yes, it follows the GDC API pagination to collect all matching records up to your limit.
Are the results flattened?	Yes, nested JSON fields are flattened into a single row per record for easy analysis.
Can I schedule this actor to run regularly?	Yes, you can set up a schedule in Apify to run it daily or at any interval.
Is this actor affiliated with the NCI?	No, it is an independent tool that accesses the public NCI GDC API.

Related actors

• clinicaltrials-scraper: Use this to scrape clinical trial data from ClinicalTrials.gov.

Browse the full ParseForge collection for more scrapers.

🆘 Need help? Email parseforge@protonmail.com with your run ID, your input, and what you expected.

⚠️ Disclaimer. This Actor is unofficial and is not affiliated with, endorsed by, or sponsored by National Cancer Institute. It collects only publicly available data. You are responsible for using the collected data in compliance with the source's terms of service and applicable data-protection laws, including GDPR, CCPA, and PIPL. Do not use it to collect personal data unlawfully.