NCI GDC Projects Scraper

NCI GDC Projects Scraper

Scrape NCI GDC project metadata from the Genomic Data Commons API. Every project comes with its disease type, primary site, dbGaP accession, and release state. No login or API key. Export to CSV, JSON, Excel, or XML.

The NCI Genomic Data Commons (GDC) data portal holds thousands of cancer research projects, but browsing them page by page is slow and manual. This Actor reads the public GDC projects API directly, filters by search term, and returns each matching project in one flat row. No authentication, no rate-limit headaches.

Who uses it	What they scrape NCI GDC Data Portal for
Bioinformatics researchers	Build a local catalog of all available cancer genomics projects for cohort discovery
Data journalists	Map the landscape of funded cancer research by disease type and primary site
Clinical trial coordinators	Identify GDC projects linked to specific dbGaP studies for participant recruitment
Cancer registry managers	Cross-reference GDC project metadata with internal registry records

What it does

This Actor collects NCI GDC project records from the official API and returns each one as a flat row with its ID, name, disease type, primary site, and release status.

• 🔍 Search term filter: narrow results to projects whose name or disease type matches your keyword
• 📊 Bulk extraction: pull project records in a single run for full-portal snapshots
• 📁 Multi-format export: save results as CSV, JSON, Excel, or XML for downstream tools

Results export to CSV, JSON, Excel, or XML, or straight from the API.

What you can do with NCI GDC Data Portal data

🧬 Map the cancer research landscape.

A bioinformatician pulls all GDC projects, groups them by disease type and primary site, and identifies underrepresented cancer types for a grant proposal.

📋 Build a project catalog for your lab.

A research coordinator scrapes all projects matching 'breast' and 'lung', exports to CSV, and shares the list with the team for cohort planning.

🔗 Link dbGaP studies to GDC projects.

A clinical data manager extracts all projects with a dbGaP accession number, cross-references them with approved studies, and validates data access compliance.

📈 Monitor newly released projects.

A data journalist runs the Actor weekly, filters for projects with state 'released', and tracks which cancer types are gaining new genomic data over time.

Why choose this scraper

	What you get
No API key required	The GDC projects endpoint is fully public; start scraping immediately
Flat, analysis-ready rows	Every project is returned in a consistent schema with no nested JSON to flatten
Search without the portal UI	Filter by disease type or project name directly in the Actor input
Full-portal snapshots	Pull every project in the GDC in one run for offline analysis or backup

What a NCI GDC Data Portal record looks like

Every record returns as one flat JSON row. Here is a real one from a run:

{
 "id": "TCGA-LUAD",
 "projectId": "TCGA-LUAD",
 "name": "Lung Adenocarcinoma",
 "releasable": true,
 "state": "open",
 "released": true,
 "scrapedAt": "2026-09-29T23:58:19.755Z"
}

Every value above comes from a real run. A field a record does not have comes back as null.

Configure the run

Drive the Actor with an optional search term to filter projects by name or disease type, and set a max items limit to control the volume of records returned. The Input tab lists every parameter.

A first run with the defaults:

{
 "maxItems": 10
}

A larger pull:

{
 "maxItems": 200
}

Free users

Free-plan runs return up to 10 results as a preview. Upgrade your Apify plan to collect more results per run.

Run it

1. Create a free Apify account.
2. Open the NCI GDC Projects Scraper.
3. Set your inputs and any filters, then click Start.
4. Export the results as CSV, Excel, JSON, or XML from the Dataset tab.

Run it programmatically through the Apify API (run-sync-get-dataset-items) or the ApifyClient for JavaScript and Python.

Use with AI agents (MCP)

Give an AI agent live access to NCI GDC Data Portal through the Model Context Protocol. Add the Actor to Claude, Cursor, or any MCP client:

$claude mcp add --transport http apify "https://mcp.apify.com?tools=acquistion-automation/nci-gdc-projects-scraper"

Then prompt it in plain language to run the scraper and read back the results.

Troubleshooting

Why am I getting no results?

Check your search term spelling. The filter matches against project name and disease type fields. Try a broader term or remove the search term entirely to see all projects.

Why did my run stop before reaching max items?

The GDC API may have fewer total projects than your max items setting. The Actor stops when the API returns no more records. Try lowering max items or removing the search term to confirm the total available.

Why are some fields empty in my output?

Not every GDC project has a dbGaP accession number. Empty fields reflect missing metadata in the source API, not a scraping error.

The Actor is taking a long time to finish. Is that normal?

For large max items values, the Actor may paginate through many API pages. This is expected. Reduce max items if you need faster preview runs.

FAQ

Question	Answer
Do I need an API key or login to scrape NCI GDC projects?	No. The GDC projects API is fully public and requires no authentication. You can start scraping immediately.
What data fields does each project record include?	Each record includes the project ID, name, disease type, primary site, dbGaP accession number, release state, and releasable flag. The exact schema is shown in the sample output on the Actor page.
Can I filter projects by a specific cancer type?	Yes. Use the search term input to filter projects whose name or disease type contains your keyword, such as 'breast' or 'lung'.
How many projects can I scrape in one run?	Free users can scrape up to 10 projects as a preview. Paid users can set max items up to 1,000,000, which covers the entire GDC projects catalog many times over.
Is the data returned in real time from the GDC API?	Yes. The Actor calls the live GDC projects API endpoint on each run, so you always get the current project metadata.
Can I export the results to CSV?	Yes. The Actor supports CSV, JSON, Excel, and XML exports. Choose your format in the output settings.
Does this Actor scrape the GDC portal website or the API?	It scrapes the official GDC API endpoint directly. The portal website is a JavaScript application with no embedded data, so the API is the reliable source for structured project records.
What is a dbGaP accession number and why is it included?	dbGaP accession numbers link GDC projects to authorized human subjects studies. The field is included so you can cross-reference projects with dbGaP for compliance or study tracking.

Related actors

Browse the full Acquisition Automation collection for more scrapers.

🆘 Need help? Open an issue in the Issues tab of this Actor with your run ID, your input, and what you expected.

Pricing

This Actor uses pay-per-result pricing: $0.004 per result collected. You are billed only for the results you receive, so a run that returns nothing costs nothing.

⚠️ Disclaimer. This Actor is unofficial and is not affiliated with, endorsed by, or sponsored by National Cancer Institute. It collects only publicly available data. You are responsible for using the collected data in compliance with the source's terms of service and applicable data-protection laws, including GDPR, CCPA, and PIPL. Do not use it to collect personal data unlawfully.