NCI GDC File Metadata Scraper

NCI GDC File Metadata Scraper

Scrape NCI GDC file metadata from the public API, up to a million records per run. Every file comes with its ID, name, size, MD5 checksum, data type, experimental strategy, and linked case ID. No login or API key. Export to CSV, JSON, Excel, or XML.

The NCI Genomic Data Commons API requires you to page through JSON responses and build filter objects by hand. This Actor reads the public GDC file metadata endpoint directly, applies your data type and data format filters as it goes, and returns each matching file as one flat row. It is the fastest way to get a GDC file manifest into a spreadsheet or a pipeline.

Who uses it	What they scrape NCI Genomic Data Commons for
Bioinformatics researchers	Build a local manifest of GDC files for a specific cancer study
Data engineers	Feed GDC file metadata into a pipeline for downstream processing
Clinical data analysts	Pull file-level metadata for a cohort before joining it with clinical tables
Cancer genomics teams	Export a filtered list of sequencing files for a project

What it does

This Actor collects NCI GDC file metadata from the public API and returns each file as a flat row with its ID, name, size, checksum, data type, experimental strategy, and linked case ID.

• 📋 Flat file rows: every file comes back as one row with file_id, file_name, file_size, md5sum, data_type, experimental_strategy, and case_id.
• 🔎 Data type filter: pass one or more data types like Aligned Reads or Raw Sequencing Data and only matching files are returned.
• 📦 Data format filter: pass formats like BAM or VCF to narrow the result set before it reaches your dataset.
• 🔁 Pagination handled: the Actor pages through the GDC API automatically up to your maxItems limit.

Results export to CSV, JSON, Excel, or XML, or straight from the API.

What you can do with NCI Genomic Data Commons data

📄 Build a GDC file manifest.

A bioinformatics researcher filters by data type and format, then exports the result to CSV as a manifest for a processing workflow.

🧬 Collect sequencing files for a cohort.

A cancer genomics team pulls all BAM files for a study and joins the file metadata with case IDs for downstream analysis.

📊 Feed a metadata pipeline.

A data engineer schedules the Actor to refresh GDC file metadata into a warehouse table on a regular cadence.

🔎 Audit file integrity.

A clinical data analyst exports file names, sizes, and MD5 checksums to verify that downloaded files match the GDC source.

Why choose this scraper

	What you get
No API key	The GDC endpoint is public, so you can start a run immediately
Filtered output	Data type and data format filters run before results are stored
Flat schema	Every file is one row, ready for CSV, JSON, Excel, or XML export
Large runs	Paid users can pull up to 1,000,000 file records in a single run

What a NCI Genomic Data Commons record looks like

Every record returns as one flat JSON row. Here is a real one from a run:

{
 "id": "af0d3d92-f8b5-4612-805b-d6773595262b",
 "file_id": "af0d3d92-f8b5-4612-805b-d6773595262b",
 "file_name": "25ab28a6-8841-4048-8950-6ca724320b41.wxs.Pindel.aliquot.maf.gz",
 "file_size": 35992,
 "md5sum": "b7273f961881a3556adbbcabf69a5c5b",
 "data_type": "Annotated Somatic Mutation",
 "experimental_strategy": "WXS",
 "case_id": "1e5c5b1b-3450-4a1f-9e92-bf522b47f23f",
 "scrapedAt": "2026-09-24T05:52:29.867Z"
}

Every value above comes from a real run. A field a record does not have comes back as null.

Configure the run

Drive the Actor with optional data type and data format filters, and set maxItems to cap how many file records are returned. Filters are applied as the API is read, so only matching files reach your dataset. The Input tab lists every parameter.

A first run with the defaults:

{
 "maxItems": 10,
 "dataTypeFilter": [],
 "dataFormatFilter": []
}

A larger pull:

{
 "maxItems": 200,
 "dataTypeFilter": [],
 "dataFormatFilter": []
}

Free users

Free-plan runs return up to 10 results as a preview. Upgrade your Apify plan to collect up to 1,000,000 results per run.

Run it

1. Create a free Apify account with $5 in credit.
2. Open the NCI GDC File Metadata Scraper.
3. Set your inputs and any filters, then click Start.
4. Export the results as CSV, Excel, JSON, or XML from the Dataset tab.

Run it programmatically through the Apify API (run-sync-get-dataset-items) or the ApifyClient for JavaScript and Python.

Use with AI agents (MCP)

Give an AI agent live access to NCI Genomic Data Commons through the Model Context Protocol. Add the Actor to Claude, Cursor, or any MCP client:

$claude mcp add --transport http apify "https://mcp.apify.com?tools=acquistion-automation/nci-gdc-file-metadata-scraper"

Then prompt it in plain language to run the scraper and read back the results.

Troubleshooting

Why am I getting no results?

Check your data type and data format filters. If you pass a value that does not match any file, the API returns an empty result set. Try clearing the filters and running again.

Why did my run stop at 10 items?

Free users are limited to 10 items as a preview. Upgrade to a paid plan and set maxItems higher to pull more records.

Why is my run slow?

The GDC API pages through results in chunks. Large maxItems values mean more requests. Narrow your data type or data format filters to reduce the number of pages.

Why are some files missing from my export?

The GDC API only returns files that match your filters and are accessible through the public endpoint. Verify that the file IDs exist in the GDC Data Portal.

FAQ

Question	Answer
Do I need an API key or GDC credentials?	No. The Actor reads the public GDC API endpoint, which does not require authentication.
What filters can I apply?	You can filter by data type, such as Aligned Reads or Raw Sequencing Data, and by data format, such as BAM or VCF. Leave both empty to return all file types.
How many file records can I get in one run?	Free users are limited to 10 items as a preview. Paid users can set maxItems up to 1,000,000.
What fields are returned for each file?	Each row includes file_id, file_name, file_size, md5sum, data_type, experimental_strategy, and the linked case_id.
Can I export the results?	Yes. The dataset can be exported to CSV, JSON, Excel, or XML from the Apify platform.
Does the Actor handle pagination?	Yes. It pages through the GDC API automatically until it reaches your maxItems limit or the end of the result set.
Can I filter by case ID or project?	The current input schema supports data type and data format filters. Case ID is returned in the output, but case-level filtering is not exposed as an input.
Is this the same as the GDC Data Portal?	The Actor reads the same public API that powers the GDC Data Portal, but returns file metadata as structured rows instead of a web interface.

Related actors

Browse the full Acquisition Automation collection for more scrapers.

🆘 Need help? Open an issue in the Issues tab of this Actor with your run ID, your input, and what you expected.

Pricing

This Actor uses pay-per-result pricing: $0.0395 per result collected. You are billed only for the results you receive, so a run that returns nothing costs nothing.

⚠️ Disclaimer. This Actor is unofficial and is not affiliated with, endorsed by, or sponsored by National Cancer Institute. It collects only publicly available data. You are responsible for using the collected data in compliance with the source's terms of service and applicable data-protection laws, including GDPR, CCPA, and PIPL. Do not use it to collect personal data unlawfully.