# NCI GDC File Metadata Scraper (`acquistion-automation/nci-gdc-file-metadata-scraper`) Actor

Pull file-level metadata from the NCI Genomic Data Commons public API. Retrieve file IDs, file names, file sizes, MD5 checksums, data types, experimental strategies, and associated case IDs. Ideal for researchers and bioinformaticians building curated datasets for cancer genomics analysis.

- **URL**: https://apify.com/acquistion-automation/nci-gdc-file-metadata-scraper.md
- **Developed by:** [Acquisition Automation Co.](https://apify.com/acquistion-automation) (community)
- **Categories:** Other
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $29.62 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

[![Acquisition Automation](https://api.apify.com/v2/key-value-stores/AOdPHdOpeDpzEPS5f/records/banner.jpg)](https://apify.com/acquistion-automation)

### NCI GDC File Metadata Scraper

**Scrape NCI GDC file metadata from the public API, up to a million records per run.** Every file comes with its ID, name, size, MD5 checksum, data type, experimental strategy, and linked case ID. No login or API key. Export to CSV, JSON, Excel, or XML.

The NCI Genomic Data Commons API requires you to page through JSON responses and build filter objects by hand. This Actor reads the public GDC file metadata endpoint directly, applies your data type and data format filters as it goes, and returns each matching file as one flat row. It is the fastest way to get a GDC file manifest into a spreadsheet or a pipeline.

| Who uses it | What they scrape NCI Genomic Data Commons for |
|---|---|
| Bioinformatics researchers | Build a local manifest of GDC files for a specific cancer study |
| Data engineers | Feed GDC file metadata into a pipeline for downstream processing |
| Clinical data analysts | Pull file-level metadata for a cohort before joining it with clinical tables |
| Cancer genomics teams | Export a filtered list of sequencing files for a project |

### What it does

This Actor collects NCI GDC file metadata from the public API and returns each file as a flat row with its ID, name, size, checksum, data type, experimental strategy, and linked case ID.

- 📋 **Flat file rows:** every file comes back as one row with file\_id, file\_name, file\_size, md5sum, data\_type, experimental\_strategy, and case\_id.
- 🔎 **Data type filter:** pass one or more data types like Aligned Reads or Raw Sequencing Data and only matching files are returned.
- 📦 **Data format filter:** pass formats like BAM or VCF to narrow the result set before it reaches your dataset.
- 🔁 **Pagination handled:** the Actor pages through the GDC API automatically up to your maxItems limit.

Results export to CSV, JSON, Excel, or XML, or straight from the API.

### What you can do with NCI Genomic Data Commons data

**📄 Build a GDC file manifest.**

A bioinformatics researcher filters by data type and format, then exports the result to CSV as a manifest for a processing workflow.

**🧬 Collect sequencing files for a cohort.**

A cancer genomics team pulls all BAM files for a study and joins the file metadata with case IDs for downstream analysis.

**📊 Feed a metadata pipeline.**

A data engineer schedules the Actor to refresh GDC file metadata into a warehouse table on a regular cadence.

**🔎 Audit file integrity.**

A clinical data analyst exports file names, sizes, and MD5 checksums to verify that downloaded files match the GDC source.

### Why choose this scraper

| | What you get |
|---|---|
| **No API key** | The GDC endpoint is public, so you can start a run immediately |
| **Filtered output** | Data type and data format filters run before results are stored |
| **Flat schema** | Every file is one row, ready for CSV, JSON, Excel, or XML export |
| **Large runs** | Paid users can pull up to 1,000,000 file records in a single run |

### What a NCI Genomic Data Commons record looks like

Every record returns as one flat JSON row. Here is a real one from a run:

```json
{
 "id": "af0d3d92-f8b5-4612-805b-d6773595262b",
 "file_id": "af0d3d92-f8b5-4612-805b-d6773595262b",
 "file_name": "25ab28a6-8841-4048-8950-6ca724320b41.wxs.Pindel.aliquot.maf.gz",
 "file_size": 35992,
 "md5sum": "b7273f961881a3556adbbcabf69a5c5b",
 "data_type": "Annotated Somatic Mutation",
 "experimental_strategy": "WXS",
 "case_id": "1e5c5b1b-3450-4a1f-9e92-bf522b47f23f",
 "scrapedAt": "2026-09-24T05:52:29.867Z"
}
```

Every value above comes from a real run. A field a record does not have comes back as `null`.

### Configure the run

Drive the Actor with optional data type and data format filters, and set maxItems to cap how many file records are returned. Filters are applied as the API is read, so only matching files reach your dataset. The Input tab lists every parameter.

A first run with the defaults:

```json
{
 "maxItems": 10,
 "dataTypeFilter": [],
 "dataFormatFilter": []
}
```

A larger pull:

```json
{
 "maxItems": 200,
 "dataTypeFilter": [],
 "dataFormatFilter": []
}
```

### Free users

Free-plan runs return up to 10 results as a preview. [Upgrade your Apify plan](https://console.apify.com/sign-up) to collect up to 1,000,000 results per run.

### Run it

1. [Create a free Apify account with $5 in credit](https://console.apify.com/sign-up).
2. Open the [NCI GDC File Metadata Scraper](https://apify.com/acquistion-automation/nci-gdc-file-metadata-scraper).
3. Set your inputs and any filters, then click **Start**.
4. Export the results as CSV, Excel, JSON, or XML from the **Dataset** tab.

Run it programmatically through the [Apify API](https://docs.apify.com/api/v2) (`run-sync-get-dataset-items`) or the [ApifyClient](https://docs.apify.com/api/client/js) for JavaScript and Python.

### Use with AI agents (MCP)

Give an AI agent live access to NCI Genomic Data Commons through the Model Context Protocol. Add the Actor to Claude, Cursor, or any MCP client:

```bash
claude mcp add --transport http apify "https://mcp.apify.com?tools=acquistion-automation/nci-gdc-file-metadata-scraper"
```

Then prompt it in plain language to run the scraper and read back the results.

### Troubleshooting

**Why am I getting no results?**

Check your data type and data format filters. If you pass a value that does not match any file, the API returns an empty result set. Try clearing the filters and running again.

**Why did my run stop at 10 items?**

Free users are limited to 10 items as a preview. Upgrade to a paid plan and set maxItems higher to pull more records.

**Why is my run slow?**

The GDC API pages through results in chunks. Large maxItems values mean more requests. Narrow your data type or data format filters to reduce the number of pages.

**Why are some files missing from my export?**

The GDC API only returns files that match your filters and are accessible through the public endpoint. Verify that the file IDs exist in the GDC Data Portal.

### FAQ

| Question | Answer |
|---|---|
| Do I need an API key or GDC credentials? | No. The Actor reads the public GDC API endpoint, which does not require authentication. |
| What filters can I apply? | You can filter by data type, such as Aligned Reads or Raw Sequencing Data, and by data format, such as BAM or VCF. Leave both empty to return all file types. |
| How many file records can I get in one run? | Free users are limited to 10 items as a preview. Paid users can set maxItems up to 1,000,000. |
| What fields are returned for each file? | Each row includes file\_id, file\_name, file\_size, md5sum, data\_type, experimental\_strategy, and the linked case\_id. |
| Can I export the results? | Yes. The dataset can be exported to CSV, JSON, Excel, or XML from the Apify platform. |
| Does the Actor handle pagination? | Yes. It pages through the GDC API automatically until it reaches your maxItems limit or the end of the result set. |
| Can I filter by case ID or project? | The current input schema supports data type and data format filters. Case ID is returned in the output, but case-level filtering is not exposed as an input. |
| Is this the same as the GDC Data Portal? | The Actor reads the same public API that powers the GDC Data Portal, but returns file metadata as structured rows instead of a web interface. |

### Related actors

Browse the full [Acquisition Automation collection](https://apify.com/acquistion-automation) for more scrapers.

🆘 **Need help?** Open an issue in the Issues tab of this Actor with your run ID, your input, and what you expected.

### Pricing

This Actor uses **pay-per-result** pricing: **$0.0395 per result** collected. You are billed only for the results you receive, so a run that returns nothing costs nothing.

⚠️ **Disclaimer.** This Actor is unofficial and is not affiliated with, endorsed by, or sponsored by National Cancer Institute. It collects only publicly available data. You are responsible for using the collected data in compliance with the source's terms of service and applicable data-protection laws, including GDPR, CCPA, and PIPL. Do not use it to collect personal data unlawfully.

# Actor input Schema

## `maxItems` (type: `integer`):

Free users: Limited to 10 items (preview). Paid users: Optional, max 1,000,000

## `dataTypeFilter` (type: `array`):

Filter by data type (e.g., Aligned Reads, Raw Sequencing Data). Leave empty for all types.

## `dataFormatFilter` (type: `array`):

Filter by data format (e.g., BAM, VCF). Leave empty for all formats.

## Actor input object example

```json
{
  "maxItems": 10,
  "dataTypeFilter": [],
  "dataFormatFilter": []
}
```

# Actor output Schema

## `results` (type: `string`):

Complete dataset of all scraped records.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "maxItems": 10,
    "dataTypeFilter": [],
    "dataFormatFilter": []
};

// Run the Actor and wait for it to finish
const run = await client.actor("acquistion-automation/nci-gdc-file-metadata-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "maxItems": 10,
    "dataTypeFilter": [],
    "dataFormatFilter": [],
}

# Run the Actor and wait for it to finish
run = client.actor("acquistion-automation/nci-gdc-file-metadata-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "maxItems": 10,
  "dataTypeFilter": [],
  "dataFormatFilter": []
}' |
apify call acquistion-automation/nci-gdc-file-metadata-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,acquistion-automation/nci-gdc-file-metadata-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/FrV43pxsZLfhxoljl/builds/0wI5wtBAaPTnH3hMg/openapi.json