🤗 Hugging Face Dataset Scraper

Structured export of Hugging Face datasets — size, splits, features schema, license, and download counts. Built for AI engineers, data scientists, and ML teams performing dataset curation, compliance audits, and benchmark tracking at scale.

---

📑 Table of Contents

• Why Use This Scraper?
• Quick Start
• Programmatic Integration (API)
• Input Parameters
• Output Format
• Use Cases
• Pricing (Pay-Per-Event)
• Frequently Asked Questions (FAQ)
• Related Actors

---

🌟 Why Use This Scraper?

Hugging Face Hub hosts over 150,000 datasets, but finding the right data, evaluating license risks, and tracking training splits manually is time-consuming.

✅ Complete Structural Metadata – Extracts dataset ID, author, total downloads, likes, license, dataset size tier, splits (train/test/validation), and tag categories.

✅ Parsed Features & Schemas – Directly inspects dataset configuration and column features without downloading terabytes of raw data files.

✅ Rich Filtering & Sorting – Query datasets by keywords, sort by popularity (downloads, likes, trending, created), and paginate up to 500 records per run.

✅ Pay-Per-Event Economy – Zero compute waste. Pay only for the datasets you actually retrieve ($0.015 per item).

✅ Zero Setup & Authentication Free – Works immediately out-of-the-box without requiring a Hugging Face API token (optional token supported for private/gated access).

---

⚡ Quick Start

1. Open the Actor page on Apify.
2. Set your Search Query (e.g. code, instruction, vision, or medical).
3. Choose your Sort Order (downloads, likes, trending, or created).
4. Click Start and export your clean dataset catalog in JSON, CSV, or Excel.

---

💻 Programmatic Integration (API)

Integrate dataset intelligence into your AI pipelines, automated retraining workflows, or research dashboards.

Python Example

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_API_TOKEN")

run_input = {
    "query": "finetune instruct",
    "sort": "downloads",
    "maxResults": 100
}

run = client.actor("lovely_radiologist/hf-dataset-scraper").call(run_input=run_input)

# Fetch results from the dataset
dataset_items = client.dataset(run["defaultDatasetId"]).list_items().items
for item in dataset_items:
    print(f"{item.get('datasetId')} | Downloads: {item.get('downloads')} | License: {item.get('license')}")

JavaScript / Node.js Example

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_APIFY_API_TOKEN' });

const input = {
    query: 'code generation',
    sort: 'likes',
    maxResults: 50
};

const run = await client.actor('lovely_radiologist/hf-dataset-scraper').call(input);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(`Found ${items.length} datasets:`, items);

---

⚙️ Input Parameters

Field	Type	Description	Default
query	String	Search keyword or filter term (e.g., multimodal, python, gsm8k).	""
sort	String (Select)	Sort criteria: downloads, likes, trending, or created.	"downloads"
maxResults	Integer	Maximum number of datasets to extract (1 to 500).	50
hfToken	String	Optional Hugging Face User Access Token (to access private/gated datasets).	""

---

📊 Output Format

Each dataset is saved as an individual item in your Apify dataset:

{
  "datasetId": "HuggingFaceTB/smollm-corpus",
  "author": "HuggingFaceTB",
  "downloads": 4100000,
  "likes": 890,
  "size": "100B<n<1T",
  "splits": ["train"],
  "features": {
    "text": { "dtype": "string" },
    "metadata": { "dtype": "string" }
  },
  "license": "odbl",
  "tags": [
    "text",
    "code",
    "en",
    "license:odbl",
    "size_categories:100B<n<1T"
  ],
  "description": "SmolLM-Corpus is a curated collection of high-quality educational and synthetic data..."
}

---

🎯 Use Cases

• AI Training & Pretraining: Identify high-quality, high-download corpora for LLM or SLM training.
• License & Compliance Audits: Screen datasets for restrictive or copyleft licenses (e.g. GPL, CC-BY-NC) before ingestion.
• Trend & Ecosystem Intelligence: Monitor emerging benchmarks and datasets trending within the open-source AI community.
• RAG & Knowledge Bases: Populate metadata catalogs for internal enterprise model/data registries.

---

💰 Pricing (Pay-Per-Event)

Predictable and cost-effective pricing based strictly on usage:

Event	Price	Description
actor-start	$0.01	Charged once per run start
dataset-item	$0.0005	Per dataset exported into your dataset ($0.50 per 1,000 datasets)

---

❓ Frequently Asked Questions (FAQ)

Do I need a Hugging Face API key?

No! Public datasets on the Hugging Face Hub are queried directly without any token. You only need to provide an hfToken if querying private datasets or gated datasets tied to your HF account.

Does this scraper download the full dataset content?

No, this scraper extracts structured dataset metadata, features schema, splits, and download metrics. This lets you inspect and filter thousands of datasets without spending gigabytes of bandwidth and hours of download time.

Can I schedule automatic daily/weekly monitoring?

Yes! Use Apify's built-in Schedule tab to run this scraper on a cron schedule (e.g. daily) to track download growth and new trending datasets.

---

🔗 Related Actors

• Hugging Face Model Explorer — Discover and filter Hugging Face models with structured YAML card metadata
• GitHub Trending Radar — Track trending AI and software repositories with delta growth
• arXiv Paper Extractor — Extract academic papers introducing these datasets