# OpenML Dataset Scraper (`automation-lab/openml-scraper`) Actor Scrape ML datasets, tasks, flows, and runs from OpenML - the open science platform for machine learning - **URL**: https://apify.com/automation-lab/openml-scraper.md - **Developed by:** [Stas Persiianenko](https://apify.com/automation-lab) (community) - **Categories:** AI - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, NaN bookmarks - **User rating**: No ratings yet ## Pricing Pay per event This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## OpenML Scraper Extract ML datasets, benchmark tasks, and algorithm flows from [OpenML](https://www.openml.org) — the open science platform for machine learning. Get structured metadata for thousands of public ML benchmark datasets including feature counts, instance counts, class distributions, quality metrics, tags, download URLs, and more. No API key required. No proxy needed. Pure HTTP access to OpenML's public REST API. ### What does it do? OpenML Scraper connects to the OpenML public REST API (`openml.org/api/v1/json`) and extracts structured data for three resource types: - **📊 Datasets** — ML benchmark datasets with quality metrics (features, instances, classes, missing values), tags, descriptions, download URLs, and metadata - **🎯 Tasks** — Supervised classification and regression tasks defining evaluation procedures and target attributes - **⚙️ Flows** — Algorithm and pipeline implementations (scikit-learn, Weka, R packages, etc.) uploaded by the community Results are pushed to an Apify dataset in clean, flat JSON format — ready for analysis, filtering, or export to CSV/Excel. ### Who is it for? **ML researchers** who want to browse and discover datasets for benchmarking without clicking through the OpenML web UI. Filter by name, status, or type and get all metadata in a single structured output. **AutoML engineers** building dataset recommendation systems or experiment tracking pipelines. Use the scraper to programmatically catalog available benchmark datasets and their properties. **Data scientists** who need to audit which OpenML datasets match their constraints (minimum features, instances, classes) for reproducible research. **Platform builders** creating dataset directories or ML curriculum tools who need a machine-readable catalog of public benchmark datasets. **Students and educators** exploring the landscape of ML datasets for teaching purposes — quickly find datasets by name, size, or domain tag. ### Why use it? OpenML's REST API is public and powerful, but integrating it into workflows requires building custom fetch/pagination/normalization code. This actor handles all of that: - ✅ **Pagination built-in** — fetches all matching results up to your `maxResults` limit, automatically handling page offsets - ✅ **Rich metadata** — goes beyond the list API to fetch full dataset descriptions, upload dates, licence info, and download URLs - ✅ **Quality metrics extracted** — flattens the nested `quality` array into named fields (`numberOfFeatures`, `numberOfInstances`, etc.) - ✅ **No auth needed** — OpenML's public API requires no API key - ✅ **Retry logic** — configurable retry count for transient failures - ✅ **Clean flat output** — no nested objects, ready for Apify datasets table view and CSV export ### Data extracted #### Datasets | Field | Description | |-------|-------------| | `id` | OpenML dataset ID | | `name` | Dataset name | | `version` | Dataset version number | | `status` | Active / deactivated / in_preparation | | `format` | File format (ARFF, CSV, etc.) | | `url` | OpenML dataset page URL | | `downloadUrl` | Direct ARFF file download URL | | `numberOfFeatures` | Total number of attributes/columns | | `numberOfInstances` | Total number of rows/samples | | `numberOfClasses` | Number of target classes (classification datasets) | | `numberOfMissingValues` | Count of missing values across all cells | | `uploadDate` | When the dataset was uploaded | | `description` | Dataset description (up to 500 chars) | | `licence` | Licence (Public, CC BY, etc.) | | `defaultTargetAttribute` | Default prediction target column name | | `tags` | Array of tags (domain, study labels, source) | #### Tasks | Field | Description | |-------|-------------| | `id` | Task ID | | `name` | Task name (usually dataset name) | | `taskType` | Task type (Supervised Classification, Supervised Regression, etc.) | | `taskTypeId` | Numeric task type ID | | `datasetId` | Source dataset ID | | `status` | Task status | | `targetFeature` | Target column to predict | | `estimationProcedure` | Cross-validation procedure ID | | `evaluationMeasures` | Primary evaluation metric | | `numberOfFeatures` | Features in the underlying dataset | | `numberOfInstances` | Instances in the underlying dataset | | `url` | OpenML task page URL | #### Flows | Field | Description | |-------|-------------| | `id` | Flow ID | | `name` | Flow name (e.g., `sklearn.ensemble.forest.RandomForestClassifier`) | | `fullName` | Full name with version (e.g., `sklearn...RandomForestClassifier(8)`) | | `version` | Flow version number | | `externalVersion` | External library version tag | | `uploaderId` | User ID of the uploader | | `url` | OpenML flow page URL | ### How much does it cost to scrape OpenML datasets? 💡 **Free plan estimate:** ~100 free results per month on the Apify Free plan. The actor uses Pay-Per-Event (PPE) pricing: | Event | BRONZE | SILVER | GOLD | PLATINUM | DIAMOND | |-------|--------|--------|------|----------|---------| | Run started | flat fee | flat fee | flat fee | flat fee | flat fee | | Per result | ~$0.000029 | ~$0.0000225 | ~$0.0000173 | ~$0.0000115 | ~$0.0000081 | **Example costs:** - 100 datasets: ~$0.008 - 500 datasets: ~$0.019 - 1,000 datasets: ~$0.034 OpenML has ~6,000 active datasets, ~100,000 tasks, and ~20,000 flows. A full catalog export at BRONZE pricing costs ~$0.18–$2.89 depending on resource type. ### How to use it #### Step 1 — Choose your resource type Select whether you want **Datasets**, **Tasks**, or **Flows** from the "What to scrape" section. #### Step 2 — Filter (optional) For datasets, enter a name filter in **Search by name** (e.g., `iris`, `mnist`, `breast cancer`) and set the **Dataset status filter** to `active`. #### Step 3 — Set a result limit Set **Max results** to control how many items to return. Start small (20–50) to preview the output before running a large batch. #### Step 4 — Run and export Click **Save & Run**. Results appear in the **Dataset** tab. Export to JSON, CSV, or Excel from the **Export** button. ### Input parameters | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | `resourceType` | string | `datasets` | What to scrape: `datasets`, `tasks`, or `flows` | | `searchQuery` | string | `` | Filter by name (datasets: API-side; flows: client-side) | | `status` | string | `active` | Dataset status: `active`, `deactivated`, `in_preparation`, `any` | | `maxResults` | integer | `100` | Maximum results to return (1–10,000) | | `maxRequestRetries` | integer | `3` | Retry attempts per failed request | ### Output example ```json { "resourceType": "dataset", "id": 61, "name": "iris", "version": 1, "status": "active", "format": "ARFF", "url": "https://www.openml.org/d/61", "downloadUrl": "https://openml.org/data/v1/download/61/iris.arff", "numberOfFeatures": 5, "numberOfInstances": 150, "numberOfClasses": 3, "numberOfMissingValues": 0, "uploadDate": "2014-04-06T23:23:39", "description": "Fisher's Iris Plants Database...", "licence": "Public", "defaultTargetAttribute": "class", "tags": ["Botany", "Machine Learning", "uci"] } ```` ### Tips for best results - 🔍 **Name search is exact-prefix for datasets** — search for `iris` returns `iris`, `iris-2`, etc. Use short, common dataset names. - ⚙️ **Flow search is substring match** — searching for `sklearn` matches any flow whose name contains `sklearn`. It scans all flows (up to 20,000), which takes ~30–60 seconds. - 📊 **Use `status: any`** to include deactivated and in-preparation datasets in your catalog. - ⚡ **Set maxResults to 100** for quick previews. For full catalogs, set it to 10,000 and expect 2–5 minutes of runtime. - 🔄 **Tasks don't support name filtering** — all tasks are returned in order of task ID. Filter by task type in your downstream pipeline. ### Integrations #### 🔗 Export to Google Sheets Use the [Google Sheets integration](https://apify.com/integration/google-sheets) to automatically push extracted datasets to a spreadsheet for collaborative review or ML experiment planning. #### 📊 Connect to Power BI or Tableau Export the dataset as CSV from the Apify console and import it into your BI tool to build dashboards comparing dataset sizes, feature counts, and class distributions. #### 🤖 AutoML pipeline seeding Run this actor on a schedule to keep a local database of OpenML datasets fresh. Use the dataset list to auto-select benchmark datasets for your AutoML framework's evaluation suite. #### 🔔 Monitor new datasets via webhook Configure an Apify webhook to trigger your downstream pipeline whenever new datasets matching your filter are found. Useful for ML research groups that want to stay current with new public benchmarks. ### API usage #### Node.js ```javascript import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: 'YOUR_API_TOKEN' }); const run = await client.actor('automation-lab/openml-scraper').call({ resourceType: 'datasets', searchQuery: 'mnist', status: 'active', maxResults: 50, }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); console.log(items); ``` #### Python ```python from apify_client import ApifyClient client = ApifyClient(token="YOUR_API_TOKEN") run = client.actor("automation-lab/openml-scraper").call(run_input={ "resourceType": "datasets", "searchQuery": "mnist", "status": "active", "maxResults": 50, }) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) ``` #### cURL ```bash curl -X POST \ "https://api.apify.com/v2/acts/automation-lab~openml-scraper/runs?token=YOUR_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "resourceType": "datasets", "searchQuery": "iris", "status": "active", "maxResults": 10 }' ``` ### Use with Claude and MCP (AI agent access) This actor is available as an **MCP (Model Context Protocol) tool**, letting AI agents like Claude query OpenML datasets directly in conversation. #### Claude Code (terminal) ```bash claude mcp add --transport http apify "https://mcp.apify.com?tools=automation-lab/openml-scraper" ``` #### Claude Desktop / Cursor / VS Code Add to your MCP config file: ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com?tools=automation-lab/openml-scraper", "headers": { "Authorization": "Bearer YOUR_API_TOKEN" } } } } ``` **Example prompts for Claude:** - *"Find all active OpenML datasets with 'breast cancer' in the name"* - *"Get 100 OpenML benchmark datasets with at least 1000 instances"* - *"List the first 20 supervised classification tasks on OpenML"* - *"Find all scikit-learn algorithm flows on OpenML"* ### Legality and terms of service OpenML data is publicly available under the [OpenML terms of service](https://openml.org/terms). The datasets themselves are shared under various open licences (Public Domain, CC BY, etc.) which are included in the `licence` field. This actor only accesses the public REST API using documented endpoints — no scraping of HTML content. Commercial use of the data depends on individual dataset licences. ### FAQ **Q: Why does the flow search take a long time?** A: OpenML's API doesn't support server-side name filtering for flows. The actor paginates through all flows and filters client-side. With 20,000+ flows, this can take 30–120 seconds. For fast results on flows, set `maxResults` to 50–100 and omit the `searchQuery` to get the latest flows by ID. **Q: The actor returned fewer results than my maxResults — why?** A: OpenML may not have that many resources matching your filter. For example, searching for `iris` as a dataset name returns ~5 datasets (multiple versions). This is expected behavior. **Q: How do I get the actual dataset file (ARFF/CSV)?** A: Each result includes a `downloadUrl` field with the direct ARFF download link. You can use this in your ML framework (e.g., `arff.load()` in Python, or pass directly to OpenML Python client). **Q: Can I filter datasets by minimum number of instances or features?** A: Not directly via the actor input. Run the actor with no filter to get all datasets, then filter in your downstream pipeline using the `numberOfInstances` and `numberOfFeatures` fields. **Q: The description field is truncated — can I get the full description?** A: The description is truncated at 500 characters to keep dataset sizes manageable. OpenML descriptions can be several kilobytes of text. If you need full descriptions, use the `id` field to call `https://www.openml.org/api/v1/json/data/{id}` directly. ### Related scrapers - [ACL Anthology Scraper](https://apify.com/automation-lab/acl-anthology-scraper) — scrape NLP/ML research papers from the ACL Anthology - [ArXiv Paper Scraper](https://apify.com/automation-lab/arxiv-scraper) — extract ML and AI paper metadata from arXiv # Actor input Schema ## `resourceType` (type: `string`): Which type of OpenML resource to scrape: datasets (ML benchmark datasets), tasks (supervised classification/regression tasks), or flows (ML algorithm implementations). ## `searchQuery` (type: `string`): Filter datasets or flows by name. Leave empty to return all. Example: 'iris', 'mnist', 'sklearn'. ## `status` (type: `string`): Filter datasets by status. Only applies when Resource type is 'datasets'. ## `maxResults` (type: `integer`): Maximum number of results to return. Larger values cost proportionally more. ## `maxRequestRetries` (type: `integer`): Number of retry attempts for failed HTTP requests. ## Actor input object example ```json { "resourceType": "datasets", "searchQuery": "iris", "status": "active", "maxResults": 20, "maxRequestRetries": 3 } ``` # Actor output Schema ## `overview` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "resourceType": "datasets", "searchQuery": "iris", "status": "active", "maxResults": 20, "maxRequestRetries": 3 }; // Run the Actor and wait for it to finish const run = await client.actor("automation-lab/openml-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "resourceType": "datasets", "searchQuery": "iris", "status": "active", "maxResults": 20, "maxRequestRetries": 3, } # Run the Actor and wait for it to finish run = client.actor("automation-lab/openml-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "resourceType": "datasets", "searchQuery": "iris", "status": "active", "maxResults": 20, "maxRequestRetries": 3 }' | apify call automation-lab/openml-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=automation-lab/openml-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "OpenML Dataset Scraper", "description": "Scrape ML datasets, tasks, flows, and runs from OpenML - the open science platform for machine learning", "version": "0.1", "x-build-id": "miEJX9cD0w0i9SnBW" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/automation-lab~openml-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-automation-lab-openml-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/automation-lab~openml-scraper/runs": { "post": { "operationId": "runs-sync-automation-lab-openml-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/automation-lab~openml-scraper/run-sync": { "post": { "operationId": "run-sync-automation-lab-openml-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "resourceType": { "title": "📦 Resource type", "enum": [ "datasets", "tasks", "flows" ], "type": "string", "description": "Which type of OpenML resource to scrape: datasets (ML benchmark datasets), tasks (supervised classification/regression tasks), or flows (ML algorithm implementations).", "default": "datasets" }, "searchQuery": { "title": "🔍 Search by name", "type": "string", "description": "Filter datasets or flows by name. Leave empty to return all. Example: 'iris', 'mnist', 'sklearn'." }, "status": { "title": "🟢 Dataset status filter", "enum": [ "active", "deactivated", "in_preparation", "any" ], "type": "string", "description": "Filter datasets by status. Only applies when Resource type is 'datasets'.", "default": "active" }, "maxResults": { "title": "📏 Max results", "minimum": 1, "maximum": 10000, "type": "integer", "description": "Maximum number of results to return. Larger values cost proportionally more.", "default": 100 }, "maxRequestRetries": { "title": "🔁 Max request retries", "minimum": 1, "maximum": 10, "type": "integer", "description": "Number of retry attempts for failed HTTP requests.", "default": 3 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```