# Apify Dataset Quality Gate & Validation Integration (`quanmatrix/dataset-quality-gate-integration`) Actor

Validate any Apify dataset before downstream use with required-field, duplicate, null-rate and minimum-size checks, producing a structured pass/fail quality score for Actor-to-Actor workflows.

- **URL**: https://apify.com/quanmatrix/dataset-quality-gate-integration.md
- **Developed by:** [Rafael Barreto Haddad](https://apify.com/quanmatrix) (community)
- **Categories:** Developer tools, Automation, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $3.50 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Apify Dataset Quality Gate & Validation Integration

Validate an Apify dataset before it is sent to a database, spreadsheet, CRM, AI/RAG pipeline, webhook workflow, or another Actor. The Actor is designed as a reusable quality gate inside Apify rather than as a generic JSON validator.

### Why use this Actor

Scrapers and extraction Actors can succeed technically while still returning an empty dataset, missing required fields, duplicate identifiers, or a sudden rise in null values. Those problems often become more expensive after the data reaches a database or an AI pipeline. This Actor adds a simple decision point between collection and downstream use. It evaluates the dataset and returns a structured `passed` decision, a 0–100 quality score, and explicit reasons when the gate fails.

### Key features

- Native Actor-to-Actor integration workflow.
- Automatically reads `payload.resource.defaultDatasetId` when invoked from an Apify integration.
- Accepts a direct `datasetId` for manual or advanced runs.
- Supports inline JSON items for deterministic testing and small manual checks.
- Processes remote datasets in configurable batches instead of loading the whole dataset into memory.
- Validates minimum item count, required-field completeness, duplicate rate, and an overall minimum quality score.
- Returns field-level completeness percentages and explicit issues/warnings.
- Does not modify the source dataset.

### Input

The most important fields are `datasetId`, `requiredFields`, `uniqueField`, `maxDuplicateRatePct`, `maxNullRatePct`, `minItems`, `minimumQualityScore`, `maxItems`, and `batchSize`. When the Actor is triggered as an integration after another Actor run, `datasetId` can be omitted because the upstream default dataset ID is read from the integration payload automatically.

### Output

The Actor writes one quality report to its own default dataset. The report includes `passed`, `quality_score`, `items_evaluated`, required-field null rate, duplicate count/rate, per-field completeness, issues, warnings, source dataset ID, and evaluation timestamp.

### Example

A typical pipeline is:

`Scraper -> Dataset Quality Gate -> PostgreSQL / Google Sheets / CRM / AI pipeline`

For a product dataset, require `id`, `name`, and `price`, set `id` as the unique field, allow at most 1% duplicates, and require a quality score of at least 90. If the upstream Actor is used through Apify Integrations, the dataset ID is supplied automatically.

### Use cases

- Pre-database validation before inserts or upserts.
- RAG and AI ingestion quality control.
- Scheduled scraper regression detection.
- CRM and lead-data quality gates.
- Product, jobs, review, real-estate, and market-data validation.
- Release QA for Actors whose output schema or completeness can drift.
- Protection against empty datasets silently propagating through automation.

### Pricing

The Actor uses pay-per-event pricing with one clear billable unit: one completed dataset quality-gate report written to the default dataset. The design intentionally avoids charging separately for individual checks because they are parts of the same quality decision.

### Limitations

- The first version evaluates top-level fields only; nested dot-path validation is not yet implemented.
- It evaluates up to the configured `maxItems` safety limit, so the score represents that evaluated portion when the source dataset is larger.
- It reports quality problems but does not repair or mutate the source dataset.
- A quality score is a transparent operational heuristic, not a legal, compliance, or statistical certification.

### Schema drift and contract fingerprints

Each evaluation creates a deterministic SHA-256 fingerprint from the observed top-level field/type contract. Store that fingerprint from a healthy run and supply it as `previousSchemaFingerprint` later. The Actor reports drift and can optionally make drift a gate failure. This catches a common pipeline failure mode where a scraper still returns rows but the shape of those rows changed.

### Type and format rules

Use `expectedSchema` to require JSON types and `formatRules` for URL, email, date, or datetime checks. Multiple `uniqueFields` can be evaluated in the same run. Field profiles expose completeness and observed type distributions, making the quality decision auditable instead of opaque.

### Pipeline stopping

With `failRunOnError=false`, the Actor behaves as a reporting audit. With `failRunOnError=true`, it writes the diagnostic dataset item first and then fails the run if the gate did not pass, allowing webhook and integration pipelines to stop before bad data is written downstream.

# Actor input Schema

## `datasetId` (type: `string`):

Optional Apify dataset ID. Leave empty when the Actor is triggered as an integration; it will use payload.resource.defaultDatasetId automatically.

## `items` (type: `array`):

Optional inline JSON objects for testing or small manual checks. If supplied, these are evaluated instead of a remote dataset.

## `requiredFields` (type: `array`):

Field names that must be present and non-null in every evaluated item. Dot-path nesting is not interpreted; use top-level fields.

## `uniqueField` (type: `string`):

Optional top-level field whose duplicate rate should be measured, such as id, url, email, or product\_id.

## `maxDuplicateRatePct` (type: `number`):

Fail the quality gate when the duplicate rate for uniqueField exceeds this percentage.

## `maxNullRatePct` (type: `number`):

Fail when the combined missing/null rate across required fields exceeds this percentage.

## `minItems` (type: `integer`):

Minimum number of evaluated items required for the dataset to pass.

## `minimumQualityScore` (type: `number`):

Minimum final 0–100 quality score required to pass.

## `maxItems` (type: `integer`):

Safety cap for this run. Remote datasets are processed in batches up to this many items.

## `batchSize` (type: `integer`):

Number of remote dataset items read per batch.

## `uniqueFields` (type: `array`):

Top-level fields that should be unique across evaluated records, such as id, url, sku, or email.

## `expectedSchema` (type: `object`):

Optional JSON object mapping field names to expected JSON types: string, number, integer, boolean, object, array, or null.

## `formatRules` (type: `object`):

Optional JSON object mapping fields to format checks: url, email, date, or datetime.

## `maxFormatErrorRatePct` (type: `number`):

Fail if configured format-rule errors exceed this percentage of checked values.

## `previousSchemaFingerprint` (type: `string`):

Optional fingerprint from a previous successful run. When supplied, the Actor reports whether the observed top-level schema drifted.

## `failOnSchemaDrift` (type: `boolean`):

Treat a changed schema fingerprint as a quality-gate failure when previousSchemaFingerprint is supplied.

## `failRunOnError` (type: `boolean`):

After writing the quality report, fail the Actor run if the quality gate did not pass. Useful for CI/CD and downstream integration stopping.

## `itemOrder` (type: `string`):

Evaluate earliest or latest stored records first.

## `itemOffset` (type: `integer`):

Skip this many records before evaluation.

## `validateAllItems` (type: `boolean`):

When enabled, keep reading batches until the dataset ends or maxItems is reached. When disabled, evaluate one batch only.

## Actor input object example

```json
{
  "requiredFields": [],
  "maxDuplicateRatePct": 1,
  "maxNullRatePct": 5,
  "minItems": 1,
  "minimumQualityScore": 80,
  "maxItems": 10000,
  "batchSize": 500,
  "uniqueFields": [],
  "expectedSchema": {},
  "formatRules": {},
  "maxFormatErrorRatePct": 0,
  "failOnSchemaDrift": false,
  "failRunOnError": false,
  "itemOrder": "first",
  "itemOffset": 0,
  "validateAllItems": true
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("quanmatrix/dataset-quality-gate-integration").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("quanmatrix/dataset-quality-gate-integration").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call quanmatrix/dataset-quality-gate-integration --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,quanmatrix/dataset-quality-gate-integration"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/0Pmh13MSYanieLDAl/builds/miK9DZTdJbCYB5goM/openapi.json