Apify Dataset Quality Gate & Validator

Check Apify datasets before bad records reach a CRM, database or AI pipeline. Detect missing fields, duplicate identifiers, schema drift and low-quality data. Get an explainable quality score and pass/fail decision; optionally stop a failing pipeline. Supports Apify integrations, APIs and agents.

Who this helps

For developers and data operations teams that cannot afford a scraper's malformed output to silently reach a CRM, database or AI system. The Actor checks data quality and returns an explicit decision; it does not modify the original dataset.

Example request for an AI agent

Run the Apify Dataset Quality Gate & Validator on this dataset. Require the fields id and name, check that id is unique, and require a minimum quality score of 90. Return the pass/fail result, reasons and any schema-drift signal before continuing my pipeline.

Supply datasetId when checking an existing Apify dataset or items for a small inline test. Set failRunOnError only if the pipeline is designed to stop on validation failure.

Business use and first result

For: Apify dataset engineers.

Why run it: Apply explainable validation rules to an authorized upstream Dataset before downstream use.

What to inspect: Schema-drift and rule-validation diagnostics with pass/fail outcomes.

First run: use the documented sample input below, inspect the Dataset rows and source links, then reuse the saved output for a later comparison only where snapshots are supported. Confirm the input scope and current Pay Per Event pricing before scheduling ongoing use.

Scope and limitations: Current checks focus on top-level fields; nested validation may require additional checks.

Quick start

Start with this working example and replace the target values with your own:

{
  "items": [
    {
      "id": 1,
      "name": "Alpha"
    },
    {
      "id": 2,
      "name": "Beta"
    }
  ],
  "requiredFields": [
    "id",
    "name"
  ],
  "uniqueField": "id",
  "maxDuplicateRatePct": 0,
  "maxNullRatePct": 0,
  "minimumQualityScore": 80
}

The Actor writes structured results to the default Apify Dataset and can be used from the Store, API, schedules, Tasks, automations and MCP-compatible AI workflows.

Input

• datasetId — Dataset ID (advanced/manual): Optional Apify dataset ID. Leave empty when the Actor is triggered as an integration; it will use payload.resource.defaultDatasetId automatically.
• items — Inline items (test/manual): Optional inline JSON objects for testing or small manual checks. If supplied, these are evaluated instead of a remote dataset.
• requiredFields — Required fields: Field names that must be present and non-null in every evaluated item. Dot-path nesting is not interpreted; use top-level fields.
• uniqueField — Unique field: Optional top-level field whose duplicate rate should be measured, such as id, url, email, or product_id.
• maxDuplicateRatePct — Maximum duplicate rate (%): Fail the quality gate when the duplicate rate for uniqueField exceeds this percentage.
• maxNullRatePct — Maximum required-field null rate (%): Fail when the combined missing/null rate across required fields exceeds this percentage.

All integration and advanced analysis fields are optional. The default example is intentionally runnable without configuring MCP or a previous-run baseline.

Output

The Dataset exposes predictable machine-readable output. Representative fields include results.

Pricing

This Actor uses Pay Per Event. The current factory base price is $0.005000 per result event. The Apify Store remains the source of truth for the price and plan/tier details shown to the buyer.

Use cases

• Run the buyer-ready workflows exposed as Apify Tasks without preparing a custom integration first.
• Use the Actor from API or schedules for recurring collection, comparison or monitoring.
• Feed the structured Dataset output into spreadsheets, databases, automations or AI agents.
• Compare repeat runs when the product supports snapshots or previous-run inputs.

Automation and AI

Use the same Actor through Apify API, schedules, public Tasks and the Apify MCP server. Outputs are structured for downstream workflows and AI agents rather than requiring manual copy/paste.

Limitations

• Public websites and upstream APIs can change markup, access rules, rate limits or field availability without notice.
• Fields that are not publicly available are returned as unavailable or omitted rather than fabricated.
• Analytical outputs depend on the quality and coverage of the supplied or collected source data.
• Treat marketplace, reputation, workforce, safety or commercial signals as decision support and validate material decisions against the underlying source evidence.

Detailed documentation

Validate an Apify dataset before it is sent to a database, spreadsheet, CRM, AI/RAG pipeline, webhook workflow, or another Actor. The Actor is designed as a reusable quality gate inside Apify rather than as a generic JSON validator.

Why use this Actor

Scrapers and extraction Actors can succeed technically while still returning an empty dataset, missing required fields, duplicate identifiers, or a sudden rise in null values. Those problems often become more expensive after the data reaches a database or an AI pipeline. This Actor adds a simple decision point between collection and downstream use. It evaluates the dataset and returns a structured passed decision, a 0–100 quality score, and explicit reasons when the gate fails.

Key features

• Native Actor-to-Actor integration workflow.
• Automatically reads payload.resource.defaultDatasetId when invoked from an Apify integration.
• Accepts a direct datasetId for manual or advanced runs.
• Supports inline JSON items for deterministic testing and small manual checks.
• Processes remote datasets in configurable batches instead of loading the whole dataset into memory.
• Validates minimum item count, required-field completeness, duplicate rate, and an overall minimum quality score.
• Returns field-level completeness percentages and explicit issues/warnings.
• Does not modify the source dataset.

Input

The most important fields are datasetId, requiredFields, uniqueField, maxDuplicateRatePct, maxNullRatePct, minItems, minimumQualityScore, maxItems, and batchSize. When the Actor is triggered as an integration after another Actor run, datasetId can be omitted because the upstream default dataset ID is read from the integration payload automatically.

Output

The Actor writes one quality report to its own default dataset. The report includes passed, quality_score, items_evaluated, required-field null rate, duplicate count/rate, per-field completeness, issues, warnings, source dataset ID, and evaluation timestamp.

Example

A typical pipeline is:

Scraper -> Dataset Quality Gate -> PostgreSQL / Google Sheets / CRM / AI pipeline

For a product dataset, require id, name, and price, set id as the unique field, allow at most 1% duplicates, and require a quality score of at least 90. If the upstream Actor is used through Apify Integrations, the dataset ID is supplied automatically.

Use cases

• Pre-database validation before inserts or upserts.
• RAG and AI ingestion quality control.
• Scheduled scraper regression detection.
• CRM and lead-data quality gates.
• Product, jobs, review, real-estate, and market-data validation.
• Release QA for Actors whose output schema or completeness can drift.
• Protection against empty datasets silently propagating through automation.

Pricing

The Actor uses pay-per-event pricing with one clear billable unit: one completed dataset quality-gate report written to the default dataset. The design intentionally avoids charging separately for individual checks because they are parts of the same quality decision.

Limitations

• The first version evaluates top-level fields only; nested dot-path validation is not yet implemented.
• It evaluates up to the configured maxItems safety limit, so the score represents that evaluated portion when the source dataset is larger.
• It reports quality problems but does not repair or mutate the source dataset.
• A quality score is a transparent operational heuristic, not a legal, compliance, or statistical certification.

Schema drift and contract fingerprints

Each evaluation creates a deterministic SHA-256 fingerprint from the observed top-level field/type contract. Store that fingerprint from a healthy run and supply it as previousSchemaFingerprint later. The Actor reports drift and can optionally make drift a gate failure. This catches a common pipeline failure mode where a scraper still returns rows but the shape of those rows changed.

Type and format rules

Use expectedSchema to require JSON types and formatRules for URL, email, date, or datetime checks. Multiple uniqueFields can be evaluated in the same run. Field profiles expose completeness and observed type distributions, making the quality decision auditable instead of opaque.

Pipeline stopping

With failRunOnError=false, the Actor behaves as a reporting audit. With failRunOnError=true, it writes the diagnostic dataset item first and then fails the run if the gate did not pass, allowing webhook and integration pipelines to stop before bad data is written downstream.

Gen2 decision intelligence

This Actor preserves its original analysis and adds a decision layer with baseline awareness, regression detection, confidence, GO/WARN/BLOCK executive output, and an optional economic-impact estimate. Economic estimates are produced only when the user supplies valuePerImpactUnitUsd; the result states the calculation basis instead of inventing monetary value.

Agent execution and AI discovery (October 2026)

Use this Actor when: Apply explainable validation rules to an authorized upstream Dataset before downstream use. Output: Schema-drift and rule-validation diagnostics with pass/fail outcomes.

For AI agents and automations:

• Find this Actor in Apify Store by its use case and verify its actual input and pricing through Apify MCP (search-actors and fetch-actor-details). Availability and ranking in Apify AI are controlled by Apify; discovery is not guaranteed.
• Read the Input tab or input schema. Recognized fields include datasetId, items, requiredFields, uniqueField, maxDuplicateRatePct, maxNullRatePct, minItems, minimumQualityScore. Required fields: See the Input tab for valid example values. Do not invent records or credentials.
• Supply a small representative input first. Use Apify API, Tasks, or MCP to run and retrieve the structured dataset; inspect the output schema rather than assuming all inputs produce the same rows.
• For programmatic calls, set timeouts and maxTotalChargeUsd in run options to limit exposure. Check the Apify Pricing tab for billable events and discounts.
• Payment compatibility: this Actor is published with limited permissions and pay-per-event pricing. Apify AGI prepaid x402 tokens can execute limited-permission Actors. Direct per-request payment is a separate, conditional feature that may require additional eligibility checks and creator verification. Do not interpret this paragraph as proof of approved KYC or guaranteed payment acceptance.
• This Actor does not independently guarantee real-time access to a third-party source. Verify input provenance and important conclusions, and do not supply confidential personal data unnecessarily.

Reliability update — 2026-10-24

• Correct completeness counts for fields first encountered after the first row.