Apify Dataset Quality Gate & Validation Integration

Validate an Apify dataset before it is sent to a database, spreadsheet, CRM, AI/RAG pipeline, webhook workflow, or another Actor. The Actor is designed as a reusable quality gate inside Apify rather than as a generic JSON validator.

Why use this Actor

Scrapers and extraction Actors can succeed technically while still returning an empty dataset, missing required fields, duplicate identifiers, or a sudden rise in null values. Those problems often become more expensive after the data reaches a database or an AI pipeline. This Actor adds a simple decision point between collection and downstream use. It evaluates the dataset and returns a structured passed decision, a 0–100 quality score, and explicit reasons when the gate fails.

Key features

• Native Actor-to-Actor integration workflow.
• Automatically reads payload.resource.defaultDatasetId when invoked from an Apify integration.
• Accepts a direct datasetId for manual or advanced runs.
• Supports inline JSON items for deterministic testing and small manual checks.
• Processes remote datasets in configurable batches instead of loading the whole dataset into memory.
• Validates minimum item count, required-field completeness, duplicate rate, and an overall minimum quality score.
• Returns field-level completeness percentages and explicit issues/warnings.
• Does not modify the source dataset.

Input

The most important fields are datasetId, requiredFields, uniqueField, maxDuplicateRatePct, maxNullRatePct, minItems, minimumQualityScore, maxItems, and batchSize. When the Actor is triggered as an integration after another Actor run, datasetId can be omitted because the upstream default dataset ID is read from the integration payload automatically.

Output

The Actor writes one quality report to its own default dataset. The report includes passed, quality_score, items_evaluated, required-field null rate, duplicate count/rate, per-field completeness, issues, warnings, source dataset ID, and evaluation timestamp.

Example

A typical pipeline is:

Scraper -> Dataset Quality Gate -> PostgreSQL / Google Sheets / CRM / AI pipeline

For a product dataset, require id, name, and price, set id as the unique field, allow at most 1% duplicates, and require a quality score of at least 90. If the upstream Actor is used through Apify Integrations, the dataset ID is supplied automatically.

Use cases

• Pre-database validation before inserts or upserts.
• RAG and AI ingestion quality control.
• Scheduled scraper regression detection.
• CRM and lead-data quality gates.
• Product, jobs, review, real-estate, and market-data validation.
• Release QA for Actors whose output schema or completeness can drift.
• Protection against empty datasets silently propagating through automation.

Pricing

The Actor uses pay-per-event pricing with one clear billable unit: one completed dataset quality-gate report written to the default dataset. The design intentionally avoids charging separately for individual checks because they are parts of the same quality decision.

Limitations

• The first version evaluates top-level fields only; nested dot-path validation is not yet implemented.
• It evaluates up to the configured maxItems safety limit, so the score represents that evaluated portion when the source dataset is larger.
• It reports quality problems but does not repair or mutate the source dataset.
• A quality score is a transparent operational heuristic, not a legal, compliance, or statistical certification.

Schema drift and contract fingerprints

Each evaluation creates a deterministic SHA-256 fingerprint from the observed top-level field/type contract. Store that fingerprint from a healthy run and supply it as previousSchemaFingerprint later. The Actor reports drift and can optionally make drift a gate failure. This catches a common pipeline failure mode where a scraper still returns rows but the shape of those rows changed.

Type and format rules

Use expectedSchema to require JSON types and formatRules for URL, email, date, or datetime checks. Multiple uniqueFields can be evaluated in the same run. Field profiles expose completeness and observed type distributions, making the quality decision auditable instead of opaque.

Pipeline stopping

With failRunOnError=false, the Actor behaves as a reporting audit. With failRunOnError=true, it writes the diagnostic dataset item first and then fails the run if the gate did not pass, allowing webhook and integration pipelines to stop before bad data is written downstream.