Dataset Contract & Regression Tester

Stop broken dataset changes before they break a pipeline.

This Actor infers or enforces a versioned dataset contract, validates every JSON item, detects duplicate composite keys, and compares the current contract with a previous version. It produces compact, evidence-rich item results and a suite result that can gate a deployment, scraper release, ETL job, dashboard, or RAG ingestion workflow.

It is deterministic, requires no external API key, does not browse the web, and does not send your data to an AI model.

What it catches

• missing required fields
• undeclared extra fields
• string, number, integer, boolean, object, and array type drift
• null values in non-nullable fields
• duplicate single-field or composite keys
• fields added or removed between contract versions
• type, required, and nullability changes
• duplicate-key or extra-field policy changes
• structural contract changes made without a version bump

Each finding identifies the rule, field path, expected state, actual state, and a short explanation. Optional mismatched-value previews are capped at 120 characters. Duplicate key values are represented only by a short SHA-256 fingerprint.

Quick start

Run with {} or no input to use the safe built-in sample. It demonstrates valid records, a missing field, an unexpected field, a type change, null handling, a duplicate key, and schema changes between contract versions.

For a real dataset, select it in Dataset to test and provide a contract:

{
  "datasetId": "YOUR_DATASET_ID",
  "contract": {
    "schemaVersion": "dataset-contract-v1",
    "name": "orders",
    "version": "2.0.0",
    "fields": {
      "id": { "type": "string", "required": true, "nullable": false },
      "total": { "type": "number", "required": true, "nullable": false },
      "currency": { "type": "string", "required": true, "nullable": false },
      "paid": { "type": "boolean", "required": true, "nullable": false }
    },
    "keyFields": ["id"],
    "allowExtraFields": false
  },
  "maxItems": 500
}

You can paste items, choose datasetId, or combine both. Inline items are evaluated first. Work stops at maxItems or the run's pay-per-event spending limit, whichever comes first.

Infer a contract

Omit contract to infer one from the selected items:

{
  "datasetId": "YOUR_DATASET_ID",
  "contractName": "customer-export",
  "contractVersion": "1.0.0",
  "requiredPresenceRatio": 1,
  "keyFields": ["customerId"],
  "allowExtraFields": true
}

Inference observes all items loaded for the run. A field becomes required when its presence ratio reaches requiredPresenceRatio. Observed non-null types become its accepted types, and observed nulls make it nullable. If no keys are supplied, the Actor tries a fully present, unique id, _id, url, email, or key field.

Save the normalized CONTRACT output as a reviewed baseline. For stable production and CI checks, pass that explicit contract on later runs; do not rely on fresh inference to catch drift in the same batch.

Compare contract versions

Pass baselineContract alongside the current contract. The suite result reports additive and breaking changes with before/after evidence:

{
  "items": [{ "id": "a", "score": 42, "active": true }],
  "contract": {
    "name": "scores",
    "version": "2.0.0",
    "fields": {
      "id": { "type": "string", "required": true },
      "score": { "type": "integer", "required": true },
      "active": { "type": "boolean", "required": true }
    },
    "keyFields": ["id"],
    "allowExtraFields": false
  },
  "baselineContract": {
    "name": "scores",
    "version": "1.0.0",
    "fields": {
      "id": { "type": "string", "required": true },
      "score": { "type": "number", "required": true }
    },
    "keyFields": ["id"],
    "allowExtraFields": false
  }
}

The comparison is semantic rather than textual. Field ordering does not matter.

Output

The default dataset contains:

1. One recordType: "item" result for each evaluated item.
2. One uncharged recordType: "suite" result with totals and schema changes.

The OUTPUT key-value-store record contains the same suite metrics plus source information, the effective contract, run timestamps, and a budget-limit flag. CONTRACT contains only the normalized effective contract.

A failing contract check is still a successful evaluation:

{
  "recordType": "item",
  "itemIndex": 2,
  "itemKey": "sha256:70cd1f58c55d65cb",
  "status": "failed",
  "violationCount": 2,
  "violations": [
    {
      "rule": "type-mismatch",
      "path": "score",
      "expected": "integer",
      "actual": "string",
      "valuePreview": "unknown",
      "evidence": "Field \"score\" has type string; expected integer."
    }
  ],
  "valueEvent": "item-validated"
}

Use the suite status, itemsFailed, processingErrors, totalViolations, or error-severity schemaChanges as CI gates.

Pricing and charging

Contract violations are successful paid evaluations because the finding is the product. Internal processing failures are returned with valueEvent: null and are uncharged. The suite dataset record, OUTPUT summary, effective CONTRACT, input validation, and schema comparison are also uncharged.

Apify tier	Price per evaluated item
Free	$0.0020
Bronze	$0.0018
Silver	$0.0016
Gold	$0.0014
Platinum and Diamond	$0.0012

A one-time $0.0003 Actor-start event covers measured startup overhead, including runs stopped by invalid input. A 500-item Free-tier test has a maximum event price of $1.0003. The Actor calculates the allowed value-event count before processing and sets limitedByBudget: true when a run limit truncates work.

Privacy and permissions

• Limited permissions: read only the dataset you select; write only the run's default output dataset and key-value store.
• No web requests, proxies, cookies, credentials, tracking pixels, or external analytics.
• No external AI or third-party enrichment service.
• Original source rows are never copied to output.
• Only violating values can produce a preview, capped at 120 characters; set includeValueEvidence: false to suppress all previews.
• Composite duplicate keys are hashed before output and logs never include item contents.
• Input and output remain subject to your Apify account's storage and retention settings.

If your dataset contains private, regulated, or customer data, disable value evidence, use appropriate Apify storage retention, and confirm your organization's data-handling requirements before running it.

Important limitations

• Contracts validate top-level fields. Nested objects and arrays are type-checked as objects or arrays, but nested paths and array element schemas are not recursively validated in version 0.1.
• A number contract accepts integers; an integer contract rejects non-integer numbers.
• Inference describes the loaded sample, not the unseen source population. Increase maxItems, use representative data, and review the inferred contract before adopting it.
• Duplicate detection is exact after JSON serialization of configured key values. It does not fuzzy-match names, normalize emails, or merge near-duplicates.
• Structural validity does not prove that values are current, truthful, legally usable, or semantically correct.

API and automation

curl -X POST \
  "https://api.apify.com/v2/acts/gifted_wagon~dataset-contract-tester/runs" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"datasetId":"YOUR_DATASET_ID","contractName":"nightly-export","contractVersion":"1.0.0"}'

Run it after a scraper, on a schedule, from GitHub Actions, or through Apify webhooks, Make, Zapier, n8n, an API client, or the hosted Apify MCP server.

Support

Open an issue from the Actor's Issues tab and include the run ID, contract version, redacted contract, and redacted violation. Never post private rows, credentials, access tokens, or unhashed personal identifiers in a public issue.