Apify Dataset Release QA Gate

Audit pasted Apify-style dataset rows before you publish or trust an Actor output. The Actor produces one structured release-readiness report with blockers, warnings, scores, and sampled problem rows.

What it does

• Checks required fields for missing or empty values.
• Detects duplicate rows using one or more top-level unique key fields.
• Normalizes domain and URL-like keys for duplicate detection.
• Reports field coverage for inspected fields.
• Flags type drift across rows, such as a field switching between number, string, and null.
• Validates configured URL/domain fields and email fields.
• Pushes a single summary QA report item to the default dataset.

Input

This MVP supports items mode only.

Required:

• items: non-empty array of dataset row objects to audit.

Optional:

• requiredFields: top-level field names expected to be present and non-empty.
• uniqueKeys: top-level field names used as a composite duplicate key.
• urlFields: top-level field names expected to contain URLs or domains.
• emailFields: top-level field names expected to contain emails.
• maxRows: maximum rows to inspect. Default: 1000.
• strictMode: when true, warnings become release blockers.

Example input:

{
  "items": [
    {
      "name": "Acme AI",
      "domain": "https://acme.ai",
      "email": "hello@acme.ai"
    },
    {
      "name": "Acme AI Inc.",
      "domain": "acme.ai",
      "email": ""
    }
  ],
  "requiredFields": ["name", "domain"],
  "uniqueKeys": ["domain"],
  "urlFields": ["domain"],
  "emailFields": ["email"]
}

Output

The Actor always emits one summary dataset item for any valid non-empty items input.

The default dataset is documented by .actor/dataset_schema.json for the Store/API output tab. The overview table highlights the release grade, score, row count, duplicate group count, blockers, warnings, recommendations, sample problem rows, and check timestamp.

Key output fields:

• rowCount
• overallScore
• grade: PASS, WARN, or FAIL
• blockers
• warnings
• recommendations
• duplicateGroups
• missingRequiredCounts
• fieldCoverage
• typeDriftWarnings
• urlWarnings
• emailWarnings
• sampleProblemRows

Example output shape:

{
  "rowCount": 5,
  "overallScore": 58,
  "grade": "FAIL",
  "blockers": ["Required field \"name\" is missing in 40% of inspected rows."],
  "duplicateCount": 1,
  "missingRequiredCounts": { "name": 2, "domain": 0 }
}

Scoring

The score starts at 100 and subtracts capped penalties for missing required values, duplicate groups, URL/domain warnings, email warnings, and type drift. A report fails when blockers exist or the score drops below 60.

MVP limitations

• It does not fetch Apify datasets by ID.
• It does not call the Apify cloud API.
• It does not scrape websites.
• It does not use AI/LLM.
• It does not clean or mutate data; it reports quality signals only.
• It does not guarantee Store approval, legal compliance, deliverability, lead accuracy, or commercial usefulness.
• It supports top-level fields only for required-field, unique-key, URL/domain, and email checks.

Running it

In Apify Console, start with the prefilled input in Apify Console or paste rows using the example input shape above. The prefilled sample intentionally contains quality issues, so a FAIL report is expected and proves the audit path is working.

For local source checkout verification:

npm install
npm test
apify validate-schema

To run locally without Apify storage, save the example input as input.json and run:

$node src/main.js --input input.json --output output.json