Data Validator & Profiler

Validate and profile CSV, JSON, and XML data — infer schema, detect anomalies, get stats.

Drop in raw CSV, JSON, or XML and get back a full column-by-column profile: types, null ratios, uniqueness, min/max/mean/median/stdev for numeric columns, and anomaly flags. Batch mode handles up to 50 datasets in a single run — each one parsed and profiled independently.

I built this because I was tired of manually inspecting data files. You get a CSV from a client, a JSON dump from an API, or an XML export from some legacy system — and you need to know what's in it before you write a line of ETL. This actor answers those questions in seconds.

---

What does it do?

You give it raw data — a CSV string, a JSON array, or a flat XML structure. It auto-detects the format, parses the rows, then profiles every column.

For each column you get:

• Row count — how many rows of data.
• Null count & percentage — counts both null and empty strings as missing.
• Unique values — count and percentage. Tells you if a column is a candidate key.
• Detected type — int, float, str, bool, number (mixed int/float), mixed, or unknown (all-null).
• Numeric stats — min, max, mean, median, and standard deviation when the column has enough numeric values.
• Anomaly flags — marks columns with >50% nulls (high_nulls, severity: warn) and columns with mixed types (mixed_types, severity: info).

If you turn off schema inference (inferSchema: false), you get a lighter result — just row count, column count, column names, and validity. Faster if you only need a parse check.

Batch mode takes an array of datasets (up to 50), profiles each one independently, and pushes results as separate dataset items. One broken dataset won't kill the run for the rest.

---

Features

1. Format auto-detection — JSON (starts with { or [), XML (starts with <), everything else assumed CSV. You can override with the format field.
2. Schema inference — column-level type detection, null ratio, uniqueness scoring, numeric stats.
3. Anomaly flagging — high_nulls at >50% nulls, mixed_types when a column holds values of multiple incompatible types.
4. Numeric profiling — min, max, mean, median, stdev for columns where at least 2 values are numeric (stdev needs 4+ values with variance). Non-numeric values are skipped.
5. Batch mode — profile up to 50 datasets in one run. Each gets its own output item with a dataset_index.
6. Error resilience — parse failures produce valid: false + error message. Batch mode keeps going.

---

Who is it for?

Persona	What they use it for
Data Engineer	Validating upstream data quality before ingestion pipelines
QA Engineer	Checking data exports for regressions and format issues
Data Analyst	Understanding unknown datasets from clients or partners
ETL Developer	Profiling source data before writing transformation logic
DevOps Engineer	Integrating data validation into CI/CD quality gates
Data Onboarding Specialist	Quickly assessing the shape and quality of new data deliveries

---

Input Parameters

Field	Type	Required	Description
data	string	No*	Raw CSV/JSON/XML string (single dataset)
datasets	array	No*	Array of raw data strings (max 50)
format	enum	No	auto (default), csv, json, xml
inferSchema	boolean	No	Column-level profiling (default: true)
batchMode	boolean	No	Enable batch charge event

* One of data or datasets must be provided.

Example Input JSON

{
  "data": "name,age,email\nAlice,30,alice@example.com\nBob,,bob@example.com\nCarol,25,\n",
  "inferSchema": true
}

Example Batch Input

{
  "datasets": [
    "name,age,city\nAlice,30,London\nBob,25,Paris",
    "name,age,city\nCarol,35,Berlin\nDan,28,Madrid"
  ],
  "inferSchema": true
}

---

Output Format

Field	Type	Description
dataset_index	integer	1-based index of the dataset in the batch
format	string	Detected or specified format (csv, json, xml)
valid	boolean	Whether the dataset parsed successfully
error	string or null	Error message if validation failed
row_count	integer	Number of data rows detected
column_count	integer	Number of columns detected
columns	array	List of column names
schema	array or null	Per-column field profiles (when inferSchema=true)
anomaly_count	integer	Total anomaly flags across all columns

Example Output JSON

{
  "dataset_index": 1,
  "format": "csv",
  "valid": true,
  "row_count": 150,
  "column_count": 8,
  "columns": ["id", "name", "email", "age", "city", "salary", "joined", "active"],
  "anomaly_count": 2,
  "schema": [
    {
      "name": "age",
      "type": "number",
      "nullable": true,
      "unique": false,
      "stats": {
        "null_pct": 5.3,
        "min": 18,
        "max": 72,
        "mean": 34.5,
        "stdev": 12.8,
        "anomalies": []
      }
    }
  ]
}

---

FAQ

Q: What formats are supported? A: CSV (comma-separated), JSON (object array — a single JSON array of objects), and XML (flat element structures). Format is auto-detected by default. NDJSON (newline-delimited JSON) is not currently supported — wrap your lines in a JSON array or feed them as separate batch items.

Q: How does type detection work? A: It inspects all non-null, non-empty values in a column and checks their Python types. If every value is an int, the column type is int. Mixed ints and floats become number. A mix of incompatible types becomes mixed. All-null columns get unknown. String values that look numeric (parse to float) are treated as float.

Q: What anomalies are detected? A: Two types. high_nulls (severity: warn) fires when more than 50% of values are null or empty. mixed_types (severity: info) fires when a column contains values of multiple incompatible types — like strings mixed with numbers in the same column.

Q: Can I process large datasets? A: The actor processes data in memory. Very large datasets may hit the memory limit (default 512 MB on Apify). If you're profiling files over roughly 100 MB, split them into chunks or process one at a time.

Q: What happens if a dataset fails to parse? A: The result comes back with valid: false, an error string explaining what went wrong, and row_count: 0. In batch mode, the remaining datasets keep processing.

Q: Is there a limit on batch size? A: Maximum 50 datasets per run. Input arrays longer than 50 get truncated to 50.

Q: Can I use this in automated workflows? A: Yes. The API is documented below with cURL and Python examples. It works in CI/CD pipelines, ETL workflows, and monitoring setups — anything that can make an HTTP request or use the Apify client SDK.

Q: Does the actor require any external API keys? A: No. All processing happens inside the actor. No external services called. You just need an Apify account to run it.

---

API Usage

cURL

curl -X POST "https://api.apify.com/v2/acts/perryay~data-validator-profiler/runs?token=YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"data": "name,age\nAlice,30\nBob,25", "inferSchema": true}'

Python (ApifyClient)

from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")
run = client.actor("perryay~data-validator-profiler").call(
    run_input={"data": "name,age\nAlice,30\nBob,25", "inferSchema": True}
)
dataset = client.dataset(run["defaultDatasetId"]).list_items()
for item in dataset.items:
    print(f'Dataset {item["dataset_index"]}: {item["row_count"]} rows, {item["column_count"]} cols, {item["anomaly_count"]} anomalies')

---

Use Cases

1. CI/CD data quality gates — Profile CSV exports in your pipeline. Fail the build if anomaly counts cross a threshold, or if column counts change unexpectedly between releases.

2. ETL input validation — Run this before ingestion to catch schema drift, unexpected null spikes, or type changes. You don't want a varchar column suddenly full of JSON blobs mid-pipeline.

3. Data onboarding — Client sends you a file and says "here's the data." Profile it and you know the schema, null ratios, and numeric ranges in seconds instead of poking around in Excel.

4. QA automation — Profile data exports from each build. Compare row counts, column counts, and anomaly flags between releases to catch regressions.

5. Data migration validation — Profile source and target independently before migration. If column counts, types, or null ratios don't match, you find out before the migration runs.

6. API response validation — Feed JSON API responses straight into the actor to verify the response structure hasn't drifted from what you expect.

7. CSV export auditing — Profile every CSV your application generates. Catch silently dropped columns, empty exports, or format drift.

8. Data catalog population — Run this against incoming datasets and pipe the column-level metadata directly into your data catalog or warehouse documentation.