# Data Validator & Profiler — CSV JSON XML schema inference (`perryay/data-validator-profiler`) Actor

Validate and profile CSV, JSON, and XML datasets. Auto-detects schema (column types, null ratios, uniqueness), flags anomalies (mixed types, high nulls), and produces a detailed profile report. Supports batch mode for multiple datasets.

- **URL**: https://apify.com/perryay/data-validator-profiler.md
- **Developed by:** [Perry AY](https://apify.com/perryay) (community)
- **Categories:** Developer tools, AI
- **Stats:** 2 total users, 1 monthly users, 33.3% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.02 / actor start

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md):

```bash
npm install apify-client
```

In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md):

```bash
pip install apify-client
```

In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md):

````bash
# MacOS / Linux
curl -fsSL https://apify.com/install-cli.sh | bash
# Windows
irm https://apify.com/install-cli.ps1 | iex
```bash

In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md).

If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).


# README

## Data Validator & Profiler

### Validate and profile CSV, JSON, and XML data — infer schema, detect anomalies, get stats.

Drop in raw CSV, JSON, or XML and get back a full column-by-column profile: types, null ratios, uniqueness, min/max/mean/median/stdev for numeric columns, and anomaly flags. Batch mode handles up to 50 datasets in a single run — each one parsed and profiled independently.

I built this because I was tired of manually inspecting data files. You get a CSV from a client, a JSON dump from an API, or an XML export from some legacy system — and you need to know what's in it before you write a line of ETL. This actor answers those questions in seconds.

---

### What does it do?

You give it raw data — a CSV string, a JSON array, or a flat XML structure. It auto-detects the format, parses the rows, then profiles every column.

For each column you get:

- **Row count** — how many rows of data.
- **Null count & percentage** — counts both `null` and empty strings as missing.
- **Unique values** — count and percentage. Tells you if a column is a candidate key.
- **Detected type** — `int`, `float`, `str`, `bool`, `number` (mixed int/float), `mixed`, or `unknown` (all-null).
- **Numeric stats** — min, max, mean, median, and standard deviation when the column has enough numeric values.
- **Anomaly flags** — marks columns with >50% nulls (`high_nulls`, severity: warn) and columns with mixed types (`mixed_types`, severity: info).

If you turn off schema inference (`inferSchema: false`), you get a lighter result — just row count, column count, column names, and validity. Faster if you only need a parse check.

Batch mode takes an array of datasets (up to 50), profiles each one independently, and pushes results as separate dataset items. One broken dataset won't kill the run for the rest.

---

### Features

1. **Format auto-detection** — JSON (starts with `{` or `[`), XML (starts with `<`), everything else assumed CSV. You can override with the `format` field.
2. **Schema inference** — column-level type detection, null ratio, uniqueness scoring, numeric stats.
3. **Anomaly flagging** — `high_nulls` at >50% nulls, `mixed_types` when a column holds values of multiple incompatible types.
4. **Numeric profiling** — min, max, mean, median, stdev for columns where at least 2 values are numeric (stdev needs 4+ values with variance). Non-numeric values are skipped.
5. **Batch mode** — profile up to 50 datasets in one run. Each gets its own output item with a `dataset_index`.
6. **Error resilience** — parse failures produce `valid: false` + error message. Batch mode keeps going.

---

### Who is it for?

| Persona | What they use it for |
|---------|---------------------|
| Data Engineer | Validating upstream data quality before ingestion pipelines |
| QA Engineer | Checking data exports for regressions and format issues |
| Data Analyst | Understanding unknown datasets from clients or partners |
| ETL Developer | Profiling source data before writing transformation logic |
| DevOps Engineer | Integrating data validation into CI/CD quality gates |
| Data Onboarding Specialist | Quickly assessing the shape and quality of new data deliveries |

---

### Input Parameters

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `data` | string | No* | Raw CSV/JSON/XML string (single dataset) |
| `datasets` | array | No* | Array of raw data strings (max 50) |
| `format` | enum | No | `auto` (default), `csv`, `json`, `xml` |
| `inferSchema` | boolean | No | Column-level profiling (default: `true`) |
| `batchMode` | boolean | No | Enable batch charge event |

*\* One of `data` or `datasets` must be provided.*

#### Example Input JSON

```json
{
  "data": "name,age,email\nAlice,30,alice@example.com\nBob,,bob@example.com\nCarol,25,\n",
  "inferSchema": true
}
````

#### Example Batch Input

```json
{
  "datasets": [
    "name,age,city\nAlice,30,London\nBob,25,Paris",
    "name,age,city\nCarol,35,Berlin\nDan,28,Madrid"
  ],
  "inferSchema": true
}
```

***

### Output Format

| Field | Type | Description |
|-------|------|-------------|
| `dataset_index` | integer | 1-based index of the dataset in the batch |
| `format` | string | Detected or specified format (`csv`, `json`, `xml`) |
| `valid` | boolean | Whether the dataset parsed successfully |
| `error` | string or null | Error message if validation failed |
| `row_count` | integer | Number of data rows detected |
| `column_count` | integer | Number of columns detected |
| `columns` | array | List of column names |
| `schema` | array or null | Per-column field profiles (when `inferSchema=true`) |
| `anomaly_count` | integer | Total anomaly flags across all columns |

#### Example Output JSON

```json
{
  "dataset_index": 1,
  "format": "csv",
  "valid": true,
  "row_count": 150,
  "column_count": 8,
  "columns": ["id", "name", "email", "age", "city", "salary", "joined", "active"],
  "anomaly_count": 2,
  "schema": [
    {
      "name": "age",
      "type": "number",
      "nullable": true,
      "unique": false,
      "stats": {
        "null_pct": 5.3,
        "min": 18,
        "max": 72,
        "mean": 34.5,
        "stdev": 12.8,
        "anomalies": []
      }
    }
  ]
}
```

***

### FAQ

**Q: What formats are supported?**
A: CSV (comma-separated), JSON (object array — a single JSON array of objects), and XML (flat element structures). Format is auto-detected by default. NDJSON (newline-delimited JSON) is not currently supported — wrap your lines in a JSON array or feed them as separate batch items.

**Q: How does type detection work?**
A: It inspects all non-null, non-empty values in a column and checks their Python types. If every value is an `int`, the column type is `int`. Mixed ints and floats become `number`. A mix of incompatible types becomes `mixed`. All-null columns get `unknown`. String values that look numeric (parse to float) are treated as `float`.

**Q: What anomalies are detected?**
A: Two types. `high_nulls` (severity: warn) fires when more than 50% of values are null or empty. `mixed_types` (severity: info) fires when a column contains values of multiple incompatible types — like strings mixed with numbers in the same column.

**Q: Can I process large datasets?**
A: The actor processes data in memory. Very large datasets may hit the memory limit (default 512 MB on Apify). If you're profiling files over roughly 100 MB, split them into chunks or process one at a time.

**Q: What happens if a dataset fails to parse?**
A: The result comes back with `valid: false`, an `error` string explaining what went wrong, and `row_count: 0`. In batch mode, the remaining datasets keep processing.

**Q: Is there a limit on batch size?**
A: Maximum 50 datasets per run. Input arrays longer than 50 get truncated to 50.

**Q: Can I use this in automated workflows?**
A: Yes. The API is documented below with cURL and Python examples. It works in CI/CD pipelines, ETL workflows, and monitoring setups — anything that can make an HTTP request or use the Apify client SDK.

**Q: Does the actor require any external API keys?**
A: No. All processing happens inside the actor. No external services called. You just need an Apify account to run it.

***

### API Usage

#### cURL

```bash
curl -X POST "https://api.apify.com/v2/acts/perryay~data-validator-profiler/runs?token=YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"data": "name,age\nAlice,30\nBob,25", "inferSchema": true}'
```

#### Python (ApifyClient)

```python
from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")
run = client.actor("perryay~data-validator-profiler").call(
    run_input={"data": "name,age\nAlice,30\nBob,25", "inferSchema": True}
)
dataset = client.dataset(run["defaultDatasetId"]).list_items()
for item in dataset.items:
    print(f'Dataset {item["dataset_index"]}: {item["row_count"]} rows, {item["column_count"]} cols, {item["anomaly_count"]} anomalies')
```

***

### Use Cases

1. **CI/CD data quality gates** — Profile CSV exports in your pipeline. Fail the build if anomaly counts cross a threshold, or if column counts change unexpectedly between releases.

2. **ETL input validation** — Run this before ingestion to catch schema drift, unexpected null spikes, or type changes. You don't want a varchar column suddenly full of JSON blobs mid-pipeline.

3. **Data onboarding** — Client sends you a file and says "here's the data." Profile it and you know the schema, null ratios, and numeric ranges in seconds instead of poking around in Excel.

4. **QA automation** — Profile data exports from each build. Compare row counts, column counts, and anomaly flags between releases to catch regressions.

5. **Data migration validation** — Profile source and target independently before migration. If column counts, types, or null ratios don't match, you find out before the migration runs.

6. **API response validation** — Feed JSON API responses straight into the actor to verify the response structure hasn't drifted from what you expect.

7. **CSV export auditing** — Profile every CSV your application generates. Catch silently dropped columns, empty exports, or format drift.

8. **Data catalog population** — Run this against incoming datasets and pipe the column-level metadata directly into your data catalog or warehouse documentation.

# Actor input Schema

## `data` (type: `string`):

Raw CSV, JSON, or XML data as a string. Use for single dataset validation.

## `datasets` (type: `array`):

Array of raw data strings for batch validation (max 50).

## `format` (type: `string`):

Input data format. 'auto' detects format automatically.

## `inferSchema` (type: `boolean`):

Perform column-level type inference and profiling.

## `batchMode` (type: `boolean`):

Treat input as batch (enables batch-profile charge event). Auto-enabled when using datasets array.

## Actor input object example

```json
{
  "data": "name,age,email\nAlice,30,alice@example.com\nBob,25,bob@example.com\nCarol,35,carol@example.com",
  "datasets": [],
  "format": "auto",
  "inferSchema": true,
  "batchMode": false
}
```

# Actor output Schema

## `results` (type: `string`):

Dataset profiling results

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "data": `name,age,email
Alice,30,alice@example.com
Bob,25,bob@example.com
Carol,35,carol@example.com`,
    "datasets": [],
    "format": "auto",
    "inferSchema": true,
    "batchMode": false
};

// Run the Actor and wait for it to finish
const run = await client.actor("perryay/data-validator-profiler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "data": """name,age,email
Alice,30,alice@example.com
Bob,25,bob@example.com
Carol,35,carol@example.com""",
    "datasets": [],
    "format": "auto",
    "inferSchema": True,
    "batchMode": False,
}

# Run the Actor and wait for it to finish
run = client.actor("perryay/data-validator-profiler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "data": "name,age,email\\nAlice,30,alice@example.com\\nBob,25,bob@example.com\\nCarol,35,carol@example.com",
  "datasets": [],
  "format": "auto",
  "inferSchema": true,
  "batchMode": false
}' |
apify call perryay/data-validator-profiler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=perryay/data-validator-profiler",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

```json
{
    "openapi": "3.0.1",
    "info": {
        "title": "Data Validator & Profiler — CSV JSON XML schema inference",
        "description": "Validate and profile CSV, JSON, and XML datasets. Auto-detects schema (column types, null ratios, uniqueness), flags anomalies (mixed types, high nulls), and produces a detailed profile report. Supports batch mode for multiple datasets.",
        "version": "1.0",
        "x-build-id": "mZZKTefJalFsbMTpC"
    },
    "servers": [
        {
            "url": "https://api.apify.com/v2"
        }
    ],
    "paths": {
        "/acts/perryay~data-validator-profiler/run-sync-get-dataset-items": {
            "post": {
                "operationId": "run-sync-get-dataset-items-perryay-data-validator-profiler",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK"
                    }
                }
            }
        },
        "/acts/perryay~data-validator-profiler/runs": {
            "post": {
                "operationId": "runs-sync-perryay-data-validator-profiler",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor and returns information about the initiated run in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK",
                        "content": {
                            "application/json": {
                                "schema": {
                                    "$ref": "#/components/schemas/runsResponseSchema"
                                }
                            }
                        }
                    }
                }
            }
        },
        "/acts/perryay~data-validator-profiler/run-sync": {
            "post": {
                "operationId": "run-sync-perryay-data-validator-profiler",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK"
                    }
                }
            }
        }
    },
    "components": {
        "schemas": {
            "inputSchema": {
                "type": "object",
                "properties": {
                    "data": {
                        "title": "Data (single dataset)",
                        "type": "string",
                        "description": "Raw CSV, JSON, or XML data as a string. Use for single dataset validation."
                    },
                    "datasets": {
                        "title": "Datasets (batch)",
                        "type": "array",
                        "description": "Array of raw data strings for batch validation (max 50).",
                        "default": []
                    },
                    "format": {
                        "title": "Data Format",
                        "enum": [
                            "auto",
                            "csv",
                            "json",
                            "xml"
                        ],
                        "type": "string",
                        "description": "Input data format. 'auto' detects format automatically.",
                        "default": "auto"
                    },
                    "inferSchema": {
                        "title": "Infer Schema",
                        "type": "boolean",
                        "description": "Perform column-level type inference and profiling.",
                        "default": true
                    },
                    "batchMode": {
                        "title": "Batch Mode",
                        "type": "boolean",
                        "description": "Treat input as batch (enables batch-profile charge event). Auto-enabled when using datasets array.",
                        "default": false
                    }
                }
            },
            "runsResponseSchema": {
                "type": "object",
                "properties": {
                    "data": {
                        "type": "object",
                        "properties": {
                            "id": {
                                "type": "string"
                            },
                            "actId": {
                                "type": "string"
                            },
                            "userId": {
                                "type": "string"
                            },
                            "startedAt": {
                                "type": "string",
                                "format": "date-time",
                                "example": "2025-01-08T00:00:00.000Z"
                            },
                            "finishedAt": {
                                "type": "string",
                                "format": "date-time",
                                "example": "2025-01-08T00:00:00.000Z"
                            },
                            "status": {
                                "type": "string",
                                "example": "READY"
                            },
                            "meta": {
                                "type": "object",
                                "properties": {
                                    "origin": {
                                        "type": "string",
                                        "example": "API"
                                    },
                                    "userAgent": {
                                        "type": "string"
                                    }
                                }
                            },
                            "stats": {
                                "type": "object",
                                "properties": {
                                    "inputBodyLen": {
                                        "type": "integer",
                                        "example": 2000
                                    },
                                    "rebootCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "restartCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "resurrectCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "computeUnits": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            },
                            "options": {
                                "type": "object",
                                "properties": {
                                    "build": {
                                        "type": "string",
                                        "example": "latest"
                                    },
                                    "timeoutSecs": {
                                        "type": "integer",
                                        "example": 300
                                    },
                                    "memoryMbytes": {
                                        "type": "integer",
                                        "example": 1024
                                    },
                                    "diskMbytes": {
                                        "type": "integer",
                                        "example": 2048
                                    }
                                }
                            },
                            "buildId": {
                                "type": "string"
                            },
                            "defaultKeyValueStoreId": {
                                "type": "string"
                            },
                            "defaultDatasetId": {
                                "type": "string"
                            },
                            "defaultRequestQueueId": {
                                "type": "string"
                            },
                            "buildNumber": {
                                "type": "string",
                                "example": "1.0.0"
                            },
                            "containerUrl": {
                                "type": "string"
                            },
                            "usage": {
                                "type": "object",
                                "properties": {
                                    "ACTOR_COMPUTE_UNITS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_WRITES": {
                                        "type": "integer",
                                        "example": 1
                                    },
                                    "KEY_VALUE_STORE_LISTS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_INTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_EXTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_SERPS": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            },
                            "usageTotalUsd": {
                                "type": "number",
                                "example": 0.00005
                            },
                            "usageUsd": {
                                "type": "object",
                                "properties": {
                                    "ACTOR_COMPUTE_UNITS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_WRITES": {
                                        "type": "number",
                                        "example": 0.00005
                                    },
                                    "KEY_VALUE_STORE_LISTS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_INTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_EXTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_SERPS": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}
```