# Advanced JSON / CSV Data Cleaner (`flyerlin/advanced-data-cleaner`) Actor

Clean, deduplicate, validate and convert JSON, JSONL, CSV and XLSX files. Includes field mapping, optional type inference and heuristic PII redaction.

- **URL**: https://apify.com/flyerlin/advanced-data-cleaner.md
- **Developed by:** [林 吴](https://apify.com/flyerlin) (community)
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.10 / 1,000 cleaned records

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Advanced JSON / CSV Data Cleaner

Turn messy tabular data into validated, reusable records. Convert between JSON, JSONL, CSV and modern Excel (.xlsx), remove duplicates, rename fields, and validate records against JSON Schema.

### Quick start

Paste a JSON array, JSONL or CSV into `raw_text`, or provide one publicly downloadable `source_url`. Choose your output format and run.

```json
{"raw_text":"name,age\n Alice ,30\n Alice ,30","source_format":"csv","target_format":"json","trim_strings":true,"dedupe":true}
```

The result is one record: `{"name":"Alice","age":"30"}`. Enable `infer_types` to convert age to a number. Type inference is off by default to preserve identifiers and leading zeroes.

### Features

- JSON / JSONL / CSV / TSV / XLSX input; JSON / JSONL / CSV / XLSX output
- Conservative JSON repair: curly quotes and trailing commas; full Markdown code fences accepted
- Exact record deduplication, top-level whitespace trimming and empty-field removal
- Top-level field mapping (`{"old":"new"}`), dotted-header nesting
- JSON Schema validation with up to 100 diagnostic messages
- Optional heuristic email, phone, card-like number and IPv4 redaction; this is not guaranteed anonymization

### Results

The default dataset contains cleaned records. The key-value store contains `OUTPUT` (selected format) and `SUMMARY` (input/output counts, duplicates, validation errors and redactions). Obtain download links in the run's Storage tab. Standard dataset export is also available.

Malformed JSONL fails the run rather than silently discarding lines. Invalid Schema records are retained with diagnostics unless `drop_invalid` is enabled. CSV and XLSX are exported with formula execution prevention. Input limits: 30 MB and up to 100,000 records; default 20,000. Inputs above the record limit fail rather than truncate. XLSX formulas are read as cached values; legacy .xls is unsupported. CSV requires a header row. Do not submit duplicate headers.

### Pricing

$0.10 per 1,000 output records ($0.0001 each). Failed conversions have no custom record event. Store pricing is authoritative.

### Suitable workflows

Clean scraped datasets before CRM imports, normalize automation output, validate LLM records, and prepare structured data for analysis.

### Platform costs

Apify platform compute, storage and transfer usage is paid separately by the user, in addition to the custom event price. No separate custom startup or dataset insertion event is configured.

# Actor input Schema

## `raw_text` (type: `string`):

Input text. See README for behavior and limits.

## `source_url` (type: `string`):

Public downloadable file, up to 30 MB. Provide this OR input text.

## `source_format` (type: `string`):

Source format. See README for behavior and limits.

## `target_format` (type: `string`):

Output format. See README for behavior and limits.

## `sheet_name` (type: `string`):

Excel sheet name. See README for behavior and limits.

## `delimiter` (type: `string`):

CSV delimiter. See README for behavior and limits.

## `field_mapping` (type: `string`):

JSON object, e.g. {"old_name":"new_name"}. Top-level fields only.

## `expected_schema` (type: `string`):

JSON Schema. See README for behavior and limits.

## `trim_strings` (type: `boolean`):

Trim top-level string values. See README for behavior and limits.

## `dedupe` (type: `boolean`):

Remove identical records. See README for behavior and limits.

## `infer_types` (type: `boolean`):

Infer booleans and numbers (may remove leading zeroes). See README for behavior and limits.

## `drop_empty` (type: `boolean`):

Remove empty top-level fields. See README for behavior and limits.

## `nest_headers` (type: `boolean`):

Convert dotted headers to nested objects. See README for behavior and limits.

## `redact_pii` (type: `boolean`):

Heuristic PII redaction. See README for behavior and limits.

## `drop_invalid` (type: `boolean`):

Drop records that fail Schema validation. See README for behavior and limits.

## `max_records` (type: `integer`):

Maximum records. See README for behavior and limits.

## Actor input object example

```json
{
  "raw_text": "[{\"name\":\" Alice \",\"email\":\"alice@example.com\"},{\"name\":\" Alice \",\"email\":\"alice@example.com\"}]",
  "source_format": "auto",
  "target_format": "json",
  "trim_strings": true,
  "dedupe": true,
  "infer_types": false,
  "drop_empty": false,
  "nest_headers": false,
  "redact_pii": false,
  "drop_invalid": false,
  "max_records": 20000
}
```

# Actor output Schema

## `records` (type: `string`):

Cleaned records

## `file` (type: `string`):

Converted file

## `summary` (type: `string`):

Processing summary

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "raw_text": "[{\"name\":\" Alice \",\"email\":\"alice@example.com\"},{\"name\":\" Alice \",\"email\":\"alice@example.com\"}]"
};

// Run the Actor and wait for it to finish
const run = await client.actor("flyerlin/advanced-data-cleaner").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "raw_text": "[{\"name\":\" Alice \",\"email\":\"alice@example.com\"},{\"name\":\" Alice \",\"email\":\"alice@example.com\"}]" }

# Run the Actor and wait for it to finish
run = client.actor("flyerlin/advanced-data-cleaner").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "raw_text": "[{\\"name\\":\\" Alice \\",\\"email\\":\\"alice@example.com\\"},{\\"name\\":\\" Alice \\",\\"email\\":\\"alice@example.com\\"}]"
}' |
apify call flyerlin/advanced-data-cleaner --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,flyerlin/advanced-data-cleaner"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/YsEd8cKJAozSsdgIF/builds/PwReM5e9rOZ2CzgYC/openapi.json