# Dataset Filter, Dedupe & Convert – JSON/CSV/XLSX (`polychrome_haymow/dataset-filter-dedupe-convert`) Actor

Filter, deduplicate, select and rename fields of any Apify dataset or public JSON/CSV URL, then export to JSON, CSV and XLSX. No code needed. By NYC Data Works.

- **URL**: https://apify.com/polychrome_haymow/dataset-filter-dedupe-convert.md
- **Developed by:** [Levani Zurabiani](https://apify.com/polychrome_haymow) (community)
- **Categories:** Developer tools, Integrations
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 1,000 input rows processeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Dataset Filter, Dedupe & Convert – JSON / CSV / XLSX

By **NYC Data Works**. A no-code post-processing step for any Apify scraper's output, or for any JSON/CSV file.

### What it does

1. **Loads** an Apify dataset (picked in the `datasetId` field, read-only) or a public JSON array / JSONL / CSV URL (`source`) (delimiter auto-detected).
2. **Filters** rows with simple expressions (AND/OR).
3. **Dedupes** on one or more keys, with optional normalization (case, whitespace, `http(s)://www.` prefix, trailing slash).
4. **Selects and renames** fields; optionally flattens nested objects.
5. **Exports** `output.json`, `output.csv` and `output.xlsx` to the key-value store, pushes rows to the dataset, and writes a `SUMMARY` record (rows in/out, duplicates removed, file links).

### Filter syntax

`field OP value`, with dotted paths for nested data (`address.city`, `tags.0`).

| OP | Example |
|---|---|
| `== !=` | `status == active` (numeric when both sides are numbers) |
| `> >= < <=` | `price >= 9.99` |
| `contains !contains` | `title contains engineer` (case-insensitive; list membership for arrays) |
| `startswith endswith` | `url startswith https://shop` |
| `regex` | `email regex @(gmail|yahoo)\.com$` |
| `in` | `country in US,CA,MX` |
| `exists !exists empty !empty` | `phone !empty` |

### Example input

```json
{"datasetId":"abcDEF1234567890x","filters":["rating >= 4","city == Berlin"],"dedupeKeys":["website"],"fields":["name","website","rating"],"rename":{"website":"url"}}
```

### FAQ

**Which datasets can it read?** Only the one dataset you select in the `datasetId` field. The actor runs with limited permissions and gets read-only access to exactly that dataset; it cannot see or change anything else in your account. Pasting a dataset ID into `source` is no longer supported; `source` is for public CSV/JSON URLs only.
**How large a dataset can it handle?** Rows are processed in memory, so the practical size depends on row width and the run's memory (not benchmarked; a 2,500-row test runs easily at 512 MB). For large sources raise memory, or use `maxItems` (max 1,000,000).
**Is fuzzy dedupe supported?** Normalized exact matching only (case, spaces, URL prefix). Edit-distance fuzzy matching is not included.
**Why are some CSV values JSON strings?** Arrays are serialized to JSON in CSV/XLSX, and nested objects are flattened to dotted columns.
**Does it change my source dataset?** No, it is read-only.
**Is any external service used?** No. Only the URL you give in `source` is fetched.

### Limits

- CSV values are read as strings; numeric comparison still works because values are converted when possible.
- XLSX is limited to 1,048,576 rows (an Excel limit).
- XLSX cannot store control characters (e.g. `\x01`); they are removed in the XLSX file only (JSON/CSV keep the original value).
- Formula-injection guard: in CSV/XLSX, text values starting with `=`, `+`, `-`, `@` (except plain numbers like `-5`) are prefixed with `'` so spreadsheets do not run them as formulas. JSON output is unchanged.

### Permissions

Runs with **limited permissions**. Pick the input dataset in the **Dataset** field (`datasetId`): the actor is then granted **read-only** access to that one dataset and nothing else in your account. `Source URL` (`source`) is only for public CSV/JSON/JSONL URLs.

### Safety limits

- URL sources: max 200 MB (checked before download, then streamed with a byte cap).
- `maxItems` up to 1,000,000 rows. Regex filters: pattern ≤ 200 characters, matched against the first 10,000 characters of a value.
- **Regex time limit:** each regex search on one value may take at most **0.05 s** (Python [`regex`](https://pypi.org/project/regex/) module with a timeout). If a pattern exceeds it (catastrophic backtracking, e.g. `(a|aa)+$`), the run **fails fast with a clear "regex too slow" error and nothing is charged**. We chose failing over silently treating the value as "no match", because a silent skip would give you a wrong, incomplete result that you would still pay for. Simplify the pattern (avoid nested quantifiers) and rerun.
- Renaming two fields to the same name is rejected instead of silently overwriting data. Rows that are not objects are wrapped as `{"value": …}`.

### Security

- Only public `http(s)` URLs are fetched. Every URL and every redirect hop (max 5) is resolved and rejected if it points to a private, loopback, link-local (incl. the `169.254.169.254` cloud-metadata address), multicast or reserved address; other schemes (`file:`, `ftp:` …) are refused. The connection is pinned to the checked IP.
- Size limits are checked from `Content-Length` before downloading and enforced again while streaming, with an overall per-file deadline.
- Query strings and credentials in URLs are stripped from logs, dataset items and `SUMMARY`.

### Spending limit

Nothing is charged until the whole source has loaded and parsed **and** all output files (JSON/CSV/XLSX) were built successfully. A source that fails to load (bad URL, blocked address, broken JSON/JSONL line, …) or an export that cannot be built ends the run with a clear error and **$0 charged**. Then `actor-start` and the input-row blocks are charged together. If your **Max cost per run** only covers part of the source, the actor processes only the rows that fit (whole 1,000-row blocks after the start fee) and says so in the status message and `SUMMARY.stoppedByChargeLimit`; nothing beyond the limit is charged.

### Data retention & consent

- **Retention:** results are stored only in this run's own default dataset and key-value store, and are kept under Apify's standard storage retention for your plan (unnamed run storages are deleted automatically after the retention period). Nothing is copied elsewhere; no third-party API is called.
- **Your responsibility:** you must have the rights to the content you submit and, where it contains personal data, a lawful basis and any required consent (e.g. under GDPR/CCPA). Do not submit data you are not allowed to process.

### Credits & licences

- [httpx](https://www.python-httpx.org/) – BSD-3-Clause
- [openpyxl](https://openpyxl.readthedocs.io/) – MIT
- [Apify SDK for Python](https://github.com/apify/apify-sdk-python) – Apache-2.0
- [regex](https://github.com/mrabarnett/mrab-regex) – Apache-2.0 (user regex filters)
- Python standard library `csv`/`json`/`re` – PSF License

### Pricing

Pay-per-event – you pay only for what you process:

| Event | Price |
|---|---|
| Actor start (`actor-start`) | $0.002 per run |
| Rows processed (`rows-processed-1k`) | $0.002 per 1,000 input rows (rounded up) |

Charged on **input** rows (before filter/dedupe), and only after loading and building the exports succeeded. Example: a 2,500-row dataset costs $0.002 + 3 × $0.002 = $0.008.

# Actor input Schema

## `datasetId` (type: `string`):

Apify dataset to process (read-only access is granted to this dataset only). Use either this or Source URL.

## `source` (type: `string`):

Public http(s) URL of a JSON array / JSONL / CSV file (max 200 MB). For Apify datasets use "Dataset" above. Private/internal addresses are rejected.

## `filters` (type: `array`):

All must match (AND). Syntax: field OP value. OPs: == != > >= < <= contains !contains startswith endswith regex in exists !exists empty !empty. Dotted paths ok (a.b.0.c). Examples: price > 10 · country in US,CA · email exists · title regex ^Senior. Regex patterns are limited to 200 characters and match the first 10,000 characters of a value; a regex taking over 0.05 s on one value stops the run with an error (nothing charged).

## `filterMode` (type: `string`):

AND = all filters must match; OR = any.

## `dedupeKeys` (type: `array`):

Fields that together identify a duplicate. Empty = no dedupe. Use '\*' for whole-record dedupe.

## `dedupeNormalize` (type: `boolean`):

Lowercase, trim and collapse whitespace; strips URL scheme/www/trailing slash and email case before comparing.

## `dedupeKeep` (type: `string`):

Which record to keep.

## `fields` (type: `array`):

Output only these fields (dotted paths allowed). Empty = keep all.

## `rename` (type: `object`):

Map old name → new name, applied after field selection. Example: {"ISO3166-1-Alpha-2":"iso2"} Two fields may not be renamed to the same name.

## `flatten` (type: `boolean`):

Flatten nested objects into dotted keys (CSV/XLSX always flatten).

## `outputFormats` (type: `array`):

Files saved into the key-value store.

## `pushToDataset` (type: `boolean`):

Also write the output rows to the run's default dataset.

## `maxItems` (type: `integer`):

Stop reading after this many rows (max 1,000,000).

## Actor input object example

```json
{
  "source": "https://raw.githubusercontent.com/datasets/country-codes/main/data/country-codes.csv",
  "filters": [
    "Continent == EU"
  ],
  "filterMode": "and",
  "dedupeNormalize": true,
  "dedupeKeep": "first",
  "rename": {},
  "flatten": false,
  "outputFormats": [
    "json",
    "csv",
    "xlsx"
  ],
  "pushToDataset": true,
  "maxItems": 100000
}
```

# Actor output Schema

## `rows` (type: `string`):

Output rows (when pushToDataset is on).

## `json` (type: `string`):

No description

## `csv` (type: `string`):

No description

## `xlsx` (type: `string`):

No description

## `summary` (type: `string`):

Rows in/out, duplicates removed, file links.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "source": "https://raw.githubusercontent.com/datasets/country-codes/main/data/country-codes.csv",
    "filters": [
        "Continent == EU"
    ],
    "rename": {}
};

// Run the Actor and wait for it to finish
const run = await client.actor("polychrome_haymow/dataset-filter-dedupe-convert").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "source": "https://raw.githubusercontent.com/datasets/country-codes/main/data/country-codes.csv",
    "filters": ["Continent == EU"],
    "rename": {},
}

# Run the Actor and wait for it to finish
run = client.actor("polychrome_haymow/dataset-filter-dedupe-convert").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "source": "https://raw.githubusercontent.com/datasets/country-codes/main/data/country-codes.csv",
  "filters": [
    "Continent == EU"
  ],
  "rename": {}
}' |
apify call polychrome_haymow/dataset-filter-dedupe-convert --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,polychrome_haymow/dataset-filter-dedupe-convert"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/KoEk5oAMOdakcf4aq/builds/kg7CilCLpAawoJQN0/openapi.json