Dataset Filter, Dedupe & Convert – JSON / CSV / XLSX

By NYC Data Works. A no-code post-processing step for any Apify scraper's output, or for any JSON/CSV file.

What it does

1. Loads an Apify dataset (picked in the datasetId field, read-only) or a public JSON array / JSONL / CSV URL (source) (delimiter auto-detected).
2. Filters rows with simple expressions (AND/OR).
3. Dedupes on one or more keys, with optional normalization (case, whitespace, http(s)://www. prefix, trailing slash).
4. Selects and renames fields; optionally flattens nested objects.
5. Exports output.json, output.csv and output.xlsx to the key-value store, pushes rows to the dataset, and writes a SUMMARY record (rows in/out, duplicates removed, file links).

Filter syntax

field OP value, with dotted paths for nested data (address.city, tags.0).

OP	Example
== !=	status == active (numeric when both sides are numbers)
> >= < <=	price >= 9.99
contains !contains	title contains engineer (case-insensitive; list membership for arrays)
startswith endswith	url startswith https://shop
regex	`email regex @(gmail
in	country in US,CA,MX
exists !exists empty !empty	phone !empty

Example input

{"datasetId":"abcDEF1234567890x","filters":["rating >= 4","city == Berlin"],"dedupeKeys":["website"],"fields":["name","website","rating"],"rename":{"website":"url"}}

FAQ

Which datasets can it read? Only the one dataset you select in the datasetId field. The actor runs with limited permissions and gets read-only access to exactly that dataset; it cannot see or change anything else in your account. Pasting a dataset ID into source is no longer supported; source is for public CSV/JSON URLs only. How large a dataset can it handle? Rows are processed in memory, so the practical size depends on row width and the run's memory (not benchmarked; a 2,500-row test runs easily at 512 MB). For large sources raise memory, or use maxItems (max 1,000,000). Is fuzzy dedupe supported? Normalized exact matching only (case, spaces, URL prefix). Edit-distance fuzzy matching is not included. Why are some CSV values JSON strings? Arrays are serialized to JSON in CSV/XLSX, and nested objects are flattened to dotted columns. Does it change my source dataset? No, it is read-only. Is any external service used? No. Only the URL you give in source is fetched.

Limits

• CSV values are read as strings; numeric comparison still works because values are converted when possible.
• XLSX is limited to 1,048,576 rows (an Excel limit).
• XLSX cannot store control characters (e.g. \x01); they are removed in the XLSX file only (JSON/CSV keep the original value).
• Formula-injection guard: in CSV/XLSX, text values starting with =, +, -, @ (except plain numbers like -5) are prefixed with ' so spreadsheets do not run them as formulas. JSON output is unchanged.

Permissions

Runs with limited permissions. Pick the input dataset in the Dataset field (datasetId): the actor is then granted read-only access to that one dataset and nothing else in your account. Source URL (source) is only for public CSV/JSON/JSONL URLs.

Safety limits

• URL sources: max 200 MB (checked before download, then streamed with a byte cap).
• maxItems up to 1,000,000 rows. Regex filters: pattern ≤ 200 characters, matched against the first 10,000 characters of a value.
• Regex time limit: each regex search on one value may take at most 0.05 s (Python regex module with a timeout). If a pattern exceeds it (catastrophic backtracking, e.g. (a|aa)+$), the run fails fast with a clear "regex too slow" error and nothing is charged. We chose failing over silently treating the value as "no match", because a silent skip would give you a wrong, incomplete result that you would still pay for. Simplify the pattern (avoid nested quantifiers) and rerun.
• Renaming two fields to the same name is rejected instead of silently overwriting data. Rows that are not objects are wrapped as {"value": …}.

Security

• Only public http(s) URLs are fetched. Every URL and every redirect hop (max 5) is resolved and rejected if it points to a private, loopback, link-local (incl. the 169.254.169.254 cloud-metadata address), multicast or reserved address; other schemes (file:, ftp: …) are refused. The connection is pinned to the checked IP.
• Size limits are checked from Content-Length before downloading and enforced again while streaming, with an overall per-file deadline.
• Query strings and credentials in URLs are stripped from logs, dataset items and SUMMARY.

Spending limit

Nothing is charged until the whole source has loaded and parsed and all output files (JSON/CSV/XLSX) were built successfully. A source that fails to load (bad URL, blocked address, broken JSON/JSONL line, …) or an export that cannot be built ends the run with a clear error and $0 charged. Then actor-start and the input-row blocks are charged together. If your Max cost per run only covers part of the source, the actor processes only the rows that fit (whole 1,000-row blocks after the start fee) and says so in the status message and SUMMARY.stoppedByChargeLimit; nothing beyond the limit is charged.

Data retention & consent

• Retention: results are stored only in this run's own default dataset and key-value store, and are kept under Apify's standard storage retention for your plan (unnamed run storages are deleted automatically after the retention period). Nothing is copied elsewhere; no third-party API is called.
• Your responsibility: you must have the rights to the content you submit and, where it contains personal data, a lawful basis and any required consent (e.g. under GDPR/CCPA). Do not submit data you are not allowed to process.

Credits & licences

• httpx – BSD-3-Clause
• openpyxl – MIT
• Apify SDK for Python – Apache-2.0
• regex – Apache-2.0 (user regex filters)
• Python standard library csv/json/re – PSF License

Pricing

Pay-per-event – you pay only for what you process:

Event	Price
Actor start (actor-start)	$0.002 per run
Rows processed (rows-processed-1k)	$0.002 per 1,000 input rows (rounded up)

Charged on input rows (before filter/dedupe), and only after loading and building the exports succeeded. Example: a 2,500-row dataset costs $0.002 + 3 × $0.002 = $0.008.