LLM Dataset Cleaner: Near-Duplicate Dedupe, Language, Tokens

Clean crawled or scraped text before it goes into a RAG index or a fine-tuning set. The Actor reads a dataset (for example from Website Content Crawler or the PDF to Markdown Actor), a JSONL, JSON or CSV file, and in one pass removes exact and near duplicates, detects the language, flags low-quality text, counts or masks personal data, and counts tokens exactly. It can also split the kept records into RAG chunks, or compare a new crawl with the last one and tell you which items to re-embed and which to delete.

What it does

For each record, in input order:

1. Normalise the text: Unicode NFKC, control characters removed, runs of spaces collapsed. Line breaks and Markdown headings stay. A record longer than maxChars (default and maximum 200,000 characters, about 50,000 tokens) is rejected here as TOO_LONG and goes through none of the later steps.
2. Exact duplicates: identical normalised text to an earlier record is rejected as EXACT_DUPLICATE.
3. Near duplicates: MinHash with 128 permutations over 5-word shingles and LSH, verified on the full signature. A record whose estimated similarity to an earlier one reaches the threshold (default 0.85) is rejected as NEAR_DUPLICATE. The first occurrence is always kept; each duplicate gets cluster_id and duplicate_of.
4. Language with lingua on the first 2,000 characters. Optionally keep only the languages you list.
5. Quality flags, Gopher/C4-style rules of thumb: too short, low share of words with letters (Markdown markup and code are not counted as words), many symbols or digits, repeated lines, placeholder text and cookie banners. By default they are only flagged.
6. Personal data: emails, phone numbers in international format, IBANs (with the mod-97 check), card numbers (Luhn and brand prefix), IPv4 and IPv6 addresses. Count them in the text, or mask them as [EMAIL], [PHONE] and so on in the text, url, title and the keepFields values.
7. Exact token counts with OpenAI's o200k_base or cl100k_base tokenizer.
8. Optional RAG chunks that keep the Markdown heading path (Website Content Crawler and PDF chunk presets).

Everything runs inside the Actor's container. Nothing is sent to a language model or any other service.

What you get

• Dataset rows (record_type tells them apart):
  • record: one per kept record with its text, language, token count, quality flags and personal-data counts. Rejected records too, if you switch that on.
  • chunk: one per chunk, when chunking is on. These replace the record rows of kept records.
  • estimate: the record count and projected price, in estimate mode.
  • manifest_item: one per item, in manifest mode.
• Files in the run's key-value store:
  • cleaned.jsonl: one {"text": ...} line per kept record, ready for fine-tuning. Or chunks.jsonl with every chunk row.
  • rejected.jsonl: every rejected record with its reasons.
  • clusters.json: every duplicate group, with the record that was kept.
  • SUMMARY: counts by reason, a language histogram, token totals with p50/p90/p99 (within about 1%) and the exact maximum, personal-data counts by type, records read and charged, and why the run stopped early, if it did.

Long runs split the export files into parts of up to 20,000 records (cleaned-001.jsonl, cleaned-002.jsonl, ...); SUMMARY.exports lists the keys.

How to use it

1. Run Website Content Crawler or PDF to Markdown & RAG Chunks first.
2. Start this Actor, pick that run's dataset under Dataset to clean, and choose the matching Input preset.
3. Leave the defaults or adjust the steps, then start the run.
4. Open the Output tab: the Records view shows the kept text, the key-value store holds the export files.

Not sure what a big dataset will cost? Run it once with Mode: estimate. It counts the records and shows the projected price, and charges none of this Actor's events.

With no input at all, the Actor cleans a bundled sample of 300 synthetic records (with planted duplicates, four languages, low-quality text and fake personal data) so you can see the output first. The sample is charged none of this Actor's events; Apify's standard Actor start fee still applies, as it does to every run.

Input

{
  "datasetId": "YOUR_CRAWLER_RUN_DATASET_ID",
  "preset": "website-content-crawler",
  "languages": ["en"],
  "pii": "mask",
  "chunking": true,
  "chunkSize": 1500,
  "chunkOverlap": 150,
  "exportFormat": "rag-chunks"
}

Every field is optional. The main ones:

Field	Default	What it does
datasetId	none	Dataset to clean. Give exactly one source: this, keyValueStoreId with keyValueStoreKey, or sourceUrl.
keyValueStoreId, keyValueStoreKey	none	A JSONL, JSON array or CSV record in a key-value store.
sourceUrl	none	A direct link to a JSONL, JSON array or CSV file (UTF-8, UTF-16 or Windows-1252).
preset	generic	website-content-crawler reads markdown (else text), url and metadata.title. pdf-actor-chunks reads the PDF Actor's chunk rows and skips its other rows without charging them.
textFields	["text"]	Generic preset: the fields with the text, joined with a blank line. Dotted paths reach nested fields.
idField, keepFields	none, []	Your own record ID, and fields copied through into each row.
mode	clean	clean, estimate (free count and price) or manifest (incremental update plan, below).
exactDedup, nearDedup, nearDupThreshold	on, on, 0.85	Duplicate removal. Lower thresholds catch looser copies and use more memory.
languages, minLanguageConfidence	[], 0.5	ISO 639-1 codes to keep. Empty: detect only. no keeps both Norwegian forms (nb, nn).
quality, minChars, maxChars	flag, 50, 200000	off, flag or reject. Shorter records get the TOO_SHORT flag. Longer records are always rejected as TOO_LONG, whatever quality says; maxChars can be lowered, not raised above 200,000.
pii, piiTypes	flag, all six	off, flag (count) or mask (text, url, title and keepFields values).
tokenizer	o200k_base	Or cl100k_base.
chunking, chunkSize, chunkOverlap, chunkMaxTokens	off, 1500, 150, none	Chunk sizes are in characters. chunkMaxTokens (at least 64) re-chunks with smaller sizes until every chunk fits, down to 100 characters; a chunk that still does not fit (dense scripts, long unbroken strings) is delivered with over_max_tokens: true and counted in SUMMARY.chunks_over_max_tokens.
exportFormat	text-jsonl	text-jsonl (cleaned.jsonl), rag-chunks (chunks.jsonl, needs chunking) or none.
includeRejectedInDataset	off	Also put rejected records in the dataset (they always go to rejected.jsonl).
maxRecords	100000	Up to 500,000 per run.

Output

Rows from the bundled sample, trimmed where marked with .... A kept record with pii: "mask":

{
  "record_type": "record",
  "record_id": "31",
  "source_ref": "bundled-sample:sample.jsonl#31",
  "kept": true,
  "reasons": [],
  "cluster_id": null,
  "duplicate_of": null,
  "similarity_estimate": null,
  "language": "en",
  "language_confidence": 1.0,
  "chars": 1754,
  "tokens": 322,
  "quality_flags": [],
  "pii_counts": {"email": 1},
  "text": "A neighbourhood repair cafe reduced prices for early visitors after a long and wet winter....",
  "url": null,
  "title": null,
  "fields": {}
}

A rejected near duplicate (shown because includeRejectedInDataset was on):

{
  "record_type": "record",
  "record_id": "38",
  "kept": false,
  "reasons": ["NEAR_DUPLICATE"],
  "cluster_id": "c35",
  "duplicate_of": "35",
  "similarity_estimate": 1.0,
  "...": "..."
}

A chunk:

{
  "record_type": "chunk",
  "document_id": "0ae621f51c189fe2",
  "chunk_id": "56658acb9f9e97508372582f",
  "chunk_index": 1,
  "text": "Un nouveau jardin partagé a repeint l'entrée principale avec l'aide des habitant...",
  "char_start": 297,
  "char_end": 675,
  "heading_path": [],
  "page_start": null,
  "page_end": null,
  "approx_tokens": 95,
  "record_id": "0",
  "source_url": null,
  "tokens": 83,
  "source_ref": "bundled-sample:sample.jsonl#0"
}

Chunk rows have the same core fields (document_id, chunk_id, chunk_index, text, char_start, char_end, heading_path, page_start, page_end, approx_tokens) as the other Actors listed below, so their chunks can share one vector index. char_start and char_end are offsets into the record's cleaned text, tokens is the exact count with your tokenizer, and approx_tokens is the shared rough estimate (characters / 4).

source_ref is <source>#<position> with the record's 0-based position in the source. record_id is your idField value, or that position.

A run that cannot start or read its source fails with a message: INVALID_INPUT (the message names every bad field), SOURCE_NOT_FOUND, SOURCE_PARSE_ERROR (with the line number; the records before it are delivered and charged), FIELD_MISSING (more than 20% of the first records have no text in your text fields; nothing is charged), DOWNLOAD_FAILED, FILE_TOO_LARGE, UNSUPPORTED_FORMAT or BLOCKED_DESTINATION. A run that stops early still succeeds, and SUMMARY.stop_reason says why: CHARGE_LIMIT, TIMEOUT or RECORD_CAP_REACHED.

Manifest mode: incremental updates for a RAG index

Re-crawling a site every week and re-embedding everything is slow and costs money. Manifest mode compares this run's records with the manifest the previous run wrote and returns a plan: what is new, what changed, what is unchanged, what can be deleted, plus the next manifest. It never touches your vector database; you apply the plan.

Each item has two identities: its location (item_key, the source ID plus an optional locator such as the chunk index) and its content (content_hash, a hash of the normalised text). Keeping them apart means one passage can be cited from several places, a moved page reuses its embedding, and deleting a location never deletes content that is still cited elsewhere.

States per item: new, changed, unchanged, reindex_required (your declared parser or chunker version changed), unknown, removed, moved, conflict (the same item key twice in the input; the first wins) and other_scope. The action is upsert, delete, none or review. Upsert rows carry the text, ready to embed, and embedding_reusable tells you when that exact content was already embedded. With pii: "mask" the text is masked before it is hashed, so rows, the next manifest and the hashes all hold the masked text; keep the same pii and piiTypes from run to run, or every item shows as changed. (pii: "flag" does nothing in manifest mode.)

Deletions are conservative. An item missing from the input is proposed for deletion only when you set completeSnapshot to true for that manifestScope and the run read the whole input: no record cap, charge limit, timeout or source error. Otherwise missing items stay unknown and are carried into the next manifest unchanged. Items of a PDF document that failed, partly failed or was skipped in the PDF Actor run stay unknown too, as do items whose record has no text or no source ID.

Files: NEXT_MANIFEST.jsonl (pass it to the next run), and MANIFEST_PLAN.json with embed_contents (content hashes that need an embedding), delete_contents (hashes no longer cited anywhere), duplicate_groups and, when deletion was blocked, the reasons.

A weekly recipe with the PDF Actor, in plain shell with the Apify REST API (the same works with Website Content Crawler and preset website-content-crawler). It needs curl, jq and an APIFY_TOKEN:

API=https://api.apify.com/v2; AUTH="Authorization: Bearer $APIFY_TOKEN"
run_actor() {  # $1 = Actor, $2 = input JSON; prints the finished run
  local run id
  run=$(curl -s -X POST "$API/acts/$1/runs" -H "$AUTH" -H "Content-Type: application/json" -d "$2")
  id=$(echo "$run" | jq -r .data.id)
  while :; do
    run=$(curl -s "$API/actor-runs/$id?waitForFinish=60" -H "$AUTH")
    case $(echo "$run" | jq -r .data.status) in READY|RUNNING) ;; *) break ;; esac
  done
  echo "$run"
}

# 1. Convert this week's PDFs to chunks.
pdf=$(run_actor bongo-seakhoa~pdf-to-markdown-rag-chunks '{"sources": ["https://example.com/handbook.pdf"], "outputs": ["chunks"]}')
dataset=$(echo "$pdf" | jq -r .data.defaultDatasetId)

# 2. Compare the chunks with last week's manifest (the first week LAST_WEEK_STORE is empty: no previous manifest).
plan=$(run_actor bongo-seakhoa~llm-dataset-cleaner "$(jq -n --arg ds "$dataset" --arg prev "$LAST_WEEK_STORE" '{
  mode: "manifest", preset: "pdf-actor-chunks", datasetId: $ds, manifestScope: "handbooks",
  parserVersion: "pdf-actor-1", chunkerVersion: "1500/150", completeSnapshot: true,
  previousManifestKeyValueStoreId: $prev}')")

# 3. Apply the plan with your own code: embed and upsert the "upsert" rows, remove the "delete" rows.
curl -s "$API/datasets/$(echo "$plan" | jq -r .data.defaultDatasetId)/items?clean=true" -H "$AUTH"   | jq -c '.[] | select(.record_type == "manifest_item" and (.action == "upsert" or .action == "delete"))
           | {action, item_key, content_hash, embedding_reusable, text}' > changes.jsonl
LAST_WEEK_STORE=$(echo "$plan" | jq -r .data.defaultKeyValueStoreId)   # keep it for next week

The handoff is plain JSON rows, so any framework or vector database works. How it compares with LlamaIndex's built-in docstore and upserts: an IngestionPipeline with a docstore already skips unchanged documents by hash, and with upserts_and_delete it also deletes documents missing from the batch. That is simpler if your whole pipeline is LlamaIndex and every run sees the full corpus. This Actor's differences: deletions need an explicit complete-snapshot declaration and an error-free run, so a partial crawl or a failed PDF never wipes data; locations and content are tracked separately, so moved or duplicated passages reuse embeddings; parser and chunker versions are part of the state; and the plan is an inspectable file you can review before applying it, independent of any framework.

Pricing

This Actor is priced per event: you pay for what was processed.

Event	Price	When it is charged
record-processed	$0.0001 per record ($0.10 per 1,000)	Every record read and evaluated, kept or rejected (duplicates, filtered and TOO_LONG records included). Records with no text in your text fields are not charged. In manifest mode: every current record read plus every entry read from the previous manifest.
chunk-written	$0.00002 per chunk ($0.02 per 1,000)	Every chunk row, only when chunking is on.

Apify adds the Actor start fee to every run ($0.00005 per run at up to 1 GB of memory, once more for each extra GB). Dataset rows and files are not charged separately: this Actor does not use the apify-default-dataset-item event.

Worked examples:

• A 50,000-page Website Content Crawler dataset: 50,000 × $0.0001 = $5.00. It takes about 80 minutes at 1 GB, so raise the run timeout above the default hour, or use 2 GB.
• The same with chunking into about 200,000 chunks: add 200,000 × $0.00002 = $4.00.
• A weekly manifest run over 10,000 chunks with a 10,000-entry previous manifest: 20,000 × $0.0001 = $2.00.

Never charged by this Actor: the bundled sample, estimate mode, rows a preset skips (the PDF Actor's document and table rows), records with no text in your text fields (NO_TEXT) or with text the Actor cannot read (INVALID_TEXT), runs that fail before reading (invalid input, source not found, FIELD_MISSING), and records not read because of your maximum charge, the record cap or the run timeout. Only Apify's start fee applies to those runs.

Your maximum charge is respected. Before each batch of 500 records the Actor checks what is left of your run's maximum total charge and evaluates only the records and chunks it can charge for, each at its own price, keeping one record's price unspent below your limit. It then stops, writes the files for what it delivered, and the run finishes as Succeeded with stop_reason: "CHARGE_LIMIT". In manifest mode the previous manifest's entries are reserved first, so the plan and the next manifest are always written.

Limits: read before you buy

• Near-duplicate detection is approximate (MinHash). Pairs right around the threshold can be missed or merged. The first occurrence is always kept, even if a later copy is longer.
• What counts as a near duplicate: with 5-word shingles at 0.85, copies with a sentence added, removed or cut off, a word changed, or different case, punctuation and spacing are caught. Text where about 5% of the words are changed scattered throughout usually falls below 0.85 and is kept. This holds for records of about 100 words or more: in shorter records (product blurbs, reviews, Q&A) one changed word can drop the similarity below 0.85, so lower nearDupThreshold (for example to 0.7) for short-text datasets.
• Deduplication only sees records in the same run. Running twice on overlapping data does not dedupe across runs; manifest mode compares runs, but does not remove duplicates across them.
• Language detection reads the first 2,000 characters and is unreliable below about 20 words. Very short records get language null, and with a language filter they are rejected as LANGUAGE_FILTERED. Norwegian Bokmål and Nynorsk are often confused with each other; filter with no to keep both.
• Personal data: only emails, international phone numbers (with a leading +), IBANs, card numbers and IP addresses, found with patterns and checksums. Names, postal addresses and national IDs are not detected. This reduces personal data; it does not remove it. Do not rely on it for GDPR or similar compliance without your own review. It errs both ways: a version number such as 1.2.3.4 is masked as an IPv4 address, and a + followed by 8 to 15 digits as a phone number. Personal data inside other fields of your input is not looked for unless that field is your text, url, title or a keepFields value. In manifest mode mask applies to the text (and its content hash); source_id, the locator fields and item_key are identifiers and are copied as they are.
• Quality heuristics are rules of thumb and can flag good text. They only flag by default. The share-of-words-with-letters rule is skipped for text written mostly without spaces between words (Chinese, Japanese, Thai and similar).
• Record length: records longer than maxChars (at most 200,000 characters after normalising) are rejected as TOO_LONG without further processing, carry no text in their row, and are charged as rejected records. Split longer documents before cleaning, for example into the PDF Actor's chunks.
• At most 500,000 records per run. Split larger datasets. Memory: the default 1 GB is enough for about 100,000 records of typical web-page length; use 2 GB above that and 4 GB near 500,000 (manifest mode with a large previous manifest also needs more).
• Speed and run timeout: on Apify the Actor cleaned about 10 web pages per second at the default 1 GB and about 20 per second at 2 GB (synthetic pages averaging 3,600 characters, measured 2026-10-02). Apify's default run timeout of one hour therefore covers about 35,000 such pages at 1 GB. For bigger datasets raise the timeout in the run options, or the memory, which also adds CPU. Near the timeout the Actor stops reading (about 90 seconds before it; 2.5 minutes in manifest mode), delivers and charges only what it read, writes its files and SUMMARY, and the run succeeds with stop_reason: "TIMEOUT". In manifest mode the plan and the next manifest are still written, with deletions blocked. Manifest mode only normalises and hashes the text, so it is much faster (2,000 pages in under 10 seconds at 1 GB).
• Token counts are exact only for the chosen OpenAI tokenizer. Other models tokenise differently.
• Every record evaluated is charged, including duplicates and rejected records. Records with no text in your text fields are not.
• It does not crawl websites. Run Website Content Crawler first and pass its dataset. File links must point at a JSONL, JSON array or CSV file; web pages, PDFs and archives fail with UNSUPPORTED_FORMAT.
• Links: only public http(s) addresses are fetched, and every redirect is checked the same way. Links to private, local or cloud-metadata addresses, and links carrying a user name or password, fail with BLOCKED_DESTINATION and are not charged.
• Files: one source per run, up to a quarter of the run's memory (256 MB at 1 GB).
• CSV: the Actor reads CSV but writes no CSV files; its outputs are JSON and JSONL, which keep values exactly as they were.
• Restarts: if Apify restarts a run (a server migration), the Actor re-reads the records it already charged to rebuild its duplicate index, without charging them again. The records of the batch in progress may appear twice in the dataset, with the same record_id (and, for chunk rows, the same chunk_id); nothing is charged twice. Dedupe chunks by chunk_id and record rows by record_id.

FAQ

Is my data sent to an AI service? No. Language detection (lingua), tokenization (tiktoken with the vocabularies bundled in the image) and every other step run inside the Actor's container. There is no network access during processing apart from reading your source.

Which record is kept from a duplicate group? The first one in input order. clusters.json lists every group.

Why was a record I like rejected? The reasons field says why. With quality: "flag" (the default) quality rules never reject anything; check languages and the duplicate threshold.

Can I read the output from code? Yes. Use the dataset items endpoint and filter on record_type, or download cleaned.jsonl / chunks.jsonl from the run's key-value store.

Something does not work. Open an issue on the Actor's Issues tab with the run link. Issues are read on a best-effort basis; there is no guaranteed response time.

Privacy and data retention

The Actor reads only the dataset, key-value store record or URL you give it. Records are processed inside your run and are not sent anywhere else. The log contains counts, status and error codes, never record content, field values or URLs. The Actor keeps nothing after the run ends: the outputs live in your run's dataset and key-value store under your Apify account, and Apify's storage retention applies to them.

More Actors from the same developer

These Actors share one output format for chunks, so their results can go into the same index:

• PDF to Markdown & RAG Chunks: PDFs to Markdown, tables to CSV, offline OCR. Its chunk rows are this Actor's pdf-actor-chunks preset.

About the sample and the tests

The bundled sample and every test file used to develop this Actor are synthetic: generated for the purpose by scripts in the source and labelled synthetic (see sample/PROVENANCE.md). The personal data in them is fake: documentation IBANs, published test card numbers and example domains. No third-party text was used.

Licences

The Actor is proprietary software by Bongo Seakhoa. It is built on open-source components under permissive licences (the Apify SDK, numpy, xxhash, lingua-language-detector, tiktoken, charset-normalizer and httpx, among others). certifi (MPL-2.0) is included unmodified. The OpenAI tokenizer vocabularies are bundled unmodified; they carry no explicit licence statement. The full list and licence texts are in the THIRD_PARTY_NOTICES file shipped with the Actor image.