# LLM Dataset Cleaner: Near-Duplicate Dedupe, Language, Tokens (`bongo-seakhoa/llm-dataset-cleaner`) Actor Clean crawled or scraped text for RAG and fine-tuning: exact and MinHash near-duplicate removal, language filter, quality flags, PII masking, token counts. - **URL**: https://apify.com/bongo-seakhoa/llm-dataset-cleaner.md - **Developed by:** [Bongo Seakhoa](https://apify.com/bongo-seakhoa) (community) - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 1 bookmarks - **User rating**: No ratings yet ## Pricing from $0.10 / 1,000 record processeds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## LLM Dataset Cleaner: Near-Duplicate Dedupe, Language, Tokens Clean crawled or scraped text before it goes into a RAG index or a fine-tuning set. The Actor reads a dataset (for example from Website Content Crawler or the PDF to Markdown Actor), a JSONL, JSON or CSV file, and in one pass removes exact and near duplicates, detects the language, flags low-quality text, counts or masks personal data, and counts tokens exactly. It can also split the kept records into RAG chunks, or compare a new crawl with the last one and tell you which items to re-embed and which to delete. ### What it does For each record, in input order: 1. **Normalise** the text: Unicode NFKC, control characters removed, runs of spaces collapsed. Line breaks and Markdown headings stay. A record longer than `maxChars` (default and maximum 200,000 characters, about 50,000 tokens) is rejected here as `TOO_LONG` and goes through none of the later steps. 2. **Exact duplicates**: identical normalised text to an earlier record is rejected as `EXACT_DUPLICATE`. 3. **Near duplicates**: MinHash with 128 permutations over 5-word shingles and LSH, verified on the full signature. A record whose estimated similarity to an earlier one reaches the threshold (default 0.85) is rejected as `NEAR_DUPLICATE`. The first occurrence is always kept; each duplicate gets `cluster_id` and `duplicate_of`. 4. **Language** with lingua on the first 2,000 characters. Optionally keep only the languages you list. 5. **Quality flags**, Gopher/C4-style rules of thumb: too short, low share of words with letters (Markdown markup and code are not counted as words), many symbols or digits, repeated lines, placeholder text and cookie banners. By default they are only flagged. 6. **Personal data**: emails, phone numbers in international format, IBANs (with the mod-97 check), card numbers (Luhn and brand prefix), IPv4 and IPv6 addresses. Count them in the text, or mask them as `[EMAIL]`, `[PHONE]` and so on in the text, `url`, `title` and the `keepFields` values. 7. **Exact token counts** with OpenAI's `o200k_base` or `cl100k_base` tokenizer. 8. **Optional RAG chunks** that keep the Markdown heading path (Website Content Crawler and PDF chunk presets). Everything runs inside the Actor's container. Nothing is sent to a language model or any other service. ### What you get - **Dataset rows** (`record_type` tells them apart): - `record`: one per kept record with its text, language, token count, quality flags and personal-data counts. Rejected records too, if you switch that on. - `chunk`: one per chunk, when chunking is on. These replace the record rows of kept records. - `estimate`: the record count and projected price, in estimate mode. - `manifest_item`: one per item, in manifest mode. - **Files** in the run's key-value store: - `cleaned.jsonl`: one `{"text": ...}` line per kept record, ready for fine-tuning. Or `chunks.jsonl` with every chunk row. - `rejected.jsonl`: every rejected record with its reasons. - `clusters.json`: every duplicate group, with the record that was kept. - `SUMMARY`: counts by reason, a language histogram, token totals with p50/p90/p99 (within about 1%) and the exact maximum, personal-data counts by type, records read and charged, and why the run stopped early, if it did. Long runs split the export files into parts of up to 20,000 records (`cleaned-001.jsonl`, `cleaned-002.jsonl`, ...); `SUMMARY.exports` lists the keys. ### How to use it 1. Run [Website Content Crawler](https://apify.com/apify/website-content-crawler) or [PDF to Markdown & RAG Chunks](https://apify.com/bongo-seakhoa/pdf-to-markdown-rag-chunks) first. 2. Start this Actor, pick that run's dataset under **Dataset to clean**, and choose the matching **Input preset**. 3. Leave the defaults or adjust the steps, then start the run. 4. Open the **Output** tab: the **Records** view shows the kept text, the key-value store holds the export files. Not sure what a big dataset will cost? Run it once with **Mode: estimate**. It counts the records and shows the projected price, and charges none of this Actor's events. With no input at all, the Actor cleans a bundled sample of 300 synthetic records (with planted duplicates, four languages, low-quality text and fake personal data) so you can see the output first. The sample is charged none of this Actor's events; Apify's standard Actor start fee still applies, as it does to every run. ### Input ```json { "datasetId": "YOUR_CRAWLER_RUN_DATASET_ID", "preset": "website-content-crawler", "languages": ["en"], "pii": "mask", "chunking": true, "chunkSize": 1500, "chunkOverlap": 150, "exportFormat": "rag-chunks" } ``` Every field is optional. The main ones: | Field | Default | What it does | |---|---|---| | `datasetId` | none | Dataset to clean. Give exactly one source: this, `keyValueStoreId` with `keyValueStoreKey`, or `sourceUrl`. | | `keyValueStoreId`, `keyValueStoreKey` | none | A JSONL, JSON array or CSV record in a key-value store. | | `sourceUrl` | none | A direct link to a JSONL, JSON array or CSV file (UTF-8, UTF-16 or Windows-1252). | | `preset` | `generic` | `website-content-crawler` reads `markdown` (else `text`), `url` and `metadata.title`. `pdf-actor-chunks` reads the PDF Actor's chunk rows and skips its other rows without charging them. | | `textFields` | `["text"]` | Generic preset: the fields with the text, joined with a blank line. Dotted paths reach nested fields. | | `idField`, `keepFields` | none, `[]` | Your own record ID, and fields copied through into each row. | | `mode` | `clean` | `clean`, `estimate` (free count and price) or `manifest` (incremental update plan, below). | | `exactDedup`, `nearDedup`, `nearDupThreshold` | on, on, `0.85` | Duplicate removal. Lower thresholds catch looser copies and use more memory. | | `languages`, `minLanguageConfidence` | `[]`, `0.5` | ISO 639-1 codes to keep. Empty: detect only. `no` keeps both Norwegian forms (`nb`, `nn`). | | `quality`, `minChars`, `maxChars` | `flag`, `50`, `200000` | `off`, `flag` or `reject`. Shorter records get the `TOO_SHORT` flag. Longer records are always rejected as `TOO_LONG`, whatever `quality` says; `maxChars` can be lowered, not raised above 200,000. | | `pii`, `piiTypes` | `flag`, all six | `off`, `flag` (count) or `mask` (text, `url`, `title` and `keepFields` values). | | `tokenizer` | `o200k_base` | Or `cl100k_base`. | | `chunking`, `chunkSize`, `chunkOverlap`, `chunkMaxTokens` | off, `1500`, `150`, none | Chunk sizes are in characters. `chunkMaxTokens` (at least 64) re-chunks with smaller sizes until every chunk fits, down to 100 characters; a chunk that still does not fit (dense scripts, long unbroken strings) is delivered with `over_max_tokens: true` and counted in `SUMMARY.chunks_over_max_tokens`. | | `exportFormat` | `text-jsonl` | `text-jsonl` (cleaned.jsonl), `rag-chunks` (chunks.jsonl, needs chunking) or `none`. | | `includeRejectedInDataset` | off | Also put rejected records in the dataset (they always go to rejected.jsonl). | | `maxRecords` | `100000` | Up to 500,000 per run. | ### Output Rows from the bundled sample, trimmed where marked with `...`. A kept record with `pii: "mask"`: ```json { "record_type": "record", "record_id": "31", "source_ref": "bundled-sample:sample.jsonl#31", "kept": true, "reasons": [], "cluster_id": null, "duplicate_of": null, "similarity_estimate": null, "language": "en", "language_confidence": 1.0, "chars": 1754, "tokens": 322, "quality_flags": [], "pii_counts": {"email": 1}, "text": "A neighbourhood repair cafe reduced prices for early visitors after a long and wet winter....", "url": null, "title": null, "fields": {} } ``` A rejected near duplicate (shown because `includeRejectedInDataset` was on): ```json { "record_type": "record", "record_id": "38", "kept": false, "reasons": ["NEAR_DUPLICATE"], "cluster_id": "c35", "duplicate_of": "35", "similarity_estimate": 1.0, "...": "..." } ``` A chunk: ```json { "record_type": "chunk", "document_id": "0ae621f51c189fe2", "chunk_id": "56658acb9f9e97508372582f", "chunk_index": 1, "text": "Un nouveau jardin partagé a repeint l'entrée principale avec l'aide des habitant...", "char_start": 297, "char_end": 675, "heading_path": [], "page_start": null, "page_end": null, "approx_tokens": 95, "record_id": "0", "source_url": null, "tokens": 83, "source_ref": "bundled-sample:sample.jsonl#0" } ``` Chunk rows have the same core fields (`document_id`, `chunk_id`, `chunk_index`, `text`, `char_start`, `char_end`, `heading_path`, `page_start`, `page_end`, `approx_tokens`) as the other Actors listed below, so their chunks can share one vector index. `char_start` and `char_end` are offsets into the record's cleaned text, `tokens` is the exact count with your tokenizer, and `approx_tokens` is the shared rough estimate (characters / 4). `source_ref` is `#` with the record's 0-based position in the source. `record_id` is your `idField` value, or that position. A run that cannot start or read its source fails with a message: `INVALID_INPUT` (the message names every bad field), `SOURCE_NOT_FOUND`, `SOURCE_PARSE_ERROR` (with the line number; the records before it are delivered and charged), `FIELD_MISSING` (more than 20% of the first records have no text in your text fields; nothing is charged), `DOWNLOAD_FAILED`, `FILE_TOO_LARGE`, `UNSUPPORTED_FORMAT` or `BLOCKED_DESTINATION`. A run that stops early still succeeds, and `SUMMARY.stop_reason` says why: `CHARGE_LIMIT`, `TIMEOUT` or `RECORD_CAP_REACHED`. ### Manifest mode: incremental updates for a RAG index Re-crawling a site every week and re-embedding everything is slow and costs money. Manifest mode compares this run's records with the manifest the previous run wrote and returns a plan: what is new, what changed, what is unchanged, what can be deleted, plus the next manifest. It never touches your vector database; you apply the plan. Each item has two identities: its **location** (`item_key`, the source ID plus an optional locator such as the chunk index) and its **content** (`content_hash`, a hash of the normalised text). Keeping them apart means one passage can be cited from several places, a moved page reuses its embedding, and deleting a location never deletes content that is still cited elsewhere. States per item: `new`, `changed`, `unchanged`, `reindex_required` (your declared parser or chunker version changed), `unknown`, `removed`, `moved`, `conflict` (the same item key twice in the input; the first wins) and `other_scope`. The action is `upsert`, `delete`, `none` or `review`. Upsert rows carry the text, ready to embed, and `embedding_reusable` tells you when that exact content was already embedded. With `pii: "mask"` the text is masked before it is hashed, so rows, the next manifest and the hashes all hold the masked text; keep the same `pii` and `piiTypes` from run to run, or every item shows as changed. (`pii: "flag"` does nothing in manifest mode.) **Deletions are conservative.** An item missing from the input is proposed for deletion only when you set `completeSnapshot` to true for that `manifestScope` and the run read the whole input: no record cap, charge limit, timeout or source error. Otherwise missing items stay `unknown` and are carried into the next manifest unchanged. Items of a PDF document that failed, partly failed or was skipped in the PDF Actor run stay `unknown` too, as do items whose record has no text or no source ID. Files: `NEXT_MANIFEST.jsonl` (pass it to the next run), and `MANIFEST_PLAN.json` with `embed_contents` (content hashes that need an embedding), `delete_contents` (hashes no longer cited anywhere), `duplicate_groups` and, when deletion was blocked, the reasons. A weekly recipe with the PDF Actor, in plain shell with the Apify REST API (the same works with Website Content Crawler and preset `website-content-crawler`). It needs `curl`, `jq` and an `APIFY_TOKEN`: ```bash API=https://api.apify.com/v2; AUTH="Authorization: Bearer $APIFY_TOKEN" run_actor() { # $1 = Actor, $2 = input JSON; prints the finished run local run id run=$(curl -s -X POST "$API/acts/$1/runs" -H "$AUTH" -H "Content-Type: application/json" -d "$2") id=$(echo "$run" | jq -r .data.id) while :; do run=$(curl -s "$API/actor-runs/$id?waitForFinish=60" -H "$AUTH") case $(echo "$run" | jq -r .data.status) in READY|RUNNING) ;; *) break ;; esac done echo "$run" } ## 1. Convert this week's PDFs to chunks. pdf=$(run_actor bongo-seakhoa~pdf-to-markdown-rag-chunks '{"sources": ["https://example.com/handbook.pdf"], "outputs": ["chunks"]}') dataset=$(echo "$pdf" | jq -r .data.defaultDatasetId) ## 2. Compare the chunks with last week's manifest (the first week LAST_WEEK_STORE is empty: no previous manifest). plan=$(run_actor bongo-seakhoa~llm-dataset-cleaner "$(jq -n --arg ds "$dataset" --arg prev "$LAST_WEEK_STORE" '{ mode: "manifest", preset: "pdf-actor-chunks", datasetId: $ds, manifestScope: "handbooks", parserVersion: "pdf-actor-1", chunkerVersion: "1500/150", completeSnapshot: true, previousManifestKeyValueStoreId: $prev}')") ## 3. Apply the plan with your own code: embed and upsert the "upsert" rows, remove the "delete" rows. curl -s "$API/datasets/$(echo "$plan" | jq -r .data.defaultDatasetId)/items?clean=true" -H "$AUTH" | jq -c '.[] | select(.record_type == "manifest_item" and (.action == "upsert" or .action == "delete")) | {action, item_key, content_hash, embedding_reusable, text}' > changes.jsonl LAST_WEEK_STORE=$(echo "$plan" | jq -r .data.defaultKeyValueStoreId) # keep it for next week ``` The handoff is plain JSON rows, so any framework or vector database works. How it compares with LlamaIndex's built-in docstore and upserts: an `IngestionPipeline` with a docstore already skips unchanged documents by hash, and with `upserts_and_delete` it also deletes documents missing from the batch. That is simpler if your whole pipeline is LlamaIndex and every run sees the full corpus. This Actor's differences: deletions need an explicit complete-snapshot declaration and an error-free run, so a partial crawl or a failed PDF never wipes data; locations and content are tracked separately, so moved or duplicated passages reuse embeddings; parser and chunker versions are part of the state; and the plan is an inspectable file you can review before applying it, independent of any framework. ### Pricing This Actor is priced per event: you pay for what was processed. | Event | Price | When it is charged | |---|---|---| | `record-processed` | $0.0001 per record ($0.10 per 1,000) | Every record read and evaluated, kept or rejected (duplicates, filtered and `TOO_LONG` records included). Records with no text in your text fields are not charged. In manifest mode: every current record read plus every entry read from the previous manifest. | | `chunk-written` | $0.00002 per chunk ($0.02 per 1,000) | Every chunk row, only when chunking is on. | Apify adds the Actor start fee to every run ($0.00005 per run at up to 1 GB of memory, once more for each extra GB). Dataset rows and files are not charged separately: this Actor does not use the `apify-default-dataset-item` event. Worked examples: - A 50,000-page Website Content Crawler dataset: 50,000 × $0.0001 = $5.00. It takes about 80 minutes at 1 GB, so raise the run timeout above the default hour, or use 2 GB. - The same with chunking into about 200,000 chunks: add 200,000 × $0.00002 = $4.00. - A weekly manifest run over 10,000 chunks with a 10,000-entry previous manifest: 20,000 × $0.0001 = $2.00. Never charged by this Actor: the bundled sample, estimate mode, rows a preset skips (the PDF Actor's document and table rows), records with no text in your text fields (`NO_TEXT`) or with text the Actor cannot read (`INVALID_TEXT`), runs that fail before reading (invalid input, source not found, `FIELD_MISSING`), and records not read because of your maximum charge, the record cap or the run timeout. Only Apify's start fee applies to those runs. **Your maximum charge is respected.** Before each batch of 500 records the Actor checks what is left of your run's maximum total charge and evaluates only the records and chunks it can charge for, each at its own price, keeping one record's price unspent below your limit. It then stops, writes the files for what it delivered, and the run finishes as *Succeeded* with `stop_reason: "CHARGE_LIMIT"`. In manifest mode the previous manifest's entries are reserved first, so the plan and the next manifest are always written. ### Limits: read before you buy - **Near-duplicate detection is approximate** (MinHash). Pairs right around the threshold can be missed or merged. The first occurrence is always kept, even if a later copy is longer. - **What counts as a near duplicate**: with 5-word shingles at 0.85, copies with a sentence added, removed or cut off, a word changed, or different case, punctuation and spacing are caught. Text where about 5% of the words are changed *scattered* throughout usually falls below 0.85 and is kept. This holds for records of about 100 words or more: in shorter records (product blurbs, reviews, Q\&A) one changed word can drop the similarity below 0.85, so lower `nearDupThreshold` (for example to 0.7) for short-text datasets. - **Deduplication only sees records in the same run.** Running twice on overlapping data does not dedupe across runs; manifest mode compares runs, but does not remove duplicates across them. - **Language detection** reads the first 2,000 characters and is unreliable below about 20 words. Very short records get language `null`, and with a language filter they are rejected as `LANGUAGE_FILTERED`. Norwegian Bokmål and Nynorsk are often confused with each other; filter with `no` to keep both. - **Personal data**: only emails, international phone numbers (with a leading `+`), IBANs, card numbers and IP addresses, found with patterns and checksums. Names, postal addresses and national IDs are not detected. This reduces personal data; it does not remove it. Do not rely on it for GDPR or similar compliance without your own review. It errs both ways: a version number such as `1.2.3.4` is masked as an IPv4 address, and a `+` followed by 8 to 15 digits as a phone number. Personal data inside other fields of your input is not looked for unless that field is your text, `url`, `title` or a `keepFields` value. In manifest mode `mask` applies to the text (and its content hash); `source_id`, the locator fields and `item_key` are identifiers and are copied as they are. - **Quality heuristics** are rules of thumb and can flag good text. They only flag by default. The share-of-words-with-letters rule is skipped for text written mostly without spaces between words (Chinese, Japanese, Thai and similar). - **Record length**: records longer than `maxChars` (at most 200,000 characters after normalising) are rejected as `TOO_LONG` without further processing, carry no text in their row, and are charged as rejected records. Split longer documents before cleaning, for example into the PDF Actor's chunks. - **At most 500,000 records per run.** Split larger datasets. Memory: the default 1 GB is enough for about 100,000 records of typical web-page length; use 2 GB above that and 4 GB near 500,000 (manifest mode with a large previous manifest also needs more). - **Speed and run timeout**: on Apify the Actor cleaned about 10 web pages per second at the default 1 GB and about 20 per second at 2 GB (synthetic pages averaging 3,600 characters, measured 2026-10-02). Apify's default run timeout of one hour therefore covers about 35,000 such pages at 1 GB. For bigger datasets raise the timeout in the run options, or the memory, which also adds CPU. Near the timeout the Actor stops reading (about 90 seconds before it; 2.5 minutes in manifest mode), delivers and charges only what it read, writes its files and SUMMARY, and the run succeeds with `stop_reason: "TIMEOUT"`. In manifest mode the plan and the next manifest are still written, with deletions blocked. Manifest mode only normalises and hashes the text, so it is much faster (2,000 pages in under 10 seconds at 1 GB). - **Token counts** are exact only for the chosen OpenAI tokenizer. Other models tokenise differently. - **Every record evaluated is charged**, including duplicates and rejected records. Records with no text in your text fields are not. - **It does not crawl websites.** Run Website Content Crawler first and pass its dataset. File links must point at a JSONL, JSON array or CSV file; web pages, PDFs and archives fail with `UNSUPPORTED_FORMAT`. - **Links**: only public http(s) addresses are fetched, and every redirect is checked the same way. Links to private, local or cloud-metadata addresses, and links carrying a user name or password, fail with `BLOCKED_DESTINATION` and are not charged. - **Files**: one source per run, up to a quarter of the run's memory (256 MB at 1 GB). - **CSV**: the Actor reads CSV but writes no CSV files; its outputs are JSON and JSONL, which keep values exactly as they were. - **Restarts**: if Apify restarts a run (a server migration), the Actor re-reads the records it already charged to rebuild its duplicate index, without charging them again. The records of the batch in progress may appear twice in the dataset, with the same `record_id` (and, for chunk rows, the same `chunk_id`); nothing is charged twice. Dedupe chunks by `chunk_id` and record rows by `record_id`. ### FAQ **Is my data sent to an AI service?** No. Language detection (lingua), tokenization (tiktoken with the vocabularies bundled in the image) and every other step run inside the Actor's container. There is no network access during processing apart from reading your source. **Which record is kept from a duplicate group?** The first one in input order. `clusters.json` lists every group. **Why was a record I like rejected?** The `reasons` field says why. With `quality: "flag"` (the default) quality rules never reject anything; check `languages` and the duplicate threshold. **Can I read the output from code?** Yes. Use the dataset items endpoint and filter on `record_type`, or download `cleaned.jsonl` / `chunks.jsonl` from the run's key-value store. **Something does not work.** Open an issue on the Actor's Issues tab with the run link. Issues are read on a best-effort basis; there is no guaranteed response time. ### Privacy and data retention The Actor reads only the dataset, key-value store record or URL you give it. Records are processed inside your run and are not sent anywhere else. The log contains counts, status and error codes, never record content, field values or URLs. The Actor keeps nothing after the run ends: the outputs live in your run's dataset and key-value store under your Apify account, and Apify's storage retention applies to them. ### More Actors from the same developer These Actors share one output format for chunks, so their results can go into the same index: - [PDF to Markdown & RAG Chunks](https://apify.com/bongo-seakhoa/pdf-to-markdown-rag-chunks): PDFs to Markdown, tables to CSV, offline OCR. Its chunk rows are this Actor's `pdf-actor-chunks` preset. ### About the sample and the tests The bundled sample and every test file used to develop this Actor are synthetic: generated for the purpose by scripts in the source and labelled synthetic (see `sample/PROVENANCE.md`). The personal data in them is fake: documentation IBANs, published test card numbers and example domains. No third-party text was used. ### Licences The Actor is proprietary software by Bongo Seakhoa. It is built on open-source components under permissive licences (the Apify SDK, numpy, xxhash, lingua-language-detector, tiktoken, charset-normalizer and httpx, among others). certifi (MPL-2.0) is included unmodified. The OpenAI tokenizer vocabularies are bundled unmodified; they carry no explicit licence statement. The full list and licence texts are in the `THIRD_PARTY_NOTICES` file shipped with the Actor image. # Actor input Schema ## `datasetId` (type: `string`): Pick a dataset, for example the output of Website Content Crawler or of the PDF to Markdown Actor. Give exactly one source: a dataset, a key-value store record, or a file URL. With no source at all, the Actor cleans a small bundled synthetic sample (300 records) so you can see the output; the sample is never charged. ## `keyValueStoreId` (type: `string`): A key-value store holding a JSONL, JSON array or CSV file. Use it together with the record key below. ## `keyValueStoreKey` (type: `string`): Key of the JSONL, JSON or CSV record in the store picked above. ## `sourceUrl` (type: `string`): A direct http(s) link to a JSONL, JSON array or CSV file (UTF-8, UTF-16 or Windows-1252). Only public internet hosts are fetched. The Actor does not crawl web pages: run Website Content Crawler first and pick its dataset above. ## `preset` (type: `string`): 'website-content-crawler' reads 'markdown' (falling back to 'text'), 'url' and 'metadata.title' and keeps the Markdown heading path for chunks. 'pdf-actor-chunks' reads the chunk rows of the PDF to Markdown Actor (record_type 'chunk'); its other rows are skipped and not charged. 'generic' reads the text fields below. ## `textFields` (type: `array`): Fields holding the text (generic preset). Several fields are joined with a blank line. Dotted paths such as 'metadata.description' reach into nested objects. ## `idField` (type: `string`): Field with each record's own ID, copied to record_id. Without it, record_id is the record's position in the source (0-based). ## `keepFields` (type: `array`): Other fields copied unchanged into each output row under 'fields'. ## `mode` (type: `string`): 'clean' runs the pipeline. 'estimate' only counts the records and shows the projected price; it charges none of this Actor's events (Apify's standard Actor start fee still applies). 'manifest' compares the records with a previous run's manifest and returns an incremental update plan for a RAG index (see the README). ## `normalize` (type: `boolean`): Unicode NFKC, control characters removed, runs of spaces collapsed, line structure kept. Off: output text is the original text (dedup still compares normalised text). ## `exactDedup` (type: `boolean`): Records whose normalised text is identical to an earlier record are rejected as EXACT_DUPLICATE. The first occurrence is kept. ## `nearDedup` (type: `boolean`): MinHash (128 permutations, 5-word shingles) with LSH. Records at or above the similarity threshold to an earlier record are rejected as NEAR_DUPLICATE. The first occurrence is kept. ## `nearDupThreshold` (type: `number`): Estimated Jaccard similarity of 5-word shingles at which a record counts as a near duplicate. Lower values catch looser copies and use more memory. ## `languages` (type: `array`): ISO 639-1 codes such as 'en' or 'de' ('no' keeps both Norwegian forms, nb and nn). Empty: detect the language only, filter nothing. Records in other languages, or too short to tell, are rejected as LANGUAGE_FILTERED. ## `minLanguageConfidence` (type: `number`): Below this confidence the language is reported as null. ## `quality` (type: `string`): Rules of thumb (length, letter, symbol and digit ratios, repeated lines, placeholder text, cookie banners). 'flag' only reports them in quality_flags; 'reject' also rejects the record. ## `minChars` (type: `integer`): Shorter records get the TOO_SHORT flag. ## `maxChars` (type: `integer`): Longer records (after normalising) are always rejected as TOO_LONG, whatever 'Quality rules' says, and are not processed further. At most 200,000 characters (about 50,000 tokens). ## `pii` (type: `string`): Emails, phone numbers in international format, IBANs (checksum), card numbers (Luhn) and IP addresses. 'flag' counts them per record; 'mask' also replaces them with \[EMAIL], \[PHONE] and so on, in the text, url, title and kept fields (in manifest mode: in the text). This reduces personal data; it does not remove names, addresses or national IDs. ## `piiTypes` (type: `array`): Which kinds of personal data to look for. ## `tokenizer` (type: `string`): Exact token counts with this OpenAI tokenizer (GPT-4o family: o200k_base; GPT-4 and GPT-3.5: cl100k_base). ## `chunking` (type: `boolean`): Kept records become chunk rows (record_type 'chunk') instead of record rows. Each chunk is charged as 'chunk-written' on top of the per-record price. ## `chunkSize` (type: `integer`): Target maximum length of each chunk. Chunks break at headings, paragraphs and sentences. ## `chunkOverlap` (type: `integer`): Characters repeated between consecutive chunks. At most half the chunk size. ## `chunkMaxTokens` (type: `integer`): When a chunk has more tokens than this (with the tokenizer above), the record is chunked again with a smaller size until every chunk fits, down to 100 characters. A chunk that still does not fit is delivered with over_max_tokens: true. ## `exportFormat` (type: `string`): 'text-jsonl' writes cleaned.jsonl with one {"text": ...} line per kept record (fine-tuning format). 'rag-chunks' writes chunks.jsonl with every chunk row (needs chunking). rejected.jsonl with the reasons is always written. ## `includeRejectedInDataset` (type: `boolean`): Rejected records always go to rejected.jsonl. On: they also become dataset rows with kept = false. ## `maxRecords` (type: `integer`): Stop after this many records (up to 500,000 per run). The SUMMARY then says RECORD_CAP_REACHED. Split larger datasets over several runs; deduplication only sees records within one run. ## `manifestScope` (type: `string`): Manifest mode: a name for the collection this input covers, for example 'docs-site'. Only previous entries of the same scope are compared; others are carried forward untouched. ## `sourceIdField` (type: `string`): Manifest mode: the field identifying each item's source. Defaults: 'url' (Website Content Crawler preset), 'source' (PDF Actor chunks), else the ID field. ## `locatorFields` (type: `array`): Manifest mode: fields that locate an item inside its source; joined with '|'. Default: 'chunk_index' for PDF Actor chunks, none otherwise. ## `parserVersion` (type: `string`): Manifest mode: the version of whatever produced the text (for example 'pdf-markdown-actor 1.0'). When it differs from the previous manifest, unchanged items are marked reindex_required. ## `chunkerVersion` (type: `string`): Manifest mode: the version of your chunking settings. A change marks unchanged items reindex_required. ## `completeSnapshot` (type: `boolean`): Manifest mode: only when this is on (and the whole input was read without errors, caps or charge limits) are items missing from the input proposed for deletion. Off: missing items stay 'unknown' and are carried forward. ## `previousManifestKeyValueStoreId` (type: `string`): Manifest mode: the key-value store of an earlier manifest run (its NEXT_MANIFEST.jsonl). Leave empty for the first run. ## `previousManifestKey` (type: `string`): Key of the manifest in the store above. ## `previousManifestDatasetId` (type: `string`): Manifest mode, instead of the store: the dataset of an earlier manifest run. Its manifest_item rows with in_next_manifest = true are read. ## `includeTextInManifest` (type: `boolean`): Manifest mode: put the normalised text into rows whose action is 'upsert', ready to embed. ## Actor input object example ```json { "preset": "generic", "textFields": [ "text" ], "keepFields": [], "mode": "clean", "normalize": true, "exactDedup": true, "nearDedup": true, "nearDupThreshold": 0.85, "languages": [], "minLanguageConfidence": 0.5, "quality": "flag", "minChars": 50, "maxChars": 200000, "pii": "flag", "piiTypes": [ "email", "phone", "iban", "card", "ipv4", "ipv6" ], "tokenizer": "o200k_base", "chunking": false, "chunkSize": 1500, "chunkOverlap": 150, "exportFormat": "text-jsonl", "includeRejectedInDataset": false, "maxRecords": 100000, "parserVersion": "unspecified", "chunkerVersion": "unspecified", "completeSnapshot": false, "previousManifestKey": "NEXT_MANIFEST.jsonl", "includeTextInManifest": true } ``` # Actor output Schema ## `results` (type: `string`): Kept records (record_type 'record'), RAG chunks ('chunk'), the estimate ('estimate') or manifest rows ('manifest_item'). ## `files` (type: `string`): cleaned.jsonl or chunks.jsonl, rejected.jsonl, clusters.json, NEXT_MANIFEST.jsonl, MANIFEST_PLAN.json and the run SUMMARY, in the run's key-value store. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = {}; // Run the Actor and wait for it to finish const run = await client.actor("bongo-seakhoa/llm-dataset-cleaner").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = {} # Run the Actor and wait for it to finish run = client.actor("bongo-seakhoa/llm-dataset-cleaner").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{}' | apify call bongo-seakhoa/llm-dataset-cleaner --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,bongo-seakhoa/llm-dataset-cleaner" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/qGzJjEoMTI0m7anJE/builds/6FSYmhydBHT0fG4d8/openapi.json