Text Cleaner For RAG

Store Positioning

Store title: Text Cleaner For RAG

Short description: Clean web text for RAG by removing boilerplate, normalizing whitespace, and producing chunk-ready text.

SEO title: Text Cleaner For RAG — fast deterministic data utility

SEO description: Clean web text for RAG by removing boilerplate, normalizing whitespace, and producing chunk-ready text. Use it for fast, low-cost normalization or validation at scale with predictable pay-per-event pricing.

Categories: DEVELOPER_TOOLS

Keywords: text, cleaner, rag, web audit, tiny deterministic utility

Pay-Per-Event Pricing

This actor uses pay-per-event pricing. Event prices include Apify platform usage; users are not expected to pay a separate platform-usage pass-through charge for the configured pricing model.

• Primary event: identifier-validated at $0.00195 base

Event set:

• actor-start: $0.00176 when run setup is accepted.
• identifier-validated: $0.00195 for each cleaned chunk emitted to the dataset. This event name is fixed by the actor pricing contract.
• report-generated: $0.02000 when cleaned Markdown, chunk JSON, summary, and report files are created.

Public Task Concepts

• Normalize Text Cleaner For RAG records
• Validate Text Cleaner For RAG values in bulk
• Generate a small utility report for Text Cleaner For RAG
• Clean HTML into retrieval-ready text
• Chunk documents for a RAG pipeline

Clean web text for RAG by removing boilerplate, normalizing whitespace, and producing chunk-ready text from supplied text or HTML.

What This Actor Does

Text Cleaner For RAG accepts user-supplied text, HTML, or document objects and produces dataset rows that are ready for embedding or retrieval preprocessing:

• Strips script/style/svg/no-script blocks from HTML.
• Converts HTML structure into readable text.
• Removes common cookie, newsletter, social, navigation, and footer boilerplate lines.
• Deduplicates repeated normalized lines inside each document.
• Normalizes whitespace while preserving useful headings by default.
• Splits cleaned text into size-bounded chunks with optional overlap.
• Writes cleaned Markdown, chunk JSON, and Markdown/JSON summary reports.

What This Actor Does Not Do

• It does not crawl pages or fetch remote URLs.
• It does not call an embedding model, LLM, vector database, or paid API.
• It does not classify factual accuracy or content safety.
• It does not guarantee legal, privacy, or copyright compliance for uploaded text.

Best Use Cases

• Clean scraped page text before embedding.
• Remove repeated navigation or cookie banner text from documentation exports.
• Turn supplied HTML snippets into retrieval chunks.
• Create traceable chunk rows for retrieval QA and prompt experiments.
• Validate a small batch before running a larger ingestion pipeline.

Input Fields

• documents: Array of document objects with id, title, sourceUrl, text, or html.
• text: Optional single plain-text document.
• html: Optional single HTML document.
• chunkSize: Target maximum characters per chunk.
• chunkOverlap: Approximate character overlap between adjacent chunks.
• minChunkChars: Threshold for warning about short chunks.
• removeBoilerplate: Remove common web boilerplate lines.
• dedupeRepeatedLines: Drop repeated normalized lines.
• stripMarkdownLinks: Convert markdown links to visible text.
• preserveHeadings: Keep markdown heading markers for retrieval context.
• maxDocuments: Maximum documents to process. Default is 25 and hard cap is 1,000.
• maxItems: Maximum output chunks across all documents. Default is 100 and hard cap is 10,000.
• maxTextBytes: Maximum UTF-8 bytes cleaned from each document. Default is 1,000,000.
• includeReport: Store cleaned Markdown, chunk JSON, and summary report artifacts.

Example Input

{
  "text": "Cookie settings\n# Product Documentation\nThis guide explains production setup.\nThis guide explains production setup.",
  "chunkSize": 900,
  "chunkOverlap": 120,
  "removeBoilerplate": true,
  "dedupeRepeatedLines": true,
  "includeReport": true
}

Output Fields

Each dataset item represents one cleaned chunk:

• chunkId: Stable chunk identifier.
• documentId, title, sourceUrl: Source metadata.
• chunkIndex: Zero-based chunk index for that document.
• chunkText: Cleaned text ready for embedding or retrieval indexing.
• charCount, wordCount, tokenEstimate: Chunk size metrics.
• status: ready, short, or empty.
• removedLineCount, duplicateLineCount, boilerplateLineCount: Cleaning audit metrics.
• issues: Structured warnings or errors.
• recommendation: Short next step.

The key-value store also contains:

• CLEANED_TEXT.md: Cleaned Markdown grouped by document.
• RAG_CHUNKS.json: Full chunk array.
• TEXT_CLEANER_REPORT.md: Human-readable summary.
• TEXT_CLEANER_SUMMARY.json: Machine-readable summary.

Cost-Control Tips

• Start with one or two documents and default chunk settings.
• Keep maxDocuments low until cleaning rules are confirmed.
• Larger chunkSize usually emits fewer chunks.
• Turn reports off only when you do not need cleaned Markdown or chunk JSON artifacts.

FAQ

Does this fetch URLs?

No. Provide the text or HTML content directly. This avoids crawler cost and keeps tests deterministic.

Does this embed chunks?

No. It prepares chunks for downstream embeddings, vector stores, or RAG pipelines.

Can I keep boilerplate?

Yes. Set removeBoilerplate to false.

Troubleshooting

• Empty output: check whether the source only contains boilerplate or scripts/styles.
• Too many short chunks: increase chunkSize or lower minChunkChars.
• Important repeated text removed: set dedupeRepeatedLines to false.
• Markdown links look wrong: try stripMarkdownLinks.

Limitations

• Boilerplate detection is pattern-based, not ML-based.
• Chunking is character-oriented with paragraph awareness, not model-token exact.
• HTML extraction is intentionally simple and deterministic.

Source And Safety Notes

This actor processes user-provided text and HTML. Only upload text you are authorized to process in Apify and downstream systems.

Changelog

• 1.0.0: Initial production build with HTML stripping, boilerplate and duplicate-line cleanup, chunk generation, PPE billing, examples, and fixture tests.