Fix Broken Text in Scraped Data - Mojibake, Entities, Language

Repairs the broken text that scrapers leave behind: Café de la Paix, don’t, Tom & Jerry, zero-width characters, odd spaces, and adds a detected-language column. For anyone who feeds scraped data to an LLM, a CRM, a spreadsheet or a search index and is tired of garbled characters.

Quick start

1. Run your scraper, then start this Actor and pick its dataset in Dataset(s) to fix.
2. Every text field (including nested ones) is cleaned in place; textFixes lists what changed per field.
3. Download JSON, CSV or Excel. Run with empty input to see a demo of 6 broken rows.

Input examples

1. Dataset from a scraper

{ "datasets": ["<pick with the dataset picker>"] }

2. Pasted rows, language from the description only

{ "rows": [{ "title": "Café de Paris", "description": "À bientôt! Tom & Jerry" }], "languageFields": ["description"] }

3. Keep the originals, straighten quotes, skip a raw-HTML field

{ "rows": [{ "text": "don’t", "html": "<p>&amp;</p>" }], "keepOriginals": true, "straightenQuotes": true, "skipFields": ["html"] }

Input guide

Field	What it does
datasets, rows, rowsCsv	Where the rows come from.
fixMojibake	Repair UTF-8 read as Windows-1252/Latin-1 (also double-encoded, Cyrillic, emoji).
decodeHtmlEntities	&, ', é,  ... to characters.
unicodeNormalization	NFC (default), NFKC or none.
keepNewlines	Keep line breaks (3+ become 2) or flatten to single spaces.
straightenQuotes	Curly to straight quotes (off by default).
detectLanguage, languageFields	Add language columns; limit detection to chosen fields.
skipFields	Fields that must not be touched.
keepOriginals	Store the original text of changed fields in originalText.

What gets fixed

• Mojibake: Café -> Café, don’t -> don’t, € -> €, Привет -> Привет, 😀 -> 😀. Only sequences that decode to valid UTF-8 are changed.
• HTML entities, also double-escaped ones (&).
• Invisible characters: zero-width spaces, BOM, soft hyphens, bidirectional override marks (also a security risk), control characters. Zero-width joiners are kept next to emoji and in Indic/Arabic text.
• Whitespace: no-break and exotic spaces become normal spaces, runs of spaces collapse, line endings unify, text is trimmed.
• URL and e-mail values are left alone.

Output

Your fields (cleaned) plus:

Field	Meaning
textFixes	{ "title": ["mojibake"], "description": ["html-entities","whitespace"] }. Codes: mojibake, html-entities, invisible-chars, whitespace, unicode-normalized, quotes-straightened, unrecoverable-replacement-chars.
fixedFieldCount	Number of fields that changed.
language, languageName	ISO 639-1 code (or und) and name.
languageConfidence	0-100, how far the best guess is ahead of the runner-up. Low for short text and close languages.
originalText	Only with keepOriginals.

Sample row:

{ "title": "Café de la Paix", "description": "À bientôt!", "language": "fr", "languageName": "French", "languageConfidence": 21, "textFixes": { "title": ["mojibake"], "description": ["mojibake"] }, "fixedFieldCount": 2 }

Use it with

Any scraper (Google Maps, reviews, jobs, news) -> this Actor -> LLM pipeline, CRM or warehouse export. Pairs with the dataset-to-Parquet Actor from the same author.

FAQ

Is it AI? No. It is deterministic rules plus the franc-min language detector (82 languages), so results are repeatable and offline.

Will it damage correct text? Only strings that decode as valid UTF-8 from a Windows-1252 misreading are changed; correct accented text such as Café is untouched. Check textFixes and use keepOriginals if unsure.

Honest limits

• Text where the original bytes were already lost (? or � replacing characters) cannot be recovered; such fields get the unrecoverable-replacement-chars flag.
• Mojibake from other legacy encodings (Shift-JIS, GBK, KOI8-R read as Latin-1) is not repaired.
• Language detection needs about 20 characters and confuses close languages (Russian/Bulgarian, Spanish/Portuguese, Norwegian/Danish) on short text.
• Entity decoding also affects text that talks about HTML; switch it off for that.

Price

Pay per event: $0.0003 per row cleaned (every row delivered, changed or not). 100,000 rows = $30.