# Fix Broken Text in Scraped Data - Mojibake, Entities, Language (`that_red_bird/scraped-text-fixer`) Actor

Scraped text full of Café, don’t, & and invisible characters breaks LLMs and CRMs. Pick the dataset and get repaired text in every field plus a detected-language column, with a per-field report of what changed. Offline.

- **URL**: https://apify.com/that_red_bird/scraped-text-fixer.md
- **Developed by:** [mohamed alaya](https://apify.com/that_red_bird) (community)
- **Categories:** AI, Automation, Developer tools
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$0.30 / 1,000 row cleaneds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Fix Broken Text in Scraped Data - Mojibake, Entities, Language

Repairs the broken text that scrapers leave behind: `Café de la Paix`, `don’t`, `Tom & Jerry`, zero-width characters, odd spaces, and adds a detected-language column.
For anyone who feeds scraped data to an LLM, a CRM, a spreadsheet or a search index and is tired of garbled characters.

### Quick start

1. Run your scraper, then start this Actor and pick its dataset in **Dataset(s) to fix**.
2. Every text field (including nested ones) is cleaned in place; `textFixes` lists what changed per field.
3. Download JSON, CSV or Excel. Run with empty input to see a demo of 6 broken rows.

### Input examples

**1. Dataset from a scraper**

```json
{ "datasets": ["<pick with the dataset picker>"] }
```

**2. Pasted rows, language from the description only**

```json
{ "rows": [{ "title": "Café de Paris", "description": "À bientôt! Tom &amp; Jerry" }], "languageFields": ["description"] }
```

**3. Keep the originals, straighten quotes, skip a raw-HTML field**

```json
{ "rows": [{ "text": "don’t", "html": "<p>&amp;</p>" }], "keepOriginals": true, "straightenQuotes": true, "skipFields": ["html"] }
```

### Input guide

| Field | What it does |
|---|---|
| `datasets`, `rows`, `rowsCsv` | Where the rows come from. |
| `fixMojibake` | Repair UTF-8 read as Windows-1252/Latin-1 (also double-encoded, Cyrillic, emoji). |
| `decodeHtmlEntities` | `&amp;`, `&#39;`, `&eacute;`, `&nbsp;`... to characters. |
| `unicodeNormalization` | `NFC` (default), `NFKC` or `none`. |
| `keepNewlines` | Keep line breaks (3+ become 2) or flatten to single spaces. |
| `straightenQuotes` | Curly to straight quotes (off by default). |
| `detectLanguage`, `languageFields` | Add language columns; limit detection to chosen fields. |
| `skipFields` | Fields that must not be touched. |
| `keepOriginals` | Store the original text of changed fields in `originalText`. |

### What gets fixed

- **Mojibake:** `Café` -> `Café`, `don’t` -> `don’t`, `€` -> `€`, `Привет` -> `Привет`, `😀` -> 😀. Only sequences that decode to valid UTF-8 are changed.
- **HTML entities**, also double-escaped ones (`&amp;amp;`).
- **Invisible characters:** zero-width spaces, BOM, soft hyphens, bidirectional override marks (also a security risk), control characters. Zero-width joiners are kept next to emoji and in Indic/Arabic text.
- **Whitespace:** no-break and exotic spaces become normal spaces, runs of spaces collapse, line endings unify, text is trimmed.
- URL and e-mail values are left alone.

### Output

Your fields (cleaned) plus:

| Field | Meaning |
|---|---|
| `textFixes` | `{ "title": ["mojibake"], "description": ["html-entities","whitespace"] }`. Codes: `mojibake`, `html-entities`, `invisible-chars`, `whitespace`, `unicode-normalized`, `quotes-straightened`, `unrecoverable-replacement-chars`. |
| `fixedFieldCount` | Number of fields that changed. |
| `language`, `languageName` | ISO 639-1 code (or `und`) and name. |
| `languageConfidence` | 0-100, how far the best guess is ahead of the runner-up. Low for short text and close languages. |
| `originalText` | Only with `keepOriginals`. |

Sample row:

```json
{ "title": "Café de la Paix", "description": "À bientôt!", "language": "fr", "languageName": "French", "languageConfidence": 21, "textFixes": { "title": ["mojibake"], "description": ["mojibake"] }, "fixedFieldCount": 2 }
```

### Use it with

Any scraper (Google Maps, reviews, jobs, news) -> **this Actor** -> LLM pipeline, CRM or warehouse export. Pairs with the dataset-to-Parquet Actor from the same author.

### FAQ

**Is it AI?** No. It is deterministic rules plus the franc-min language detector (82 languages), so results are repeatable and offline.

**Will it damage correct text?** Only strings that decode as valid UTF-8 from a Windows-1252 misreading are changed; correct accented text such as `Café` is untouched. Check `textFixes` and use `keepOriginals` if unsure.

### Honest limits

- Text where the original bytes were already lost (`?` or � replacing characters) cannot be recovered; such fields get the `unrecoverable-replacement-chars` flag.
- Mojibake from other legacy encodings (Shift-JIS, GBK, KOI8-R read as Latin-1) is not repaired.
- Language detection needs about 20 characters and confuses close languages (Russian/Bulgarian, Spanish/Portuguese, Norwegian/Danish) on short text.
- Entity decoding also affects text that talks about HTML; switch it off for that.

### Price

Pay per event: **$0.0003 per row cleaned** (every row delivered, changed or not). 100,000 rows = $30.

# Actor input Schema

## `datasets` (type: `array`):

Pick the dataset of any scraper run. Every text field, including nested ones, is cleaned. Use the picker rather than typing an ID: it gives this Actor permission to read your dataset. Leave empty to use the pasted rows below, or the built-in demo.

## `rows` (type: `array`):

Optional. Paste rows as JSON objects, e.g. {"title":"Café de Paris","text":"don’t miss it"}. Plain strings are treated as a text field.

## `rowsCsv` (type: `string`):

Optional. Paste a CSV (first row = column names) or a JSON array.

## `fixMojibake` (type: `boolean`):

Repair text where UTF-8 was read with the wrong encoding (Café becomes Café, don’t becomes don’t, Cyrillic and emoji too), including double-encoded text.

## `decodeHtmlEntities` (type: `boolean`):

Turn &, ', é,   and similar into real characters. Turn off if your text legitimately talks about HTML.

## `unicodeNormalization` (type: `string`):

NFC joins combining accents into single characters (safe for everything). NFKC also folds compatibility characters such as full-width letters and ligatures. none leaves characters alone.

## `keepNewlines` (type: `boolean`):

When on, line breaks are kept (3 or more in a row become 2). When off, all whitespace including line breaks becomes a single space.

## `straightenQuotes` (type: `boolean`):

Replace curly quotes with plain ' and " characters. Off by default.

## `detectLanguage` (type: `boolean`):

Add language (ISO 639-1 code), languageName and languageConfidence columns, detected offline from the text fields. Needs about 20 characters of text; shorter text is reported as und.

## `languageFields` (type: `array`):

Optional list of field names (for example description, reviewText) to detect the language from. Empty = all text fields of 20 or more characters.

## `skipFields` (type: `array`):

Optional list of field names that must not be changed, for example rawHtml or code. Nested fields are skipped with their parent.

## `keepOriginals` (type: `boolean`):

When on, the text before fixing is stored in an originalText object for every field that was changed. Makes rows bigger.

## `maxRows` (type: `integer`):

Safety limit on the number of rows processed in one run.

## Actor input object example

```json
{
  "rows": [],
  "rowsCsv": "",
  "fixMojibake": true,
  "decodeHtmlEntities": true,
  "unicodeNormalization": "NFC",
  "keepNewlines": true,
  "straightenQuotes": false,
  "detectLanguage": true,
  "languageFields": [],
  "skipFields": [],
  "keepOriginals": false,
  "maxRows": 100000
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `downloadCsv` (type: `string`):

No description

## `summary` (type: `string`):

No description

## `count` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "rows": [],
    "rowsCsv": "",
    "unicodeNormalization": "NFC",
    "languageFields": [],
    "skipFields": [],
    "maxRows": 100000
};

// Run the Actor and wait for it to finish
const run = await client.actor("that_red_bird/scraped-text-fixer").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "rows": [],
    "rowsCsv": "",
    "unicodeNormalization": "NFC",
    "languageFields": [],
    "skipFields": [],
    "maxRows": 100000,
}

# Run the Actor and wait for it to finish
run = client.actor("that_red_bird/scraped-text-fixer").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "rows": [],
  "rowsCsv": "",
  "unicodeNormalization": "NFC",
  "languageFields": [],
  "skipFields": [],
  "maxRows": 100000
}' |
apify call that_red_bird/scraped-text-fixer --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,that_red_bird/scraped-text-fixer"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/TpOxL3C2W1heGXJLL/builds/Bm3Jf2qic4aPwt6nk/openapi.json