Unicode Text Inspector & Normalizer

Reveal every character hiding in your strings — invisible controls, confusable homoglyphs, and encoding artifacts.

Most text tools treat a string as a black box. This one breaks it down to individual codepoints. It tells you which characters are zero-width, which ones are Cyrillic letters that look exactly like Latin ones, whether your data has mojibake from a bad encoding conversion, and what normalisation will actually change.

---

What does it do?

A string looks simple until you inspect it codepoint by codepoint. The letter "A" at position 0 is U+0041 (Latin). But the identical-looking "А" is U+0410 (Cyrillic) — and you can register domain names with it. A zero-width space at position 12 is invisible but breaks database queries and string comparisons. A file declared as UTF-8 might actually contain Latin-1 mojibake from a broken pipeline.

This actor runs four complementary analysis passes on every text you give it:

Inspect mode — Walk through every character. Shows the codepoint as U+XXXX, the Unicode name, which block it belongs to, whether it's visible or printable, and if it's invisible, exactly what kind of invisible character it is (zero-width space, BOM, RTL override, soft hyphen, combining grapheme joiner, etc.). Counts characters per Unicode block — from Basic Latin through CJK Extension H to Supplementary Private Use. Suggests the best-fit encoding and flags mixed-script text.

Normalize mode — Apply Unicode normalisation (NFC, NFD, NFKC, NFKD) and report whether the text actually changed, along with the length difference. NFC composes, NFD decomposes, and the K forms add compatibility decomposition for things like fullwidth Latin → ASCII.

Security mode — Scan for homoglyph characters across 40+ confusable pairs: Cyrillic and Greek letters that impersonate Latin ones for domain spoofing and phishing. Each match gets a risk level (high for phishing-grade confusables, medium for potentially misleading, low for purely visual). The actor assigns an overall security risk level based on the worst match found.

Mojibake detection — Identify replacement characters (U+FFFD), common Latin-1/UTF-8 mismatch patterns (é for é, “ for left double quote, â€" for en dash), Windows-1252 decoding artifacts, and high C1 control character ratios.

All four modes combine these checks. "all" mode runs everything in a single pass.

---

Features

1. Per-character codepoint inspection — Position, character, U+XXXX codepoint, Unicode name, block, category (Lu, Ll, Cf, Zs, etc.), visible/printable flags

2. Invisible character detection — Zero-width spaces, zero-width joiners and non-joiners, BOM, RTL/LTR marks and overrides (LRM, RLM, LRE, RLE, LRO, RLO), bidi isolates, soft hyphens, combining grapheme joiners, variation selectors, format characters, interlinear annotation anchors, C0 and C1 control characters

3. Homoglyph/confusable detection — 40+ pairs across Cyrillic, Greek, fullwidth, and special Unicode characters that impersonate Latin letters, digits, and punctuation. Each flagged with a risk level (high/medium/low)

4. Unicode normalisation — NFC, NFD, NFKC, NFKD with change detection. Only charges for normalisation when the text actually changes

5. Unicode block frequency — Counts characters per Unicode block across 340+ blocks from BMP to Supplementary Private Use Area B

6. Encoding suggestion — Recommends best-fit encoding: us-ascii, Latin-1, Windows-1252, CJK-specific (Japanese/Chinese/Korean), BMP-only UTF-16, or supplementary plane UTF-8

7. Mixed script detection — Flags text containing Latin, Cyrillic, Greek, Arabic, Armenian, CJK, Kana, or Hangul together — potential spoofing or encoding issue indicator

8. Mojibake detection — Replacement characters (U+FFFD), Latin-1/UTF-8 mismatch patterns (é, …, “, â€, â€", €), high C1 control character proportion

9. Supplementary plane detection — Flags characters outside the Basic Multilingual Plane (U+10000+) that require 4-byte UTF-8 or surrogate pairs

---

Who is it for?

Persona	What they use it for
Security engineer	Auditing domain names and user input for homoglyph spoofing and bidi override attacks
Backend developer	Finding the invisible character that's breaking a string comparison or database unique constraint
QA engineer	Verifying that text normalisation doesn't change user-submitted data unexpectedly
Localisation engineer	Checking character coverage and encoding compatibility before launching in new markets
Data engineer	Profiling text datasets for encoding issues and mixed scripts before loading into a warehouse
Developer debugging copy-paste bugs	Finding the zero-width characters that users accidentally pasted along with visible text

---

Input Parameters

Field	Type	Required	Default	Description
text	string	No*	—	A single text string to analyse
texts	array	No*	—	Multiple text strings (max 20)
mode	enum	No	inspect	inspect, normalize, security, or all
normalization_form	enum	No	NFC	NFC, NFD, NFKC, or NFKD
show_codepoints	boolean	No	true	Include the full per-character codepoint array in the output

* At least one of text or texts must be provided.

---

Example Input

Inspect a single string

{
  "text": "Héllo Wörld",
  "mode": "inspect",
  "show_codepoints": true
}

Normalise multiple texts

{
  "texts": ["こんにちは", "Привет мир", "مرحبا"],
  "mode": "normalize",
  "normalization_form": "NFKC",
  "show_codepoints": false
}

Security audit for phishing confusables

{
  "text": "www.example.com",
  "mode": "security",
  "show_codepoints": true
}

---

Output Format

Per-codepoint detail (when show_codepoints is true)

Field	Type	Description
position	integer	Zero-based position in the original text
char	string	The character, or [invisible] for invisible characters
codepoint	string	Unicode codepoint as U+XXXX
name	string	Unicode character name (from Python stdlib unicodedata)
block	string	Unicode block name
visible	boolean	Whether the character has independent visual representation
printable	boolean	Whether the character's category is not Cc, Cf, Cs, Co, or Cn
category	string	Unicode general category (Lu, Ll, Nd, Cf, Zs, etc.)
invisible_type	string	Human-readable description of the invisible type (hidden characters only)

Top-level fields

Field	Type	Description
original_length	integer	Number of Unicode codepoints in the input text
visible_char_count	integer	Count of characters that are independently visible
invisible_char_count	integer	Count of invisible/control characters detected
codepoints	array	Per-character detail array (omitted when show_codepoints is false)
hidden_characters	array	Subset of codepoints that are invisible, each with invisible_type
unicode_blocks	object	Count of characters per Unicode block name
encoding_suggestion	string	Best-fit encoding recommendation
warnings	array of strings	Human-readable warnings: invisible characters, mixed scripts, mojibake, supplementary plane characters
homoglyph_warnings	array	Security findings with position, character, confusable pair, and risk level (only in security/all modes)
homoglyph_count	integer	Total number of confusable characters found
security_risk	string	low, medium, or high based on the worst homoglyph risk level found
normalized_text	string	Normalised text output (only in normalize/all modes)
normalization_form	string	Which normalisation form was applied
normalized_length	integer	Length of normalised text
normalization_changed	boolean	Whether normalisation actually modified the text
normalization_diff_length	integer	Difference in length after normalisation (positive = longer, negative = shorter)
normalization_error	string	Error message if normalisation failed

Example output (inspect mode)

{
  "original_length": 12,
  "visible_char_count": 11,
  "invisible_char_count": 1,
  "codepoints": [
    {
      "position": 0,
      "char": "H",
      "codepoint": "U+0048",
      "name": "LATIN CAPITAL LETTER H",
      "block": "Basic Latin",
      "visible": true,
      "printable": true,
      "category": "Lu"
    },
    {
      "position": 4,
      "char": "[invisible]",
      "codepoint": "U+200B",
      "name": "ZERO WIDTH SPACE",
      "block": "General Punctuation",
      "visible": false,
      "printable": false,
      "category": "Cf",
      "invisible_type": "Zero-width space"
    }
  ],
  "hidden_characters": [
    {
      "position": 4,
      "char": "[invisible]",
      "codepoint": "U+200B",
      "name": "ZERO WIDTH SPACE",
      "block": "General Punctuation",
      "visible": false,
      "printable": false,
      "category": "Cf",
      "invisible_type": "Zero-width space"
    }
  ],
  "unicode_blocks": {
    "Basic Latin": 10,
    "Latin-1 Supplement": 1,
    "General Punctuation": 1
  },
  "encoding_suggestion": "UTF-8 (also fits iso-8859-1 / windows-1252)",
  "warnings": [
    "Text contains 1 invisible/control character(s) (8.3% of text). These may cause unexpected behavior in processing or display."
  ]
}

---

FAQ

Q: What counts as an invisible character? A: Characters with zero display width (zero-width spaces, joiners, non-joiners), control characters (C0 and C1 ranges), bidirectional marks and overrides, format characters, variation selectors, and non-spacing combining marks. Rephrased: if you can't see it and it changes text behaviour, it's caught.

Q: How accurate is the homoglyph detection? A: The homoglyph map covers 40+ well-known confusable pairs across Cyrillic, Greek, and fullwidth scripts. Each match has a risk level — high flags domain spoofing and phishing concerns, medium means it could mislead, low is purely visual. It does not include every entry from the full Unicode confusables table.

Q: Does normalisation charge even if the text doesn't change? A: No. Normalisation only charges when normalization_changed is true — meaning the text was actually modified by the operation.

Q: Can this handle emoji? A: Yes. Emoji characters have Unicode codepoints like any other character. They'll appear in the codepoint array with their block (typically "Emoticons", "Miscellaneous Symbols and Pictographs", or supplementary blocks for newer emoji) and their Unicode name. Some emoji use zero-width joiners to combine — those will flag as invisible characters.

Q: How does encoding suggestion work? A: The actor checks the highest codepoint in the text. If everything is U+007F or below, it suggests us-ascii. Up to U+00FF looks at Latin-1 or Windows-1252 specifics. CJK ranges get detected by U+4E00–U+9FFF, U+3040–U+30FF, U+AC00–U+D7AF. BMP vs supplementary plane is a simple threshold check at U+FFFF. The suggestion is a practical recommendation, not an exhaustive analysis.

Q: Can I process very long texts? A: The actor processes the full text you provide. Per-character analysis is O(n) on text length. For texts over 100,000 characters, Unicode block lookup dominates runtime. The input schema caps at 20 texts per run.

Q: Why would I use normalisation? A: Two strings can look identical but have different underlying representations. NFC composes combined characters (é as one codepoint), NFD decomposes them (e + combining acute accent as two). NFKC and NFKD add compatibility decomposition — fullwidth Latin becomes halfwidth, ligatures become individual letters. Use this before comparing, hashing, or storing user-submitted text.

---

API Usage

cURL

curl -X POST "https://api.apify.com/v2/acts/perryay~unicode-inspector/runs?token=YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "text": "Héllo — check for encoding issues",
    "mode": "inspect",
    "show_codepoints": true
  }'

Python

from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")
run = client.actor("perryay~unicode-inspector").call(
    run_input={
        "text": "Héllo — check for encoding issues",
        "mode": "inspect",
        "show_codepoints": True,
    }
)
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["encoding_suggestion"], "-", item["invisible_char_count"], "hidden")

Node.js

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_API_TOKEN' });
const run = await client.actor('perryay~unicode-inspector').call({
    text: 'Héllo — check for encoding issues',
    mode: 'inspect',
    show_codepoints: true,
});
const dataset = client.dataset(run.defaultDatasetId);
for await (const item of dataset.iterateItems()) {
    console.log(item.encoding_suggestion, '-', item.invisible_char_count, 'hidden chars');
}

---

Use Cases

• Finding the zero-width space a user accidentally pasted that's making your database unique constraint fail
• Auditing domain names for Cyrillic or Greek homoglyph characters before registering or blocking them
• Checking whether a text file declared as UTF-8 actually contains mojibake from a Latin-1 pipeline
• Normalising user-submitted text to NFC before storing it so search and comparison behave consistently
• Verifying that a localisation file covers all the Unicode blocks used in the target language
• Detecting bidirectional override attacks in user-generated content before it reaches your frontend
• Profiling a text corpus to decide whether UTF-8, UTF-16, or a legacy encoding is appropriate for storage
• Debugging why two seemingly identical strings don't compare equal — normalisation diff will show you
• Checking what encoding a mystery text file actually uses by looking at its codepoint distribution