Arabic Text Normalizer

Deterministic Arabic text normalization for search indexes, non-personal product catalogs, and text-processing pipelines. No LLM, spelling correction, scraping, or external API in the normalization core. Original source; MIT licensed.

Version 0.1.0. Use it to make Arabic search keys consistent without rewriting the display text in your original catalog. Preserve your originals separately: search normalization is intentionally lossy and is not an HTML sanitizer, spelling corrector, or anonymizer.

Quick start on Apify

1. Open the Actor's Input form. Submit synthetic or public non-personal text only.
2. Set texts to ["أَهْلًا بِالْعَالَمِ ١٢٣", "مدرسة ؤ ئ"], profile to search, and digitMode to ascii.
3. Check the live Pricing tab and Maximum cost per run before starting. The current event price is $0.001 per result; a $0.002 result budget allows at most two results. A smaller budget can return only part of a batch. No extra usage charge or paid startup event is configured.
4. Read the default dataset. The example's normalized text values are اهلا بالعالم 123 and مدرسة ؤ ئ. Each row also includes its index, settings, input/output code-point counts, warnings, and tool/Unicode version. Download JSON or CSV using Apify's dataset export.
5. Check the OUTPUT summary in the default key-value store. complete means all accepted inputs were published; budget_limited means only the indicated publishedCount were published. Check the dataset before retrying an interrupted or ambiguous run.

بداية سريعة بالعربية

الأداة توحّد النص العربي لاستخدامه كمفتاح بحث، ولا تصحح الإملاء ولا تخفي البيانات الشخصية. احتفظ بالنص الأصلي منفصلًا. أدخل نصوصًا اصطناعية أو عامة غير شخصية في texts، واختر search للبحث أو conservative للحفاظ على الحروف والتشكيل مع NFC. اختر ascii لتحويل الأرقام العربية والفارسية إلى 0–9 عند الحاجة. السعر الحالي 0.001 دولار لكل نتيجة، حتى إن لم يتغير النص؛ حد التكلفة قد يوقف الدفعة جزئيًا. النتائج في dataset والملخص في OUTPUT؛ لا تكرر تشغيلًا غامضًا قبل فحصه. لا تستخدم التطبيع الذي يحذف التشكيل أو يوحّد الحروف على نص ديني أو قانوني معتمد.

Run locally

Python 3.11+; no third-party package is required for the local core or unit tests. The Unicode database depends on the Python version; output includes unicodeVersion. Acceptance here used Python 3.14.7, not every supported Python version.

python arabic_normalizer.py < examples/input.json > result.json
python -m unittest discover -s tests -v

Input is UTF-8 JSON on stdin; output is JSON on stdout. Success exits 0. Invalid input exits 2 with a fixed error.code, error.path, and error.message, without echoing supplied values. No file paths, URLs, or arbitrary code are accepted as input options. Output is plain text data; consumers must escape it when rendering HTML.

Input and exact behavior

• texts: required array of 1–100 strings. Each string has 1–8192 Unicode code points, and the batch has at most 65536 code points. The CLI accepts at most 1048576 UTF-8 JSON bytes. Whole-batch validation occurs before any result processing or billing. Empty strings and unpaired surrogates are rejected. A whitespace-only string is allowed and can normalize to an empty result.
• profile: conservative by default. Applies Unicode NFC only; preserves Arabic marks, Tatweel, presentation-form letters, Alef/Yeh distinctions, and digits unless another option explicitly changes them.
• profile: search: NFC, then compatibility expansion of Arabic presentation-form characters in U+FB50–U+FDFF and U+FE70–U+FEFC, NFC again, removal of the exact marks listed below and U+0640 Tatweel, then أ إ آ ٱ → ا and ى → ي. This is lossy and can merge different words. It does not fold ة to ه, change ؤ/ئ/ء, translate text, stem, correct spelling, or normalize Persian Kaf/Yeh. Compatibility expansion is restricted to the Arabic blocks; full-width Latin and other scripts are not NFKC-folded.
• digitMode: preserve by default. ascii maps U+0660–U+0669 and U+06F0–U+06F9 to ASCII 0–9. Decimal and percent separators are unchanged; this is not number parsing.
• whitespace: preserve by default. collapse uses Python Unicode whitespace rules to trim both ends and replace every whitespace run with one ASCII space. This removes line breaks and can alter formatting.
• removeBidiControls: false by default. true removes only U+061C, U+200E, U+200F, U+202A–U+202E, U+2066–U+2069. These 12 characters affect visual ordering. ZWJ, ZWNJ, and BOM are not removed. This option is not a security sanitizer.

Search profile removes these explicitly enumerated mark ranges: U+0610–U+061A, U+064B–U+065F, U+0670, U+06D6–U+06DC, U+06DF–U+06E4, U+06E7–U+06E8, U+06EA–U+06ED. It does not claim to remove every Arabic mark in every Unicode extension or preserve religious recitation annotations. Do not use lossy normalization on authoritative legal or religious text.

Example (synthetic): أَهْلًا بِالْعَالَمِ ١٢٣ with search + ascii + collapse produces اهلا بالعالم 123. مدرسة ؤ ئ remains unchanged. examples/input.json and examples/output.json provide a complete runnable example.

Output

Local output is described by schemas/output.schema.json and contains settings, tool/Unicode version, processed count, total input code points, and results. Each result has a zero-based index, normalized text, changed, lossy, input/output code-point counts, and warning codes. No separate original-text field is retained.

Warnings: bidi_controls_present, bidi_controls_removed, control_characters_preserved, not_html_sanitized, lossy_transformation. Warnings are not a PII detector or a complete safety assessment. Canonical NFC equivalence alone is reported as changed but not lossy.

Input contract: schemas/input.schema.json. Apify UI input contract: .actor/input_schema.json. Runtime enforces aggregate limits, strict boolean types and surrogate rejection in addition to the JSON schema. JSON duplicate keys, non-finite constants, invalid UTF-8 and malformed JSON are rejected by the local CLI. The Apify platform parses JSON before the adapter, so duplicate-key detection at that upstream boundary is not claimed.

Privacy and limitations

Do not submit personal data, customer records, credentials, bank identifiers, or confidential text. This tool does not detect or automatically anonymize personal information. Synthetic or public non-personal text only.

The local CLI does not make network requests, write files, or log inputs. Shell redirection saves exactly the output path you choose. Normalized text may retain all sensitive content from the input; losing a diacritic is not anonymization.

When run on Apify, Apify stores the run input and dataset under its storage/retention settings. The adapter also stores a summary as OUTPUT. It does not delete or promise immediate erasure of platform storage. The developer's source does not send text to external services; this is not a promise that cloud infrastructure has no access. Consumers must review Apify privacy/retention settings and permissions.

This is a structural text-processing tool, not tax or legal advice, a ZATCA certification, or a spelling/grammar guarantee. AI-assisted development was used; the runtime itself uses deterministic code, not AI.

Apify deployment candidate

apify_main.py uses the official Apify Python SDK. requirements-cloud.txt identifies the SDK and requirements-lock.txt pins the resolved 30-package runtime closure. The SDK was exercised in a network-disabled namespace without credentials: full, zero-budget, partial-budget, invalid-input, missing-event, extra-paid-event and non-PPE cases passed real local-storage assertions. These packages are not required for the CLI. Local SDK charging logs are simulations, not live cloud billing or revenue evidence.

The existing cloud build 0.0.1 was additionally checked with four synthetic cases: small batch (2 results), aggregate-invalid input (rejected with no dataset results), maximum aggregate input (100 texts / 65536 code points, stopped at 2 results by a $0.002 event budget), and a 2-text batch limited to 1 result by a $0.001 event budget. This does not establish complete 100-row cloud output, external customer use, payout eligibility, or an exact all-in platform cost. Cloud tests used 256 MB and a 60-second timeout; actual output includes Unicode version 16.0.0.

Use limited permissions, no proxies, no schedules, no Standby. Read only default input and write only default dataset/summary. Each accepted text-normalized event corresponds to one output item, including an unchanged result or a whitespace-only input that becomes empty. Invalid batches produce no text result events. An interrupted run may have partial charged results; its dataset is authoritative, not an absent final summary. Do not automatically retry an ambiguous run: read its state and dataset first to avoid duplicate processing/charges.

Configured price: $0.001 per text result through the single text-normalized event, no paid startup or default-dataset synthetic event, and no extra usage charge. Check the live Pricing tab before running; it is the current price reference. apify_main.py requires PPE configuration and stops at its event budget; no manual repeated charge/retry logic is present. Invalid input is rejected before text-result charging. An unchanged result or a whitespace-only input that becomes empty still counts as one result.

Use the Actor's API options in Apify Console for the account-authorized request examples. MCP discovery and external-agent integration have not been independently verified; they are not guaranteed by this README. For problems with this Actor, use its Apify Issues tab. Do not include private text, credentials, or personal data in an issue.

License and technical references

Our code/docs: MIT (LICENSE). Python standard library follows the Python/Unicode distribution licenses. Optional Apify SDK: Apache-2.0 per official PyPI metadata for 4.1.0. Resolved package license metadata and the OSV query were checked; this is not a full third-party source audit. See DEPENDENCIES.md.

• Python Unicode normalization: https://docs.python.org/3/library/unicodedata.html
• Apify PPE: https://docs.apify.com/actors/publishing/monetize/pay-per-event
• SDK charging and limits: https://docs.apify.com/sdk/python/docs/concepts/pay-per-event
• Limited permissions: https://docs.apify.com/actors/development/permissions

Arabic documentation: README.ar.md.