Japanese Text Normalizer & RAG Chunker

Clean Japanese text before you give it to a RAG pipeline or an LLM: full-width letters and digits, half-width katakana, separated voiced marks, look-alike dashes and long vowels, letter-spaced text, invisible characters and lines broken by PDF text extraction. URLs, e-mail addresses, inline code and fenced code blocks are kept as they are. Optionally, split the result into sentence-aware chunks with overlap.

Typical uses:

• Text extracted from Japanese PDFs: join lines cut in the middle of a sentence, remove page numbers and running headers, fix ｶﾞｲﾄﾞ, コ−ヒ― and 日 本 語.
• Scraped Japanese web pages: one consistent form of the same words before embedding, search or deduplication.
• RAG ingestion: chunks of whole sentences (never cut inside 「」 or （）) with their position in the document and your metadata (URL, title).
• Mixed Japanese and English documents: English words keep their spaces, code blocks keep their indentation.

Try it in one click

Click Start without choosing a dataset and without texts. The Actor cleans three built-in sample documents that it ships with (no characters-processed charge for them): text copied from a 3-page PDF report, a scraped FAQ page with half-width katakana, and a technical note with English, a URL and a code block. You get 3 records; turn on Split into chunks to see chunks instead, and Remove page numbers and repeated headers to see the PDF page numbers (－ 1 －) and the repeated header line disappear.

How it works

1. You choose the input: a dataset (for example the output of your scraper or PDF extractor) and the text fields that hold the text, or a list of texts typed into the input.
2. Before anything is charged, the Actor looks at the first 100 items. If fewer than half of them have text in the text fields, it stops (nothing charged) and lists the fields it found, so a wrong field name costs nothing. It also checks that the largest document of the first page fits in the run's memory.
3. Each document is normalized with safe defaults (table below). Every step can be switched off under Normalization (advanced).
4. Protected parts come out unchanged: URLs and e-mail addresses (also half-width katakana inside a URL), inline code and fenced code blocks (```). Outside fenced blocks, a line is kept as code only when it is plain ASCII and looks like code (a guess): code lines that contain Japanese, full-width characters or full-width spaces are cleaned like text. Put code in ``` fences to keep it exactly.
5. Optionally the document is split into chunks: whole sentences (split after 。！？, never inside brackets) up to Maximum chunk length, with Chunk overlap.
6. The Actor writes one record per document (or per chunk) to its dataset, and a summary with the counts to the OUTPUT record of its key-value store.

Normalizing twice gives the same text as normalizing once (tested on thousands of random texts made of the tricky characters below).

What changes and what stays (default settings)

Input	Output	Why
ＡＢＣ１２３	ABC123	Full-width letters and digits become ASCII.
１，０００円 １０：３０ ｈｔｔｐｓ：／／ｅｘａｍｐｌｅ．ｃｏｍ	1,000円 10:30 https://example.com	Full-width symbols become ASCII only between ASCII letters or digits.
（注） 本当！？ １００％	（注） 本当！？ 100％	Japanese punctuation stays full width.
①②③ ㈱ ㌔ Ⅳ x² ℃	unchanged	NFKC would change these to 123, (株), キロ, IV, x2 (NFKC is an option).
ｶﾞｲﾄﾞ ｡｢｣､	ガイド 。「」、	Half-width katakana become full width.
か゛ (separate voiced mark)	が	Voiced and semi-voiced marks are joined when a combined character exists.
コ−ヒ― サ－ビス	コーヒー サービス	After katakana, a dash look-alike is a long vowel.
03−1234−5678 −5℃ Wi‐Fi	03-1234-5678 -5℃ Wi-Fi	Between letters or digits, or before a number, it is a hyphen or minus.
ゲーム――その歴史 メール―お問い合わせ	unchanged	Doubled dashes, and a dash before hiragana or kanji, are punctuation.
10～20	10〜20	Both wave dashes become 〜.
日 本 語 の 文 章 精 度 を 上 げ る 日 月 火 水 木 金 土	日本語の文章 精度を上げる 日月火水木金土	Letter-spaced text (three or more one-letter words in a row, often left by PDF extraction) is joined. A list of one-letter words, like the days of the week, is joined too.
山田 太郎 林 修 私 は 学生 です 品名 数量 単価 合計	unchanged	Spaces between longer Japanese words, and between only one or two one-letter words, stay (names, table headers). Option Spaces between Japanese words: remove all, or keep all.
日本語 。 Apify 「Actor」	日本語。 Apify「Actor」	Spaces next to Japanese punctuation and brackets are removed.
Hello World	Hello World	Spaces between English words are kept (runs become one).
Apify の Actor	unchanged	Optional: Also remove spaces between Japanese and English.
テ(zero-width space)スト, BOM, soft hyphen, control characters	テスト	Invisible characters are removed.
👨‍👩‍👧 👍🏽 🇯🇵 葛󠄀 (variation selector)	unchanged	Emoji sequences and variation selectors are kept.
﹁縦書き﹂ ︵注︶	「縦書き」 （注）	Vertical and small presentation forms from vertical PDF text.
A sentence broken over two lines by the PDF	one line	Japanese lines are joined without a space, English lines with one. Paragraphs, lists, headings and lines ending with ： stay.
－ 3 －, a header repeated on every page	unchanged	Optional: Remove page numbers and repeated headers (a guess, so off by default). Repeated lines are removed only next to a page break.
， ．	unchanged	Optional: Japanese comma and period (，． → 、。 or the other way).
https://example.com/ｶﾞｲﾄﾞ, e-mail addresses, `inline code`, ``` blocks	unchanged	Protected. Plain-ASCII code-like lines outside fences are protected too (a guess). A full-width backtick ｀ is not a code mark: it stays as it is (ボタン｀ｶﾞｲﾄﾞ｀を押す → ボタン｀ガイド｀を押す), except in the all and NFKC modes, where it becomes ` before code is found.

NFC (on by default) also turns CJK compatibility ideographs into the usual character (神 U+FA19 → 神 U+795E), as every NFC implementation does.

Input

Field	What it does
datasetId	Dataset with your text. Leave empty (and leave texts empty) to try the sample documents.
texts	Instead of a dataset: a JSON list of strings, one per document. The whole input of a run can be at most 9 MB (9,437,184 bytes, an Apify limit), about 3 million Japanese characters; Apify refuses a larger input before the run starts (413 request-too-large). Inputs up to that size were processed with the minimum memory of 1,024 MB (measured through the API with 3,027 texts of 1,000 characters, 9.0 MB). Put more text in a dataset.
textFields	Fields that hold the text (default ["text"]). Dots for nested fields: page.content. Several fields are joined with a blank line; a field may hold a list of strings.
metadataFields	Fields copied unchanged into metadata of every record, for example ["url", "title"].
splitIntoChunks	Split each document into chunks. Default false (one record per document).
maxChunkChars	Longest chunk in characters (default 500, at least 200).
chunkOverlapChars	Overlap between chunks in characters (default 50, at most a third of maxChunkChars).
unicodeNormalization	nfc (default), nfkc or none.
fullwidthAscii	alphanumeric (default), all or none.
halfwidthKatakana, composeVoicedMarks, presentationForms, unifyDashes, removeInvisibleCharacters, normalizeWhitespace, joinBrokenLines	Steps that are on by default (true).
spacesBetweenJapanese	letterSpaced (default: remove spaces only in letter-spaced text), all (also 山田 太郎 → 山田太郎) or keep.
removeSpacesBetweenJapaneseAndLatin, removePageNumbersAndHeaders	Steps that are off by default (false).
punctuationStyle	keep (default), japanese (，． → 、。) or western (、。 → ，．).

Example input:

{
    "datasetId": "YOUR_DATASET_ID",
    "textFields": ["text"],
    "metadataFields": ["url", "title"],
    "splitIntoChunks": true,
    "maxChunkChars": 500,
    "chunkOverlapChars": 50,
    "removePageNumbersAndHeaders": true
}

Output

Each record is one document, or one chunk when Split into chunks is on. Every record has the same fields:

{
    "documentIndex": 1,
    "chunkIndex": 0,
    "chunkCount": 1,
    "text": "よくある質問（FAQ）\n\nQ. ログインできません。\nA. パスワードを再設定してください。手順は次のとおりです。\n① 設定画面を開く\n②「パスワードを忘れた」を選ぶ\n③ 届いたメールのリンクを押すと、再設定が完了します 👍 …",
    "charCount": 255,
    "startOffset": 0,
    "endOffset": 255,
    "originalCharCount": 263,
    "metadata": { "id": "faq", "title": "よくある質問（Web ページから取得）", "url": "https://example.com/faq" }
}

• documentIndex: position of the document in your dataset or in texts, from 0.
• chunkIndex, chunkCount: position of the chunk and number of chunks of the document (0 and 1 without chunking).
• startOffset, endOffset: where the chunk is in the normalized document. Lengths and offsets count characters as Unicode code points, like Python's str: normalized[startOffset:endOffset] == text.
• originalCharCount: length of the document's text before normalization (what is charged; a document counts as at least 100 characters).
• metadata: the fields chosen in metadataFields, unchanged (null when missing; empty for texts). With chunking, the metadata is copied into every chunk, so choose small fields such as url and title.
• Items without text, and documents that are empty after cleaning, produce no record. A document that cannot be processed is skipped (not charged) and its index is listed in failedDocumentIndexes of the summary; the run goes on.

The OUTPUT record of the run's key-value store holds the summary. It is updated after every page ("status": "RUNNING") and at the end:

• COMPLETE: all documents were processed.
• PARTIAL: not the whole input. The run stopped early (the maximum cost per run was reached, you aborted it, or 30 seconds before its timeout), or more than 1% of the documents had to be skipped (too large for the memory or for one dataset item, or failed). The message says which and how many. The records and charges of the pages finished before that are kept.
• FAILED: an input problem (nothing charged) or an unexpected error; the message says what happened.

In counts, inputCharacters is the length of the documents that produced records (before cleaning) and billedCharacters what the characters-processed event counts (each document at least 100 characters); billedCharacters is 0 when nothing is charged, as for the built-in sample.

{
    "status": "COMPLETE",
    "source": "sample",
    "message": "Processed 3 documents (1,200 characters) and wrote 3 records. (Built-in sample documents. …)",
    "counts": {
        "documentsRead": 3, "documentsWithText": 3, "documentsWithoutText": 0, "documentsEmptyAfterNormalization": 0,
        "documentsTooLarge": 0, "documentsFailed": 0, "recordsWritten": 3, "inputCharacters": 1200, "billedCharacters": 0, "normalizedCharacters": 1142,
        "joinedLineBreaks": 6, "paragraphBreaksAdded": 0, "removedPageNumberLines": 0, "removedRepeatedLines": 0
    },
    "failedDocumentIndexes": [],
    "chargedEvents": 0
}

Run it after your scraper or PDF extractor

1. Open the Actor (or saved task) that produces the text and go to its Integrations tab.
2. Add this Actor as an integration that runs when that run succeeds.
3. In the integration's input, set:

{
    "datasetId": "{{resource.defaultDatasetId}}",
    "textFields": ["text"],
    "metadataFields": ["url"],
    "splitIntoChunks": true
}

{{resource.defaultDatasetId}} is replaced with the dataset of the run that just finished. Always set datasetId like this: the Actor cannot read a dataset that is passed only in the integration's implicit payload, so a run with an empty datasetId and a dataset in the payload stops at once with a message (nothing charged). Use the field that holds the page text in that dataset, for example text or markdown. Then add your next step (for example a vector database integration) on this Actor's own Integrations tab.

Pricing

This Actor uses pay-per-event pricing. The current prices are on the Pricing tab of this Actor.

Event	Price	When it is charged
Actor start	$0.00005 per run (once more for every GB beyond 1 GB)	Set by Apify at the start of every run (also for the sample): once for a run with up to 1 GB of memory (the default and the minimum of this Actor), and once more for every further GB.
1,000 characters processed (characters-processed)	$0.0001 (= $0.10 per million characters)	Once for every 1,000 characters of input text, rounded up once per run. Characters are Unicode code points of the text read from your text fields (or your texts), counted before cleaning. Each document counts as at least 100 characters.

• The cost of a run is easy to predict: it depends only on the length of your input text (each document counts as at least 100 characters), not on the number of records or chunks. For example, 1,000 documents of 30 characters count as 100,000 characters.
• Each page of documents is charged right after its records are saved, so a run that is aborted or times out pays only for the pages it finished.
• Not charged: the built-in sample documents, items without text, documents that are empty after cleaning, documents skipped as too large, and documents that could not be processed.
• Checked before anything is charged: if fewer than half of the first 100 items have text in your text fields (for example a wrong field name), or the largest document of the first page does not fit in memory, the run stops with a message and nothing is charged.
• Maximum cost per run: if you set one, the log tells you at the start how many characters it covers. The Actor stops cleanly before the first document that would go over it; the records written so far are kept and the summary says "status": "PARTIAL".

Limits

Measured on a desktop PC (Node.js 24, Windows), in memory, without Apify storage:

• 100,000 documents of 1,000 characters (100 million characters), normalized and split into chunks: 14–19 seconds (5–7 million characters per second); the heap stayed under 25 MB, because documents are processed page by page and records are saved in batches of at most 8 MB.
• One document of 10 million characters: 2.5–2.9 seconds (normalize and chunk).

Memory: one document is normalized in memory at a time, and its records are saved in batches. What a document needs depends more on its shape than on its length (measured with scripts/memory-probe.js, above what the run itself uses): about 2–13 bytes per character for ordinary text (a report with 50-character lines: 8–9), 27 for letter-spaced text, 39 for text full of e-mail addresses, and 60–71 for text with one character per line (vertical PDF text). The Actor estimates each document from its lines, spaces, URLs and sentences. If a document does not seem to fit, the Actor first saves the records it holds and frees memory, then checks again; only then is the document skipped (not charged), with advice on how much memory to give the run. If the largest document of the first page does not fit, the run stops before charging anything. The minimum and default memory is 1 GB (one Actor start). Node is started with a heap limit of 3/4 of the run's memory (plus a small young generation), so it collects garbage in time, and the Actor plans with that limit. The log shows the run's memory and the heap limit at the start of every run.

Measured on Apify (October 2026): started without a heap limit, V8 reported a heap limit (heap_size_limit) of about 30,000 MB with 512, 1,024 and 4,096 MB of memory and collected garbage late: one document of 2 million characters with one character per line (vertical PDF text), split into chunks, peaked at 277 and 411 MB in two runs of 512 MB. Since version 0.1.3, Node starts with 3/4 of the run's memory as its heap (816 MB in all with 1,024 MB), and the minimum memory is 1,024 MB. With 1,024 MB, that document peaked at 246 MB, and one of 3 million such characters (about the largest this shape can be in 1,024 MB) at 322 MB. In local tests with the same settings, the largest accepted document of each shape tested (one dataset item of at most 9 MB: vertical text, letter-spaced text, e-mail addresses, URLs, full-width symbols, reports, lines of digits) peaked at about 470 MB or less. A larger document stops the run before anything is charged and asks for 2,048 MB. 300 documents of 20,000 characters and one report of 500,000 characters were processed without skipping any (already with 512 MB on an earlier version).

On Apify, most of the time is spent reading your dataset and writing the records through the Apify API. Measured: 10,000 documents of 1,000 characters (10 million characters, 10,000 records, without chunking) took 16 seconds with 1,024 MB of memory (version 0.1.1). 100,000 documents were not measured on Apify; expect about ten times as long (an estimate).

Other limits:

• Without Split into chunks, one record can hold at most 9 MB (an Apify limit), about 3 million Japanese characters. Larger documents are skipped (not charged): turn on chunking. Metadata larger than 9 MB is skipped the same way; metadata over 64 KB is copied into every chunk and makes the output large (the log warns).
• Join lines broken by PDF extraction and Remove page numbers and repeated headers are guesses. Joining needs at least one long line (15+ full-width characters) in the document and never joins after 。 or ：, before a list item or heading, or after a short line; a document made only of short lines (lists, key-value data) is not changed. Header removal removes short lines that appear 3+ times at least 5 lines apart and within 3 lines of a page break (a form feed, a page number line, or the start or end of the document), and number-only lines that go up like page numbers. Check the result on your documents.
• Sentence splitting uses 。！？ (and . ! ? followed by a space in English) and every line break. A single sentence longer than the maximum chunk length is cut, preferably after 、 or a space, never inside a surrogate pair or an emoji sequence.
• Dataset fields whose name starts with # (hidden fields) are not read.
• If you abort the run, or 30 seconds before its timeout, the run stops at the next page and saves a PARTIAL summary. This needs a graceful abort (in the API: abort with gracefully=true), which gives the run 30 seconds. An immediate abort stops the Actor at once: the OUTPUT keeps its last "status": "RUNNING" summary, whose counts and chargedEvents are right up to the last saved page; records of the page that was being saved may be in the dataset without being charged.

Privacy

• The Actor does not keep your data. It reads the dataset you choose (or your texts), writes the records to its own run dataset in your account, and stops.
• The run's key-value store holds the summary (OUTPUT) and, on Apify, a progress record (STATE) with counters only, used to continue if the platform moves the run to another server. Neither contains your text.
• The Actor does not contact any website, AI service or other service outside your Apify account. It only reads datasets that you choose in the input (it runs with limited permissions).

Example tasks

Ready-made inputs for common jobs. Open one, then click Try for free to copy it to your account and change the input:

• Clean Japanese text before sending it to an LLM
• Split Japanese documents into RAG chunks with overlap
• Fix broken line breaks in text copied from Japanese PDFs
• 全角英数字と半角カナをまとめて正規化（日本語テキスト）

More from Quiet Works Lab

• Dataset Change Tracker: compare two runs of your scraper and get only the new, changed and removed items.
• Image Converter & Compressor: convert images to WebP, AVIF, JPEG or PNG, resize them and remove EXIF and GPS metadata.

Support

This Actor is developed and maintained with the help of AI (Claude Code). Issues are answered within a few days.

Please open an issue on the Issues tab with the run ID and, if possible, a short example text (remove anything private first) and the output you expected.

日本語の要約

日本語の文章を、RAG や LLM に読ませる前に整えます。全角英数→半角、半角カナ→全角、濁点の合成（か＋゛→が）、長音・ハイフン・マイナスの統一（コ−ヒ―→コーヒー、03−1234→03-1234）、～ と 〜 の統一（〜 にそろえる）、ゼロ幅の文字の除去、字間を空けた文字の空白の除去（日 本 語→日本語。山田 太郎 のような語の間の空白は残す）、PDF の抽出で途中で切れた行の連結を行います。①・㈱・（）・！？ は変えません（NFKC は選択式）。URL・メールアドレス・インラインのコード・囲み（```）のコードは変えません。囲みの外では、ASCII だけのコードらしい行を推定で守ります。ページ番号とくり返すヘッダーの除去、句読点の統一は任意です。文の区切り（。！？、かっこの中では切らない）を守った分割もできます。課金は入力の文字数 1,000 文字ごとです（1文書は最低100文字と数えます。同梱の見本は無料。テキストのある件が最初の100件の半分未満なら、課金の前に止まります）。