# Changelog of Japanese Text Normalizer & RAG Chunker (`quietworkslab/japanese-text-normalizer`) Actor

- **URL**: https://apify.com/quietworkslab/japanese-text-normalizer/changelog.md
- **Full Actor documentation**: https://apify.com/quietworkslab/japanese-text-normalizer.md

## Changelog

### 0.1 (not published yet)

- First version: normalize Japanese text from a dataset (text fields with dotted paths, several fields, lists of strings) or from texts in the input; built-in sample documents for a free first run.
- Safe defaults for Japanese: NFC (NFKC optional), full-width letters and digits to ASCII, half-width katakana to full width, voiced sound marks joined, vertical and small presentation forms, long vowels / hyphens / minus signs by context, wave dashes unified, invisible characters removed, spaces cleaned (spaces between English words kept), lines broken by PDF extraction joined.
- Optional: page numbers and repeated headers removed, spaces between Japanese and English removed, Japanese comma and period unified.
- URLs, e-mail addresses, inline code, fenced code blocks and code-like lines are never changed. Normalizing twice gives the same text.
- Optional sentence-aware chunking (never inside brackets) with overlap; offsets in Unicode code points; metadata fields copied from the input items.
- Pay-per-event: one `characters-processed` event per 1,000 input characters (rounded up once per run); each page is charged right after it is saved; stops cleanly at the maximum cost per run; continues after a platform migration without charging twice; stops with a PARTIAL summary when aborted or 30 seconds before the timeout.
- After an independent review (2026-10-29): spaces between Japanese words are kept by default (only letter-spaced text such as 日 本 語 is joined; option to remove all); text that already contains Unicode noncharacters no longer breaks the run; a document that fails is skipped and listed instead of failing the run; memory is estimated from the shape of the text (lines, spaces, URLs, sentences) and records are saved in batches of at most 8 MB; half-width katakana inside URLs are kept; a dash before hiragana is no longer turned into ー; lines ending with ： are not joined; repeated lines are removed only next to page breaks; several quadratic cases are now linear.
- After the second review (2026-10-29): the memory estimate no longer adds a fixed 32 MB per document and is calibrated on the heap a document needs above the run itself; when a document does not seem to fit, the records held are saved and memory is freed before it is checked again; skipped documents are counted in the message, and more than 1% skipped makes the summary PARTIAL; default memory 1 GB; the heap limit is logged at the start; letter-spaced text needs three or more one-letter words in a row (林 修 stays); a full-width backtick is kept as it is and never starts inline code (except in the "all" and NFKC modes); an e-mail address must start with a letter or digit.
- 0.1.2 (2026-10-30, after the checks on Apify): each document counts as at least 100 characters for `characters-processed` (every record pays for its dataset write; the summary shows `inputCharacters` and `billedCharacters`); the maximum chunk length is at least 200 characters; a run with an empty `datasetId` and a dataset only in the integration payload stops before anything is charged and says to set `datasetId` to `{{resource.defaultDatasetId}}`; the memory plan uses at most the run's memory as the heap limit (on Apify V8 reports about 30,000 MB) and the log says so; the `texts` description gives the measured limit (about 9 MB).
- 0.1.3 (2026-10-30, after the second review): the minimum memory is 1,024 MB (one Actor start, as before); Node starts with 3/4 of the run's memory as its heap, a small young generation and garbage collection on demand, and the memory plan uses that heap limit even when V8 reports more (on Apify a 3-million-character vertical document now peaks at 322 MB instead of 586 MB); the chunk overlap is at most a third of the maximum chunk length; messages say "billed" and `billedCharacters` is above 0 only when characters are really charged; the README price table shows the prices.
- README only (2026-11-01): links to the example tasks (ready-made inputs with their own pages) and to the other Actors of Quiet Works Lab.
- Next build (2026-11-01): the dataset is read until a page comes back empty or shorter than asked, instead of stopping at the item count the API reports. Right after items are pushed (for example when an integration starts this Actor the moment a scraper finishes) that count can be behind, even 0, and earlier builds then silently left out the documents after it; a maximum cost reached exactly at the end of a page past that count also wrote the final summary instead of PARTIAL. A last page that is exactly full costs one extra (empty) request.