Carsensor Japan Used Car Scraper (Listings & Details)
Pricing
Pay per event
Scrape public Carsensor (carsensor.net) Japan used-car list cards and optional detail pages into structured dataset rows: prices in JPY, mileage, shops, equipment, and pay-per-event pricing for list-row or detail-row exports.
feat: scaffold package, .actor schemas (input/dataset/PPE), Dockerfile, and schema tests (red: redlogs/01-scaffold-schemas.red.log)
feat: robots.txt parser and injectable HTTP client with 429/403 >=30s backoff (red: redlogs/02-robots-http.red.log)
feat: list page parser (30 cars/page, AU7162516349 + AU7352699859 fixture fields) (red: redlogs/03-list-parser.red.log)
feat: detail page parser (AU7352699859 prices/phone/address/equipment) (red: redlogs/04-detail-parser.red.log)
feat: input normalisation edge-case tests (defaults, freeword URLs, validation) (red: redlogs/05-input.red.log)
feat: run orchestration with pagination, dedupe, maxItems, Crawl-delay (red: redlogs/06-run-orchestration.red.log)
feat: billing/OUTPUT empty+fail paths, detail fallback, 429 backoff, charge-limit (red: redlogs/07-billing-output.red.log)
feat: CLI main with injectable Actor deps and failMessage (red: redlogs/08-cli-main.red.log)
test: mutation checks (429 backoff, dedupe, charge-on-empty) saved under mutation-logs/
fix: list mainImageUrl/imageUrls use CSphoto car photos; keep shopImageUrl separate (red: redlogs/09-list-car-photos.red.log)
fix: planPrices labels are letter+プラン with separate A/B prices (red: redlogs/10-plan-prices.red.log)
fix: list labels from top-level labelIcon spans only, deduped (red: redlogs/11-labels-dedupe.red.log)
fix: detail imageUrls CSphoto-only prefer L; imageCount from (n/N枚) (red: redlogs/12-detail-images.red.log)
fix: loan 実質年率 + first/bonus/total/monthly payment fields (red: redlogs/13-loan-fields.red.log)
fix: dealerTel full-width + normalized, dealerName/Address from loan block (red: redlogs/14-dealer-tel.red.log)
fix: rename inspectionDate to carInspection; add qualityInspectionDate (red: redlogs/15-inspection-fields.red.log)
fix: strip inquiry/button UI text from warrantyDetail and legalMaintenance (red: redlogs/16-strip-ui-text.red.log)
fix: fullTitle nbsp becomes space between model and grade (red: redlogs/17-fulltitle-nbsp.red.log)
fix: catalogSpecs/shopRatings/shopCoupons; reviews remain NOT_CAPTURED (red: redlogs/18-catalog-ratings-coupons.red.log)
fix: list totalCount from resultBar; totalCountIsLowerBound for 以上 (red: redlogs/19-totalcount-lowerbound.red.log)
fix: dataset_schema lists all list+detail fields; coverage test (red: redlogs/20-dataset-schema-fields.red.log)
test: mutation check (revert list image selector → shopinfo mainImageUrl) under mutation-logs/04-revert-list-image-selector.log
docs: live samples fetched 2026-11-01 ~19:13 SGT (list/detail rows + OUTPUT)
docs: buyer-facing English README with live samples, pricing, legal notes (red: redlogs/21-readme.red.log)
docs: SEO English actor.json title and description (red: redlogs/22-actor-seo.red.log)
test: input_schema title/type/description + README titles (317e783; red: redlogs/23-input-readme.red.log)
docs: COST.md measured cost estimate per 1000 rows (81fe544).
fix: derive crawl deadline from platform timeoutAt with safety margin + force-exit (red: redlogs/24-platform-deadline.red.log)
fix: shopAddress from shop info block only (never photo captions); add live detail fixtures (red: redlogs/25-shop-address.red.log)
fix: dealerTel full-width ＴＥＬ from loan block only; shopTel when shop block publishes regular phone (red: redlogs/26-dealer-tel-fw.red.log)
test: mutation-kill coverage for pagination/detail billing/deadline/retry/timeout/OUTPUT/push (red: redlogs/27-mutation-kill.red.log)
fix: pushData before charge; push failure not charged, recorded in errors (red: redlogs/28-push-before-charge.red.log)
fix: wire proxyConfiguration (default direct; datacenter only; strip RESIDENTIAL) (red: redlogs/29-proxy.red.log)
docs: README same-basis pricing vs serp.cheap.ofc $3.00/$5.00 (red: redlogs/30-pricing-numbers.red.log)
docs: live-sample-list-20261101.json now has all 60 list rows (red: redlogs/31-live-sample-60.red.log)
fix: remove always-null priceNote from parser, schema, README, samples (red: redlogs/32-remove-pricenote.red.log)
fix: maxPagesPerSearch maximum 200 (default 20) (red: redlogs/33-maxpages-200.red.log)
docs: CHANGELOG references commits 317e783 and 81fe544 (red: redlogs/34-changelog-shas.red.log)
fix: entry/run/http/proxy/billing default to apify log (not console); buildEntryOptions (red: redlogs/35-entry-apify-log.red.log)
test: mutation buildEntryOptions → console killed (mutation-logs/15-entry-log-console.log)
fix: force-exit writes OUTPUT stopReason=deadline then Actor.exit; cooperative stop ~105s + capped backoff (red: redlogs/36-force-exit-output.red.log)
fix: dealerName by TEL position; shopTel from dealerBlock/shopCaption (skip mobile); 11 prefecture fixtures (red: redlogs/37-shop-tel-fixtures.red.log)
fix: extract buildGotOptions/gotImpl; caption cut + proxy default-off + retry bound mutations (red: redlogs/38-got-options.red.log)
fix: warn/error log helpers with warn/info/console fallbacks (red: redlogs/39-warn-helper.red.log)
fix: proxyUrls without useApifyProxy use custom proxies via createProxyConfiguration (red: redlogs/40-proxy-urls.red.log)
fix: rename chargedButNotPushed → pushedButNotCharged in billing/OUTPUT/README (red: redlogs/41-pushed-but-not-charged.red.log)
docs: note that the 4 detail fixtures were actually added in 894b9af not d0ff468 (red: redlogs/42-fixture-changelog.red.log)
Fix: merged detail rows always carry every detail key (null when the page has no value, list value kept when present), so shopTel/dealerTel* keys no longer vanish from JSON. Red: redlogs/43-merge-stable-keys.red.log.
Docs: regenerated docs/live-sample-detail-20261101.json (5 live rows, 20:44 SGT, 7 requests at ≤1 req/s) and README full output example with the current parser: clean shopAddress, shopTel/shopTelNormalized/shopTelSource/dealerTelNormalized present (review f0dd367 B-2). Old sample moved to /workspace/_archive. Red: redlogs/44-fresh-detail-sample.red.log.
test: got-options asserts opts.timeout exists before reading timeout.request, so removing timeout fails on AssertionError not TypeError; re-ran mutations 17–21, all AssertionError (review 5d328da A-2; red: mutation-logs/17-remove-got-timeout.log, 18–21 re-run; old logs in /workspace/_archive/carsensor-mutlogs-5d328da)
chore: af7c988 (follow-up to 6c26732) only added the trailing newline to src/detail.js, regenerated redlogs/37-shop-tel-fixtures.green.log after the fact, and rewrote mutation-logs/16-address-osaka-only.log; no behavior change (review 5d328da A-3; red: redlogs/45-af7c988-changelog.red.log)
fix: shop/dealer phones 070/080/090 are kept as published (jade: business-contact info stays verbatim); loan block with a mobile TEL no longer drops dealerName/dealerAddress/dealerTel/shopTel (山田モータース repro); only 0078 free-dial stays out of shopTel; schema + README dealerTel/shopTel descriptions updated (review 5d328da A-1; red: redlogs/46-keep-mobile-tel.red.log)
fix: finish() stops the biller before writing OUTPUT (biller.stop: no pushData/charge afterwards) and the crawl loops bail out once finished, so a force-exit OUTPUT stays final while Actor.exit waits; test proves no pushData/charge after OUTPUT (review 5d328da M-1; red: redlogs/47-force-exit-stops-writes.red.log)
test: OUTPUT is written exactly once when force-exit and the normal finish both run; removing the finished guard turns it red (review 5d328da M-2; red: mutation-logs/22-remove-finish-guard.log)
docs: dataset_schema gives every one of the 76 fields a title (review 5d328da N-1; red: redlogs/48-dataset-titles.red.log)
docs: README + input_schema now say custom proxyUrls are used as given and not filtered (a residential URL there is the buyer's own choice and cost); only the Apify RESIDENTIAL group is stripped (review 5d328da N-2; red: redlogs/49-proxyurls-readme.red.log)
test: 0078 free-dial never lands in shopTel (shop block or caption); allowing it turns the test red (review 5d328da N-3; red: mutation-logs/23-0078-into-shoptel.log)
fix: when too little run time is left to fetch anything, OUTPUT says so (stopReason=deadline, isEmpty=false, warning + log line, statusMessage 'not enough run time left…') instead of 'No cars matched' (review 5d328da N-4; red: redlogs/50-deadline-zero-rows.red.log)
docs: detail sample time in the CHANGELOG corrected from ~21:00 to 20:44 SGT (startedAt 12:44Z) (review 5d328da N-5; red: redlogs/51-sample-time.red.log)
test: npm test runs with --test-timeout=30000 so a hung test fails instead of stalling the suite; 8 back-to-back full runs all passed, no hang reproduced (review 5d328da N-6; red: redlogs/52-test-timeout.red.log)
docs: meaning change (含义变更) of OUTPUT.isEmpty: was rows===0 && errors===0; since c3358e8 it is also false when stopReason is 'deadline'. isEmpty=true only means the search really matched 0 cars; a deadline stop gives isEmpty=false with rows possibly 0, so check rows and stopReason. Documented in README (OUTPUT record) and .actor/key_value_store_schema.json (review 4c59b54 B-1; red: redlogs/53-isempty-doc.red.log)
fix: a 0078 free-dial TEL in the loan/dealer block now nulls only dealerTel/dealerTelNormalized; dealerName and dealerAddress are still output (review 4c59b54 N-1; red: redlogs/54-loan-0078.red.log)
feat: new field shopTels = every public shop phone on the detail page as {number, normalized, source} (0078 relay excluded, deduped), priority dealerBlock → shopInfo → shopCaption, documented in README and dataset_schema; existing shopTel/shopTelSource unchanged. Yamada repro now keeps the caption 080 number. README full output example and docs/live-sample-detail rows that have a committed fixture (AU7394561852, AU7387965737, AU7455908016) got shopTels recomputed offline from those fixtures; AU7401891307 and AU7422611469 have no fixture and were left without shopTels (review 4c59b54 N-3; red: redlogs/55-shoptels.red.log)
test: billing unit tests 'emit after stop() pushes/charges nothing' and 'in-flight rows are not charged', plus run-level tests for finish()→biller.stop and the finished checks. Mutations that turn red with AssertionError: remove biller.stop (mutation-logs/24), emit ignores stopped (25), charge in-flight rows (26), and removing all main.js finished checks together (27). Not every layer goes red on its own: removing only the finished check before emitRow (mutation-logs/28), only the loop-top check at main.js:246, or only finished || in crawlSearch at main.js:309 each stays green, because biller.stop already blocks the row and timeShort() is also true at force-exit; these finished checks are redundant backstops (corrected in review ae36598 N-3) (review 4c59b54 N-2; red: redlogs/56-billing-stop-layers.red.log)
fix: docs/live-sample-detail rows AU7401891307 and AU7422611469 now carry shopTels=[{number:０７２－６９６－２７１１, normalized:072-696-2711, source:shopCaption}], recomputed offline from their public detail pages fetched once each on 2026-11-01 (≥1 s apart) and committed as test/fixtures/detail-AU7401891307.html / detail-AU7422611469.html. The shoptels sample test no longer skips rows without a fixture (a missing fixture fails), and a new test requires the shopTels key on every sample row. README full output example (AU7394561852) already matches (review ae36598 B-1; red: redlogs/57-sample-shoptels.red.log)
fix: removed contentTypes from the OUTPUT collection in .actor/key_value_store_schema.json so the platform cannot reject the SDK's default 'application/json; charset=utf-8' OUTPUT write (review ae36598 N-1; red: redlogs/58-kv-no-contenttypes.red.log)
fix: dataset_schema shopTels now matches the code: type 'array' only (the code never outputs null; [] when none, key absent on list-only/listing-ended/detailError rows), and items number/normalized/source have title and description; README field table says 'array' (review ae36598 N-4; red: redlogs/59-shoptels-schema.red.log)
docs: corrected the ae36598 CHANGELOG entry: it no longer says each layer turns red; removing only the finished check at main.js:246 or :309 (or before emitRow) stays green, they are redundant backstops (review ae36598 N-3; red: redlogs/60-changelog-backstops.red.log)
fix: shopTels recall — labelled numbers are now found with TEL (colon optional), 電話番号, 直通電話, 店舗直通, 携帯, 電話 and separators like ☆; planInfo text (source 'planInfo') and the 保証 warranty cell (source 'warranty') are scanned too; priority dealerBlock → shopInfo → shopCaption → planInfo → warranty; 0078 still excluded, deduped by digits; numbers must start with 0 and have 10–11 digits (no postal codes / IDs / prices). shopTel/shopTelSource unchanged. New live fixtures detail-AU7381287206/AU7216565161/AU7441677706/AU7442558945 (fetched once each 2026-11-01 22:02 SGT, ≥1 s apart, actor UA); on the 15 existing detail fixtures shopTels is byte-identical (baseline test/fixtures/shoptels-baseline-401a046.json). README + dataset_schema enum/priority updated (review 401a046 #1; red: redlogs/61-shoptels-recall.red.log)
fix: HTML entities are decoded exactly once in text built from .html(): fullTitle (AU7441677706 showed a literal 「シート&ステア」 — cheerio .html() re-encodes & once; now 「シート&ステア」) and the loan/dealer block lines (dealerName/dealerAddress). A new test checks every string field of every committed list/detail fixture for leaked entities (none left); list grade already decoded via cheerio (review 401a046 #2; red: redlogs/62-entity-decode.red.log)
fix: shopTels gets a separate source 'catchCopy' (review correction to 401a046 #1: 048-753-6633 is the dataset's catchCopy). On the detail page that text sits in .planInfo__description (same text as the list-card catchCopy), now source 'catchCopy'; the rest of .planInfo stays 'planInfo'; mergeDetail also scans the list-row catchCopy (deduped by digits). Priority dealerBlock → shopInfo → shopCaption → catchCopy → planInfo → warranty in code, README and dataset_schema enum. AU7442558945 now gives 048-753-6633 source catchCopy; no other fixture or docs sample changes, list fixtures' catchCopy yield no numbers (red: redlogs/63-shoptels-catchcopy.red.log)
fix: a short run timeout (e.g. 120 s) no longer reports an empty search as deadline. List page 1 now only needs one request's budget (REQUEST_TIMEOUT_MS + FINISH_MARGIN_MS) instead of the worst-case retry budget; when page 1 confirms 0 results the run ends stopReason=empty, isEmpty=true. Non-empty searches with too little time still stop as deadline with 0 rows charged; with <30 s left nothing is fetched. test/main.test.js 'timeShort stops before list fetch' now expects exactly 1 list fetch (page 1) and 0 rows. README notes this and a minimum suggested timeout (review 401a046 #3; red: redlogs/64-empty-short-timeout.red.log)
fix (bug fix): shopTel / shopTelNormalized / shopTelSource now take shopTels[0] (number / normalized / source) when the single pick was null but shopTels is non-empty — this only fills values that were wrongly null; rows that already had shopTel are unchanged (parseDetailPage and mergeDetail incl. list catchCopy). Affects AU7381287206, AU7216565161, AU7340667437, AU7442558945, AU7328650264. New baseline test/fixtures/detail-baseline-95b2667.json (full parse of the 17 original detail fixtures at 95b2667): shopTels byte-identical, shopTel* only null→shopTels[0], all other fields identical. README + dataset_schema describe the fallback and the shopTel/shopTels relation (rescore 95b2667 error B, review N6; red: redlogs/65-shoptel-fill.red.log)
fix: labelled-phone recall adds the inquiry label お問い合わせ (also お問合せ / 問い合わせ / 問合せ / 〜先, optional は), so 「お問い合わせは0537-62-6130まで」 is captured. New live fixture test/fixtures/detail-AU7328650264.html (GET once 2026-11-01 22:34 SGT, ≥1 s gap, actor UA): shopTels[0]=0537-62-6130 source shopCaption, shopTel* filled from it. 0078 / postal codes / 「お電話でのお問合せはこちら」 still give nothing; the 17 original fixtures are unchanged vs detail-baseline-95b2667.json. README + dataset_schema label lists updated (rescore 95b2667 error A; red: redlogs/66-inquiry-label.red.log)
fix: third-party phones excluded from shopTels and the shopInfo/shopCaption single pick (and the list catchCopy path): a number with 保証会社 / カーティオ / JAF / ロードサービス / ローン会社 / 信販 / オリコ / ジャックス / セディナ / アプラス / カスタマーセンター / コールセンター within ~30 chars before or ~15 after (same sentence, cut at the neighbouring number / ／、。) is dropped, unless 当店/当社/弊社/自社 follows that word before the number, so the shop's own 24h tow line 080-9408-8173 (AU7381287206/AU7216565161 warranty) stays. dealerBlock is not filtered. Synthetic negative tests for each keyword; all 18 detail fixtures re-run: no number lost (17 byte-identical vs detail-baseline-95b2667.json, AU7328650264 keeps 0537-62-6130). README + dataset_schema document it (review 95b2667 N4, rescore remaining #3; red: redlogs/67-thirdparty.red.log)
feat: startUrls accepts detail page URLs (/usedcar/detail/AU…/ or …/index.html): fetched directly, one full detail row each (all list keys present, list-card-only ones null), charged detail-row without includeDetails. List and detail URLs can be mixed; detail URLs run first, deduped by carId (host/query/fragment ignored), and list pages skip a carId already taken. Malformed detail URLs are skipped with a warning (OUTPUT.warnings); 404 / ended detail URLs give no row, no charge, a warning; 403/429/5xx go to errors. Only-detail input no longer falls back to the default Voxy search. New fields-free change (no output field renamed/removed); normaliseInput adds detailUrls/inputWarnings. input_schema description + README updated (rescore 95b2667 remaining #4; red: redlogs/68-detail-urls.red.log)
fix/test: review 95b2667 non-blocking items — N1 test for the bare 電話 label (kills M03); N2 tests for findLabelledTels 0078 exclusion and 10–11 digit check (kill M15/M16), isPublishedShopPhone is now a shared export and the list catchCopy path (withCatchCopyTels) also goes through it; N3 test for loan block 「A&B商会」→「A&B商会」 (kills M20); README states the shopTel/shopTels relation (shopTel is one number and always appears in shopTels); N7 a short timeout where list page 1 returns 5xx/timeout and there is no time to retry now ends stopReason=deadline with the HTTP error in OUTPUT.errors, 0 rows, 0 charged, not FAILED (ample time: page-1 error still fails as before), README documents it (review 95b2667 N1/N2/N3/N6/N7; red: redlogs/69-review-nonblocking.red.log)
fix: dealerBlock only from a real loan/dealer block — the loanComment, or (no loanComment) a <p> whose TEL line directly follows a prefecture address line; a bare 「…
TEL」 paragraph (e.g. 保証会社カーティオ) is no longer a dealerBlock, and every TEL line inside a dealer block goes through isThirdPartyContext (the line + the line before), so guarantee/loan company numbers never become shopTel/dealerTel/shopTels dealerBlock. README updated (no longer "not filtered"). 18 detail fixtures + 90 list rows byte-identical to 2e9327c (review 2e9327c N1 / round item A; red: redlogs/70-dealerblock.red.log)
test: maxItems on the detail-URL path — 3 detail URLs with maxItems=1/2 must give exactly 1/2 rows, charges (detail-row) and detail requests, stopReason=max-items. Code already correct; red log is the review mutation M19 (loop-top maxItems check removed in crawlDetailUrls → 3 rows / 3 charges, ERR_ASSERTION), source restored (review 2e9327c N6 / round item B; red: redlogs/71-detail-maxitems.red.log)
fix: third-party context — fewer false drops: a keyword followed by 加盟店 / 取扱(店) / 提携店 / 可 is the shop describing itself (「オリコローン取扱店 電話 …」「JAF加盟店」), and 当店/当社/弊社/自社 as the subject before the keyword (「当店は JAF 加盟店です TEL …」「当社はロードサービス完備」) keeps the number (「当店提携のローン会社 TEL」 still dropped); fewer leaks: カーコンビニ俱楽部/倶楽部 and loan/warranty desks (ローン/保証 + お問い合わせ/窓口, e.g. 「ローンのお問い合わせ TEL」) added. README + dataset_schema say it is best effort. 18 detail fixtures + 90 list rows byte-identical to 2e9327c; 080-9408-8173 kept (review 2e9327c N2/N3 / round item C; red: redlogs/72-context.red.log)
test: direct tests for the 当店/当社/弊社 keep rule (after the keyword, kills review mutation M08; and as subject before it) and third-party negatives for the shop-info-box TEL： path (保証会社カーティオ / JAF / ロードサービス / ローンのお問い合わせ next to or without a shop TEL; kills M12). Code unchanged; red log is M08+M12 applied, source restored (review 2e9327c N4/N5 / round item D; red: redlogs/73-shopown-shopinfo.red.log)
perf/fix: deadline reserve no longer over-conservative — a request is started while one request timeout + finish margin (REQUEST_RESERVE_MS = 15 s + 15 s) fits, instead of worst-case retries (WORST_CASE_REQUEST_MS 105 s, kept only for the page-1 "no time to retry" deadline-vs-FAILED decision) + margin = 120 s; list-only rows (no detail request) reserve nothing (main.js processListRows / crawlDetailUrls); the HTTP client gets retryMarginMs=FINISH_MARGIN_MS and does not start a retry that would not fit before the deadline (backoff sleeps were already capped; force-exit stays as backstop). Fake-clock tests: 180 s run with 10 s start-up, detail mode 2 s/request delivers all 30 rows (was ~24 rows / stop at 62 s), 3 s/request works until ≥140 s and ends deadline, not FAILED, charges = rows, OUTPUT once; always-timing-out details end before the deadline. Three older tests that encoded the 120 s reserve with a list-only 110 s budget (main.test.js timeShort / zero-rows message, empty-short-timeout non-empty) now use detail mode with a slow page 1 to reach the same 0-row deadline path. README OUTPUT section updated (review 2e9327c N9 / round item E; red: redlogs/74-deadline.red.log)
fix (bug fix, wrong value → correct value): nonSmoking now comes only from the 禁煙車 spec cell (th→td via tableMap), null when there is none; the free-text regex /禁煙車\s*([^\s]+)/ (detail.js:448 in 2e9327c) matched title / meta words first. Checked against the 51 QA original pages (/workspace/qa/cs2e/pages, no new fetch): exactly these 12 values change, each now equal to the page spec cell, nothing else changes; 18 detail fixtures + 90 list rows byte-identical to 59c240e. AU7193186587 「SDナビ」→「◯」; AU7275159984 「・下取直売・スマートキー&プッシュスタート・ベンチシート・ルームクリーニング」→「◯」; AU7275315505 「(大阪)の中古車詳細」→「◯」; AU7292812141 「/衝突軽減/14AW/ETC/(千葉)の中古車詳細」→「◯」; AU7340665342 「ETC車載器」→「◯」; AU7354839188 「（ディープクリスタルブルーマイカ）」→「◯」; AU7359918876 「自動ドア」→「◯」; AU7394825061 「ターボ(高知)の中古車詳細」→「◯」; AU7405102516 「SDナビ」→「◯」; AU7420198753 「ワンオーナー車」→「◯」; AU7442558870 「(岡山)の中古車詳細」→「◯」; AU7460843009 「/車いす仕様車/純正ナビ/ワンセグ/CD(愛知)の中古車詳細」→「◯」 (rescore 2e9327c #1; red: redlogs/75-nonsmoking.red.log)
fix (bug fix, wrong value → correct value): oneOwner now comes only from the ワンオーナー spec cell (th→td via tableMap), null when there is none; the free-text regex (detail.js:445 in 2e9327c) matched title / equipment words. On the 51 QA original pages exactly the 4 rescore values change, each now equal to the page spec cell: AU7340665342 「衝突軽減ブレーキ」→「◯」; AU7420198753 「車」→「◯」; AU7441677706 「(北海道)の中古車詳細」→「◯」; AU7446187159 「車・認定中古車(福岡)の中古車詳細」→「◯」. The same bug also hits 3 committed fixtures outside the rescore list (not in its 47-row sample), fixed the same way and checked against each fixture's spec cell: AU7376720181 「純正10.5型DA(東京)の中古車詳細」→「◯」; AU7387965737 「純正SDナビ(NSZT-W62G)/AM(愛知)の中古車詳細」→「◯」; AU7455908016 「純SDナビテレビフルセグテレビ(愛知)の中古車詳細」→「－」 (fixture AU7441677706 too; list rows #24/#63/#64 carry the same 3 changes). shoptel-fill regression test now pins these 4 fixture oneOwner values, all other fields still identical to the 95b2667 baseline (rescore 2e9327c #2; red: redlogs/76-oneowner.red.log)
feat: detail-URL rows (startUrls detail URLs) now fill list-card fields from the detail page itself (new export parseDetailListFields, used only on that path so list / list+detail rows are unchanged): maker, model, grade, catchCopy, year, yearJp, mileageKm, mileageText, inspection, repairHistory, warranty, maintenance, displacementCc, transmission, bodyType, prefecture, city, shopName, shopUrl, shopId, shopCatch, monthlyLoanJpy. Checked offline against 62 list+detail rows (QA run rows /workspace/qa/cs2e + fixture list rows, with the saved pages): all 22 fields equal except monthlyLoanJpy on AU7394561852 (list 9300 / page 9,800) and AU7401891307 (9600 / 10,000), where the detail page itself shows the newer amount. Still null and listed in README: color, shopImageUrl, shopReviewScore, shopReviewCount, labels/planPrices ([]), searchUrl, page. No field renamed/removed; 18 detail fixtures + 90 list rows byte-identical to 37760ee (rescore 2e9327c #3; red: redlogs/77-detail-url-fields.red.log)
fix: フリーダイヤル / フリーダイアル (and half-width ﾌﾘｰﾀﾞｲﾔﾙ) is a phone label, so 「フリーダイヤル０１２０－２１８－００７」 without TEL is captured; 0078 relay and third-party context still excluded. Shop-own check from the saved original page only (/workspace/qa/cs2e/pages/AU7446053416.html, no new fetch): the number appears only in that shop's own photo captions written in the shop's voice (「ご購入のお悩み相談は、フリーダイヤル…まで」「全国納車…ご質問などありましたらお気軽に♪フリーダイヤル…」), no third-party wording, no other non-0078 phone on the page. Changes: AU7446053416 (QA page) shopTels [] → [０１２０－２１８－００７ / 0120-218-007 / shopCaption], shopTel* null → same; 18 detail fixtures + 90 list rows byte-identical to 3938af8. README + dataset_schema label lists updated (rescore 2e9327c #4; red: redlogs/78-freedial.red.log)
fix: recordBook / recycleFee now come only from the 定期点検記録簿 / リサイクル料 spec cells (th→td), null when absent, replacing the same free-text regexes that broke nonSmoking / oneOwner. No existing value changes: 18 detail fixtures + 90 list rows byte-identical to 62a3ef9, and all 51 QA original pages give identical recordBook / recycleFee (reviewer request; red: redlogs/79-recordbook-recyclefee.red.log)
fix (review 59c240e NA1): the no-loanComment dealer-block fallback was too narrow and dropped the shop's own number in 4 layouts that 2e9327c handled (address without 都道府県, a line between address and TEL, TEL before the address, address and TEL on one line). A <p> now counts as a dealer block when it has a TEL line and an address anywhere (〒 / 都道府県 / 市区町村+number), on the TEL line or not. Paragraphs with no address (e.g. 「保証会社カーティオ
TEL」) are still rejected. New synthetic fixtures test/fixtures/synthetic-dealer-fallback-{1..4}-*.html go through this path. No change in the 18 detail fixtures, 90 list rows or 51 QA pages (red: redlogs/80-na1-fallback.red.log)
fix (review 59c240e NA2): a 保証会社 / loan-company entry with an address line containing a prefecture (「保証会社… / 東京都… / TEL」, also in loanComment 「提携ローン会社 オリコ / 東京都… / TEL」) still put its number into shopTel / dealerTel, because the third-party check only saw the TEL line and the line before it. A dealer-block TEL is now also dropped when any earlier line of its entry (the lines since the previous TEL line) names a third party (new lineNamesThirdParty; self-descriptions such as 提携店 and lines with 当店/当社 do not count). README: removed "never becomes shopTel/dealerTel" and now says best effort (an entry naming only 「株式会社○○」 can still slip through). It also documents the NA1 address rule, which the NA1 commit c7c353c left out of the README. No change in fixtures, list rows or QA pages (red: redlogs/81-na2-block-thirdparty.red.log)
fix (review 59c240e NC1): the 「当店は…」 subject rule and the 〜可 / 取扱 self-description let real third-party numbers through, which 2e9327c excluded: 「当店は提携の保証会社 TEL」, 「当社が紹介する保証会社 TEL」, 「保証会社可 TEL」, 「ジャックス取扱 窓口 TEL」. The subject rule now keeps a number only when every keyword is followed by a shop-feature word (加盟 / 完備 / 会員 / 対応). 取扱 counts only as 取扱店 / 取扱あり. Company-type keywords (保証会社, ローン会社, 信販, call centres, loan desks) count as self-description only with 〜店 (加盟店 / 取扱店 / 提携店). Tests also cover MC5 (subject scope with a second keyword), MC6 (〜可 before the number) and MC7 (〜可 after the number). Earlier keeps (F1–F5, 「当社はJAFの会員店」, 「ジャックス取り扱いあり」) are unchanged. No change in fixtures, list rows or QA pages (red: redlogs/82-nc1-subject.red.log)
fix (review 59c240e NE1): the remaining-time check ran before the request-interval wait. With requestIntervalSecs ≥ 16 (or a long Crawl-delay), a request could start with < 15 s left and a 180 s run ended at 182 s. The client now exposes nextWaitMs(). timeShort (list pages, detail rows, detail URLs) and the client's own retry check both add that wait to the reserve. Test: requestIntervalSecs 16 / 40 / 60 → the run ends by 180 s and every request starts with ≥ 30 s left. No change in fixtures or list rows (red: redlogs/83-ne1-interval.red.log)
fix (review 59c240e NE2): after a 5xx / 403 / 429 or a network error, the client slept the back-off (up to the deadline) before checking whether a retry could still fit, so one 503 near the deadline slept uselessly until force-exit. The check (noTimeForRetryAfter(backoff): back-off, request-interval wait, request timeout and margin) now runs before the sleep; if a retry cannot fit, the error is returned at once. The post-sleep check is kept. New tests cover both paths (503 and network) for: no retry fits → no sleep; plenty of time → normal retries; near deadline → no retry (also fills the ME12 gap). Existing test main.test.js 'http backoff sleep is capped…' had no back-off sleep left to observe; it now reaches the capped sleep through a second request's 60 s interval wait. Its assertions are unchanged (red: redlogs/84-ne2-retry-before-sleep.red.log)
docs (review 59c240e NE3): README said a deadline-cut request leads to "never a FAILED run". That is wrong for one case: a run given only detail URLs, where every detail request fails (5xx) and 0 rows are delivered, ends with failed=true even with stopReason=deadline (the N8 root cause, not changed here). The README now names this exception, and a test pins both the wording and the behaviour (red: redlogs/85-ne3-readme-failed.red.log)
test (review 59c240e ME2, M23/N7, MC5–7): new tests only, no code change.
fix (OOM at 256 MB, build 0.1.4 runs tKQMIwSKlt75PljuZ / WvtzbFovQLy1v1Bna): npm start now runs node --max-old-space-size=$((ACTOR_MEMORY_MBYTES*35/100)) (89 MB at 256 MB).
fix (QA 59c240e, AU7338077003): 011-896-4111 was missed. The cause was not a ローン rule: the number had no recognised label (「…詳しくはアウトレット店へご相談ください！011-896-4111」).
feat: memory guard (review ②): once RSS passes 85 % of ACTOR_MEMORY_MBYTES the run stops before its next request like a deadline — stopReason memory-limit, OUTPUT written once, charged = rows, isEmpty=false, not FAILED; guard off when the memory size is unknown (local). New enum value only, no field changes (red: redlogs/89-memory-guard.red.log, green: redlogs/89-memory-guard.green.log)
docs(changelog): addendum to 37760ee (oneOwner from the spec cell) — the committed fixture test/fixtures/detail-AU7441677706.html also changed: oneOwner 「(北海道)の中古車詳細」→「◯」 (equal to its ワンオーナー spec cell; value computed at 37760ee^ vs HEAD). No code change (review b9aca04 §2; red: redlogs/90-changelog-au7441677706.red.log, green: redlogs/90-changelog-au7441677706.green.log)
fix (NA2-R, regression vs 59c240e): a dealer-block shop-name line advertising road service as a shop feature (「24時間ロードサービス」, 「ロードサービス完備 / 対応 / あり / 付」) no longer makes the entry back-scan (lineNamesThirdParty) drop the shop's own TEL; the road-service desk / company itself (「24時間ロードサービス受付センター」, 「JAFロードサービス」, 「株式会社ロードサービスジャパン」) is still excluded. Only lineNamesThirdParty changes; fixtures, list rows, QA pages (cs2e + cs59) byte-identical to b2b56a7 (review b9aca04 NA2-R; red: redlogs/91-na2r-roadservice-name.red.log, green: redlogs/91-na2r-roadservice-name.green.log)
fix (NC1-R): the 「当店は…」 subject rule now keeps 「当店ではオリコのローンがご利用可能」-style text (keyword + の…ローン + (ご)利用, which was wrongly excluded since b9aca04), and no longer accepts 対応 as a shop feature after company-type keywords (保証会社 / ローン会社 / 信販 / call centres), so 「当店は保証会社対応 TEL」 is excluded; 対応 after service keywords (「当社はロードサービス対応」) still keeps. Fixtures, list rows, QA pages (cs2e + cs59) byte-identical to b2b56a7 (review b9aca04 NC1-R; red: redlogs/92-nc1r-subject.red.log, green: redlogs/92-nc1r-subject.green.log)
fix (NF1): a フリーダイヤル number next to a customer-centre word (お客様/お客さま相談センター・窓口・相談室, 相談センター, サポートセンター/ダイヤル, メーカー) or an audio / navi / parts maker name (パイオニア, カロッツェリア, ケンウッド, パナソニック, アルパイン, イクリプス, クラリオン, デンソー, ユピテル, コムテック, セルスター, ブリヂストン, ダンロップ) — ~30 chars before / 15 after in the same sentence — is no longer taken into shopTels / shopTel. Only the フリーダイヤル label is affected; 「ご購入のお悩み相談は、フリーダイヤル…まで」 (AU7446053416) still kept. README + shopTels description updated (description text only). Fixtures, list rows, QA pages (cs2e + cs59) byte-identical to b2b56a7 (review b9aca04 NF1; red: redlogs/93-nf1-freedial-maker.red.log, green: redlogs/93-nf1-freedial-maker.green.log)
test (NT1 / MC5): tests that kill the surviving mutants A05 (lineNamesThirdParty without the 当店 exception), A07 (without the self-description exception), E02 (post-back-off retry check without the request-interval wait — interval raised during the back-off) and MC5 (subject rule scoped to the whole pre window: 「JAF会員 当社はロードサービス完備 TEL」). Test-only, no src change. Red = the new tests run against each mutant (each fails 1 with ERR_ASSERTION, then reverted): redlogs/94-surviving-mutants.red.log; green: redlogs/94-surviving-mutants.green.log (review b9aca04 NT1, MC5)
docs(changelog): addendum (QA rescore b9aca04 §A) — 9 more existing-value corrections from the same spec-cell fix (1410170 nonSmoking / 37760ee oneOwner), on pages delivered only in the 59c240e rescore (not in the 51 QA pages used above); each equals the page's spec cell; no code change. nonSmoking: AU7282800284 「（スマートブラック）」→「－」; AU7282943939 「7インチナビ(愛知)の中古車詳細」→「◯」; AU7289415195 「(大分)の中古車詳細」→「◯」; AU7310732726 「HondaSENSING(高知)の中古車詳細」→「◯」; AU7332697428 「HondaSENSING(高知)の中古車詳細」→「－」; AU7376050461 「衝突軽減システム(福岡)の中古車詳細」→「◯」; AU7412607023 「SDナビ」→「◯」; oneOwner: AU7338077003 「(北海道)の中古車詳細」→「◯」; AU7388314307 「純正ディスプレイオーディオ(北海道)の中古車詳細」→「◯」 (values re-derived from /workspace/qa/cs59 pages, 59c240e vs b2b56a7; red: redlogs/95-changelog-rescore-9.red.log, green: redlogs/95-changelog-rescore-9.green.log)
fix (NF1-R): the フリーダイヤル maker rule no longer drops the shop's own number: the generic word メーカー is removed (「メーカー保証付き車両多数」「全メーカー取扱」); 当店/当社/弊社/自社 in the same-sentence window keeps the number (「メーカー保証付き 当店フリーダイヤル」「当店のお客様相談窓口」「フリーダイヤル… 当店お客様センター」); a maker name glued to a preceding word or followed by 〜本店/支店/店/認定店/販売 is a shop name (「カーショップパイオニア本店」「デンソーソリューション認定店」). Maker-centre numbers (「トヨタお客様相談センター」「パイオニア カスタマーサポート」「パイオニア株式会社」) are still dropped; TEL labels unchanged. Tests also lock mutants X13 (rule only for フリーダイヤル), X14 (メーカー) and X16 (sentence cut). README + shopTels description text updated. Fixtures, list rows, cs2e + cs59 pages byte-identical to ae23453 (review ae23453 NF1-R; red: redlogs/96-nf1r-shop-freedial.red.log, green: redlogs/96-nf1r-shop-freedial.green.log)
fix (NS-S1): a 「ご相談ください / ご連絡ください」 number whose sentence points at the maker or a dealer (ディーラー / メーカー within ~30 chars before, same sentence) without 当店/当社/弊社/自社 is no longer taken (「メーカー保証はディーラーへご相談ください。0120-…」); 「詳しくはアウトレット店へご相談ください！」 (AU7338077003) and 「…も当店へご相談ください」 still kept; other labels unchanged. Also adds redlogs/96-nf1r-mutants.log (NF1-R tests vs mutants X13 / X14 / X16, 1–2 ERR_ASSERTION each). README updated. Fixtures, list rows, cs2e + cs59 pages byte-identical to ae23453 (review b2b56a7 NS-S1; red: redlogs/97-ns-s1-dealer-soudan.red.log, green: redlogs/97-ns-s1-dealer-soudan.green.log)
fix (NH1): more heap headroom and a heap-aware memory guard. npm start now uses --max-old-space-size = 45 % of ACTOR_MEMORY_MBYTES (115 MB at 256 MB; was 35 % = 89 MB), --max-semi-space-size=4 (keeps RSS down) and --expose-gc. The memory guard (still off when the memory size is unknown) also stops when V8 used_heap_size / heap_size_limit > 85 % after a forced GC (garbage alone never stops a run); new run() injectables heapStats / gc. Local measurements (bHO-like: 35 details, 2 list pages, 1 s interval, 180 s timeout, cs59 pages; /workspace/_oom/nh1.txt): normal pages peak RSS 166.7–170.0 MiB (old flags 169.6–173.6), pages 4× larger 179.3–185.6 MiB and all 35 rows (old 89 MB cap: heap OOM crash), 5×/6× also finish (≤ 182.9 MiB); ≥ 7× a single page parse exceeds the heap (limit stated in README). Tests: start-flag test now asserts 115–128 MB + semi-space + --expose-gc; heap guard (stops after GC / not on collectable garbage); (test/memory-nh1.test.js) 20 detail pages 4× larger with the 256 MB flags finish < 200 MB RSS; a real child process with a 120 MB limit stops as memory-limit, not failed, charged = rows. mem-run helper gains AMP / MEM_LIMIT_MB. No dataset output change (review b2b56a7 / ae23453 NH1; red: redlogs/98-nh1-heap-headroom.red.log (3 ERR_ASSERTION) + redlogs/98-nh1-heap-headroom.red2.log (4× test vs ae23453 package.json: 1 ERR_ASSERTION; the real-process memory-limit test already passes there as a lock), green: redlogs/98-nh1-heap-headroom.green.log)
test (NT2 / X02): main's ACTOR_MEMORY_MBYTES wiring (256 → guard on; unset → guard off), the memory guard in detail-URL mode (stops between detail requests: OUTPUT once, charged = rows, not FAILED), and the ロードサービス-only road-feature exception (「保証会社完備」 still a third party). main() now forwards the optional rssBytes / heapStats injectables to run() (test seam, no behavior change). Red: redlogs/99-nt2-main-memory.red.log (main wiring test, 1 ERR_ASSERTION — the forwarding did not exist); mutants M08 (memoryLimitMb not passed), M09 (wrong env name), M26 (detail-URL loop ignores the guard), X02 each fail 1 ERR_ASSERTION: redlogs/99-nt2-main-memory.mutants.log; green: redlogs/99-nt2-main-memory.green.log (reviews b2b56a7 NT2, ae23453 X02)
docs(changelog): correction — the three indented sub-items 「New label: …」, 「Evidence that it is the shop's own …」 and 「Changes: only AU7338077003.shopTels gains 011-896-4111 …」 that appear further up (after the b9aca04 rescore addendum) belong to the 3875511 entry 「fix (QA 59c240e, AU7338077003): 011-896-4111 was missed …」, not to the b2b56a7 memory-guard entry or any later entry; they were appended at the end of the list by mistake. Left in place (history unchanged); this entry only fixes the attribution (review b2b56a7 §1; red: redlogs/100-changelog-3875511-attribution.red.log, green: redlogs/100-changelog-3875511-attribution.green.log)
test (N22): the memory guard's default gc wiring — with gc not injected (production), run() uses globalThis.gc (provided by --expose-gc in npm start), so heap pressure that a GC frees never stops the run. Test-only. Red = the test vs mutant N22 (default gc = null): 1 ERR_ASSERTION, redlogs/101-n22-default-gc.red.log; green: redlogs/101-n22-default-gc.green.log (review 2560b32 N22)
fix (NF1-R2): フリーダイヤル maker rule — the customer-centre / maker-support exclusion now runs before the 当店/弊社 rescue (「当店取扱ナビのお問合せはパイオニアカスタマーサポート」「当店でご購入のお客様もトヨタお客様相談センター」「ケンウッドお客様センター（当店取付品も対応）」 are excluded); a plain maker name is rescued only by 当店/弊社 between it and the number (「…パイオニア（弊社取扱）」 excluded); 店頭/店内/店舗 and bare 販売 after a maker name and 純正/正規/社外/製 before it no longer count as a shop name; 「自社製」 is not a shop self-reference; 「当店のお客様相談窓口」-style shop-owned centres stay kept. Fixture output (list snapshot, cs2e, cs59) unchanged. Red: redlogs/102-nf1r2.red.log (9 ERR_ASSERTION); green: redlogs/102-nf1r2.green.log (review 2560b32 NF1-R2)
fix (NS-S1 S5): 「…ご相談ください」 is no longer treated as pointing at a dealer when ディーラー only describes the car (ディーラー車/系/下取/出身/モデル), e.g. 「元ディーラー車です お気軽にご相談ください！011-…」 keeps the shop's number (a space is still not a sentence boundary). Fixture output unchanged. Red: redlogs/103-ns-s1-space.red.log (2 ERR_ASSERTION); green: redlogs/103-ns-s1-space.green.log (review 2560b32 NS-S1)
test (mutants N03/N09/N10/N14/N15): lock the glued shop-name prefix (katakana/kanji/latin), a non-maker 〜サポートセンター alone, and the NS-S1 sentence cut and 30-char window. Test-only. Red = these tests vs each surviving mutant (N03 3, N09 2, N10 1, N14 1, N15 1 ERR_ASSERTION): redlogs/104-mutants.red.log; green: redlogs/104-mutants.green.log (review 2560b32 §6)
feat (captionTels, added field): new detail-row array captionTels = phone numbers the shop wrote in photo captions, labelled or bare, as {number, normalized, source:'shopCaption'} like shopTels; collected even when they duplicate shopTels; excluded with the same rules (third-party warranty/loan/road-service/call-centre context, NF1-R/NF1-R2 maker and customer-centre context, NS-S1 「ディーラー/メーカーへご相談ください」). shopTels / shopTel unchanged. LIBERALA (リベラーラ) fixture detail-AU7271106092.html (from QA run a2spPXkh7R03gpawd, qa/cs256/pages) gives [0120-77-8440] (shopTels stays []). dataset_schema, README field table and the includeDetails input description updated (README full-output example left as the real live-sample row). Fixtures: list snapshot / cs2e / cs59 identical except the added captionTels (and the new fixture); cs59: 14/76 pages with captionTels, 19 numbers, all also in shopTels. Red: redlogs/105-captiontels.red.log (3 ERR_ASSERTION, with an empty findCaptionTels stub); green: redlogs/105-captiontels.green.log (review 2560b32 PM item / QA §4)
