TDnet Timely Disclosures

Apify Actor using Node 20, ESM, Apify SDK v3, got 14 and Cheerio. It reads the public Japanese TDnet disclosure list, preserves Japanese text, and writes one dataset record per disclosure. It resolves PDF and XBRL URLs without downloading either file.

TDnet exposes roughly one month of list history. Older dates can return 404, while weekends and holidays can return empty pages. Those dates are skipped. Schedule daily runs and retain/export their datasets to accumulate longer history. Incremental mode deduplicates recently emitted disclosures across runs; it does not create a historical archive. Today's list can change during the day.

TDnet's captured robots.txt says User-agent: * / Disallow: /. The Actor reports this restriction in its log and run summary. It does not enforce Disallow as a stop condition: its requested behavior is to fetch list pages while honoring Crawl-delay. Confirm permission from the site before production operation. This build and validation do not establish that permission.

Input

{
  "dateFrom": "2026-10-13",
  "dateTo": "2026-10-13",
  "maxItems": 1000,
  "order": "newestFirst",
  "incremental": false,
  "stateKey": "default",
  "companyCode": "4714",
  "proxy": { "useApifyProxy": false }
}

Both dates independently default to today in JST (UTC+9), inclusive. Use real YYYY-MM-DD dates; reversed ranges and invalid dates fail before requests. Future dates are skipped without a list request. Missing older dates are skipped after the first page response. Very wide past ranges therefore make one request per unavailable day; use the available month window when possible.

order defaults to newestFirst (dateTo down to dateFrom); oldestFirst visits dateFrom up to dateTo. Rows within a day always retain site order.

maxItems defaults to 1000 and caps emitted disclosures after filtering. companyCode is optional: five characters match exactly, and four characters match a TDnet code ending in 0. Codes such as 293A0 remain strings. securitiesCodes optionally accepts a list of four-character strings (for example ["4714", "293A", "587A"]); it matches the first four characters of every TDnet code, including a nonzero fifth character. categories optionally accepts a list of category names below. Empty lists mean no restriction. Values within each list are ORed, and all supplied filters are combined with AND. Both filters apply before maxItems and incremental deduplication; a page with no matches still continues pagination. No proxy is used by default; the standard Apify proxy input supports useApifyProxy, groups, country and custom proxyUrls.

Scheduled incremental runs

Set incremental: true, stateKey: "default", and order: "newestFirst" in your scheduled task input. Omit both dates to resolve the current JST day at each run, and schedule after the disclosure period. Fixed dates stay fixed across scheduled runs. To catch missed days, supply an overlapping recent date range (update it through your scheduling integration); there is no automatic lookback input. Set maxItems high enough for the expected new disclosures, and inspect truncatedAt for unfinished crawls.

Seen state is stored in the named Apify key-value store tdnet-state, under stateKey, rather than the run's default store. The same account/store and key survive separate runs and tasks. Locally it is under storage/key_value_stores/tdnet-state; keep that storage directory between runs. Use separate keys for distinct company, securities-code or category filters, or independent streams. stateKey defaults to default and accepts 1–256 letters, digits, underscores or hyphens. incremental defaults to false; ordinary runs neither read nor write seen state.

Keys hash the disclosure's date, time, company code and title. Only successfully pushed rows are remembered; seen rows are never pushed or charged through pushData. State retains the current JST day and preceding 34 days, pruned when loaded and persisted after successful page pushes or newest-first completion. Completed newest-first runs also persist a contiguousCompleted watermark (oldest completed date, newest date, and last disclosure key), even when nothing new is emitted. Successful page pushes clear that watermark until the requested range finishes, so capped or failed runs continue through seen pages on retry. Legacy state without a watermark also scans through seen pages. The watermark is deliberately conservative: it advances only after the full requested range completes. Old disclosures outside that window can be emitted again. Changing a title creates a new key; changing only a PDF URL does not. Fully-seen-page stopping assumes the site's newest-first pages remain ordered: a late addition on an older page may be missed. Run with oldestFirst to scan all pages when that matters.

Avoid overlapping runs using the same key: state updates are not transactional across concurrent runs. A successful dataset push followed by a failed state write can produce duplicates on retry. Keep/export each run's dataset separately; persistent seen state is not a combined dataset or a guarantee of complete daily coverage.

Output

Dataset records contain securitiesCode4, disclosureId, category, sourcePageUrl, date, time, companyCode, companyName, title, absolute pdfUrl, nullable absolute xbrlUrl, exchange, updateHistory, and UTC ISO scrapedAt. Dates are the JST date encoded in the list URL. Empty update history is an empty string; exchange can contain multiple Japanese market abbreviations, for example 東福. The dataset schema's fields is a JSON Schema object with properties, required fields and numeric-compatible unions. Original codes remain unchanged. securitiesCode4 is the first four characters of the five-character TDnet code, including letters and regardless of the fifth character. disclosureId is the PDF filename without .pdf (an 18-digit string); its embedded date need not equal the disclosure date. sourcePageUrl is the absolute list-page URL where the row appeared. companyName and title preserve full-width spaces (U+3000), repeated internal whitespace and original text; only ASCII whitespace at their ends is trimmed. Other text fields retain their existing whitespace handling. The schema accepts updateHistory as string, number or null; this parser emits strings, including an empty string for a blank history cell.

The default key-value store contains OUTPUT with the scrape summary (items, truncatedAt, datesCovered, datesNotReached, skippedDates, and other counters), plus RUN_SUMMARY with status, date range, count (items and newItems are successfully pushed rows), skippedSeen (matching seen rows encountered), pages, skipped dates, request attempts, robots restriction, and timestamps. datesCovered lists dates whose first page was attempted, including empty/missing days and a partially crawled cutoff day. skippedDates includes missing/empty days and future dates. When the cap is reached, truncatedAt records the last emitted disclosure (date, time, companyCode, title), logs the cutoff, and datesNotReached lists subsequent unvisited dates in crawl order. Without a cutoff, truncatedAt is null and datesNotReached is empty. Reaching the cap stops immediately, so a cutoff does not prove additional matching disclosures exist. Failed runs write a failure summary and fail the Actor; their dataset may already contain earlier pages. Logs report memory in MiB at start and finish.

Price

What 1,000 disclosures cost you: $25.00 in total. Disclosures are charged at $0.025 per saved record ($25 per 1,000) plus a $0.0025 run-start fee per run, so 1,000 records collected in one run cost $25.0025. For comparison, the main TDnet scraper on Apify (nexgendata) charges $0.05 per record plus a $0.005 run-start fee — $50.005 for the same 1,000 records, about twice as much.

Truncation and completeness

When maxItems stops the crawl, the last pushed disclosure has truncatedAfter: true and a truncation object containing reason: "maxItems", maxItems, truncatedAt, datesCovered, datesNotReached, and skippedDates. Other disclosures omit these optional fields. Dataset exports therefore carry the cutoff marker without an extra marker row. OUTPUT and RUN_SUMMARY retain the run summary.

With the default newestFirst order, a capped run can include only part of a cutoff day and omit earlier requested days entirely. truncatedAt identifies the last emitted disclosure; datesNotReached names unvisited dates. datesCovered indicates attempted dates, not complete days. A non-null cutoff means the cap was reached, not proof that more matching rows exist. Raise maxItems or continue from the cutoff date (using incremental state to skip already emitted rows) when completeness is needed. With oldestFirst, later dates can be omitted instead.

Title categories

Classification is a deterministic title heuristic, not an official TDnet classification. The first matching rule below wins; unmatched titles become other. English patterns are case-insensitive. Titles are preserved unchanged. Mixed subjects receive one category according to this precedence:

Priority	Category	Title patterns
1	tenderOffer	公開買付, 公開買い付け, “tender offer”, standalone TOB
2	mergerAcquisition	合併, 買収, 会社分割, 株式交換, 株式移転, 株式交付, 子会社化, 事業譲受, 事業譲渡, 資本業務提携, “merger”, “acquisition”
3	buyback	自己株式 followed by 取得/買付/消却/処分; 自社株買い; “buyback”; “share repurchase”
4	forecastRevision	業績/通期/四半期/利益/売上 followed by 予想 then 修正; 予想 followed by 上方修正/下方修正; “forecast” and “revision” in either order
5	dividend	配当, 剰余金の分配, “dividend”
6	earnings	決算短信, 決算説明, 決算補足, 決算概要, 決算発表, 決算報告, “earnings”, “financial results”
7	governance	コーポレート plus optional one character then ガバナンス; 定款, 役員, 取締役, 監査役, 株主総会, 人事異動; 代表者 followed by 異動; “governance”
8	other	No preceding rule matches

For example, 業績予想及び配当予想の修正 is forecastRevision, while 配当予想の修正 is dividend. Heuristics can misclassify ambiguous titles; inspect the original title for downstream decisions.

Requests and pagination

Requests are sequential with no concurrency, start at least one second apart, and use the larger of that minimum and robots.txt Crawl-delay for the Actor's user agent. got's automatic retry and redirect handling are disabled. 429 and 5xx responses get at most five attempts with exponential backoff (1, 2, 4, 8 seconds), honoring longer Retry-After seconds or HTTP dates. Transport errors and other HTTP errors fail visibly. Robots 404/410 means no available directives; other robots errors fail the run.

List URLs are https://www.release.tdnet.info/inbs/I_list_001_YYYYMMDD.html, then 002 and so on (normally 100 disclosures per page). A 404/410 or recognized empty list ends that date; an unexpected HTML page fails rather than silently counting as empty. Filtering alone never ends pagination prematurely. In incremental newestFirst mode, a fully seen page ends that day only below the persisted completed watermark date and when the requested newest date includes its completed region; capped or failed runs without a completed watermark scan through seen pages; subsequent dates are still visited. oldestFirst continues through seen pages. The item cap ends the run immediately. No scripts, images, PDFs or ZIPs are fetched.

Local use and verification

npm ci --cache /tmp/tdnet-npm-cache
npm test
npm run validate
mkdir -p storage/key_value_stores/default
# Put your input JSON into storage/key_value_stores/default/INPUT.json.
npm start

npm run validate validates all 100 records parsed from the live-captured fixture against dataset_schema.json.fields using local Ajv. npm run validate -- docs/live-run-output.json validates the fresh live check output. Tests use node --test and stored live HTML plus mocked HTTP responses and clocks; tests do not access TDnet. scripts/live-check.js fetches fresh robots and list responses with a hard five-request-attempt budget, writes up to five records through the Apify SDK, validates them with Ajv, and records memory/run artifacts. It also inspects up to three additional list pages for nonempty update history within that budget. It is a manual check, not part of the test suite.

The build captured four responses and made one additional list request for validation (five TDnet requests total). See test/fixtures/README.md, docs/tests.log, docs/validation.log, docs/memory-before.log, docs/memory-after.log, docs/phase.log and docs/run-summary.json. The supplied Dockerfile uses apify/actor-node:20; metadata requests 128 MiB. Measured local RSS is documented in the memory artifacts, rather than a claim that every hosted run fits within that setting. The container itself has not been built in this workspace.

Iteration 4 used five live requests and Ajv-validated five newly emitted records; all four October 13 list pages had blank update history. The nonempty-history fixture uses an actual captured disclosure row with a synthetic history link/value, explicitly documented in test/fixtures/README.md. Evidence: docs/iteration4-live-check.json, docs/iteration4-live.log, docs/iteration4-tests.log, and docs/iteration4-validation.log. Existing incremental keys from earlier versions may emit whitespace-affected titles once again because the corrected original title changes their hash.