# Changelog of Toutiao News Scraper – 今日头条 China News, Hot Board & Comments (`deadc0dex/toutiao-scraper`) Actor

- **URL**: https://apify.com/deadc0dex/toutiao-scraper/changelog.md
- **Full Actor documentation**: https://apify.com/deadc0dex/toutiao-scraper.md

## Changelog

### 2.1 (2026-10-16) – keywordResearch / sorted search pick from a deeper pool

Prices, events, input and output keys unchanged.

- `sortBy` other than relevance on `search` / `keywordResearch` now ranks at least 27 candidates (3 search pages, fetched in one parallel batch) instead of 3 × maxArticles: `keywordResearch` maxArticles 3 + `mostCommented` on 新能源汽车 returned 11 comments instead of 4 (local, same hour), +2 requests.
- keywordResearch adds a summary warning when fewer comments than maxArticles × maxCommentsPerArticle came back (with candidates count and what to change).
- Capabilities manifest reports version 2.1.

### 2.0 (2026-10-15) – research workflow, trust fields, metrics, snapshots, insights

Prices and events unchanged; every v1.x input and the compact/lean default output keep working (new keys only added). Platform suite: see REPORT_v2.0.md.

- **`operation: "keywordResearch"`**: keyword → top `maxArticles` articles (search + filters + `sortBy`) with full text → up to `maxCommentsPerArticle` comments (+ `repliesPerComment`) → one free heuristic `insights` row per keyword. `matchTerms[]` keeps only comments containing a word (after fetch: lowers charges, not requests).
- **Trust fields** on compact/full data rows: `isPartial`, `unavailableFields[]`, `collectionWarnings[]`; summary adds `tasksReport[]` (requested vs returned, pages, stop reason), `pagesFetched`, `nullRule`.
- **Computed metrics** (documented formulas, null-safe): `engagementRate`, `likesPer1kReads`, `commentsPer1kReads`, `sharesPer1kReads`; comments `commentReplyRatio`.
- **Snapshot mode**: `trackMetrics: true` + `deltaKey` → `readsDelta`, `hotValueDelta`, `rankDelta`, … vs the previous run (compact metrics in a named KV store).
- **Insights row** (free, no LLM): top terms (CJK n-grams), lexicon sentiment labelled heuristic, engagement medians, top articles/comments/sources, commenter IP regions. `include: ["insights"]` on other operations.
- Comment rows (full mode / `fields`): `authorVerifiedReason`, `hasAuthorReplied` (2.0.2).
- Fixes (2.0.2): capabilities row key `v2` is now declared in the dataset schema; the operation field text says to use exact names (Apify rejects aliases such as "research" with HTTP 400 before the run).
- Billing event per row kind behind one constant map (src/v2.js `BILLING_EVENTS`), so a future price split is a config change.

### 1.4 (2026-10-13) – listing/docs

- Capabilities manifest reports version 1.4 and the measured search depth (~150 per keyword); input schema explains all 24 channel ids and the maxTextChars range in text (MCP drops enum titles and min/max).
- Docs (same scraping code as 1.3.8): README names every operation with the words agents use (hot list / hot search, keyword search, article details, Toutiao MCP tool) and has a copy-paste input per operation in "For AI agents". Store description/SEO fields updated to match. No input, output, price or event changes.

### 1.3 (2026-10-13)

- **Profile token links resolve:** `users` accepts `https://www.toutiao.com/c/user/token/<token>/` (mobile-UA redirect to the numeric id; unknown tokens → NOT_FOUND row). Was an INVALID_INPUT error.
- **followerCount on userProfile rows** (live, from the author's newest article/post payload; the profile API encrypts its counters). +1–2 requests per profile; skipped when `fields` omits it.
- **`include: ["video"]`:** playable MP4 stream URL (`videoUrl`, best rendition), `videoUrlExpiresAt` (~1 h), width/height/definition and `videoStreams` (all renditions) via the visitor play token (ByteDance VOD GetPlayInfo). Feed/related video cells use their embedded play info (no extra request).
- **Deeper search:** the all tab is paged to its end after the news tab (v1.2 stopped after 2 low-yield pages): 83 → 99 rows (人工智能), 160 rows (新能源汽车); empty mid-list pages are retried once on a fresh session; 3 news-tab pages in parallel. dryRun search cap 90 → 150.
- **Faster:** channel feed asks all three pools in the first round (feed 20 rows 8.3 → ~3–4 s on Apify); related pages fetched 4 at a time (50 rows 9.7 → ~6 s).
- `include: ["images"]` alone now opens only rows without a working image.
- Summary row: `latencyMs` {p50, p90, max, hedged, hedgeWins}, `profileTokensResolved`, `videoRowsWithStreams`.
- Request hedging implemented (env `TT_HEDGE_MS`) but off by default: 0 wins in 13 hedges during the bench.
- Hot Board: if www.toutiao.com has not answered after 1.5 s the same JSON is also requested from the mirror host api.toutiaoapi.com; first answer wins (single www answers took 4.7–5.7 s in 2 of 8 bench runs).
- Default compact output unchanged except the new `followerCount` key on user rows; new article video keys only with `include: ["video"]` or `outputMode: "full"`.

### 1.2 (2026-10-13) – token-efficient output, audited fields, parity features

Prices and events unchanged. Every v1.0/v1.1 input and the default (compact) output keep working; new behaviour is opt-in.

- **`outputMode: "lean"`** (new): type + id + curated fields per row type, no null/empty/false keys, envelope (operation, input, queryUsed, fetchedAt) once in the summary (`tasks`), url only when not derivable (`urlPatterns`), compact UTC dates `YYYY-MM-DDTHH:MMZ`, text cut to `maxTextChars` (default 300, `truncated: true`). Runs with several inputs keep `input` per row.
- **`leanFormat: "table"`** (new): also writes key-value record `LEAN_TABLE` (TSV, header once per row type), copied into the summary row when < 20 KB. Rows still pushed and billed the same.
- **`maxTextChars`** (new, all modes; default only in lean).
- **Field audit fixes:** 19-digit ids in related / v88 / micro-post payloads were rounded by JSON.parse (dead links in ~1/3 of related rows) → big-int-safe parsing; dead `*.pstatp.com` image URLs (PC feed, author lists) dropped instead of returned. New field `videoPlayCount` (details / fullText).
- **Competitor parity:** `profileTab` (articles | posts = 微头条 micro-posts | all) for userArticles; `textContains[]` filter (before billing); `onlyNew` + `memoryName` monitoring (skip and do not bill rows earlier runs returned; named KV store in the caller's account, 30 days).
- **Efficiency:** shipped as one esbuild bundle (`dist/main.mjs`, no npm install in the image; process start ~270 → ~70 ms locally); a third channel pool (web feed v88) – channel feeds return ~50 instead of ~37–43 unique rows, with read counts; end-of-run writes (dataset flush, charges, RUN_SUMMARY, optional LEAN_TABLE/memory) run in parallel. RUN_SUMMARY kept (default behaviour).
- **dryRun** estimates use measured caps (search ~90 per keyword, feed ~55 per channel) and warn when maxItems exceeds them.
- Input schema: 6 new fields (descriptions ≤ 500 chars, values explained in text); dataset schema: 10 new keys; output schema: LEAN_TABLE link; capabilities: outputModes, leanFields, urlPatterns, monitoring.

### 1.1.0 – 2026-10-13 (efficiency + discoverability release; build 1.1.x)

Same inputs, operations, output fields, events and prices as 1.0 (fully backward compatible). Measured on Apify, see REPORT_v1.1.md.

- **No Apify SDK:** `src/runtime.js` (≈200 lines) does input, dataset pushes, pay-per-event charging, status messages, KV and Apify Proxy URLs over the public API. Process ready in ~0.6 s instead of ~1.7 s; peak memory ~22–30 MB instead of ~70 MB.
- **Batched writes:** rows are pushed in batches (one request in flight carries everything queued) and charges are sent in batches; v1.0 made 2 sequential API calls per row. Hot Board 50 rows: 8.6 s → ~1.7 s.
- **HTTP:** undici with keep-alive connection pools replaces got-scraping (+ header-generator); Chrome-like header sets for so.toutiao.com are built in code. Captcha retries rotate immediately (short pause) instead of exponential back-off; other back-offs capped at ~6 s; per-host request spacing (`requestDelayMs` default 300 → 100 ms, now per host).
- **Scope:** search pages stop as soon as enough rows passed the filters (v1.0 fetched 3× the needed candidates); next search page prefetched only when another page is needed; feed fetches the PC and mobile pools in parallel after the first page and drops a pool after 2 dry pages; early stop on `dateFrom` for feeds; comments 50 per request (was 20); replies page size follows the cap; enrichment (`include`) is streamed in order with 8 parallel requests and stops as soon as `maxItems`/`maxCostUsd` is reached; rows dropped by a post-enrichment filter are replaced by the next candidates; `fields` asking for page-only fields (text, textLength, authorFollowerCount, authorVerification, contentHtml) open the article automatically; article HTML is not parsed when `fields` need no text/images.
- **Article text:** single-pass HTML → text tokenizer replaces cheerio (identical output on 23 real articles, ~27× less CPU).
- **Run deadline:** new requests stop 20 s before the run timeout, so the run ends with rows + summary instead of TIMED-OUT.
- **Summary row:** new `downloadedKb` and `timings` (`startupMs`, `firstRowMs`, `totalMs`).
- **Image/memory:** `node:22-alpine`, one dependency (undici), `node` started directly (no npm wrapper). Default run memory 512 → 256 MB; minimum 256 → 128 MB.
- **Store text (SEO):** title "Toutiao News Scraper – 今日头条 Hot Board, Comments & Authors"; description leads with "Toutiao news (今日头条)… Chinese news Hot Board"; new SEO title/description; README sections Toutiao news search, Hot Board, Full article text, Toutiao comments and replies, Authors, Pricing, FAQ on ByteDance.

### 1.0.0 – 2026-10-12 (agent-native release; build 1.0.3, ID ek2F0KMEJZSJ59CW0)

- **Operations** (`operation`, default `auto`): hotBoard, search, feed, articleDetails, **comments** (+ `repliesPerComment`), **replies**, **userProfile**, **userArticles**, **related**, **capabilities** (free manifest). In auto mode every filled input runs (urls/ids → articleDetails, users → userArticles, keywords → search, channels → feed; nothing → hotBoard).
- New inputs: `ids[]`, `users[]`; `include[]` (fullText, images, author, comments + `commentsPerItem`), `fields[]` projection, `outputMode` compact|full, filters `dateFrom`/`dateTo` (dates, ISO, unix, 24h/7d), `minReads`/`minLikes`/`minComments`, `sortBy`, `contentType`, `dedupe`, `maxItemsPerInput`, `maxCostUsd`, `dryRun`, `expandQuery`.
- English→Chinese query expansion with the shared AsiaData dictionary (`src/data/en_zh.json`, copy of `work/actors/_shared/query_expansion/en_zh.json`, 951 entries; +66 news/finance/politics entries added by this release). Rows show `queryUsed`.
- Output contract v1: every row has `type, operation, input, queryUsed, id, url, fetchedAt`; new row types `comment`, `reply`, `user`, `plan`, `capabilities`, `summary` (always last, free); error rows now `errorCode, message, fix` (free). `outputMode: "full"` keeps v0 names `mode, query, groupId, scrapedAt`.
- Pricing: new PPE event `comment` $0.0015 (comments + replies, also nested). `result` $0.004 and start $0.00005 unchanged. Added 2026-10-12 23:12 IST, effective immediately (no paying users → no 14-day notice).
- Search rows: `authorName` falls back to the publisher name, `coverImageUrl` to the first image.
- Explicit operations report inputs they don't use (INVALID_INPUT row) instead of dropping them; `maxCostUsd` stop adds a COST_LIMIT row.
- Dataset schema: all row types declared; views Articles, Hot board, Full text, Comments, Authors, Errors, Summary. Output schema: results, articles, comments views + RUN_SUMMARY.
- Removed: `searchVideos` (tested: cookie-free so.toutiao.com returns no Toutiao videos; video tab only lists off-site Bilibili links). Videos available via feed/userArticles/related + `contentType: "video"`.
- Backward compatible: `mode`, `searchKeywords`, `articleUrls`, `feedChannels`, `maxItemsPerQuery`, `fetchDetails` still accepted (verified on platform). Note: `keywords` as a plain string is rejected by the platform's schema validation (array required) before the run starts.

### 0.3.0 – 2026-10-12 (agent optimisation; build 0.3.2)

- Every input optional (`mode` auto); misplaced inputs auto-corrected (site URL in `keywords` → fetched, plain word in `urls` → searched); explicit mode with missing input falls back to the given input.
- Out-of-range numbers clamped instead of rejected (schema max/min removed; ranges in descriptions) – agents don't see schema min/max.
- New `type: "error"` dataset rows with `errorCode` / `errorMessage` / `suggestedFix` (never billed); input, not-found and no-result problems now end SUCCEEDED with these rows; status message lists codes + first fix; RUN_SUMMARY has `errorRows`, `notes`.
- Dataset schema: error fields, `type` enum incl. `error`, Errors view; output schema descriptions updated.
- Input field descriptions rewritten (enum values explained inline because MCP drops enumTitles; formats and examples); flagship-only prefill.
- Store description/SEO description rewritten (input → output first; correct prices); exampleRunInput set; default run memory 512 MB.
- README: "For AI agents" section (MCP tool name, task → input table, cost, runtime, error codes); fixed stale `affectionate_greengage` username in API/MCP examples.
- PPE event names and prices unchanged.

### 0.2.0 – 2026-10-12 (agent-readiness release, build 0.2.3)

- **Inputs are arrays with agent-friendly names:** `keywords[]`, `urls[]`, `channels[]`. Old names (`searchKeywords`, `articleUrls`, `feedChannels`) still accepted; single or comma-separated strings are split; channel aliases (`tech`, `finance`, `world`…) mapped when not schema-validated.
- **`mode` is now optional, default `auto`**: urls → articleDetails, keywords → search, channels → feed, nothing → hotBoard. Nothing is in `required`; no defaults on primary arrays (so `auto` is never fooled by an injected default).
- **Default `maxItems` 50 → 20** so default calls finish within the MCP 30 s wait (measured 7–13 s; ≈30 s with fetchDetails). `maxItemsPerQuery` has no default: the run budget is split evenly between keywords/channels.
- **fetchDetails runs 4 requests in parallel** (throttle now reserves start slots): 20 items with full text 42 s → 31 s on platform.
- **Stable output:** every item of a type has the same keys (null when unknown, `[]` for lists), consistent types (ids are strings, counts integers, flags booleans). New `query` field (keyword / channel / input URL) replaces `channel`, `searchKeyword`, `inputUrl`. Removed rarely filled fields: `tags`, `sourceAvatarUrl`, `mediaId`, `isHot`, `labelIconUrl`, `videoId`, `videoPlayCount`; hot topics no longer carry article-only keys.
- **Status messages:** start, progress every 10 items, and a final message with item count, stop reason and failed inputs. Invalid input fails fast with an example (`Invalid input: Input "keywords" is required for mode "search", e.g. …`).
- **Schemas:** input field descriptions rewritten (≤500 chars, examples, important channels first); dataset schema with title/description/example per field matching real output; new `.actor/output_schema.json` (results + RUN_SUMMARY), required for Store publishing.
- Agent-first Store description (264 chars, leads with outputs), SEO title, README rewritten (what it returns, input/output JSON from real runs, pricing, measured speed, limits, MCP).
- Unit tests 5 → 7 (output shape, input normalisation).

### 0.1.0 – 2026-10-12

- First release (private).
- Modes: Hot Board (头条热榜, 50 topics + optional pinned item), channel news feed (24 channels; desktop feed + mobile list combined and de-duplicated), keyword article search (so.toutiao.com News tab, continues in All tab), article details by URL (articles, videos, 微头条 micro-posts).
- Article details: plain text (paragraphs kept), all image URLs, author name/id/profile/followers/verification, publish time, read/like/comment/share/favorite counts, optional raw HTML. `fetchDetails` adds these to feed/search items.
- Search anti-captcha: cookie-less requests + generated Chrome header set per session; slide-captcha pages detected and retried with a new session/IP.
- Retries with exponential backoff and session/proxy rotation on 403/429/5xx, empty or non-JSON responses.
- Pay-per-event charging: `actor-start` (only if defined) + `result` per item; stops at the run's max charge.
- Dataset views: Articles, Hot board, Full text.