2.1 (2026-10-16) – keywordResearch / sorted search pick from a deeper pool
Prices, events, input and output keys unchanged.
sortBy other than relevance on search / keywordResearch now ranks at least 27 candidates (3 search pages, fetched in one parallel batch) instead of 3 × maxArticles: keywordResearch maxArticles 3 + mostCommented on 新能源汽车 returned 11 comments instead of 4 (local, same hour), +2 requests.
keywordResearch adds a summary warning when fewer comments than maxArticles × maxCommentsPerArticle came back (with candidates count and what to change).
Capabilities manifest reports version 2.1.
2.0 (2026-10-15) – research workflow, trust fields, metrics, snapshots, insights
Prices and events unchanged; every v1.x input and the compact/lean default output keep working (new keys only added). Platform suite: see REPORT_v2.0.md.
operation: "keywordResearch": keyword → top maxArticles articles (search + filters + sortBy) with full text → up to maxCommentsPerArticle comments (+ repliesPerComment) → one free heuristic insights row per keyword. matchTerms[] keeps only comments containing a word (after fetch: lowers charges, not requests).
Trust fields on compact/full data rows: isPartial, unavailableFields[], collectionWarnings[]; summary adds tasksReport[] (requested vs returned, pages, stop reason), pagesFetched, nullRule.
Snapshot mode: trackMetrics: true + deltaKey → readsDelta, hotValueDelta, rankDelta, … vs the previous run (compact metrics in a named KV store).
Insights row (free, no LLM): top terms (CJK n-grams), lexicon sentiment labelled heuristic, engagement medians, top articles/comments/sources, commenter IP regions. include: ["insights"] on other operations.
Fixes (2.0.2): capabilities row key v2 is now declared in the dataset schema; the operation field text says to use exact names (Apify rejects aliases such as "research" with HTTP 400 before the run).
Billing event per row kind behind one constant map (src/v2.js BILLING_EVENTS), so a future price split is a config change.
1.4 (2026-10-13) – listing/docs
Capabilities manifest reports version 1.4 and the measured search depth (~150 per keyword); input schema explains all 24 channel ids and the maxTextChars range in text (MCP drops enum titles and min/max).
Docs (same scraping code as 1.3.8): README names every operation with the words agents use (hot list / hot search, keyword search, article details, Toutiao MCP tool) and has a copy-paste input per operation in "For AI agents". Store description/SEO fields updated to match. No input, output, price or event changes.
1.3 (2026-10-13)
Profile token links resolve:users accepts https://www.toutiao.com/c/user/token/<token>/ (mobile-UA redirect to the numeric id; unknown tokens → NOT_FOUND row). Was an INVALID_INPUT error.
followerCount on userProfile rows (live, from the author's newest article/post payload; the profile API encrypts its counters). +1–2 requests per profile; skipped when fields omits it.
include: ["video"]: playable MP4 stream URL (videoUrl, best rendition), videoUrlExpiresAt (~1 h), width/height/definition and videoStreams (all renditions) via the visitor play token (ByteDance VOD GetPlayInfo). Feed/related video cells use their embedded play info (no extra request).
Deeper search: the all tab is paged to its end after the news tab (v1.2 stopped after 2 low-yield pages): 83 → 99 rows (人工智能), 160 rows (新能源汽车); empty mid-list pages are retried once on a fresh session; 3 news-tab pages in parallel. dryRun search cap 90 → 150.
Faster: channel feed asks all three pools in the first round (feed 20 rows 8.3 → ~3–4 s on Apify); related pages fetched 4 at a time (50 rows 9.7 → ~6 s).
include: ["images"] alone now opens only rows without a working image.
Request hedging implemented (env TT_HEDGE_MS) but off by default: 0 wins in 13 hedges during the bench.
Hot Board: if www.toutiao.com has not answered after 1.5 s the same JSON is also requested from the mirror host api.toutiaoapi.com; first answer wins (single www answers took 4.7–5.7 s in 2 of 8 bench runs).
Default compact output unchanged except the new followerCount key on user rows; new article video keys only with include: ["video"] or outputMode: "full".
1.2 (2026-10-13) – token-efficient output, audited fields, parity features
Prices and events unchanged. Every v1.0/v1.1 input and the default (compact) output keep working; new behaviour is opt-in.
outputMode: "lean" (new): type + id + curated fields per row type, no null/empty/false keys, envelope (operation, input, queryUsed, fetchedAt) once in the summary (tasks), url only when not derivable (urlPatterns), compact UTC dates YYYY-MM-DDTHH:MMZ, text cut to maxTextChars (default 300, truncated: true). Runs with several inputs keep input per row.
leanFormat: "table" (new): also writes key-value record LEAN_TABLE (TSV, header once per row type), copied into the summary row when < 20 KB. Rows still pushed and billed the same.
maxTextChars (new, all modes; default only in lean).
Field audit fixes: 19-digit ids in related / v88 / micro-post payloads were rounded by JSON.parse (dead links in ~1/3 of related rows) → big-int-safe parsing; dead *.pstatp.com image URLs (PC feed, author lists) dropped instead of returned. New field videoPlayCount (details / fullText).
Competitor parity:profileTab (articles | posts = 微头条 micro-posts | all) for userArticles; textContains[] filter (before billing); onlyNew + memoryName monitoring (skip and do not bill rows earlier runs returned; named KV store in the caller's account, 30 days).
Efficiency: shipped as one esbuild bundle (dist/main.mjs, no npm install in the image; process start ~270 → ~70 ms locally); a third channel pool (web feed v88) – channel feeds return ~50 instead of ~37–43 unique rows, with read counts; end-of-run writes (dataset flush, charges, RUN_SUMMARY, optional LEAN_TABLE/memory) run in parallel. RUN_SUMMARY kept (default behaviour).
dryRun estimates use measured caps (search ~90 per keyword, feed ~55 per channel) and warn when maxItems exceeds them.
Input schema: 6 new fields (descriptions ≤ 500 chars, values explained in text); dataset schema: 10 new keys; output schema: LEAN_TABLE link; capabilities: outputModes, leanFields, urlPatterns, monitoring.
Same inputs, operations, output fields, events and prices as 1.0 (fully backward compatible). Measured on Apify, see REPORT_v1.1.md.
No Apify SDK:src/runtime.js (≈200 lines) does input, dataset pushes, pay-per-event charging, status messages, KV and Apify Proxy URLs over the public API. Process ready in ~0.6 s instead of ~1.7 s; peak memory ~22–30 MB instead of ~70 MB.
Batched writes: rows are pushed in batches (one request in flight carries everything queued) and charges are sent in batches; v1.0 made 2 sequential API calls per row. Hot Board 50 rows: 8.6 s → ~1.7 s.
HTTP: undici with keep-alive connection pools replaces got-scraping (+ header-generator); Chrome-like header sets for so.toutiao.com are built in code. Captcha retries rotate immediately (short pause) instead of exponential back-off; other back-offs capped at ~6 s; per-host request spacing (requestDelayMs default 300 → 100 ms, now per host).
Scope: search pages stop as soon as enough rows passed the filters (v1.0 fetched 3× the needed candidates); next search page prefetched only when another page is needed; feed fetches the PC and mobile pools in parallel after the first page and drops a pool after 2 dry pages; early stop on dateFrom for feeds; comments 50 per request (was 20); replies page size follows the cap; enrichment (include) is streamed in order with 8 parallel requests and stops as soon as maxItems/maxCostUsd is reached; rows dropped by a post-enrichment filter are replaced by the next candidates; fields asking for page-only fields (text, textLength, authorFollowerCount, authorVerification, contentHtml) open the article automatically; article HTML is not parsed when fields need no text/images.
Article text: single-pass HTML → text tokenizer replaces cheerio (identical output on 23 real articles, ~27× less CPU).
Run deadline: new requests stop 20 s before the run timeout, so the run ends with rows + summary instead of TIMED-OUT.
Summary row: new downloadedKb and timings (startupMs, firstRowMs, totalMs).
Image/memory:node:22-alpine, one dependency (undici), node started directly (no npm wrapper). Default run memory 512 → 256 MB; minimum 256 → 128 MB.
Store text (SEO): title "Toutiao News Scraper – 今日头条 Hot Board, Comments & Authors"; description leads with "Toutiao news (今日头条)… Chinese news Hot Board"; new SEO title/description; README sections Toutiao news search, Hot Board, Full article text, Toutiao comments and replies, Authors, Pricing, FAQ on ByteDance.
1.0.0 – 2026-10-12 (agent-native release; build 1.0.3, ID ek2F0KMEJZSJ59CW0)
English→Chinese query expansion with the shared AsiaData dictionary (src/data/en_zh.json, copy of work/actors/_shared/query_expansion/en_zh.json, 951 entries; +66 news/finance/politics entries added by this release). Rows show queryUsed.
Output contract v1: every row has type, operation, input, queryUsed, id, url, fetchedAt; new row types comment, reply, user, plan, capabilities, summary (always last, free); error rows now errorCode, message, fix (free). outputMode: "full" keeps v0 names mode, query, groupId, scrapedAt.
Pricing: new PPE event comment $0.0015 (comments + replies, also nested). result $0.004 and start $0.00005 unchanged. Added 2026-10-12 23:12 IST, effective immediately (no paying users → no 14-day notice).
Search rows: authorName falls back to the publisher name, coverImageUrl to the first image.
Explicit operations report inputs they don't use (INVALID_INPUT row) instead of dropping them; maxCostUsd stop adds a COST_LIMIT row.
Dataset schema: all row types declared; views Articles, Hot board, Full text, Comments, Authors, Errors, Summary. Output schema: results, articles, comments views + RUN_SUMMARY.
Removed: searchVideos (tested: cookie-free so.toutiao.com returns no Toutiao videos; video tab only lists off-site Bilibili links). Videos available via feed/userArticles/related + contentType: "video".
Backward compatible: mode, searchKeywords, articleUrls, feedChannels, maxItemsPerQuery, fetchDetails still accepted (verified on platform). Note: keywords as a plain string is rejected by the platform's schema validation (array required) before the run starts.
Every input optional (mode auto); misplaced inputs auto-corrected (site URL in keywords → fetched, plain word in urls → searched); explicit mode with missing input falls back to the given input.
Out-of-range numbers clamped instead of rejected (schema max/min removed; ranges in descriptions) – agents don't see schema min/max.
New type: "error" dataset rows with errorCode / errorMessage / suggestedFix (never billed); input, not-found and no-result problems now end SUCCEEDED with these rows; status message lists codes + first fix; RUN_SUMMARY has errorRows, notes.
Inputs are arrays with agent-friendly names:keywords[], urls[], channels[]. Old names (searchKeywords, articleUrls, feedChannels) still accepted; single or comma-separated strings are split; channel aliases (tech, finance, world…) mapped when not schema-validated.
mode is now optional, default auto: urls → articleDetails, keywords → search, channels → feed, nothing → hotBoard. Nothing is in required; no defaults on primary arrays (so auto is never fooled by an injected default).
Default maxItems 50 → 20 so default calls finish within the MCP 30 s wait (measured 7–13 s; ≈30 s with fetchDetails). maxItemsPerQuery has no default: the run budget is split evenly between keywords/channels.
fetchDetails runs 4 requests in parallel (throttle now reserves start slots): 20 items with full text 42 s → 31 s on platform.
Stable output: every item of a type has the same keys (null when unknown, [] for lists), consistent types (ids are strings, counts integers, flags booleans). New query field (keyword / channel / input URL) replaces channel, searchKeyword, inputUrl. Removed rarely filled fields: tags, sourceAvatarUrl, mediaId, isHot, labelIconUrl, videoId, videoPlayCount; hot topics no longer carry article-only keys.
Status messages: start, progress every 10 items, and a final message with item count, stop reason and failed inputs. Invalid input fails fast with an example (Invalid input: Input "keywords" is required for mode "search", e.g. …).
Schemas: input field descriptions rewritten (≤500 chars, examples, important channels first); dataset schema with title/description/example per field matching real output; new .actor/output_schema.json (results + RUN_SUMMARY), required for Store publishing.
Agent-first Store description (264 chars, leads with outputs), SEO title, README rewritten (what it returns, input/output JSON from real runs, pricing, measured speed, limits, MCP).
Unit tests 5 → 7 (output shape, input normalisation).
0.1.0 – 2026-10-12
First release (private).
Modes: Hot Board (头条热榜, 50 topics + optional pinned item), channel news feed (24 channels; desktop feed + mobile list combined and de-duplicated), keyword article search (so.toutiao.com News tab, continues in All tab), article details by URL (articles, videos, 微头条 micro-posts).
Article details: plain text (paragraphs kept), all image URLs, author name/id/profile/followers/verification, publish time, read/like/comment/share/favorite counts, optional raw HTML. fetchDetails adds these to feed/search items.
Search anti-captcha: cookie-less requests + generated Chrome header set per session; slide-captcha pages detected and retried with a new session/IP.
Retries with exponential backoff and session/proxy rotation on 403/429/5xx, empty or non-JSON responses.
Pay-per-event charging: actor-start (only if defined) + result per item; stops at the run's max charge.
