Weibo Scraper

Search Weibo (微博) posts by keyword, or pass post links directly, and optionally collect each post's comments and reply threads. Everything is read the way a visitor who is not signed in reads it: no account, no cookie of yours, nothing behind a login.

Every row says whether its collection was complete. When Weibo stops serving a visitor part of a list, the rows say so in partial and partialReason — an incomplete collection is never written as "no results".

Input

Field	Default	What it does
keywords	—	Words or phrases to search for, one search each. Chinese works best.
postUrls	—	Posts to collect directly: m.weibo.cn/detail/<id>, weibo.com/<user id>/<code>, or a bare post ID.
maxPostsPerKeyword	20	Posts to keep per keyword.
includeComments	false	Also collect each post's comments, one row per comment.
maxCommentsPerPost	50	Comment rows per post, replies included.
includeReplies	true	Follow reply threads. Each reply carries its parentCommentId.
maxUpstreamRequests	derived	Hard ceiling on requests to Weibo. Left empty, it is worked out from your other settings; a run is never unlimited.
requestDelayMs	700	Pause between two requests to Weibo.

Give at least one keyword or one post link. A post two keywords both return is written once.

{
    "keywords": ["咖啡"],
    "maxPostsPerKeyword": 20,
    "includeComments": true,
    "maxCommentsPerPost": 50
}

Output

One row per post (recordType: "post") and one per comment or reply (recordType: "comment"), plus a keyword_summary row per keyword and a post_not_collected row for any link that has no public post behind it.

A post row carries postId, mid, bid, url, the full text (the complete text is fetched when Weibo cut it off), publishedAt (the platform's publish time), publishedRegion, author (uid, name, verified, followersCount), stats (reposts, comments, likes), images, repostOfPostId, the keyword and rank it was found at, collectedAt (when this Actor read it) and, when comments were asked for, commentsCollected and commentsPartialReason.

A comment row carries commentId, postId, isReply, parentCommentId, rootCommentId, text, likes, replyCount, floor, publishedAt, collectedAt and author (uid, name, ipLocation). Only commenter fields Weibo shows publicly on the page are kept. A field Weibo did not give is null — never a filled-in guess.

Weibo's two comment lists describe a reply differently, and the output keeps that visible rather than flattening it. The hot list returns top-level comments only; the reply list returns the replies of one comment, so their parent is known. The numbered list returns both mixed together and never says what a reply answers — such a row is isReply: true with parentCommentId: null and is marked partial with that reason, so it is never mistaken for the start of a thread. A comment that turns up in two lists is written once, from the copy that knows its parent.

A post row is marked partial when its own text was cut off or when its comments came up short, so counting partial needs no second field; commentsPartialReason says which part fell short. A keyword_summary row is marked partial when the keyword did not finish, including when the run stopped at a ceiling part-way through it.

The run's own totals are in the key-value store under RUN_SUMMARY: rows by type, how many were partial, duplicates skipped, every request sent to Weibo and what it answered, the request ceiling and where it came from, and whether the run stopped early and why.

What a visitor can and cannot read

Measured on 2026-10-15, signed out:

• Keyword search pages — readable; checked to 10 pages, 98 distinct posts for one keyword. Weibo's search widens a query it finds little for rather than returning nothing, so a keyword almost always comes back with posts; read keyword and the post text to judge relevance.
• Post detail and full text — readable.
• Comments — Weibo serves a visitor the hot comment list and the first two numbered comment pages, roughly the first 35 to 40 comments of a post. Both lists are read and merged, because they return different comments. Past that, Weibo asks for a sign-in; this Actor stops there and marks the post's comments partial with the reason.
• Reply threads — readable for the comments collected above.

This is a limit of what Weibo shows the public, not an error, and it is reported as such in every row and in RUN_SUMMARY.

When something goes wrong

A refusal, a missing post and a ceiling are three different things, and the output keeps them apart. A post Weibo has no public page for becomes a post_not_collected row with the reason. A list Weibo stops serving a visitor part-way marks the rows partial with the reason. Reaching maxCommentsPerPost, maxPostsPerKeyword or maxUpstreamRequests is named as the limit it was, and whether maxUpstreamRequests came from the input or from this Actor's own default is in RUN_SUMMARY.upstreamCapSource. Nothing is written as "no results" because collection was cut short.

Local checks

npm install
npm test

No token and no network: the tests drive the HTTP client with their own fetch.