Baidu Tieba Public Threads Scraper

Search public Baidu Tieba forums or scrape known public thread URLs. Version 2 discovers forum listings sequentially for stable pagination, then enriches selected threads concurrently and writes the dataset atomically only after successful completion.

Input modes

• query or forum: one forum name.
• queries or forums: multiple forum names, selected round-robin so one forum cannot consume the entire global limit.
• threadUrls or startUrls: one or more public https://tieba.baidu.com/p/<id> URLs.
• Search and direct-thread inputs can be combined.

Useful controls include maxItems, maxItemsPerQuery, maxPages, maxConcurrency, maxPostsPerThread, minReplies, includeSticky, featuredOnly, and post/media inclusion switches. Optional values that Tieba does not publish are omitted rather than emitted as null, empty strings, or fabricated zeroes.

The default fast mode uses Tieba's public mobile forum listing and does not open every thread. Set includeThreadDetails to true to request opening-post and loaded-reply enrichment; individual thread pages are more frequently restricted by Tieba than public forum listings.

Output

Every record has a stable baidu-tieba:<threadId> identity, a canonical tracking-free thread URL, title, source metadata, and scrape timestamp. When publicly available, records also contain forum and author identities, listing excerpt, real reply count, page count, opening content, loaded post objects, media URLs, sticky/featured status, raw time labels, normalized ISO timestamps, and search provenance.

replyCount is Tieba's published total. loadedPostCount is the number of post objects present on the rendered page; the Actor does not misrepresent the latter as the former.

Proxy and access behavior

Tieba may return HTTP 403 or an interactive verification page to some networks. The Actor detects these responses and fails closed; it never stores a block page and does not solve or bypass CAPTCHAs. Authorized Apify Residential or custom HTTP proxies can be supplied through proxyConfiguration. GOOGLE_SERP is rejected because it is restricted to ordinary Google Search traffic and is not compatible with Tieba.

Local use of Apify Proxy additionally requires the account's Proxy external access feature. If that feature is unavailable, validate live target access in Apify Cloud.

Development

npm ci
npm test
apify validate-schema .actor/input_schema.json
apify run --purge --input-file test/inputs/direct.json

The crawler uses bounded retries and timeouts, stable browser fingerprints per session, cookie persistence, media/font request reduction, strict status/content-type/challenge checks, stable-ID deduplication, and atomic final writes.