Baidu Tieba Public Threads Scraper

Search public Baidu Tieba forums or scrape known public thread URLs. Version 2.1 discovers forum listings with explicit pn pagination, then enriches selected threads concurrently with a persistent Playwright context and writes normalized dataset records. The OUTPUT key-value record gives machine-readable run status and counts.

Input modes

• query or forum: one forum name.
• queries or forums: multiple forum names, selected round-robin so one forum cannot consume the entire global limit.
• threadUrls or startUrls: one or more public https://tieba.baidu.com/p/<id> URLs.
• Search and direct-thread inputs can be combined.

Useful controls include maxItems, maxItemsPerQuery, maxPages, maxConcurrency, maxPostsPerThread, minReplies, includeSticky, featuredOnly, includeKeywords, excludeKeywords, sortBy, and post/media inclusion switches. requestDelayMs, maxRequestRetries, navigationTimeoutSecs, requestHandlerTimeoutSecs, waitAfterLoadMillis, and blockMedia are grouped as developer controls. Optional values that Tieba does not publish are omitted rather than emitted as null, empty strings, or fabricated zeroes.

The default fast mode uses Tieba's public mobile forum listing and does not open every thread. Set includeThreadDetails to true to request opening-post and loaded-reply enrichment; individual thread pages are more frequently restricted by Tieba than public forum listings.

Output

Every record has a stable baidu-tieba:<threadId> identity, a canonical tracking-free thread URL, title, source metadata, and scrape timestamp. When publicly available, records also contain forum and author identities, listing excerpt and result-count text, real reply count, page count, opening content, loaded post objects, media URLs, sticky/featured status, raw time labels, normalized ISO timestamps, and search provenance.

replyCount is Tieba's published total. loadedPostCount is the number of post objects present on the rendered page; the Actor does not misrepresent the latter as the former.

Proxy and access behavior

Tieba may return HTTP 403 or an interactive verification page to some networks. The Actor detects these responses and fails closed; it never stores a block page and does not solve or bypass CAPTCHAs. Authorized Apify Residential or custom HTTP proxies can be supplied through proxyConfiguration. GOOGLE_SERP is rejected because it is restricted to ordinary Google Search traffic and is not compatible with Tieba.

Local use of Apify Proxy additionally requires the account's Proxy external access feature. If that feature is unavailable, validate live target access in Apify Cloud. When local DNS cannot resolve a Baidu host, the Actor uses a bounded, status/content-type-validated public DNS-over-HTTPS lookup only to configure the browser host mapping.

Development

npm ci
npm test
apify validate-schema
apify run --purge --input-file test/inputs/direct.json

The Actor uses bounded retries and timeouts, one internally consistent persistent Chrome profile with shared cookies, media/font request reduction, strict status/content-type/challenge checks, stable-ID deduplication, explicit page offsets, and an OUTPUT key-value summary. A verification/403 response is reported as target_verification; challenge content is never written to the dataset.