Hupu Scraper

Scrapes 虎扑社区 (Hupu, China's largest sports forum) — without login — from the SSR forum pages.

Data source

Threads and comments come from Hupu's public web pages/APIs (bbs.hupu.com) — no login, no API key. See the Notes section for board-slug and comment-pagination caveats.

What it does

Mode	URL pattern	Output
byBoard	bbs.hupu.com/{board} + {board}-{page}	Board thread list (title, replies/views, author, time)
hot	bbs.hupu.com/hot	Hot threads (title, lights, replies, board)
byThreadUrl	bbs.hupu.com/{tid}.html	Full thread: content snippet, stats, author, board, first 20 comments

All pages verified live (HTTP 200 with real 虎扑 content) from a datacenter network. Thread pages embed the full payload in a __NEXT_DATA__ JSON script tag — no JS execution needed.

Output fields

thread records (recordType: "thread") — threadId, title, contentSnippet, contentLength, lightCount, replyCount, viewCount, recommendCount, createdAtMs, createdAtText, repliedAtMs, author, authorId, authorUrl, authorLevel, authorAvatarUrl, board, boardUrl, boardThreadCount, topicId, client, location, contentType, hasVideo, videoCover, timeText (board listings only), comments[] (up to 20: commentId, content, likeCount, replyNum, createdAtMs, createdAtText, location, client, author, authorId, authorUrl, authorAvatarUrl, isStarter), commentPageCount, sourceUrl, scrapedAt.

error records (recordType: "error") — invalid URLs, missing/blocked threads, unknown board slugs (never silently dropped).

Every data record carries sourceUrl, scrapedAt and recordType; error records carry inputValue in place of sourceUrl (the offending input, not a page URL). Null/empty fields are stripped before push.

Filters

• minReplyCount, minViewCount, minLightCount
• containsKeyword (title substring)
• maxBoardPages (board pagination depth, 40 threads per page)

Notes

• Comment pagination beyond page 1 is not served server-side (page 2+ returns empty lists) — the first 20 comments are included; deeper pagination needs login/JS and is documented as out of scope.
• Search (bbs.hupu.com/search) renders client-side only; the dedicated search API host does not resolve from datacenter networks — search is documented as out of scope.
• Board slug refreshes: Hupu changed its board slug space — csl, all-gambia, games, military, lit, film, rap now soft-404 (the site silently serves the homepage). The board enum now contains 28 slugs verified to return real thread rows; unknown slugs are detected via the homepage title and reported as a typed NOT_FOUND error record instead of a silent empty run.
• Thread videos (v.hoopchina.com.cn/*.mp4) carry expiring signed auth_key URLs — the static videoCover thumbnail is emitted instead (verified HTTP 200 image/jpeg from a clean shell); the signed mp4 is documented as out of scope.
• Author profile URLs (my.hupu.com/{uid}) redirect to passport.hupu.com login when not authenticated — the URL is correct but profile pages are login-walled; record content (threads/comments) is fully public without login.
• Auto-escalation: on 403/429 the actor lazily engages the Apify AUTO (datacenter) proxy with exponential backoff.
• Memory: 1024 MB (Tier 1, plain HTTP).