Douban Reviews Scraper — Book & Music Short Reviews

Scrape Douban short reviews (短评) for books and albums — star rating, full review text, reviewer name and profile URL, publication date, "有用" (helpful) vote count and a direct permalink to each review.

No login, no cookies, no browser. One plain HTTP request returns 20 reviews.

What you get

One row per review:

Field	Example
subjectName	小王子
subjectKind / subjectId	book / 1084336
reviewId	10287387
rating	5
reviewBody	十几岁的时候渴慕着小王子，一天之间可以看四十四次日落…
authorName / authorUrl	眠去 / https://www.douban.com/people/rebekah/
datePublished	2007-02-08
helpfulVotes	9282
reviewUrl	https://book.douban.com/comment/10287387/

Input

• Subjects to scrape — one per line. Any of these forms works:
  • book/1084336 or music/1401180
  • 1084336 (a bare id is treated as a book)
  • https://book.douban.com/subject/1084336/
• Max reviews / max pages per subject — hard caps so a subject with 200,000 reviews cannot run away with your bill.
• Delay between page requests — raise it if you start seeing 403s.

Review order

Douban's "sort by time" option returns HTTP 403 to any request that is not carrying a logged-in cookie. Verified request by request:

?sort=new_score -> 200, 38,779 bytes, 20 reviews
?sort=time      -> 403, 11,749 bytes, 0 reviews
(no sort param) -> 200, 38,779 bytes, 20 reviews   <- identical to sort=new_score

This Actor therefore uses Douban's own default order (most-useful first) and does not send a sort parameter at all. There is no input field for it, because offering a switch that always fails would be worse than not offering one.

Cost

Priced per delivered review (review-scraped event). Rows are pushed to the dataset before the charge is issued, so a failed run is never billed for data it did not deliver.

How it is built

• src/sites/douban.js — everything site-specific: URL shape, target resolution, and two independent parsers (a cheerio DOM walk over li.comment-item[data-cid], and a plain-regex path that ignores the DOM entirely).
• src/engine.js — paging, retries, de-duplication, per-page delivery. Site-agnostic.
• test/local_run.js — runs both parsers over the same live page and diffs them field by field. If the two paths ever disagree on author/date/rating/body, the test shouts before you ship.

Run it locally with no Apify account:

$node test/local_run.js book/1084336

Notes

• Douban publishes no aggregate star average on the short-review listing, so rating is per review only.
• Short reviews have no individual headline; reviewBody is the whole text.
• Dates are normalised with an explicit format matcher, never new Date(string) — that constructor parses locale-style strings at local midnight and shifts the day when converted to UTC.