# Douban Scraper (`automation-lab/douban-scraper`) Actor Scrape public Douban movie, book, music, search, top-list, review, and comment data for China media intelligence workflows. - **URL**: https://apify.com/automation-lab/douban-scraper.md - **Developed by:** [Stas Persiianenko](https://apify.com/automation-lab) (community) - **Categories:** Social media - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing Pay per event This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Douban Scraper Scrape public Douban movie, book, music, search, top-list, review, comment, and group-topic pages for China media intelligence workflows. This Apify Actor focuses on publicly accessible Douban data and does not require login, cookies, or private accounts. ### What does Douban Scraper do? Douban Scraper extracts structured records from public Douban pages. It can collect Movie Top 250 entries, hot movie JSON records, Douban search results, and metadata from public start URLs. Use it to turn Douban pages into clean JSON, CSV, Excel, or API-ready datasets. ### Who is it for? - ๐Ÿงญ China market researchers tracking audience sentiment and cultural trends. - ๐ŸŽฌ Film, TV, music, and publishing analysts comparing ratings and rankings. - ๐Ÿค– LLM and NLP teams building Chinese-language media corpora from public pages. - ๐Ÿ“Š Social listening teams monitoring public reviews, comments, and group topics. - ๐Ÿงช Data journalists and academics studying public Douban lists and search results. ### Why use this actor? Douban is a high-signal source for Chinese media, entertainment, books, music, and community discussion. Manual copy-paste is slow, inconsistent, and hard to repeat. This actor gives you repeatable public extraction with normalized fields and Apify platform integrations. ### Public data only The actor is scoped to public pages. It does not log in, bypass paywalls, use cookies, or attempt to defeat captcha/security flows. If Douban returns a security verification page for a specific detail URL, the actor skips it and continues with other public sources. ### Supported sources - `https://www.douban.com/search` public search pages. - `https://movie.douban.com/top250` Movie Top 250 pages. - `https://movie.douban.com/j/search_subjects` public hot movie JSON endpoint. - Public Douban movie, book, music, review, and group-topic URLs supplied as start URLs. ### Input modes 1. Add Douban URLs in `startUrls`. 2. Enter a `searchQuery` and choose `section`. 3. Select `topLists` such as `movie-top250` or `movie-hot`. 4. Set `maxItems` to control dataset size and cost. ### Example input ```json { "topLists": ["movie-top250"], "maxItems": 25, "proxyConfiguration": { "useApifyProxy": false } } ```` ### Search example input ```json { "searchQuery": "็ง‘ๅนป", "section": "movie", "topLists": [], "maxItems": 20 } ``` ### URL example input ```json { "startUrls": [ { "url": "https://movie.douban.com/top250" } ], "maxItems": 50 } ``` ### Output data Each dataset row is a normalized Douban record. Fields are populated when Douban exposes the value publicly. | Field | Description | | --- | --- | | `url` | Source record URL | | `canonicalUrl` | Canonical URL when available | | `id` | Douban numeric identifier when detected | | `type` | Subject, review, topic, profile, or page | | `section` | Movie, book, music, group, or all | | `source` | Input source such as `movie-top250`, `movie-hot`, `search`, or `startUrl` | | `query` | Search query for search records | | `title` | Public title | | `originalTitle` | Original title when visible | | `description` | Public description, quote, intro, or snippet | | `rating` | Numeric Douban rating when visible | | `ratingCount` | Public rating count when visible | | `rank` | List rank, such as Movie Top 250 rank | | `year` | Release/publication year when parsed | | `genres` | Public genres/tags | | `directors` | Directors or creators when visible | | `authors` | Authors when visible | | `cast` | Cast when visible | | `region` | Region/country when parsed | | `language` | Language when visible | | `duration` | Runtime when visible | | `pages` | Book page count when visible | | `imageUrl` | Main image/poster URL | | `mediaUrls` | List of media/image URLs | | `authorName` | Review/comment/topic author when visible | | `date` | Public date when visible | | `upvoteCount` | Upvotes/helpful count when visible | | `commentCount` | Comment count when visible | | `scrapedAt` | ISO timestamp of extraction | ### Example output ```json { "url": "https://movie.douban.com/subject/1292052/", "id": "1292052", "type": "subject", "section": "movie", "source": "movie-top250", "title": "่‚–็”ณๅ…‹็š„ๆ•‘่ตŽ", "originalTitle": "The Shawshank Redemption", "rating": 9.7, "rank": 1, "year": "1994", "region": "็พŽๅ›ฝ", "scrapedAt": "2026-06-20T00:00:00.000Z" } ``` ### How much does it cost to scrape Douban? The actor uses pay-per-event pricing: a small start fee plus a per-record event for each saved Douban record. Keep `maxItems` low for trial runs, then scale once the output matches your workflow. ### Tips for reliable Douban scraping - Start with Movie Top 250 or search pages because they are publicly rendered. - Keep `maxItems` small while testing your workflow. - Use datacenter proxy first if you need proxying. - Use residential proxy only when your workload is blocked and the economics still make sense. - Avoid repeatedly requesting login-only or security-check pages. ### Reviews and comments Douban may expose some review, comment, and group-topic content publicly. The actor treats these as best-effort start URL records in v0.1. It does not log in or expand private content. Future versions can add deeper public review/comment pagination if it remains commercially reliable. ### Integrations Use Douban Scraper with: - Google Sheets exports for analyst review. - Apify datasets API for data pipelines. - Webhooks to notify when a scheduled monitor finishes. - LLM enrichment actors for translation, classification, sentiment, or entity extraction. - BI tools that consume CSV, Excel, JSON, or NDJSON. ### API usage with Node.js ```js import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('automation-lab/douban-scraper').call({ topLists: ['movie-top250'], maxItems: 25, }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); console.log(items.slice(0, 3)); ``` ### API usage with Python ```python from apify_client import ApifyClient import os client = ApifyClient(os.environ['APIFY_TOKEN']) run = client.actor('automation-lab/douban-scraper').call(run_input={ 'searchQuery': '็ง‘ๅนป', 'section': 'movie', 'topLists': [], 'maxItems': 20, }) items = client.dataset(run['defaultDatasetId']).list_items().items print(items[:3]) ``` ### API usage with cURL ```bash curl -X POST "https://api.apify.com/v2/acts/automation-lab~douban-scraper/runs?token=$APIFY_TOKEN" \ -H 'Content-Type: application/json' \ -d '{"topLists":["movie-top250"],"maxItems":25}' ``` ### MCP usage Connect through Apify MCP using: `https://mcp.apify.com/?tools=automation-lab/douban-scraper` Add it to Claude Code: ```bash claude mcp add apify-douban "https://mcp.apify.com/?tools=automation-lab/douban-scraper" ``` Desktop MCP JSON configuration: ```json { "mcpServers": { "apify-douban": { "url": "https://mcp.apify.com/?tools=automation-lab/douban-scraper" } } } ``` Example prompts: - "Run Douban Scraper for Movie Top 250 and summarize the highest-rated films." - "Search Douban for ็ง‘ๅนป movies and group the results by rating." - "Extract public Douban records and prepare a sentiment-analysis input table." ### Scheduling Schedule the actor daily, weekly, or monthly to monitor public Douban list or search changes. Use Apify webhooks to send the dataset to your warehouse or notification system. ### Proxy settings Proxy is optional. Start without proxy or with Apify datacenter proxy. Residential proxy can improve access for some workloads but costs more, so test with a small `maxItems` first. ### Data quality notes Douban page layouts vary by section and anti-bot state. The actor normalizes visible values and leaves unavailable fields empty. Public list/search extraction is more reliable than individual subject pages that trigger security checks. ### FAQ #### Can I scrape private Douban pages? No. The actor is designed for public data only and skips login-only or security-check pages. #### Does it support reviews and comments? It supports public review, comment, and topic URLs on a best-effort basis. Deep login-only review expansion is intentionally excluded from v0.1. ### Troubleshooting: why did I get fewer results? You may have hit `maxItems`, supplied a URL that requires security verification, or selected a search query with few public results. Try Movie Top 250 or a broader query to validate your setup. ### Troubleshooting: why are some fields empty? Douban does not expose every field on every page type. For example, Top 250 cards expose ratings and ranks but not full star distributions. Empty optional fields mean the value was not visible publicly on that page. ### Legality This actor extracts publicly available information. You are responsible for using the data lawfully, respecting Douban terms, privacy rights, copyright rules, and applicable laws in your jurisdiction. ### Related scrapers - https://apify.com/automation-lab/bilibili-scraper - https://apify.com/automation-lab/rednote-xiaohongshu-scraper - https://apify.com/automation-lab/weibo-scraper - https://apify.com/automation-lab/website-content-crawler ### Changelog #### 0.1 Initial public-data Douban scraper with Top 250, hot movies, search, and start URL extraction. ### Development notes This actor is built with HTTP requests and Cheerio. It is intentionally lightweight and configured for 256 MB memory by default. ### Field coverage roadmap - Add deeper public review pagination if target behavior remains stable. - Add public book and music list shortcuts. - Add section-specific parsers for book pages, music pages, and group topics. - Add optional translation/enrichment workflows through companion actors. ### Support If a public Douban page returns no data, include the run ID and the input URL when reporting the issue. That makes it possible to distinguish parser changes from Douban security responses. # Actor input Schema ## `startUrls` (type: `array`): Douban subject, search, Top 250, review, group topic, book, music, or movie URLs to process. ## `searchQuery` (type: `string`): Optional Douban search query, for example a movie title, author, keyword, or media topic. ## `section` (type: `string`): Limit Douban search to all results, movie, book, music, or group. ## `topLists` (type: `array`): Built-in public Douban lists to scrape. If no input is provided, Movie Top 250 is used by default. ## `maxItems` (type: `integer`): Maximum number of dataset records to save across all selected sources. ## `includeReviews` (type: `boolean`): Best-effort flag for public review pages supplied in start URLs. The actor never logs in or uses cookies. ## `includeComments` (type: `boolean`): Best-effort flag for public comment/topic pages supplied in start URLs. Login-only content is skipped. ## `maxReviewsPerItem` (type: `integer`): Reserved for public review/comment expansion; keep low for cheap runs. ## `proxyConfiguration` (type: `object`): Optional Apify Proxy settings. Start without proxy or with datacenter proxy; use residential only if Douban blocks your workload. ## Actor input object example ```json { "startUrls": [ { "url": "https://movie.douban.com/top250" } ], "searchQuery": "็ง‘ๅนป", "section": "all", "topLists": [ "movie-top250" ], "maxItems": 20, "includeReviews": false, "includeComments": false, "maxReviewsPerItem": 10, "proxyConfiguration": { "useApifyProxy": false } } ``` # Actor output Schema ## `overview` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ { "url": "https://movie.douban.com/top250" } ], "searchQuery": "็ง‘ๅนป", "proxyConfiguration": { "useApifyProxy": false } }; // Run the Actor and wait for it to finish const run = await client.actor("automation-lab/douban-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`๐Ÿ’พ Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // ๐Ÿ“š Want to learn more ๐Ÿ“–? Go to โ†’ https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": [{ "url": "https://movie.douban.com/top250" }], "searchQuery": "็ง‘ๅนป", "proxyConfiguration": { "useApifyProxy": False }, } # Run the Actor and wait for it to finish run = client.actor("automation-lab/douban-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("๐Ÿ’พ Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # ๐Ÿ“š Want to learn more ๐Ÿ“–? Go to โ†’ https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ { "url": "https://movie.douban.com/top250" } ], "searchQuery": "็ง‘ๅนป", "proxyConfiguration": { "useApifyProxy": false } }' | apify call automation-lab/douban-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=automation-lab/douban-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Douban Scraper", "description": "Scrape public Douban movie, book, music, search, top-list, review, and comment data for China media intelligence workflows.", "version": "0.1", "x-build-id": "NY7PloKqR8h1vUbFx" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/automation-lab~douban-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-automation-lab-douban-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/automation-lab~douban-scraper/runs": { "post": { "operationId": "runs-sync-automation-lab-douban-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/automation-lab~douban-scraper/run-sync": { "post": { "operationId": "run-sync-automation-lab-douban-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "startUrls": { "title": "๐Ÿ”— Douban start URLs", "type": "array", "description": "Douban subject, search, Top 250, review, group topic, book, music, or movie URLs to process.", "items": { "type": "object", "required": [ "url" ], "properties": { "url": { "type": "string", "title": "URL of a web page", "format": "uri" } } } }, "searchQuery": { "title": "๐Ÿ”Ž Search query", "type": "string", "description": "Optional Douban search query, for example a movie title, author, keyword, or media topic." }, "section": { "title": "Search section", "enum": [ "all", "movie", "book", "music", "group" ], "type": "string", "description": "Limit Douban search to all results, movie, book, music, or group.", "default": "all" }, "topLists": { "title": "๐Ÿ† Top lists", "type": "array", "description": "Built-in public Douban lists to scrape. If no input is provided, Movie Top 250 is used by default.", "items": { "type": "string", "enum": [ "movie-top250", "movie-hot" ], "enumTitles": [ "Movie Top 250", "Hot Movies" ] }, "default": [ "movie-top250" ] }, "maxItems": { "title": "Maximum records", "minimum": 1, "maximum": 10000, "type": "integer", "description": "Maximum number of dataset records to save across all selected sources.", "default": 20 }, "includeReviews": { "title": "๐Ÿ’ฌ Include reviews when public", "type": "boolean", "description": "Best-effort flag for public review pages supplied in start URLs. The actor never logs in or uses cookies.", "default": false }, "includeComments": { "title": "Include comments when public", "type": "boolean", "description": "Best-effort flag for public comment/topic pages supplied in start URLs. Login-only content is skipped.", "default": false }, "maxReviewsPerItem": { "title": "Max reviews per item", "minimum": 1, "maximum": 100, "type": "integer", "description": "Reserved for public review/comment expansion; keep low for cheap runs.", "default": 10 }, "proxyConfiguration": { "title": "๐ŸŒ Proxy configuration", "type": "object", "description": "Optional Apify Proxy settings. Start without proxy or with datacenter proxy; use residential only if Douban blocks your workload." } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```