Baidu Baike Scraper

Baidu Baike Scraper

Search Baidu Baike and extract rich, normalized records from public encyclopedia entries. The Actor uses a rendered Playwright DOM because Baike pages depend on browser execution and may show a security-verification page to ordinary HTTP clients.

What it extracts

Each dataset item represents one deduplicated public entry and can include stable entry identity, title metrics, subtitle, summary metrics, description, rendered article text and token/paragraph counts, breadcrumb taxonomy, section headings with levels/anchors, category/keyword metadata, structured infobox values, image URLs with captions and dimensions, deduplicated references, related entry links, related-search suggestions, search-result title/snippet/position, query and search URL, canonical entry URLs, language, OpenGraph metadata, inclusion flags, source/search context, and a UTC timestamp. Optional article text, images, and references can be disabled; the corresponding counts and flags make that choice explicit. Empty values are omitted rather than written as null placeholders.

Input

Use query for one search, queries for a batch, entryUrls for direct public https://baike.baidu.com/item/... pages, or combine search and direct-entry modes. URLs are canonicalized and deduplicated globally.

maxItems is the global output limit, maxItemsPerQuery limits discovery for each query, and maxPages controls search pagination. sortBy applies to the final normalized dataset: relevance preserves Baidu's discovery order, while the other values sort by title or rendered article size. titleContains, categoryContains, and the word-count bounds are post-extraction filters; word-count filters require includeContent: true.

Minimal search input

{
  "query": "人工智能",
  "maxItems": 3,
  "proxyConfiguration": { "useApifyProxy": false }
}

Batch, pagination, filters, and developer controls

{
  "queries": ["人工智能", "机器学习"],
  "maxItems": 10,
  "maxItemsPerQuery": 5,
  "maxPages": 2,
  "titleContains": "技术",
  "categoryContains": "计算机",
  "minWordCount": 100,
  "sortBy": "wordCountDesc",
  "requestDelayMs": 500,
  "maxConcurrency": 2,
  "maxRequestRetries": 2,
  "navigationTimeoutSecs": 45,
  "requestHandlerTimeoutSecs": 90,
  "waitAfterLoadMillis": 1500,
  "includeContent": true,
  "includeImages": true,
  "includeReferences": true,
  "proxyConfiguration": { "useApifyProxy": false }
}

Direct entry input

{
  "entryUrls": ["https://baike.baidu.com/item/artificial-intelligence/9180"],
  "includeContent": false,
  "includeImages": false,
  "includeReferences": false,
  "maxConcurrency": 1,
  "proxyConfiguration": { "useApifyProxy": false }
}

Dataset output

The Actor writes one JSON dataset item per entry. The dataset JSON files are available in the run's default dataset storage and through the Apify dataset API/Console links after the run completes. A run-level OUTPUT key-value record reports the final status, stored-record count, submitted queries, selected options, and completion time.

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel.

Reliability and access boundaries

The Actor validates HTTP status and content type before DOM extraction, uses bounded retries/timeouts, persistent sessions, a consistent Chrome fingerprint, optional authorized Apify/custom proxy routing, and a DOM fallback for changed result-link tracking parameters. Media and fonts are blocked only on entry navigations so article HTML and structured data remain available. Search pagination follows public next-page links and falls back to Baike's observed pn offset parameter when the live page omits a visible next link, until maxPages or the configured limits are reached.

Baidu security-verification and CAPTCHA pages are detected and fail closed. The Actor does not solve challenges, bypass authentication, defeat access controls, or log cookies, authorization headers, proxy credentials, tokens, or raw transport payloads.

Validation

From this Actor directory:

npm test
npm run check
npm run validate:readme
npm run validate
apify validate-schema
apify run --purge --input-file test/inputs/batch.json

The repository's dataset validator checks required fields and types, safe Baike URLs, unique IDs, sequential positions, optional-field cleanliness, and the OUTPUT run summary.