Baidu Search Scraper

Playwright-based extraction of public Baidu web results with rich-result classification, stable IDs, real pagination, regional search context, fair query batches, and authorized proxy support.

Highlights

• Single query, deduplicated multiple queries, fair per-query quotas, and exact low-volume mode
• Constructed Baidu pn pagination rather than reliance on a drifting Next selector
• Organic, Baike, video, image, official, academic, finance, live, and other rich-card classification
• Source, snippet, highlighted terms, dates, thumbnails, site links, ratings, and prices when Baidu supplies them
• Stable SHA-256 IDs, canonical URL deduplication, and a typed 53-field output contract
• Persistent direct Playwright Chrome context, Chinese locale headers, shared cookies, bounded concurrency/retries/timeouts, and optional Residential proxy
• Validated DNS-over-HTTPS fallback for official Baidu hosts when the local resolver is unavailable
• Explicit no-results/challenge distinction and fail-closed CAPTCHA handling

Input

{
  "queries": ["人工智能", "机器学习"],
  "maxItems": 12,
  "maxItemsPerQuery": 6,
  "maxPages": 2,
  "sortBy": "dateDesc",
  "resultTypes": ["organic", "baike", "official"],
  "includeKeywords": ["人工"],
  "excludeKeywords": ["广告"],
  "requestDelayMs": 300,
  "market": "zh-CN",
  "includeAds": false,
  "includeRelated": false,
  "maxConcurrency": 2,
  "proxyConfiguration": { "useApifyProxy": false }
}

Use query, queries, or both. Duplicate query strings are removed and batch output is interleaved fairly. startUrls remains supported for compatibility, but every URL must be on baidu.com and include the same wd value a user would enter in Baidu's search field.

maxItems is global; maxItemsPerQuery defaults to a fair share. Each Baidu page currently supplies roughly 8–10 usable non-ad records.

See .actor/input_schema.json for the complete accepted input fields, types, defaults, and constraints.

Output

Records include stable identity, global/query/page positions, title and optional emphasized HTML, snippet, destination URL, source/domain, thumbnails and site links, rich-result classification, source-backed ad/official flags, dates, ratings/prices when present, displayed result-count text when Baidu exposes it, query/market context, transport status/content type, and run metadata. The result-count field checks Baidu's known count elements and bounded Chinese/English text fallbacks, then preserves the first valid count across paginated pages. Null, undefined, empty strings, and empty arrays/objects are removed.

For blocked runs, OUTPUT.json uses status: "blocked" and carries accessEvidence with the observed refusal: errorCode, blockedQueries, observedHost, httpStatus, responseContentType, pageTitle, a bounded bodyExcerpt, and the redirectChain host list (for example ["m.baidu.com", "wappass.baidu.com"]). Each querySummaries[] entry repeats lastHttpStatus, lastResponseContentType, lastPageHost, lastErrorCode, lastPageTitle, lastBodyExcerpt, and redirectChain, plus searchResultCountText when observed. This makes a target verification redirect (https://wappass.baidu.com/static/captcha/tuxing_v2.html) distinguishable from a transport failure without storing full response bodies, signed verification parameters, or URLs with query strings.

The Actor no longer fabricates a Google favicon URL. Optional fields absent from the Baidu card are omitted. The OUTPUT key-value record reports status (completed, partial, no-results, failed, or blocked), per-query page/record counts, filter and duplicate counts, result-type counts, and failed/no-result/blocked query positions. After a run, download the dataset JSON from the run's default dataset storage or the Apify dataset API/Console links.

The exact dataset fields, types, and requiredness are defined in .actor/dataset_schema.json.

Proxy and access behavior

Baidu may close connections or return an explicit verification page after repeated requests. Direct mode is supported, but Apify Residential proxy is recommended for cloud production runs:

{
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"],
    "apifyProxyCountry": "CN"
  }
}

GOOGLE_SERP cannot proxy Baidu and is rejected. The Actor does not solve or bypass CAPTCHA/verification pages. Challenges are marked non-retriable and never stored. Logs omit raw HTML, response bodies, destination URLs, cookies, proxy credentials, and signed verification parameters.

Performance

Browser pages run with bounded concurrency and skip media/font resources. Representative Residential cloud runs completed 15 records across two pages in about 53 crawler seconds and a fair 6+6 two-query batch in about 72 seconds, below the requested three-minute target.

Failure semantics

• Invalid input fails before navigation.
• Temporary navigation failures use bounded retries.
• Explicit verification/CAPTCHA pages fail nonzero without challenge data. Only an unambiguous signal (Baidu's own verification host, or its verification page title) is treated as a refusal before parsing; body wording such as 验证码 is consulted only when zero result cards were parsed, so a healthy SERP snippet cannot be misread as a block.
• A confirmed no-results page succeeds empty; if Baidu challenges the no-results query, the run fails rather than misclassifying it.
• Partial records buffered before a later challenge are not published as a successful complete run.

How to use

Enter a query or query batch in the Apify Input tab, choose result types and filters, and start the Actor. Inspect records in the Output dataset and status details in the OUTPUT key-value record.