Baidu Search Scraper

Playwright-based extraction of public Baidu web results with rich-result classification, stable IDs, real pagination, regional search context, fair query batches, and authorized proxy support.

Highlights

• Single query, deduplicated multiple queries, fair per-query quotas, and exact low-volume mode
• Constructed Baidu pn pagination rather than reliance on a drifting Next selector
• Organic, Baike, video, image, official, academic, finance, live, and other rich-card classification
• Source, snippet, highlighted terms, dates, thumbnails, site links, ratings, and prices when Baidu supplies them
• Stable SHA-256 IDs, canonical URL deduplication, and a typed 53-field output contract
• Persistent direct Playwright Chrome context, Chinese locale headers, shared cookies, bounded concurrency/retries/timeouts, and optional Residential proxy
• Validated DNS-over-HTTPS fallback for official Baidu hosts when the local resolver is unavailable
• Explicit no-results/challenge distinction and fail-closed CAPTCHA handling

Input

{
  "queries": ["人工智能", "机器学习"],
  "maxItems": 12,
  "maxItemsPerQuery": 6,
  "maxPages": 2,
  "sortBy": "dateDesc",
  "resultTypes": ["organic", "baike", "official"],
  "includeKeywords": ["人工"],
  "excludeKeywords": ["广告"],
  "requestDelayMs": 300,
  "market": "zh-CN",
  "includeAds": false,
  "includeRelated": false,
  "maxConcurrency": 2,
  "proxyConfiguration": { "useApifyProxy": false }
}

Use query, queries, or both. Duplicate query strings are removed and batch output is interleaved fairly. startUrls remains supported for compatibility, but every URL must be on baidu.com and include the same wd value a user would enter in Baidu's search field.

maxItems is global; maxItemsPerQuery defaults to a fair share. Each Baidu page currently supplies roughly 8–10 usable non-ad records.

Output

Records include stable identity, global/query/page positions, title and optional emphasized HTML, snippet, destination URL, source/domain, thumbnails and site links, rich-result classification, source-backed ad/official flags, dates, ratings/prices when present, displayed result-count text when Baidu exposes it, query/market context, transport status/content type, and run metadata. The result-count field checks Baidu's known count elements and bounded Chinese/English text fallbacks, then preserves the first valid count across paginated pages. Null, undefined, empty strings, and empty arrays/objects are removed.

For blocked runs, OUTPUT.json includes the last HTTP status, response content type, final page host, searchResultCountText when observed, and a sanitized lastErrorCode per query. This makes a target verification page distinguishable from a transport failure without storing response bodies.

The Actor no longer fabricates a Google favicon URL. Optional fields absent from the Baidu card are omitted. The OUTPUT key-value record reports status, per-query page/record counts, filter and duplicate counts, result-type counts, and failed/no-result query positions. After a run, download the dataset JSON from the run's default dataset storage or the Apify dataset API/Console links.

Proxy and access behavior

Baidu may close connections or return an explicit verification page after repeated requests. Direct mode is supported, but Apify Residential proxy is recommended for cloud production runs:

{
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"],
    "apifyProxyCountry": "CN"
  }
}

GOOGLE_SERP cannot proxy Baidu and is rejected. The Actor does not solve or bypass CAPTCHA/verification pages. Challenges are marked non-retriable and never stored. Logs omit raw HTML, response bodies, destination URLs, cookies, proxy credentials, and signed verification parameters.

Performance

Browser pages run with bounded concurrency and skip media/font resources. Representative Residential cloud runs completed 15 records across two pages in about 53 crawler seconds and a fair 6+6 two-query batch in about 72 seconds, below the requested three-minute target.

Failure semantics

• Invalid input fails before navigation.
• Temporary navigation failures use bounded retries.
• Explicit verification/CAPTCHA pages fail nonzero without challenge data.
• A confirmed no-results page succeeds empty; if Baidu challenges the no-results query, the run fails rather than misclassifying it.
• Partial records buffered before a later challenge are not published as a successful complete run.

Local development

npm ci
npm test
npm run check
npm run validate:readme
apify validate-schema
apify run --purge --input-file INPUT.json