Baidu News Scraper

Search Baidu's public News interface and export normalized articles with stable IDs, canonical article URLs, publisher metadata, snippets, publication dates, images, query context, and extraction provenance.

Features

• Uses the current news.baidu.com/ns search interface and follows Baidu's validated native pagination links.
• Supports single and fair batch query modes.
• Enforces total and per-query record limits, with bounded query concurrency.
• Resolves Baidu article redirects when enabled and removes common tracking parameters.
• Produces stable SHA-256 IDs and deduplicates by canonical article URL.
• Omits null, blank, empty-array, and empty-object fields recursively.
• Buffers the complete run before writing, so a later verification challenge cannot publish a partial dataset.
• Supports direct access, Apify Residential Proxy, and custom proxy URLs through Apify's proxy input editor.
• Detects CAPTCHA/verification pages and fails closed; it does not solve or bypass interactive verification.

GOOGLE_SERP Proxy is not supported because it is restricted to Google Search and Google Shopping, not Baidu.

Input

Field	Description	Default
mode	single or batch	single
query	Query used in single mode	人工智能
queries	Up to 20 deduplicated queries used in batch mode	[]
maxItems	Maximum records across the run	50
maxItemsPerQuery	Maximum records collected for each query	50
maxPages	Maximum pages followed for each query	5
resultsPerPage	Requested Baidu page size; live pagination links remain authoritative	20
maxConcurrency	Concurrent query browser pages	2
resolveRedirects	Resolve Baidu article redirect links	true
market	Locale tag used in output context	zh-CN
maxRequestRetries	Retries for temporary transport/rate-limit failures	3
navigationTimeoutSecs	Per-navigation timeout	60
requestHandlerTimeoutSecs	Per-query handler timeout	120
proxyConfiguration	Direct, Apify Proxy, or custom proxy configuration	direct
debug	Sanitized structural diagnostics only	false

Output

The documented schema contains 56 typed fields. A representative run populated 37–45 fields per record, including:

• Identity: id, type, resultType, position, globalPosition, queryPosition, resultPosition, page
• Article: title, titleHtml, snippet, snippetHtml, description, emphasis
• URLs: link, url, domain, displayUrl, sourceUrl, sourceDomain, searchUrl
• Publisher and dates: source, publishedAt, publishedAtRaw, relativeTime, date
• Media: thumbnail, thumbnailUrl, imageUrl, images, dimensions, source logo/favicon
• Context: query, searchQuery, market, country, language, searchMetadata
• Provenance: engineId, engineData, extractionMethod, snippetExtractionMethod, scrapedAt

Optional fields are omitted when Baidu does not expose them. No placeholder or fabricated values are added.

Examples

Single query:

{
  "mode": "single",
  "query": "人工智能",
  "maxItems": 20,
  "maxPages": 2,
  "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"], "countryCode": "CN" }
}

Fair batch:

{
  "mode": "batch",
  "queries": ["人工智能", "机器人"],
  "maxItems": 20,
  "maxItemsPerQuery": 10,
  "maxConcurrency": 2,
  "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"], "countryCode": "CN" }
}

Verified behavior

• Local direct run: 10 audit-clean current News records in about 12 seconds before the local IP was later challenged.
• Cloud China Residential pagination run: 20 unique records across two pages; crawler runtime about 38 seconds.
• Cloud China Residential batch run: 20 unique records in a strict 10+10 fair interleave.
• Cloud no-results run: succeeded with an empty dataset.
• Dataset audits check required fields, types, nested empty values, URL/date validity, duplicate IDs/URLs, challenge content, unsafe HTML, and sensitive/internal fields.