Baidu Scholar Scraper

Search the public Baidu Xueshu site and export normalized academic-paper records. The Actor enters one or many query phrases, crawls bounded pn pages, extracts only verified paper cards from the DOM, normalizes academic metadata, and writes dataset records plus a safe OUTPUT run summary.

Input modes

• Single query: set query.
• Batch search: set queries with up to 20 phrases. Duplicates are removed and records are interleaved fairly.
• Pagination: set maxPages; Baidu Xueshu offsets are requested independently for each query.
• Narrow results with keyword, document-type, year, citation, abstract, and DOI filters.
• Sort accepted records by search order, citations, publication year, or title.

Input

Section	Fields	Purpose
Search	query, queries, market	Academic phrase(s) and browser locale
Limits and pagination	maxItems, maxItemsPerQuery, maxPages	Total, per-query, and page limits
Filters and sorting	includeKeywords, excludeKeywords, includeDocumentTypes, excludeDocumentTypes, minYear, maxYear, minCitations, requireAbstract, requireDoi, sortBy	Restrict and order normalized papers
Developer options	maxConcurrency, maxRequestRetries, navigationTimeoutSecs, requestHandlerTimeoutSecs, requestDelayMs, blockMedia	Tune bounded concurrency, retries, timeouts, pacing, and resources
Proxy	proxyConfiguration	Direct access, authorized Apify Proxy, or custom proxy URLs

Example batch input:

{
  "queries": ["人工智能", "机器学习"],
  "maxItems": 20,
  "maxItemsPerQuery": 10,
  "maxPages": 2,
  "includeKeywords": ["模型"],
  "minYear": 2020,
  "sortBy": "citationsDesc",
  "requestDelayMs": 500,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"],
    "countryCode": "CN"
  }
}

GOOGLE_SERP is rejected because it supports Google Search and Shopping traffic, not Baidu Xueshu. Use direct access, an authorized Residential proxy, or custom proxy URLs.

The complete input contract is in .actor/input_schema.json. The accepted fields and defaults are:

Field	Type	Default
query	string	—
queries	array of strings	[]
market	string	zh-CN
maxItems	integer	20
maxItemsPerQuery	integer	20
maxPages	integer	3
includeKeywords	array of strings	[]
excludeKeywords	array of strings	[]
includeDocumentTypes	array of strings	[]
excludeDocumentTypes	array of strings	[]
minYear	integer	—
maxYear	integer	—
minCitations	integer	—
requireAbstract	boolean	false
requireDoi	boolean	false
sortBy	string	searchOrder
maxConcurrency	integer	2
maxRequestRetries	integer	1
navigationTimeoutSecs	integer	45
requestHandlerTimeoutSecs	integer	150
requestDelayMs	integer	300
blockMedia	boolean	true
proxyConfiguration	object	—

Output

Each dataset record can contain stable IDs and positions, paper title and sanitized highlighting, abstract and text counts, authors and affiliations, journal/conference/publisher, publication year/date, citations and references, DOI, document type, language, keywords, subject areas, open-access marker, paper/PDF/related/citation links, source rank, result-count metadata, query/page provenance, and ISO scrape time.

Missing optional values are omitted rather than emitted as null, empty arrays, or placeholders. Records without a safe paper URL, malformed cards, unresolved sensitive URLs, challenge content, and ordinary non-academic links are rejected.

The OUTPUT key-value record stores status, requested queries, limits, developer settings, filters, pages fetched/failed, verified cards, filtered count, per-query result counts, connection mode, DNS mode, bounded failure categories, and completion time. It also records the last HTTP status, response content type, final page host, and sanitized lastErrorCode per query. It never stores cookies, authorization headers, proxy credentials, signed URLs, or raw response bodies.

All dataset output fields, types, and requiredness are defined in .actor/dataset_schema.json.

Pagination and batch behavior

Every query is crawled across pn offsets until maxPages, the page exposes no more cards, or its per-query limit is reached. Records are deduplicated by canonical paper URL and stable ID, selected fairly across queries, then sorted and renumbered globally. A failed query is visible in OUTPUT.failedQueries; usable records from other queries produce partial, not a misleading complete status.

Proxy and access behavior

The Actor uses persistent Playwright contexts with shared cookies for a run, bounded query concurrency, optional media blocking, validated HTML status/content type, and bounded retries for temporary transport, timeout, rate-limit, and proxy errors. When local DNS cannot resolve the official hosts, it uses status/content-type-checked public DNS-over-HTTPS responses and validated Chrome host mappings.

Baidu Xueshu can redirect to an interactive verification page or return HTTP 403. The Actor detects that target-side access control, records target_verification with lastErrorCode: "TARGET_VERIFICATION", stores no challenge page as a paper, and does not solve or bypass CAPTCHA or verification.

How to use

Enter a query or batch in the Apify Input tab, set page limits and filters, and start the Actor. Review paper records in the Output dataset and run status details in the OUTPUT key-value record.

Current source limitations

Xueshu's public result DOM and target access policy can change. Academic fields only appear when the current card exposes them, so optional metadata is not inferred from a title or query. A successful live dataset requires Baidu to return public Scholar result cards to the selected direct or authorized proxy route.