# Changelog of Baidu Videos Scraper (`searchapi/baidu-videos-scraper`) Actor

- **URL**: https://apify.com/searchapi/baidu-videos-scraper/changelog.md
- **Full Actor documentation**: https://apify.com/searchapi/baidu-videos-scraper.md

## Changelog

### \[3.1.2] - 2026-09-06

- Preserve `TARGET_VERIFICATION` classification through the final no-records error path so the outer Actor log cannot relabel a target challenge as `unknown`.

### \[3.1.1] - 2026-09-06

- Return validated HTTP status/content-type metadata from the live page handler and retain final host/error-code diagnostics per query.
- Classify verification-page failures as `TARGET_VERIFICATION`, including the final all-failed run error, without storing challenge content.
- Read response headers case-insensitively before HTML extraction.

### \[3.1.0] - 2026-09-06

- Replaced the Crawlee navigation layer with a bounded direct Playwright runner so validated Baidu host mappings work when local DNS cannot resolve `www.baidu.com`.
- Added validated DNS-over-HTTPS fallback, shared persistent cookies, bounded query concurrency/retries/timeouts, request delay, and optional authorized proxy launch settings.
- Added platform, keyword, view-count, duration, short-video, and sort controls with safe run-summary diagnostics.
- Corrected no-results detection so ordinary non-video search cards are not treated as a successful empty video result.

### \[3.0.0] - 2026-08-29

#### Added

- Single-query, deduplicated batch-query, fair interleaving, single-item, bounded offset, and explicit no-results modes.
- Stable SHA-256 IDs derived from canonical watch URLs, preserving meaningful multi-part video parameters.
- A linked 59-field Dataset schema with required fields, valid numeric/date types, overview and full-detail views, and a standalone dataset audit command.
- Coherent Crawlee fingerprinting, persistent sessions, bounded retries, China Residential/custom proxy support, and explicit rejection of the Google-only `GOOGLE_SERP` group.
- Concurrent redirect resolution, query-level concurrency controls, safe resource blocking, and atomic dataset publication.
- Unit coverage for inputs, transport validation, URL/date/count normalization, stable IDs, multi-part videos, record cleaning, and fair selection.

#### Changed

- The Actor now enters queries through Baidu's search interface instead of requiring result-page URLs.
- Only cards inside verified `video` or `short_video` templates are accepted. The former generic result fallback was removed because it mislabeled ordinary web pages as videos.
- Optional unavailable values are omitted recursively instead of being stored as nulls, empty values, or fabricated flags.
- Baidu redirect URLs, opaque parameters, query URLs, and response bodies are no longer logged or stored.
- Dependencies are exact and aligned with the Playwright 1.61.1 Docker image; Docker builds now use the lockfile and non-root ownership.

#### Fixed

- Added the missing stable `id` field and corrected URL-based deduplication for multi-part videos.
- Corrected numeric types for duration and engagement fields and sanitized highlighted title HTML.
- Pagination now stops when an offset contains no verified video template instead of scraping unrelated general-search results.
- Challenge pages fail without retries or partial output; temporary network/proxy failures remain bounded.

#### Breaking

- Removed user-supplied `startUrls`; use `query` or `queries`.
- Removed fabricated `embedUrl`, favicon, technical-spec, and engagement fields that were not exposed by Baidu.

### \[2.0.1] - 2026-06-13

#### Fixed

- **Title extraction**: Title was previously matching the image wrapper link (text = just the duration like " 04:59"). Now uses a title-specific selector (`a[class*="title-default_"]` and variants) and rejects any candidate whose text matches the duration pattern. Falls back to a long-text `a[href*="link?url="]` link that has actual title content.
- **Channel/source extraction**: The source element contains both the channel name and the date joined by newlines (e.g. "哔哩哔哩\n2025-9-17"). Now strips both full-year dates (`\d{4}-\d{1,2}-\d{1,2}`) and month-day dates (`\d{1,2}-\d{1,2}`) and view-count suffixes from the source.
- **Bilibili video IDs**: `extractVideoId()` now recognizes `/video/BVxxxxxxxxxx` and `/video/av\d+` URL patterns. Previously returned null for bilibili links.
- **Date extraction**: Now handles dates without a year prefix (e.g. "3-1") in addition to full-year formats ("2025-9-17").
- **Description cleanup**: Description was being over-stripped to a single character. Now uses line-based filtering (skip duration, source, date, title) and falls back to `"<channel> · <date>"` if no descriptive text remains.
- **Uploader field**: Now populated from the source name (was previously null).

#### Verified

- 5/5 records returned on a real Baidu run via the SHADER proxy
- `title` is now the real video title (e.g. "【国家AI精品课程】零基础玩转生成式AI,全网..."), not the duration
- `channel` is correctly "哔哩哔哩" with the date separated
- `videoId` correctly extracts `BV13CptzMEWF`, `BV1sXh1zYEMr`, `BV1LeABzsEKg`, `BV12CnszQERk` for bilibili links
- `viewCount` remains null — the Baidu video SERP does not expose view counts in the visible card text. This is a source limitation, not a scraper bug.

### \[2.0.0] - 2026-06-12

#### Added

- **63 canonical output fields** grouped by category, produced via the shared `normalizeVideoRecord` in `_canonical/normalize.js`:
  - **Identity (4)**: `type`, `resultType`, `position`, `page`
  - **Title / text (7)**: `title`, `titleHtml`, `description`, `descriptionHtml`, `snippet`, `snippetHtml`, `displayUrl`
  - **URLs (5)**: `link`, `url`, `embedUrl`, `searchUrl`, `domain`
  - **Media (6)**: `thumbnail`, `thumbnailUrl`, `previewImage`, `previewImages[]`, `imageUrl`, `favicon`
  - **Duration (3)**: `duration`, `durationSeconds`, `durationFormatted`
  - **Engagement (6)**: `views`, `viewCount`, `viewCountFormatted`, `likeCount`, `dislikeCount`, `commentCount`
  - **Date / Time (3)**: `publishedAt`, `publishedAtRaw`, `date`
  - **Channel / Uploader (8)**: `channel`, `channelUrl`, `channelId`, `channelVerified`, `channelFollowers`, `uploader`, `uploaderUrl`, `uploaderVerified`
  - **Classification (3)**: `tags`, `categories`, `isShort`
  - **Flags (4)**: `isLive`, `isAd`, `isSponsored`, `isPaywall`
  - **Tech specs (6)**: `qualityOptions`, `width`, `height`, `resolution`, `aspectRatio`, `fileFormat`
  - **Identity (3)**: `videoId`, `platform`, `uploaderId`
  - **Engine identity (2)**: `engineId`, `engineData`
  - **Search context (8)**: `searchQuery`, `query`, `market`, `country`, `language`, `scrapedAt`, `searchMetadata`, `relativeTime`
- **Shared normalizer**: uses `normalizeVideoRecord` from `_canonical/normalize.js`
- **Chinese view count parser**: `parseChineseViews()` handles "1.2万", "3.4亿", "5千", "100万次播放"
- **Chinese duration parser**: `durationToSeconds()` and `formatDuration()` convert mm:ss/hh:mm:ss to seconds and back
- **Chinese date parser**: `parseChineseDate()` handles "X分钟前", "X小时前", "X天前", "刚刚", "昨天", "前天", "X年X月X日"
- **Multi-frame preview extraction**: collects all `<img>` URLs from the video card as `previewImages[]`
- **`isShort` detection**: based on `<div tpl="short_video">` ancestor or duration < 60s
- **`isAd` detection**: based on `data-tuiguang` attribute, `EC_result` class, or `[class*="ad"]` markers
- **Comment / like count parsing**: regex over card text for "1234评论" / "1234赞" patterns
- **Baidu redirect resolution**: HEAD request follows `https://www.baidu.com/link?url=...` to the real video URL
- **2 views**: `overview` (key fields) and `fullDetails` (all 63 fields)
- **New input fields**: `startUrls`, `maxPages`, `resolveRedirects`, `market`

#### Changed

- **Breaking** — actor version bumped to 2.0.0
- **Breaking** — schema expanded from ~15 fields to 63 fields
- Cleaner `main.js` — extraction moved to `routes/handlers.js`
- `package.json` no longer runs `crawlee install-playwright-browsers` postinstall
- Pagination respects `maxPages` cap

### \[1.0.0] - 2025-07-01

#### Added

- Initial release of the actor
- Basic video extraction with redirect resolution