# Changelog of Booking.com Hotel & Price Scraper (`cashmere_verdict/booking-hotel-scraper`) Actor

- **URL**: https://apify.com/cashmere_verdict/booking-hotel-scraper/changelog.md
- **Full Actor documentation**: https://apify.com/cashmere_verdict/booking-hotel-scraper.md

## Changelog

All notable changes to the Booking.com Hotel & Price Scraper are documented here.

### \[1.8.0] - 2026-02-08

#### Added

- **JSON-LD amenity fallback (Pattern 5)**: New `_extract_amenities_from_json_ld()` extracts amenities from `amenityFeature` in JSON-LD structured data. Handles both object format (`[{"name": "WiFi", "value": true}]`) and string list formats. Filters out explicitly disabled amenities (`value: false`). Used as last resort when HTML patterns return insufficient results.
- **Enrich input validation warning**: New `_get_enrich_warning()` warns users when `enrichData=true` is combined with `maxResults > 50`, alerting them about potentially slow runtime. Warning displayed in logs and Apify Console via `Actor.set_status_message()`.
- 564 tests (up from 554 in v1.7), 10 new tests for JSON-LD amenities and enrich warning logic

#### Changed

- Amenity deduplication uses shared `seen` set across all 5 extraction patterns
- `ENRICH_MAX_RESULTS_WARNING_THRESHOLD = 50` constant for configurable warning threshold

### \[1.7.0] - 2026-02-08

#### Fixed

- **Amenity extraction overhaul**: Previous `data-testid="facility-list-item"` selector is obsolete in 2026 Booking.com. Added 2 new extraction patterns:
  - Pattern 2: `property-facilities-block-container` → `facility-group-container` → `<li>` items (full facility list with group filtering)
  - Pattern 3: `property-most-popular-facilities-wrapper` → `<li>` items (popular facilities shortlist)
  - Group name filtering excludes category headers (Bathroom, Outdoors, Kitchen, etc.)
- **Description fallback**: Added JSON-LD `description` field as Pattern 4 fallback — most reliable source, survives all UI layout changes
- **WAF detection for cookie-consent pages**: Increased threshold from 10KB to 50KB with content marker validation. Pages <50KB with no hotel content markers (e.g., 15-25KB cookie-consent pages) now correctly trigger WAF retry

#### Added

- 2 new content markers for WAF detection: `facility-group-container`, `property-facilities-block-container`
- 554 tests (up from 546 in v1.6), 8 new tests for amenity extraction, JSON-LD description, and cookie-consent WAF detection

### \[1.6.0] - 2026-02-08

#### Added

- **WAF detection**: `_detect_waf_block()` identifies Booking.com WAF blocks via HTML size threshold (<10KB) + content marker absence, plus explicit WAF patterns ("access denied", "403 forbidden", "http-equiv=refresh", "cf-please-wait")
- **WAF retry with UA rotation**: On WAF block detection, waits 5-10s random delay, rotates User-Agent, reloads page, and re-extracts. Falls back gracefully to search-only data if retry also fails
- **Referer header spoofing**: Sets `Referer: https://www.booking.com/searchresults.html` before detail page extraction, simulating organic navigation from search results
- **Human-like delay**: Random 3-8s delay before detail page extraction to avoid timing-based bot detection
- **`waf_blocked` field**: New boolean field on `DetailResult` dataclass for WAF block tracking
- 546 tests (up from 534 in v1.5), 12 new tests for WAF detection, retry, and delay logic

#### Changed

- Extracted `_extract_detail_html(page)` helper for DRY reuse in main + retry extraction paths
- Expected detail enrichment rate improvement from ~33% (v1.5) to 50-65% due to WAF mitigation

### \[1.5.0] - 2026-02-08

#### Added

- **Room price per-night calculation**: Threaded `nights` parameter through the entire detail extraction pipeline. Room options now include both `price_per_night` (calculated) and `price_total` (raw from page). Uses `(check_out - check_in).days` to compute stay length.
- **`_calc_per_night()` helper**: Safe division with guard for None/zero nights — returns None if nights unavailable
- 534 tests (up from 526 in v1.4), 8 new tests for per-night price calculation

#### Changed

- `extract_room_options()` accepts optional `nights` parameter for per-night calculation
- `extract_detail_from_html()` accepts optional `nights` parameter, passes to room extraction
- `DetailContext` dataclass includes `nights: int | None` field
- All 3 room parsers (`_parse_room_row_testid`, `_parse_hprt_rows_with_carry_forward`, `_parse_block_id_rows`) now standardized: extracted prices are `price_total`, `price_per_night` is calculated

### \[1.4.0] - 2026-02-08

#### Fixed

- **Review extraction overhaul**: Added JSON-LD `aggregateRating` parser (most reliable source for `overall_score` and `total_reviews`), `review-score-right-component` fallback, and `data-review-score` attribute fallback
- **Review subscores**: Fixed `find()` → `find_all()` for `data-testid="review-subscore"` — Booking.com 2026 uses one element per category, not a single container. All 7 categories now populated (staff, cleanliness, comfort, location, facilities, value_for_money, free_wifi)
- **Room options extraction**: New `_parse_hprt_rows_with_carry_forward()` for `hprt-table` with `<th>` room names and `rowspan` carry-forward across rate plan rows. Extracts room name, bed type, occupancy, price, breakfast, and cancellation policy. Yields 32-42 room options per hotel

#### Added

- **Debug HTML capture**: `saveDebugHtml` input flag saves detail page HTML to Apify KV store for offline DOM analysis
- 526 tests (up from 511 in v1.3), 15 new tests for review and room extraction

### \[1.3.0] - 2026-02-08

#### Fixed

- **Photo URL filtering**: Added `_is_hotel_photo()` filter to reject non-hotel images from photo_urls output — only real hotel/room photos are returned
- **Photo filter hotfix**: Extended filter to catch real-world Booking.com URL patterns discovered in live testing:
  - `images-flags/` (design-assets flag icons like `Us@3x.png`, `Gb.png`)
  - `/illustrations` (TravellerQuestion.png and similar)
  - `/xphoto/` (review author photos and avatars)
  - `/design-assets/` (all Booking.com UI design assets)

#### Changed

- 511 tests (up from 500 in v1.2), 11 new tests for photo filter logic

### \[1.2.0] - 2026-02-08

#### Fixed

- **Headless mode detection**: Switched default to `headless=False` — Booking.com hides room prices, review scores, and photo galleries from headless browsers
- **Lazy-loaded content**: Added full-page scroll (bottom → wait 2s → top → wait 1s) before extracting detail page HTML, triggering IntersectionObserver for rooms, photos, and reviews
- **Photo extraction**: Enhanced `_collect_photo_urls()` with `data-highres`, `data-src`, `srcset` fallbacks and `bstatic.com` Pattern 3; filters out data: URI placeholders
- **Review summary**: Added two fallback extraction methods — flat span pairs and regex "Category X.X" text parsing for `overall_score`, `staff`, and other sub-scores
- **Room options**: Added Pattern 3 using `data-block-id` rows with multi-row name carry-forward for Booking.com's room table layout

#### Changed

- 500 tests (up from 490 in v1.1), 10 new tests covering all v1.2 fixes

### \[1.1.0] - 2026-02-08

#### Added

- **Enriched Mode** (`enrichData: true`): Visit each hotel's detail page to extract room options, full amenities, review breakdowns by category, descriptions, and photo galleries
- **Detail page extractor**: 6 extraction functions (description, amenities, room options, review summary, photos, orchestrator)
- **Detail page handler**: CAPTCHA detection, success criteria, exception-safe extraction pipeline with graceful degradation
- **Dual-mode router**: Search handler enqueues detail page requests when enriched mode is enabled; detail handler merges search + detail data
- **New filters**: `propertyType` (hotel, apartment, hostel, resort, villa, guesthouse), `minReviewScore` (0-10), `children` with `childrenAges`
- **New output fields**: `data_source` ("search" or "search+detail"), `detail` object with `description`, `all_amenities`, `room_options`, `review_summary`, `photo_urls`
- **New models**: `RoomOption`, `ReviewSummary`, `DetailInfo` Pydantic models
- **490 tests** (up from 332 in v1.0.28)

#### Changed

- `HotelRecord` now includes `data_source` and optional `detail` fields (backward compatible — defaults to `"search"` and `null`)
- `build_search_url()` supports property type, review score, and children/ages filters via `nflt` parameter
- Crawler `max_requests_per_crawl` dynamically increases when enriched mode is enabled to accommodate detail page requests
- Updated dataset schema with all new fields
- Updated README with Fast vs. Enriched mode comparison table, new parameters, and enriched output examples

### \[1.0.28] - 2026-02-07

#### Added

- `Actor.fail()` for critical errors (invalid input, zero results, CAPTCHA blocks)
- Enhanced run summary with `status`, `extraction_method`, and `success_rate` fields
- `Actor.set_status_message()` for Apify Console visibility
- CAPTCHA backoff: 30s delay after detection, abort after 2 consecutive CAPTCHAs
- Circuit breaker: abort after 3 consecutive page timeouts
- Dataset schema (`dataset_schema.json`) for Apify Store listing
- Store metadata: categories, SEO title/description, default run options
- Expanded README with use cases, integrations, and FAQ

#### Changed

- Updated User-Agent pool to Chrome 128-134 (removed outdated 120-127)
- `_validate_input()` now returns `(ActorInput, error_msg)` tuple for better error reporting
- Replaced bare `except Exception` with specific `json.JSONDecodeError` and `TimeoutError` handling
- Invalid input now triggers `Actor.fail()` instead of silent `Actor.exit()`

#### Fixed

- Zero results without CAPTCHA now correctly reports failure via `Actor.fail()`

### \[1.0.27] - 2026-02-06

#### Fixed

- Price calculation: cards show total stay price, now correctly divides by nights for `per_night`
- Star rating filter URL: uses semicolon-separated `nflt` parameter
- Stars extraction: broadened regex patterns for `data-testid="rating-stars"`
- Price high sort: client-side re-sort since Booking.com has no descending price URL param

### \[1.0.26] - 2026-02-06

#### Added

- Live testing suite: 19 tests, 17 PASS / 2 PARTIAL / 0 FAIL
- Apify deployment with residential proxy support

### \[1.0.25] - 2026-02-06

#### Fixed

- `ConcurrencySettings` requires explicit `desired_concurrency=1` (crawlee 1.3.1)
- `pre_navigation_hook` takes 1 arg, not 2 (crawlee 1.3.1 API change)
- Proxy configuration uses `Actor.create_proxy_configuration()` instead of manual URL

### \[1.0.0] - 2026-02-05

#### Added

- Initial release with hybrid API interception + HTML fallback extraction
- 14-field HotelRecord output model (price, rating, location, availability, etc.)
- Smart pagination with configurable max results (up to 1,000)
- Anti-detection: stealth scripts, UA rotation, viewport randomization
- CAPTCHA detection (skip, don't solve)
- Input validation with Pydantic (dates, price range, star ratings)
- Apify Proxy integration (RESIDENTIAL group)
- Full test suite (10 test files)