# Changelog of Google SERP Intelligence Scraper (`netdesignr/google-serp-scraper`) Actor - **URL**: https://apify.com/netdesignr/google-serp-scraper/changelog.md - **Full Actor documentation**: https://apify.com/netdesignr/google-serp-scraper.md ## Changelog All notable changes to `google-serp-scraper` are documented in this file. ### \[Unreleased] #### Removed - Removed Browserless/BQL runtime fallback code, helper modules, and tests so Google extraction no longer depends on premium third-party browser infrastructure. #### Changed - Disabled managed residential proxy usage for Google runs; the input UI now only exposes `off`, saved inputs that still pass `residentialProxyMode: "residential"` are normalized to `off`, and stale managed-proxy retry branches were removed from Google extraction paths. - Reduced default Google fallback cost by honoring explicit light mode, avoiding browser escalation for low-result pages, and removing duplicate retry layers around light/full extraction. - Blocked-only Google empty runs now exit successfully with a diagnostic note even when strict empty-run policy is requested, preventing `restartOnError` from multiplying platform start costs on unavoidable Google block pages. - Installed the matching Playwright Chromium browser in the production Docker image so legitimate full-mode fallback works on Apify cloud. ### \[0.3.8] - 2026-03-26 #### Changed - Added a clearer README overview showing the search evidence, AI engine outputs, and reporting artifacts users can expect from each run. ### \[0.3.6] - 2026-03-24 #### Fixed - Hardened global proxy fallback so blank country codes no longer generate invalid proxy configuration candidates during degraded-run recovery. - Stopped Google fallback requests from emitting an empty `gl` parameter when retrying against the global pool. ### \[0.3.1] - 2026-03-08 #### Fixed - Fixed BQL HTML capture so Browserless BQL transport passes real rendered HTML to the AI Mode parser instead of empty strings, restoring citation extraction for BQL-routed queries. - Fixed BQL UI chrome regex stripping false positives that removed common English words ("All", "News", "More", "Images") from legitimate AI Mode answers. - Fixed first-sentence drop heuristic to preserve valid short opening sentences by requiring both short length (<30 chars) and absence of common English verbs before dropping. - Added footer stripping to remove trailing Google UI boilerplate ("Generative AI is experimental") from AI Mode answers. #### Added - Added Browserless BQL integration module with GraphQL-based headless browser automation as a fallback transport layer. - Added URL-based title fallback for citations with empty anchor text, deriving "hostname - path" titles automatically. - Added citation deduplication with metadata merging that keeps the richest title and snippet per URL instead of discarding duplicates. - Added 3 new citation snippet extraction strategies (previous sibling paragraph, parent-level sibling, title/aria-label attributes) for a total of 5 strategies. - Added answer merging in BQL text fallback to pick the longer (richer) answer between HTML-parsed and BQL raw text. - Added render pause before BQL HTML capture to allow dynamic Google AI Mode content to settle. - Added usability check for BQL HTML to verify the captured HTML contains AI Mode answer content before parsing. - Added 6 new AI Mode unit tests covering false-positive preservation, short sentence preservation, chrome fragment drop, URL title fallback, dedup merging, and sibling snippet extraction. - Added benchmark reports and release checklists for v0.3.0 validation. ### \[0.3.0] - 2026-03-07 #### Changed - Removed the upstream `apify/google-search-scraper` dependency so all runs now use the actor's own direct extraction stack. - Tightened `auto` mode escalation to retry with Playwright when light mode hits block pages, consent gates, JS-required interstitials, or low-confidence extraction. - Switched the actor foundation to a sequential release-gate model so new channels and add-ons only ship after fixture, smoke, billing, and reporting checks pass. - Replaced launch-stage synthetic billing assumptions with explicit PPE events for Google pages, Google AI Mode, Perplexity, ChatGPT, ads, and leads. - Expanded the public schema from Wave 1 `web` only to a full multi-surface search visibility contract. - Moved AI answer artifacts and run reports into key-value store records so default-dataset billing only applies to Google page outputs while Apify pricing changes are locked. - Deferred custom `Actor.charge()` events until the next allowed pricing activation window to avoid mismatched-event warnings on live runs. #### Added - Added direct-only fallback tests for blocked/interstitial Google pages. - Added `outputMode` support with `raw`, `report`, and `both` modes while preserving `raw` as the backward-compatible default. - Added `visibility-report` dataset items and `REPORT.json` output for internal dashboards and scheduled monitoring workflows. - Added a Wave 1 reporting layer covering Google brand visibility, visible domains, warning counts, and confidence summaries. - Added a benchmark harness with a failure taxonomy, regression scorecard, and live smoke runner for direct-only Google validation. - Added Google vertical support for `images`, `news`, and `shopping`. - Added Google AI Mode extraction and normalization. - Added Perplexity API normalization for answers, citations, images, and related questions. - Added ChatGPT Search normalization through OpenAI web search tooling. - Added optional business leads enrichment from cited domains. - Added unit tests for Google verticals, Google AI Mode, Perplexity, ChatGPT Search, reporting, billing dedupe, and leads enrichment. - Added `AI_ENGINE_RESULTS.json` and `OUTPUT_INDEX.json` artifacts for agent-friendly retrieval of AI answers and run outputs. #### Fixed - Tightened Google AI Mode parser chrome stripping so polluted Google UI text no longer passes as answer content. - Tightened cloud-matrix AI quality gates so low-confidence or citation-free Google AI Mode outputs fail validation instead of passing on item presence alone. ### \[0.1.0] - 2026-03-07 #### Added - Initial Google SERP actor scaffold for the monorepo. - Dual-mode extraction architecture: `light` (HTTP) and `full` (browser). - Support for `web`, `images`, `news`, and `shopping` search types. - Page-level output schema with SERP features, quality warnings, and brand analysis. - Runtime validation for queries, region/language targeting, and mode selection. - Initial parser suite for organic results, ads, AI Overview, PAA, knowledge panel, video, local pack, and vertical search results. - Billing hooks, HTML snapshot support, and run metadata persistence. - Unit tests for input validation, URL building, parsers, brand analysis, billing, and light mode scraping. ### \[0.2.0] - 2026-03-07 #### Changed - Routed Apify cloud `web` runs through a managed backend token instead of the runtime `APIFY_TOKEN` to avoid customer double billing. - Disabled manual billing by default so launch pricing can use Apify synthetic events safely. - Narrowed the public launch scope to `web` search after live validation showed non-web Google verticals were not yet reliable enough to sell. - Lowered the default actor memory to 1024 MB so launch pricing charges a single actor-start event per run. - Tightened the `brandAnalysis` dataset view flattening for store-friendly table rendering. #### Fixed - Fixed official backend metadata mapping to satisfy lint and preserve stable page completion flags.