Google AI Overview Citation Scraper
Pricing
Pay per event
Scrape which domains Google's AI Overview cites for your target queries — one row per (query × cited source) with position, snippet, and selector telemetry — export to JSON or CSV. AEO / generative-SEO data for 2026, with captcha-aware retry and Pydantic-validated output.
All notable changes to this Actor are documented here.
open_browser in src/browser.py now catches PWError / Exception
on context-manager exit and suppresses teardown-only errors that contain
"connection closed" or "browser has been closed". Playwright 1.59 raises
Browser.close: Connection closed while reading from the driver when
Camoufox/Firefox terminates after processing a page whose uncaught JS
error has a null location object (a Node.js 24 / Playwright coreBundle
regression). The browsing work completes before teardown; this exception
was noise that caused exit_code 91 on every cloud run._try_proxy_groups.
Added a if url is None: return None guard after configuration.new_url()
so pyright no longer reports str | None cannot be assigned to str.Bug 1 — proxy exhaustion no longer causes FAILED runs.
_resolve_proxy_or_die now retries proxy resolution PROXY_RESOLVE_ATTEMPTS (3)
times with PROXY_RESOLVE_BACKOFF_S (2 s) backoff instead of immediately calling
raise SystemExit(1). When all retries are exhausted the function returns None;
main() emits a clear set_status_message and exits cleanly via return — the
run status is SUCCEEDED (0 rows) rather than FAILED. Traces the 2 historical FAILED
cloud runs caused by transient FREE-tier IP-pool exhaustion.
Bug 2 — Google "not available" error page no longer produces false citations.
_try_text_fallback in src/parser.py now requires the heading text to exactly
equal TEXT_FALLBACK_NEEDLE ("ai overview") rather than merely start with it.
Added _heading_signals_error guard that inspects the enclosing container text for
TEXT_FALLBACK_ERROR_PHRASES ("not available", "can't generate",
"cannot generate", "try again later", "not available for this search"); any
match suppresses the fallback hit. Prevents ai_overview_appeared=True rows with
scraped SERP links as fake citations when Google returned the error block.
Bug 3 — CAPTCHA marker is unambiguous (observability).
Added blocked_by_captcha: bool = False field to ResultRow. _marker_row sets
it True; _no_overview_row leaves it False. _is_captcha_marker now checks
rows[0].blocked_by_captcha directly instead of duck-typing on
selector_used is NonePROXY_RESOLVE_ATTEMPTS = 3 and PROXY_RESOLVE_BACKOFF_S = 2.0 module constants
in src/main.py (Bug 1 retry knobs).TEXT_FALLBACK_ERROR_PHRASES tuple constant in src/parser.py (Bug 2 guard
phrases)._heading_signals_error(node) pure helper in src/parser.py.tests/fixtures/ai_overview_error_unavailable.html — fixture reproducing the Google
error-page shape for Bug 2 regression tests.tests/test_bug_fixes.py — 20 new hermetic tests: 8 for Bug 2 (error-page parser),
6 for Bug 3 (CAPTCHA marker field), 5 for Bug 1 (proxy retry / no SystemExit).h1/h2/h3 starts-with-"AI Overview").selector_used telemetry column on every row — drift detection via dataset query./sorry/index, rotates session, retries once, emits marker row on persistent block.session_id rotation (Apify regex-safe via s_{uuid_hex}).$0.05 actor-start + $0.005/result-row (~$5.50 / 1,000 rows).Actor.charge(event, count=N) shape — no idempotency_key (SDK 3.x rejects it).SystemExit(1) + clear status message.tests/fixtures/input.qa.json (2 queries).