Google AI Overview Scraper

Read genuine AI Overview answer panels from Google Search using the configured source transport. Each result corresponds to one distinct requested query. A paragraph, citation or ordinary organic result is never expanded into a separate overview record. The Actor preserves the original query, the Google query echo, answer text, answer paragraphs, native container identity, cited URLs and source response evidence. It does not generate summaries with another language model or insert placeholder results.

The default transport is the user-configured Apify GOOGLE_SERP managed service. Its documented interface requires HTTP search URLs and does not support sessions. Earlier browser requests used an HTTPS browser flow with this HTTP-only service. The native runtime now sends supported HTTP requests through the service, while retaining the canonical HTTPS Google search URL in each output record. This uses your existing Apify proxy access and billing, with no additional provider API key. Direct/residential modes remain subject to a direct source robots preflight and may be unavailable. There is no automatic transport fallback after a failure. See https://docs.apify.com/proxy/google-serp-proxy .

Input

Supply a nonempty queries array and maxItems. The minimum maxItems/maxResults cap applies globally; duplicate queries are removed. minItems defaults to1 and can be set to10 for a strict ten-record verification. queryConcurrency defaults to2 and is limited to4. gl and hl preserve the requested source representation parameters; they do not guarantee the source's physical proxy geography. requestTimeoutSecs controls each request. Provide proxyConfiguration explicitly to select an authorized route. One query produces at most one record; a source may legitimately expose no AI Overview for a query. Queries that lack one are recorded in the run observations and never padded with other search modules.

Output and evidence

The parser selects the native turn container labelled AI Overview and its main answer column. It excludes citation-control labels and the separate organic results. It captures actual published inline citations, reference cards and answer links, deduplicates source URLs and retains the source occurrence count and relationship types. People Also Ask questions are labelled with their actual native origin and are not claimed as generated follow-up questions. Missing questions or citations are omitted instead of inventing values. No publisher page, image or citation destination is fetched.

answerCompleteness is source-response-snapshot: the text is what this response exposes. The Actor does not claim to expand every hidden section, wait for complete generation, reproduce personalized/account-only results or verify Google's statements as factual. Google-generated content may be wrong. Raw source scripts are never executed. Stored source HTML is sanitized to remove scripts, event handlers, images and unrelated token attributes while preserving the structural answer/query/citation evidence.

Native citation JSON annotations are parsed as data and matched to the exact citation widget UUID. They are used when a widget exposes its URL in the native annotation rather than an anchor. Only relevant non-executable citation annotations survive sanitization; unrelated comments and scripts are removed. Repeated anchor/annotation representations of the same widget are not double counted.

Records are buffered until the requested minimum is satisfied. Source denial or an insufficient genuine count causes a nonzero exit with no partial dataset publication. RUN_SUMMARY records individual observations and errors, the source transport, counts and timestamps. Nine meaningful regressions exercise query binding, module boundaries, native citation identity, URL safety, query/cap validation and script-free evidence. They are constructed checks rather than live-data proof. Run locally with apify run and inspect the actual dataset and same-run evidence before deployment.

Dependencies are pinned with a lockfile. A reviewed basic-ftp6.2.5 override removes the vulnerable5.x transitive proxy chain; the Actor itself never makes FTP requests. The SDK is retained at3.7.2. The dependency audit still reports three moderate SDK/Crawlee/minimatch findings; no forced SDK downgrade is applied.