# Google AI Overview Scraper – GEO - Sources & Citations (`automation_nerd/google-ai-overview-actor`) Actor Scrape Google AI Overviews and AI Mode answers at scale. Extract the full AI summary, cited sources and domains for any keyword — built for GEO & SEO monitoring. - **URL**: https://apify.com/automation\_nerd/google-ai-overview-actor.md - **Developed by:** [Egon Maier](https://apify.com/automation_nerd) (community) - **Categories:** SEO tools, Automation, AI - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $10.00 / 1,000 query processed and ai answer analyseds This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Google AI Overview Scraper – GEO, Sources & Citations **Track what Google's AI says about your brand — and who it cites.** This actor runs any list of search queries on Google and extracts the **AI Overview** answer directly from the results page. If no AI Overview is shown, it automatically switches to Google's **AI Mode** and captures the answer there. For every query you get the full response as **text and Markdown**, the **cited sources** (URL, title, domain) and an aggregated list of **cited domains** — the core metric for **Generative Engine Optimization (GEO)**. **Use it to:** - **Monitor AI search visibility** — is your brand mentioned and cited in Google's AI answers? - **Track competitors** — which domains does Google cite for your money keywords? - **GEO & SEO reporting** — measure AI Overview coverage across hundreds of keywords - **Content research** — see exactly what Google's AI summarizes for a topic ### Features - Batch processing: analyze many prompts in one run - Extracts the full (expanded) AI Overview text as plain text **and** markdown - **AI Mode fallback**: if no AI Overview is shown by default, switches to Google's AI Mode tab - Extracts all cited sources (`{ position, text, url, domain }`) plus aggregated `sourceDomains` - Detects prompts that do not trigger any AI answer (`answerSource: null`) - Language (`hl`) and country (`gl`) targeting - Residential proxies with selectable country, stealth browsing via Camoufox - Captcha handled as a real failure (retries with a fresh session; `success: false` if persistent) ### Input | Field | Type | Description | | ---------------- | ---------------- | ----------------------------------------------------------------------------------------------------------------------- | | `prompts` | array (required) | Search queries to analyze (alias: `queries`) | | `languageCode` | string | Google UI language (`hl`), e.g. `en`, `de` (default `en`) | | `countryCode` | string | Result country (`gl`), e.g. `us`, `de` (optional) | | `aiModeFallback` | boolean | If no AI Overview is shown by default, switch to Google's **AI Mode** tab and extract the answer there (default `true`) | | `maxConcurrency` | integer | How many browser sessions to run in parallel (default `3`; higher = faster, needs more memory) | | `maxRequestRetries` | integer | Retries **per session** on block/captcha — fresh browser + IP each retry; already-answered queries are skipped (default `3`) | | `promptsPerSession` | integer | Queries per browser before a fresh browser/IP (default `5`). The cold start is the risky moment; an established session handles follow-up searches fine, and a mid-session block safely retries only the remaining queries | | `proxyCountry` | string | Residential proxy country; should match `gl` (default `US`, empty also falls back to US) | ### Output One dataset item per prompt. The schema is aligned with the ChatGPT Prompt Actor (`prompt`, `success`, `responseText`, `responseMarkdown`, `sources`, `sourceDomains`, …) so results from both actors combine easily. - `prompt`: the search query - `success`: `true` if the prompt was processed, `false` on failure (e.g. persistent captcha) - `error`: `null` on success, otherwise an error marker (`CAPTCHA`, `REQUEST_FAILED_AFTER_RETRIES`, …) - `attempts`: number of tries it took (including the final one) - `hasAiOverview`: whether Google showed an AI Overview **by default** on the search results page - `answerSource`: where the answer came from — `ai_overview` (shown by default), `ai_mode` (only after switching to the AI Mode tab, i.e. NO AI summary was shown by default), or `null` (no AI answer at all) - `responseText`: the AI answer as plain text (expanded AI Overview or AI Mode answer), or `null` - `responseMarkdown`: the AI answer with formatting preserved (headings, lists, tables), or `null` - `sources`: array of `{ position, text, url, domain }` cited by the answer (URLs normalized, deduplicated) - `sourceDomains`: unique cited domains (aggregated GEO metric) - `responseTimeMs`, `promptIndex`, `totalPrompts`, `timestamp`, `url` (the AI Mode URL when the fallback was used) ### Pricing This Actor uses **pay per event + usage** pricing: - One **`query-processed`** event per **successfully processed** prompt. Failed prompts (captcha, crash) are **not** charged. - Apify **platform usage** (residential proxy traffic, compute) is billed separately to you at cost — so you always see exactly what a run consumes. ### Reliability (best-effort) Google actively rate-limits and blocks automated search. This Actor uses residential proxies with fresh IP rotation and stealth browsing to maximize success, but **scraping Google AI answers is best-effort, not guaranteed** — success rates vary by time, region and query. Blocked or captcha'd prompts are reported cleanly as `success: false` with an `error` (and are **not** charged), so you can retry them later. ### Notes - AI Overviews are not shown for every query (e.g. navigational queries) and their availability varies by country and language. - Whether an AI Overview appears can also depend on the IP location of the proxy — set `proxyCountry` to match your target market. # Actor input Schema ## `prompts` (type: `array`): Google search queries to analyze for AI Overview answers ## `languageCode` (type: `string`): Google UI language, passed as the `hl` parameter (e.g. en, de, fr). Affects which language the AI Overview is generated in. ## `countryCode` (type: `string`): Country for search results, passed as the `gl` parameter (e.g. us, de, fr). Leave empty for automatic. ## `aiModeFallback` (type: `boolean`): If no AI Overview is shown on the default search results page, switch to Google's AI Mode tab and extract the answer from there. The output marks such results with answerSource = 'ai_mode'. ## `maxConcurrency` (type: `integer`): How many prompts to process in parallel. Higher = faster, but each parallel prompt uses ~1 GB RAM and its own residential IP. Increase the Actor's memory accordingly (about (N+1) GB for N parallel prompts). Total run cost stays roughly the same — you mostly save wall-clock time. ## `maxRequestRetries` (type: `integer`): How often to retry a browser session (a chunk of 'promptsPerSession' queries) that gets blocked/captcha'd. Each retry uses a fresh browser and a new residential IP; queries already answered in the session are skipped, so effectively the failing query gets retried on a new IP. The attempts made are reported per item as 'attempts'. ## `promptsPerSession` (type: `integer`): How many search queries to run in one browser session before starting a fresh browser/IP. The risky moment is the cold start (fresh IP/browser); once past it, follow-up searches with a human-like pause behave like a normal user. A higher value also amortizes the cold start over more queries. A mid-session block safely retries only the remaining queries on a new IP. Default 5. ## `proxyCountry` (type: `string`): Country the residential proxy should be located in. Should usually match the `gl` country. Defaults to US (recommended for the most reliable AI Overviews). Empty also falls back to US. ## Actor input object example ```json { "prompts": [ "how does photosynthesis work", "best running shoes for beginners" ], "languageCode": "en", "countryCode": "", "aiModeFallback": true, "maxConcurrency": 3, "maxRequestRetries": 3, "promptsPerSession": 5, "proxyCountry": "US" } ```` # Actor output Schema ## `results` (type: `string`): All results as JSON (one item per query: hasAiOverview, aiOverviewText, sources, …) ## `run` (type: `string`): API endpoint of this run (status, stats, storages) # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "prompts": [ "how does photosynthesis work", "best running shoes for beginners" ] }; // Run the Actor and wait for it to finish const run = await client.actor("automation_nerd/google-ai-overview-actor").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "prompts": [ "how does photosynthesis work", "best running shoes for beginners", ] } # Run the Actor and wait for it to finish run = client.actor("automation_nerd/google-ai-overview-actor").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "prompts": [ "how does photosynthesis work", "best running shoes for beginners" ] }' | apify call automation_nerd/google-ai-overview-actor --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=automation_nerd/google-ai-overview-actor", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Google AI Overview Scraper – GEO - Sources & Citations", "description": "Scrape Google AI Overviews and AI Mode answers at scale. Extract the full AI summary, cited sources and domains for any keyword — built for GEO & SEO monitoring.", "version": "0.1", "x-build-id": "ZKO3I7WeajfRRN8wZ" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/automation_nerd~google-ai-overview-actor/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-automation_nerd-google-ai-overview-actor", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/automation_nerd~google-ai-overview-actor/runs": { "post": { "operationId": "runs-sync-automation_nerd-google-ai-overview-actor", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/automation_nerd~google-ai-overview-actor/run-sync": { "post": { "operationId": "run-sync-automation_nerd-google-ai-overview-actor", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "prompts" ], "properties": { "prompts": { "title": "Prompts (search queries)", "type": "array", "description": "Google search queries to analyze for AI Overview answers", "default": [ "how does photosynthesis work" ], "items": { "type": "string" } }, "languageCode": { "title": "Language (hl)", "type": "string", "description": "Google UI language, passed as the `hl` parameter (e.g. en, de, fr). Affects which language the AI Overview is generated in.", "default": "en" }, "countryCode": { "title": "Country (gl)", "type": "string", "description": "Country for search results, passed as the `gl` parameter (e.g. us, de, fr). Leave empty for automatic.", "default": "" }, "aiModeFallback": { "title": "AI Mode fallback", "type": "boolean", "description": "If no AI Overview is shown on the default search results page, switch to Google's AI Mode tab and extract the answer from there. The output marks such results with answerSource = 'ai_mode'.", "default": true }, "maxConcurrency": { "title": "Max concurrency (parallel prompts)", "minimum": 1, "maximum": 20, "type": "integer", "description": "How many prompts to process in parallel. Higher = faster, but each parallel prompt uses ~1 GB RAM and its own residential IP. Increase the Actor's memory accordingly (about (N+1) GB for N parallel prompts). Total run cost stays roughly the same — you mostly save wall-clock time.", "default": 3 }, "maxRequestRetries": { "title": "Max retries per session", "minimum": 0, "maximum": 15, "type": "integer", "description": "How often to retry a browser session (a chunk of 'promptsPerSession' queries) that gets blocked/captcha'd. Each retry uses a fresh browser and a new residential IP; queries already answered in the session are skipped, so effectively the failing query gets retried on a new IP. The attempts made are reported per item as 'attempts'.", "default": 3 }, "promptsPerSession": { "title": "Prompts per browser session", "minimum": 1, "maximum": 20, "type": "integer", "description": "How many search queries to run in one browser session before starting a fresh browser/IP. The risky moment is the cold start (fresh IP/browser); once past it, follow-up searches with a human-like pause behave like a normal user. A higher value also amortizes the cold start over more queries. A mid-session block safely retries only the remaining queries on a new IP. Default 5.", "default": 5 }, "proxyCountry": { "title": "Proxy Country", "enum": [ "US", "", "GB", "DE", "FR", "CA", "AU", "NL", "SE", "JP", "BR", "IN", "IT", "ES", "PL", "BE", "CH", "AT", "IE", "PT", "DK", "NO", "FI" ], "type": "string", "description": "Country the residential proxy should be located in. Should usually match the `gl` country. Defaults to US (recommended for the most reliable AI Overviews). Empty also falls back to US.", "default": "US" } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```