Google Search Results (SERP) Scraper - AI Answer & Its Sources

Google SERP Scraper — Organic Results, PAA & AI Overviews

Google Search Results (SERP) Scraper - AI Answer & Its Sources runs your keywords through google.com and returns the full results page as structured JSON — organic results, paid results, paid products, People Also Ask, related searches, and the AI Overview Google now prints above the blue links, split into its ordered answer blocks with every source it cites. Every response is ready to pass directly to an LLM, index into a vector store, or feed a monitoring pipeline. As AI Overviews spread across more query types, tracking what they say — and who they cite — is becoming part of any serious search-visibility workflow.

What is Google Search Results (SERP) Scraper - AI Answer & Its Sources?

It is a Google search scraper that fetches the same HTML a browser gets for google.com/search, parses the classic results page, and — its defining capability — parses the AI Overview out of that same page into an ordered list of answer blocks plus the full list of pages it cites. No Google account, Google API key, or third-party SERP API key is required; the Actor authenticates only through Apify's own proxy.

• Full classic SERP: organicResults, paidResults, paidProducts, peopleAlsoAsk, relatedQueries, resultsTotal
• AI Overview parsed into ordered aiAnswerBlocks[] (paragraph, listItem, heading)
• Every AI Overview citation returned as aiAnswerSources[] with title, URL, domain, and the exact answer block it is attached to
• Query controls: search country (country / gl), interface language (hl), content-language restriction (lr), result-country restriction (cr), raw time filter (tbs), and pagination (start / page / limit)
• Live filtering: keep only keywords whose SERP actually produced an AI Overview (onlyKeywordsWithAiAnswer)
• Automatic proxy escalation — Apify's Google SERP proxy first, residential proxy when Google pushes back

⚠️ AI Overview capture and forceWebOnlyTab are mutually exclusive. Capturing the AI Overview means Google's Web-only tab parameter (udm=14) is not sent, because that tab never renders one. forceWebOnlyTab: true reproduces the base actor's exact request shape, but every aiAnswer* field then stays null — you get the AI answer or the base actor's exact request, never both on the same request. Separately, Google does not render an AI Overview for every query: informational, question-shaped searches get one far more often than navigational, shopping, or weather queries, and when there is none, hasAiAnswer is false with every aiAnswer* field null — never a faked or empty placeholder.

What data can you get with Google Search Results (SERP) Scraper - AI Answer & Its Sources?

Every run returns one row per keyword (per page, plus an optional merged row), each carrying the result types below.

Result Type	Extracted Fields	Primary Use Case
AI Overview answer	hasAiAnswer, aiAnswerBlocks[].order/type/text, aiOverview.text, aiOverview.references[]	AEO / GEO answer monitoring
AI Overview sources	aiAnswerSources[].position/title/url/displayedUrl/domain/attachedToBlock/snippet	Citation and brand-mention tracking
Organic results	organicResults[].title/url/displayedUrl/description/emphasizedKeywords/siteLinks/productInfo/type/position	Rank tracking
Paid results	paidResults[].title/url/displayedUrl/description/emphasizedKeywords/siteLinks/productInfo/type/position	Ad monitoring
Paid products	paidProducts[].title/displayedUrl/prices	Shopping-ad price tracking
People Also Ask	peopleAlsoAsk[].question/title/url/answer/date	Content gap analysis
Related searches	relatedQueries[].title/url, suggestedResults[].title/url/type/position	Keyword expansion
Search metadata	searchQuery.term/url/device/page/type/domain/countryCode/languageCode/locationUule/resultsPerPage	Run bookkeeping
Accounting rows	errorReason, errorMessage, charged	Failed / filtered / uncharged run tracking

AI Overview answer & sources

This is the field set the base scraper does not have. aiAnswerBlocks[] holds the AI Overview's text in the exact order Google renders it — each block tagged paragraph, listItem, or heading — instead of one flattened string. aiAnswerSources[] lists every page the answer cites, and attachedToBlock points each source back at the order of the specific block it was cited from, so you can tell which sentence of the answer a given source actually supports, not just that it appeared somewhere on the page. The aiOverview object (declared by the base scraper but never filled by it) is populated from the same parse:

"hasAiAnswer": true,
"aiAnswerBlocks": [
  { "order": 1, "type": "paragraph", "text": "A heat pump moves heat instead of creating it, using a refrigerant cycle to pull warmth from outside air, water, or ground." },
  { "order": 2, "type": "listItem", "text": "Compression: the compressor raises the refrigerant's pressure and temperature." }
],
"aiAnswerSources": [
  {
    "position": 1,
    "title": "What Is A Heat Pump And How Does It Work? - Carrier",
    "url": "https://www.carrier.com/residential/en/us/products/heat-pumps/what-is-a-heat-pump/",
    "displayedUrl": "www.carrier.com/residential/en/us/products/heat-pumps/what-is-a-heat-pump",
    "domain": "www.carrier.com",
    "attachedToBlock": 2,
    "snippet": "Compression: the compressor raises the refrigerant's pressure and temperature."
  }
],
"aiOverview": { "text": "A heat pump moves heat instead of creating it…", "references": [ { "title": "What Is A Heat Pump And How Does It Work? - Carrier", "url": "https://www.carrier.com/residential/en/us/products/heat-pumps/what-is-a-heat-pump/" } ] }

Not every AI Overview prints a citation list — when Google renders none, aiAnswerSources is null, never an empty stand-in.

Organic results

organicResults[] is the classic blue-link list: title, url, displayedUrl, description, siteLinks[] (each with its own title/url/description), emphasizedKeywords[], and position. It is deduplicated by URL within a page and renumbered after every page is added, so position always reflects the current row's actual result order — the field most rank-tracking workflows key off of when comparing this Actor's output against the AI Overview sitting above it.

How does Google Search Results (SERP) Scraper - AI Answer & Its Sources differ from Google's official Custom Search JSON API?

Google's own Custom Search JSON API does not return an AI Overview at all, and — checked 2026-08-16 — its documentation states the service is "closed to new customers," so it is not an option for most new integrations in the first place.

Feature	Google Custom Search JSON API	This Actor
AI Overview data	Not returned	aiAnswerBlocks[] + aiAnswerSources[]
Search scope	A Programmable Search Engine you configure, not the open web by default	Full google.com web results
Free quota	100 queries/day	No Google-imposed cap; billed per row_result event pushed
Cost beyond free tier	$5 per 1,000 queries, up to 10,000/day	Apify pay-per-event pricing on this Actor's Store listing
New signups	Documented as closed to new customers	Open to any Apify account
Ads / shopping data	Not returned	paidResults[] + paidProducts[]
Setup	Create and configure a Programmable Search Engine, then an API key	Paste keywords, click Start

The official API is the right choice if you already hold Custom Search API access and only need results confined to a search engine you've configured. For open Google web results, People Also Ask, paid listings, and the AI Overview together, this Actor is the more direct route.

How to scrape Google with Google Search Results (SERP) Scraper - AI Answer & Its Sources?

1. Open the Actor's page on the Apify Store and start a free Apify account if you don't have one.
2. Paste your search terms into keyword — one per line. This is the only required field.
3. Set the query controls you need: country / gl for the search region, hl for interface language, limit for how many pages of organic results to walk, tbs for a time filter.
4. Click Start.
5. Open the run's Dataset tab and download the results as JSON, CSV, or Excel, or pull them through the Apify API.

{
  "keyword": "how does a heat pump work",
  "limit": "10",
  "country": "US",
  "hl": "en",
  "includeAiAnswer": true,
  "includeAiAnswerSources": true
}

How to run multiple queries in one job

Every line in keyword is its own query. Paste a list of 500 questions and each one comes back as its own dataset row (or rows, one per organic page fetched, plus a merged row per keyword if include_merged is on) — there is no separate batch-upload field or query array; the newline-separated textarea is the batching mechanism.

⬇️ Input

Only keyword is required. Everything else narrows the search or controls how the AI Overview is captured.

Parameter	Required	Type	Description	Example Value
keyword	Yes	string	One search per line. Question-shaped queries get an AI Overview far more often than navigational ones. Default: "nike".	"how does a heat pump work\nbest running shoes"
include_merged	No	boolean	Adds one extra row per keyword merging every page (searchQuery.page = "all"). Default: true.	true
limit	No	string (enum: 10, 20, 30, 40, 50, 100, all)	How many pages of organic results to walk (10 per page). The AI Overview is read from page 1 only. Default: "all".	"10"
start	No	integer	Google start offset for the first result.	10
page	No	integer	Fetch only this page number.	1
country	No	string (enum: ISO 3166-1 alpha-2 country code)	Country the search runs from (gl) and the default proxy region.	"US"
gl	No	string (enum: ISO 3166-1 alpha-2 country code)	Override just the results country without changing the proxy region.	"GB"
hl	No	string (enum: interface-language code)	Google interface language — also the language the AI answer is written in.	"en"
tbs	No	string	Raw Google time filter.	"qdr:w"
lr	No	string (enum: lang_xx code)	Restrict results to one content language.	"lang_en"
cr	No	string (enum: countryXX code)	Restrict results to one country of origin.	"countryUS"
proxy_location	No	string (enum: us, ca)	Force the proxy exit region independently of the Google country.	"us"
includeAiAnswer	No	boolean	Parse the AI Overview into aiAnswerBlocks[] and fill aiOverview. While on, the Web-only tab (udm=14) is not requested. Default: true.	true
includeAiAnswerSources	No	boolean	Return each AI Overview citation as aiAnswerSources[]. Default: true.	true
onlyKeywordsWithAiAnswer	No	boolean	Drop a keyword whose SERP carried no AI Overview instead of pushing empty AI fields. Default: false.	false
forceWebOnlyTab	No	boolean	Send udm=14 on every request — the base actor's exact request shape. No AI Overview exists on that tab. Default: false.	false
dedupeResultsAcrossPages	No	boolean	Each page row carries only that page's new organic results instead of repeating pages 1..n cumulatively. Default: false.	false

{
  "keyword": "how does a heat pump work\nbest running shoes\nis oat milk healthy",
  "include_merged": true,
  "limit": "10",
  "country": "US",
  "hl": "en",
  "includeAiAnswer": true,
  "includeAiAnswerSources": true,
  "onlyKeywordsWithAiAnswer": false
}

Common pitfall: limit: "all" walks up to 100 pages per keyword (confirmed in the Actor's source), and every page pushed is a billed row_result event. If you only need the AI Overview and its sources, set limit low (or page: 1) — the AI Overview is always read from page 1 regardless of limit, so a smaller organic crawl never costs you the AI answer.

⬆️ Output

Every row is typed, consistent JSON with the same schema across runs — base SERP fields plus the AI Overview fields, whether or not a given SERP actually carried an AI Overview. Results are downloadable as JSON, CSV, or Excel from the run's Dataset tab, or fetched through the Apify API.

Scraped results

[
  {
    "searchQuery": { "term": "how does a heat pump work", "url": "https://www.google.com/search?q=how+does+a+heat+pump+work&gl=us&hl=en", "device": "DESKTOP", "page": 1, "type": "SEARCH", "domain": "google.com", "countryCode": "US", "languageCode": "en", "locationUule": null, "resultsPerPage": 10 },
    "searchQueryTerm": "how does a heat pump work",
    "url": "https://www.google.com/search?q=how+does+a+heat+pump+work&gl=us&hl=en",
    "hasNextPage": true,
    "serpProviderCode": "O",
    "resultsTotal": 158000,
    "relatedQueries": [ { "title": "heat pump vs furnace", "url": "https://www.google.com/search?q=heat+pump+vs+furnace" } ],
    "paidResults": [],
    "paidProducts": [],
    "aiOverview": { "text": "A heat pump moves heat instead of creating it…", "references": [ { "title": "What Is A Heat Pump And How Does It Work? - Carrier", "url": "https://www.carrier.com/residential/en/us/products/heat-pumps/what-is-a-heat-pump/" } ] },
    "organicResults": [
      { "title": "How Do Heat Pumps Work? - Energy.gov", "url": "https://www.energy.gov/energysaver/heat-pump-systems", "displayedUrl": "www.energy.gov/energysaver/heat-pump-systems", "description": "A heat pump uses electricity to move heat from a cool space to a warm space...", "emphasizedKeywords": ["how does a heat pump work"], "siteLinks": [], "productInfo": {}, "type": "organic", "position": 1 },
      { "title": "What Is A Heat Pump And How Does It Work? - Carrier", "url": "https://www.carrier.com/residential/en/us/products/heat-pumps/what-is-a-heat-pump/", "displayedUrl": "www.carrier.com/residential/en/us/products/heat-pumps/what-is-a-heat-pump", "description": "Heat pumps work by transferring heat rather than generating it...", "emphasizedKeywords": ["how does a heat pump work"], "siteLinks": [], "productInfo": {}, "type": "organic", "position": 2 },
      { "title": "Heat pump - Wikipedia", "url": "https://en.wikipedia.org/wiki/Heat_pump", "displayedUrl": "en.wikipedia.org/wiki/Heat_pump", "description": "A heat pump is a device that transfers thermal energy from a source of heat...", "emphasizedKeywords": ["how does a heat pump work"], "siteLinks": [], "productInfo": {}, "type": "organic", "position": 3 }
    ],
    "suggestedResults": [ { "title": "heat pump vs furnace", "url": "https://www.google.com/search?q=heat+pump+vs+furnace", "type": "organic", "position": 1 } ],
    "peopleAlsoAsk": [ { "answer": null, "question": "Is a heat pump cheaper to run than a furnace?", "title": "Is a heat pump cheaper to run than a furnace?", "url": null, "date": null } ],
    "customData": { "perplexitySearch": { "enablePerplexity": false, "searchRecency": null, "returnImages": false, "returnRelatedQuestions": false }, "chatGptSearch": { "enableChatGpt": false }, "maximumLeadsEnrichmentRecords": 0, "leadsEnrichmentDepartments": [] },
    "htmlSnapshotUrl": null,
    "hasAiAnswer": true,
    "aiAnswerBlocks": [
      { "order": 1, "type": "paragraph", "text": "A heat pump moves heat instead of creating it, using a refrigerant cycle to pull warmth from outside air, water, or ground." },
      { "order": 2, "type": "listItem", "text": "Compression: the compressor raises the refrigerant's pressure and temperature." }
    ],
    "aiAnswerSources": [
      { "position": 1, "title": "What Is A Heat Pump And How Does It Work? - Carrier", "url": "https://www.carrier.com/residential/en/us/products/heat-pumps/what-is-a-heat-pump/", "displayedUrl": "www.carrier.com/residential/en/us/products/heat-pumps/what-is-a-heat-pump", "domain": "www.carrier.com", "attachedToBlock": 2, "snippet": "Compression: the compressor raises the refrigerant's pressure and temperature." }
    ]
  }
]

customData, htmlSnapshotUrl, and the optional html field are inherited from the base actor's contract for output-shape compatibility; this variant's input schema does not expose the base actor's advanced options (site/filetype operators, HTML snapshotting, product icons, a third-party SerpApi key), so those keys always carry their off/default values shown above.

A keyword that is blocked, unparseable, or dropped by onlyKeywordsWithAiAnswer produces an uncharged accounting row instead of a normal row:

{
  "errorReason": "FILTERED_NO_AI_ANSWER",
  "errorMessage": "onlyKeywordsWithAiAnswer=true and this SERP carried no AI Overview.",
  "charged": false,
  "searchQueryTerm": "best running shoes",
  "searchQuery": { "term": "best running shoes", "url": "", "device": "DESKTOP", "page": 1, "type": "SEARCH", "domain": "google.com", "countryCode": "US", "languageCode": "en", "locationUule": null, "resultsPerPage": 0 },
  "url": "",
  "hasAiAnswer": false,
  "aiAnswerBlocks": null,
  "aiAnswerSources": null
}

errorReason also covers SORRY_REDIRECT, JS_CHALLENGE_BOOTSTRAP, CHALLENGE_PAGE, NO_RESULT_CONTAINER, FETCH_FAILED, and NO_RESULTS_AFTER_RETRIES when Google's proxy ladder is exhausted. Only rows pushed under the row_result charged event are billed; accounting rows always carry charged: false and are visible on their own in the built-in Uncharged accounting rows dataset view.

How can I use the data extracted with Google Search Results (SERP) Scraper - AI Answer & Its Sources?

• AEO / GEO and SEO teams: Diff aiAnswerSources[].domain against your own domain across a keyword set to see which category questions already cite you — and which cite competitors instead.
• AI engineers and LLM developers: Have an agent issue a query, receive aiAnswerBlocks[] and organicResults[] as structured JSON, and pass them to the model as grounding context before it answers.
• Market researchers: Track aiAnswerSources[].domain and organicResults[].url across a keyword set over time to build a share-of-citation picture for a category.
• Content teams: Read aiAnswerBlocks[].text and peopleAlsoAsk[].question together to see the sub-topics Google's own answer and its related questions treat as part of a complete answer — a direct content-gap signal.

How do you monitor AI Overview citations over time?

Citation tracking is the discipline of watching which domains Google's AI Overview cites for a keyword, and noticing when that list changes. Because hasAiAnswer, aiAnswerBlocks[], and aiAnswerSources[] are re-parsed fresh on every run, comparing two runs of the same keyword set tells you whether an AI Overview appeared where it didn't before, whether your domain entered or dropped out of aiAnswerSources[].domain, and whether the specific block your page was attachedToBlock to changed. The fields worth diffing run-to-run are hasAiAnswer (did an answer appear at all), aiAnswerSources[].domain (who is cited now), and aiAnswerBlocks[].text (did the claim itself change).

A practical loop: run the same keyword list on a schedule, keep the previous run's dataset, and compare aiAnswerSources[].domain sets between runs — alert when a domain you track leaves the list, or when a new competitor domain enters it. Apify's built-in Schedule feature (cron-based, available on every Apify account) is the mechanism for the recurring run; the diff itself happens outside the Actor, against the datasets it produces.

Integrate Google Search Results (SERP) Scraper - AI Answer & Its Sources and automate your workflow

Google Search Results (SERP) Scraper - AI Answer & Its Sources works with any language or tool that can send an HTTP request.

REST API with Python

import requests

TOKEN = "YOUR_APIFY_TOKEN"
ACTOR = "YOUR_USERNAME~google-search-results-serp-scraper-ai-answer-and-its-sources"

resp = requests.post(
    f"https://api.apify.com/v2/acts/{ACTOR}/run-sync-get-dataset-items",
    params={"token": TOKEN},
    json={"keyword": "how does a heat pump work", "limit": "10"},
)
rows = resp.json()
for row in rows:
    if row.get("hasAiAnswer"):
        print(row["searchQueryTerm"], "->", len(row["aiAnswerSources"] or []), "sources")

MCP for query-grounded AI agents

The Actor can be exposed as a callable tool through Apify's MCP server at https://mcp.apify.com?tools=YOUR_USERNAME/google-search-results-serp-scraper-ai-answer-and-its-sources, or registered with the Apify CLI (apify mcp install claude-code / cursor / vscode). An agent issues the query, receives the AI Overview and organic results as structured JSON through the tool call, and grounds its answer in that JSON instead of its own training data. Compatible with Claude Desktop, Claude Code, Cursor, and any MCP-aware client.

Scheduled monitoring and delivery

Apify's Schedule feature runs this Actor on a cron interval without any code on your side. Combine it with Apify's dataset export or webhook integrations to have each run's results delivered — or picked up by a downstream job — the moment a run finishes.

Is it legal to scrape Google search results?

Yes — scraping publicly accessible Google search results is generally lawful, since this Actor returns only what any visitor's browser already receives when it loads the same page, without logging in or bypassing any access control. In hiQ Labs v. LinkedIn (9th Cir. 2019), the court held that scraping data a website makes publicly available does not violate the U.S. Computer Fraud and Abuse Act. Google search results are business/public data, not personal data about the person running the search, so this falls under Google's Terms of Service and database-rights considerations rather than GDPR or CCPA. Scraping for operational monitoring (rank tracking, citation tracking) and scraping to build a training dataset for a model carry different risk profiles and different ToS exposure — treat them separately. Consult your legal team for commercial use cases involving bulk data storage.

Frequently asked questions

Which countries and languages does this Actor support?

Any ISO 3166-1 alpha-2 country code through country, gl, or cr, and any of the listed interface-language codes through hl or lr — the full lists are in the input editor's dropdowns. The AI Overview and its citations differ by both, so keep gl/hl fixed when comparing keywords.

How do I control how many results come back?

limit sets how many pages of organic results are walked — 10 through 100, or all (up to 100 pages). start and page let you jump to a specific offset or fetch a single page instead of a whole crawl. None of these change the AI Overview, which is always read from page 1.

How does this Actor handle Google's anti-bot measures?

It fetches through Apify's Google SERP proxy first, and on a detected block escalates in order: a fresh Google SERP proxy session, then Apify's residential proxy pool, retrying up to the Actor's built-in retry ladder. A response is only treated as a genuine block after checking it contains none of Google's real result containers — this specifically catches Google's ~92 KB JavaScript-challenge bootstrap page (which returns HTTP 200 with zero results), instead of mistaking it for a successful, empty result.

Does this Actor extract the AI Overview?

Yes — as aiAnswerBlocks[] (the answer's ordered paragraphs, list items, and headings) and aiAnswerSources[] (every page it cites, with attachedToBlock linking each source back to its block). Both are null when Google renders no AI Overview for that query, or when forceWebOnlyTab is on.

How many results does this Actor return per query?

Up to 100 pages of organic results per keyword (10 results per page) when limit is "all", or fewer when you set limit to a specific number. The AI Overview and its sources are captured once per keyword, from page 1, regardless of limit.

How do I use this Actor to monitor AI Overview citations over time?

Run the same keyword list on a schedule, extract aiAnswerSources[].domain and hasAiAnswer from each run, and diff them against the previous run's dataset — alert when a domain you track enters or leaves the citation list. Trigger the recurring run with Apify's Schedule feature.

Does this Actor work with Claude, ChatGPT, and AI agent frameworks?

Yes. It is reachable through Apify's MCP server (https://mcp.apify.com?tools=...) for MCP-aware clients like Claude Desktop and Claude Code, and it is callable as a plain HTTP endpoint through the Apify API by any agent framework that can make a request — which enables agents that retrieve a live Google SERP and AI Overview before answering.

How does this Actor differ from the base Google Search Results (SERP) Scraper?

It is a strict superset: every base input field and every base output field is preserved, and it adds hasAiAnswer, aiAnswerBlocks[], and aiAnswerSources[] parsed from the same page. It also fills the base actor's aiOverview object (declared but never populated by the base actor) and reads the genuine "Related searches" block for relatedQueries instead of a synthesized substitute.

Can I use this Actor without managing proxies or Google credentials?

Yes. Proxy selection and escalation (Google SERP proxy, then residential) is handled automatically by the Actor. You need only an Apify account to run it — no Google account, Google Cloud project, or third-party SERP API key is required.

Your feedback

Found a bug or missing a field? Let Scrapier know through the Actor's Issues tab on its Apify Store listing — reports of selector drift or a misparsed field are what keep this Actor accurate as Google changes its markup.