Google Search Scraper — SERP, Qualified Leads & AI Overview

Google Search Results Scraper

Scrape Google Search result pages (SERPs) and get clean, structured JSON — organic results, related searches, People Also Ask, knowledge panels, and total result counts — for any query, country, and language.

Built for reliability: a dual-engine HTTP fetcher with real browser TLS impersonation, an 8-deep self-healing selector chain that survives Google's layout rotations, and structural soft-block detection that won't false-reject a valid SERP.

What it extracts

• Organic results — position, title, URL, displayed URL, description, date, emphasized keywords, sitelinks, and inline product info (rating / reviews / price)
• Paid ads — the text ads above and below the results, with placement (top or bottom), title, the advertiser's landing URL, displayed URL, description and sitelinks
• Related searches — the "people also search for" / bottom-of-page queries
• People Also Ask — the expandable Q&A questions (with answer text when present)
• Knowledge panel — title, subtitle, description, fact attributes (Born, Died, Address, Height, …), and source link
• AI Overview — detection + the cited source domains for GEO/AEO tracking (the generated prose itself is not available over the proxy — see Limitations)
• Total results — the "About N results" count
• Contact emails (opt-in) — turn a SERP into leads: scrape each result's own site for a contact email + people. See Lead enrichment below.
• Perplexity AI answer (opt-in, BYOK) — a grounded AI answer with citations per query, via Perplexity's official API with your key. See Perplexity AI answer.

Coming from another Google search scraper?

If you're switching because another Google SERP scraper let you down, here's how this one handles the things people most often run into:

What tends to go wrong elsewhere	How this scraper handles it
Runs return 0 organic results on some queries, or sit empty for minutes	Dual HTTP engine (browser-grade TLS fingerprint + a fallback), an 8-deep self-healing selector chain that survives Google layout changes, and structural soft-block detection. An optional residential transport takes over if the SERP proxy struggles.
site: and other operators return unrelated domains	site:, relatedToSite, intitle:, intext:, inurl:, filetype:, date range and exact match are real input fields, folded into the query correctly — site:github.com returns only that domain.
Setting a results language (lr=) wipes out the organic results	searchLanguage returns a full results page — tested with German (de) returning 10 organic results.
Lead enrichment bills twice for the same contact in one run	Each business domain is scraped and billed once per run. A domain that shows up on several pages or queries reuses its cached contact for free — no second scrape, no second charge, and a page retry can't re-charge it.
AI Overview text comes back truncated or missing	You get AI Overview detection plus the cited source domains reliably (useful for GEO/AEO). The generated prose is produced client-side by Google and needs a JS browser, so the output says so plainly rather than passing a truncated "Show more" preview off as the answer.

Input

Field	Type	Default	Notes
queries	string[]	—	One or more search terms. Required.
maxPagesPerQuery	integer	1	Pages to fetch per query (~10 results each).
resultsPerPage	integer	10	See the note under Limitations — Google caps this at ~10.
countryCode	string	US	Two-letter country (sets the Google domain, gl and the search location).
languageCode	string	en	Interface/results language (hl).
safeSearch	off/medium/high	off	SafeSearch level.
includeRelated	boolean	true	Extract related searches + People Also Ask.
includeKnowledgePanel	boolean	true	Extract the knowledge panel.
includeAiOverview	boolean	true	Detect AI Overview + extract its cited sources. Needs appendUdm14: false.
includeAds	boolean	true	Extract the text ads at the top and bottom of the page (see Limitations).
appendUdm14	boolean	true	Use Google's clean "Web" layout. Set false to see AI Overviews. Recommended on for max organic stability.
useAdsBotUA	boolean	false	Send the AdsBot-Google user agent.
maxConcurrency	integer	5	Parallel requests.
maxRequestRetries	integer	5	Retries per page before giving up.
proxy	object	GOOGLE_SERP	Proxy config. Leave default — see Limitations.

Advanced search filters

Google search operators exposed as structured fields (all folded into the query / URL — you can also type the operators directly inside a queries entry):

Field	Type	Operator / param	Notes
site	string	site:	Restrict to one site. Wins over relatedToSite.
relatedToSite	string	related:	Pages related to a site.
wordsInTitle	string[]	intitle:	Multi-word entries auto-quoted.
wordsInText	string[]	intext:	Words required in body text.
wordsInUrl	string[]	inurl:	Words required in the URL.
fileTypes	string[]	filetype:	OR-combined (pdf, docx, …).
forceExactMatch	boolean	"…"	Wrap the query in quotes.
searchLanguage	string	lr=lang_XX	Results language (distinct from hl).
locationUule	string	uule=	Exact location: a uule code or a place name like Berlin,Berlin,Germany. Defaults to the country.
quickDateRange	string	tbs=qdr:	Relative recency: h, d10, w, m6, y1.
afterDate / beforeDate	string	tbs=cdr:	Absolute bounds (YYYY-MM-DD or MM/DD/YYYY).
includeUnfilteredResults	boolean	filter=0	Include Google's omitted near-duplicates (off by default).
saveHtml	boolean	—	Attach raw SERP HTML to each row under html (large).
saveHtmlToKeyValueStore	boolean	—	Save HTML to the KV store; add htmlSnapshotUrl to each row.

Lead enrichment (opt-in)

Turn search results into leads. When enrichEmails: true, the top business domains in each SERP get their own site scraped for a contact email + people; the hits are attached to those organic results as contactEmail and emailEnrichment.

Field	Type	Default	Notes
enrichEmails	boolean	false	Enable contact-email enrichment.
maxEnrichedDomainsPerPage	integer	3	Cap unique business domains enriched per page.

• Best on business-intent queries ("commercial plumbers chicago", "B2B SaaS vendors").
• Non-business domains (search engines, social, Wikipedia, big marketplaces) are skipped; domains are de-duped per page.
• Best-effort and free (direct site scrape) — enrichment never fails a scrape.

Perplexity AI answer (opt-in, bring-your-own-key)

Attach a Perplexity (Sonar) AI answer to each query — grounded, with citations. This calls Perplexity's official API with your own key, so you are billed by Perplexity per request (it is not scraping). The answer lands on the page-1 row under perplexity { answer, model, citations, searchResults?, relatedQuestions? }.

Field	Type	Default	Notes
perplexitySearch	boolean	false	Enable the Perplexity answer.
perplexityApiKey	string (secret)	—	Your key from docs.perplexity.ai. Required.
perplexitySearchRecency	day/week/month/year	—	Restrict grounding to recent content.
perplexityRelatedQuestions	boolean	false	Also return follow-up questions.

Inert unless both the toggle is on and a key is supplied; a Perplexity error never breaks the SERP scrape.

Example input

{
  "queries": ["apify pricing 2026", "best web scraping tools"],
  "maxPagesPerQuery": 2,
  "countryCode": "US",
  "languageCode": "en"
}

Output

One dataset item per (query, page):

{
  "query": "apify pricing 2026",
  "page": 1,
  "searchUrl": "http://www.google.com/search?q=apify+pricing+2026&...",
  "organicResults": [
    {
      "type": "organic",
      "position": 1,
      "title": "Apify pricing - plans for data collection at any scale",
      "url": "https://apify.com/pricing",
      "displayedUrl": "https://apify.com › pricing",
      "description": "...",
      "siteLinks": []
    }
  ],
  "paidResults": [],
  "relatedQueries": [{ "title": "apify free plan", "url": "https://www.google.com/search?q=..." }],
  "peopleAlsoAsk": [{ "question": "Is Apify free to use?", "answer": "...", "url": "https://..." }],
  "knowledgePanel": null,
  "aiOverview": { "detected": true, "textAvailable": false, "sources": [{ "url": "https://ibm.com/...", "domain": "ibm.com" }] },
  "totalResults": 1230000,
  "selectorUsed": "div.tF2Cxc",
  "scrapedAt": "2026-06-25T09:00:00.000Z"
}

selectorUsed tells you which parser variant matched — handy for spotting future Google layout drift.

How it works

• Dual-engine fetch — tries impit (Rust, real Chrome TLS + HTTP/2 fingerprint) first, falling back to got-scraping. Two independent fingerprints = two chances against bot detection.
• udm=14 clean layout — by default the scraper requests Google's "Web" tab, which strips the AI Overview and returns stable, parseable markup.
• Self-healing selectors — organic results run through an 8-deep selector chain (2026 containers first, classic layouts as fallback, plus a class-agnostic a:has(h3) structural net), so a single Google class rename won't zero out a run.
• Structural soft-block detection — a page is only treated as blocked if it doesn't structurally look like a SERP, so benign /sorry/ footer links never cause a false "blocked".

Limitations & notes

• Use the default GOOGLE_SERP proxy. Google aggressively blocks datacenter and residential IPs for SERP scraping (JS-challenge / CAPTCHA interstitials). This actor is built around Apify's GOOGLE_SERP proxy group, which is purpose-built for Google and applied automatically. Overriding proxy to RESIDENTIAL or datacenter groups will fail with soft-blocks — leave the proxy at its default.

• Paid ads (paidResults) appear when Google shows them. Each entry is one text ad, in page order (position), with placement: "top" (above the organic results) or "bottom" (below them). url is the advertiser's landing page rather than Google's click link: the full landing URL when the ad carries one (Google's click IDs such as gclid removed), otherwise the advertiser's site. Sitelinks get the advertiser URL when the ad carries it, otherwise an absolute google.com click link. The actor never opens an ad link, so scraping does not register clicks the advertiser pays for. Google decides per page load whether to show ads, so the same commercial query can return ads on one run and an empty array on the next. Shopping, hotel and sponsored-places units are not included.

• AI Overview prose is not available over the proxy. Google server-renders the AI Overview shell (label + citation cards) but generates the answer text client-side, so a no-JS request receives a placeholder. aiOverview therefore returns detected + the cited sources (useful for GEO/AEO — which domains Google cites) with textAvailable: false. The prose itself needs a JS-rendering path. Requires appendUdm14: false (the default udm=14 strips AI Overviews entirely).

• resultsPerPage is effectively ~10. Google ignores the num parameter on most modern SERP layouts and returns ~10 results per page. To collect more, raise maxPagesPerQuery rather than resultsPerPage. Pagination always steps by the real page size, so no results are skipped if you set a larger value.

• Knowledge-panel images are lazy-loaded by Google's JS, so imageUrl may be absent even when the panel is otherwise fully populated.

• People Also Ask answers are only present when Google pre-expands the card; for most queries you'll get the question and destination URL but no answer text.

• AI Overview is intentionally excluded by appendUdm14: true. Set it to false if you want the default SERP layout (which can include the AI Overview and more ad/PAA surfaces), at the cost of slightly less stable organic markup.

Tips

• Keep appendUdm14 on for the most reliable organic extraction.
• For deep coverage, set maxPagesPerQuery (e.g. 5) instead of resultsPerPage.
• Use countryCode + languageCode together for accurate localized results (e.g. DE + de, JP + ja).
• Google search operators work inside queries (site:, intitle:, "exact", OR, -exclude, filetype:).

🤖 For AI Agents & LLM Apps

Compact reference for AI agents calling this actor via the Apify MCP server or the Apify API (actor: memo23/google-search-scraper).

Purpose: Scrape structured Google SERPs for any query, country and language — organic results, related searches, People Also Ask, knowledge panel, AI-Overview cited sources and total counts — with a self-healing selector chain that survives Google's layout rotations.

Minimal input:

{
  "queries": ["best web scraping tools"],
  "maxPagesPerQuery": 1
}

Output: one row per (query, page) — query, page, searchUrl, organicResults {type, position, title, url, displayedUrl, description, siteLinks}, paidResults {type, position, placement, title, url, displayedUrl, description, siteLinks}, relatedQueries {title, url}, peopleAlsoAsk {question, answer, url}, knowledgePanel, aiOverview {detected, textAvailable, sources {url, domain}}, totalResults, selectorUsed, scrapedAt.

Behaviors an agent should know:

• queries is the only required field; one row per query × page. To collect more results, raise maxPagesPerQuery (default 1) — Google caps resultsPerPage at ~10 regardless of the value set.
• Leave proxy at its default GOOGLE_SERP group; overriding to residential/datacenter groups fails with soft-blocks.
• appendUdm14 (default true) gives stable organic markup but strips AI Overview; set it false (with includeAiOverview) to get the AI-Overview sources — the generated prose is never available over the proxy.
• paidResults lists the page's text ads with placement "top" or "bottom"; url is the advertiser's landing page rather than Google's click link, and ad links are never opened. Google shows ads on some page loads and not others, so an empty array is normal even for commercial queries.
• enrichEmails (opt-in, best-effort, free) scrapes top business domains' own sites for contactEmail; perplexitySearch (opt-in, bring-your-own-key) attaches a Perplexity answer and is billed by Perplexity, not this actor.
• Pay-per-event billing — see the Pricing tab on the actor page.

⚠️ Disclaimer

This Actor is an independent tool and is not affiliated with, endorsed by, or sponsored by Google LLC or any of its subsidiaries. All trademarks mentioned, including Google and Google Search, are the property of their respective owners.

The scraper accesses only publicly available Google search results pages — no authenticated endpoints, paid features, or content behind any google.com login wall. Users are responsible for ensuring their use complies with google.com's Terms of Service, applicable data-protection law (GDPR, CCPA, etc.), and any contractual obligations of their own organization.

SEO Keywords

google search scraper, scrape google, google serp scraper, google search API, google.com scraper, Apify google, serp scraper, search results scraper, organic results scraper, seo rank tracking, keyword research data, competitor serp analysis, ai overview extraction, people also ask data, search engine intelligence, serp data export, marketing analytics data