Google SERP Scraper — Organic, AI Overviews & Leads

Scrape Google Search at scale — organic results, ads, People Also Ask, knowledge panel, local pack, shopping, and the AI Overview block with parsed source domains and citation index.

Google SERP Scraper turns any search term (or raw Google URL) into structured JSON: ranked organic results, SERP-feature blocks, and — its main differentiator — the AI Overview with each cited source parsed into a real domain plus its citation index, the exact data Google Search Console does not expose. It is built for SEO teams tracking rankings, AEO/GEO teams measuring which domains AI Overviews actually cite, and lead-gen teams turning result domains into published contacts. No Google API key is required.

Key features

• AI Overviews with parsed sources — the aiOverview block returns the answer text plus a sources[] array where every citation is resolved to a real domain, full url, title, and citationIndex, with an ordered sourceDomains[] roll-up. Most competitors return only raw AIO text (or nothing); this actor exposes which domains Google's AI cited and in what order.
• Full SERP-feature coverage — independently toggleable blocks for organicResults, ads, shopping (Popular Products carousel), peopleAlsoAsk, relatedSearches, suggestedResults ("People also search for"), featuredSnippet, knowledgePanel, and localPack.
• Rank & position monitoring — every organic result carries a position, and resultsTotal is parsed per query, so you can diff runs to track movement and new-vs-last-run changes over time.
• Optional lead enrichment — with enrichLeads, the top organic result domains are crawled for named people, emails, phones and socials. email holds only an address the site actually publishes. When a person has none, a name-derived candidate is offered separately as emailGuess (with emailGuessPattern and domainAcceptsMail) and is clearly labelled as a guess — it is never presented as a confirmed address. Results attach to the query item and mirror into separate leads and link-prospects datasets.
• Rich localization & operators — country, language, precise location targeting, Google TLD, device (desktop/mobile), SafeSearch, date filters, and the full set of search operators (site:, intitle:, filetype:, exclusions, and more).
• Keyless by default — no Google or SerpApi key to manage, and no account to keep alive. Proxying is configured automatically; leave the default in place.
• Optional multi-engine answers (BYO key) — add answers from Gemini, Perplexity, or ChatGPT for the same query when you supply your own API keys, for cross-engine AEO comparison.
• Pay-per-result pricing — you are billed for results produced, with a cheap organic-only fast mode and a full mode for every enabled block.

Input

queries is the only required field. Everything else has sensible defaults. Full parameter list is in the input schema; the main parameters:

Field	Type	Default	Description
queries	array (required)	—	Search terms or raw https://www.google.com/search?q=... URLs (used verbatim, missing params filled in).
maxResultsPerQuery	integer	10	Target organic results per page (Google num). Google deprecated num=100 in Sep 2025; values ≤ ~20 are reliable. Alias: num.
maxPagesPerQuery	integer	1	SERP pages to paginate per query.
countryCode	string	"us"	Country of search (Google gl). Alias: gl.
languageCode	string	"en"	Interface / results language (Google hl). Alias: hl.
locationName	string	—	Human location (e.g. "New York,New York,United States"), encoded for you. Advanced users can pass a pre-computed locationUule instead.
googleDomain	string	"google.com"	Google host to query (e.g. google.co.uk).
device	enum	"desktop"	desktop or mobile layout / header fingerprint.
safeSearch	enum	"off"	off | active | nfpr (no spelling auto-correct).
includeOrganic	boolean	true	Organic results block.
includeAiOverview	boolean	true	Parse the AI Overview (text + source domains + citation index).
includeAds / includeShopping / includePeopleAlsoAsk / includeRelatedSearches / includeFeaturedSnippet / includeKnowledgePanel / includeLocalPack / includeAiMode	boolean	false	Toggle each SERP-feature block independently.
aiOverviewMode	enum	"http"	http (cheap, opportunistic) or render (Playwright — reliably renders/expands the AI Overview).
enrichLeads	boolean	false	Crawl top result domains for published contacts. See maximumLeadsEnrichmentRecords, leadsEnrichmentDepartments, verifyLeadsEnrichmentEmails, linkProspecting.
includeWebsiteContent	boolean	false	Fetch each organic URL and extract title / description / text (+ websiteContentMaxPages).
geminiSearch / perplexitySearch / chatGptSearch	boolean	false	Add cross-engine answers (require geminiApiKey / perplexityApiKey / openaiApiKey or the matching env var).
proxyConfiguration	object	preconfigured	Leave as is. The default is the configuration this Actor is tuned for; other networks are refused by Google.
mode	enum	"full"	fast (organic only, cheapest) or full (all enabled blocks incl. AIO).
saveHtmlToKVS	boolean	false	Persist raw SERP HTML to the key-value store for debugging / re-parsing.

Example input

{
  "queries": [
    "best running shoes 2026",
    "site:nike.com trail shoes"
  ],
  "maxResultsPerQuery": 10,
  "maxPagesPerQuery": 1,
  "countryCode": "us",
  "languageCode": "en",
  "device": "desktop",
  "includeOrganic": true,
  "includeAiOverview": true,
  "includePeopleAlsoAsk": true,
  "includeRelatedSearches": true,
  "includeShopping": true,
  "aiOverviewMode": "http",
  "mode": "full",
  "proxyConfiguration": { "useApifyProxy": true }
}

Output

One dataset item per query-page. Trimmed real-shape sample:

{
  "searchQuery": {
    "term": "best running shoes 2026",
    "url": "https://www.google.com/search?q=best+running+shoes+2026&num=10&gl=us&hl=en",
    "device": "desktop",
    "page": 1,
    "type": "search",
    "domain": "www.google.com",
    "countryCode": "us",
    "languageCode": "en",
    "locationUule": null,
    "scrapedAt": "2026-09-02T10:14:07.512Z"
  },
  "resultsTotal": 131000000,
  "organicResults": [
    {
      "position": 1,
      "title": "The 12 Best Running Shoes of 2026, Tested by Runners",
      "url": "https://www.runnersworld.com",
      "displayedUrl": "https://www.runnersworld.com › gear › best-running-shoes",
      "snippet": "Our team logged 400+ miles across 30 models. Top picks for road, trail, and race day...",
      "domain": "runnersworld.com",
      "type": "organic",
      "emphasizedKeywords": ["running shoes", "2026"],
      "siteLinks": [
        { "title": "Best Trail Shoes", "url": "https://www.runnersworld.com/gear/trail" }
      ],
      "date": null,
      "richSnippet": { "rating": 4.6, "reviews": 218, "price": null },
      "productInfo": null
    }
  ],
  "ads": [],
  "shopping": [
    {
      "position": 1,
      "title": "Nike Pegasus 41 Men's Road Running Shoes",
      "url": "https://www.nike.com",
      "price": "$140.00",
      "oldPrice": null,
      "discount": null,
      "merchant": "Nike",
      "rating": 4.7,
      "reviews": 1032
    }
  ],
  "peopleAlsoAsk": [
    { "question": "What is the best running shoe for beginners?", "answer": null, "sourceUrl": null, "sourceTitle": null }
  ],
  "relatedSearches": [
    { "query": "best running shoes for flat feet", "url": "https://www.google.com/search?q=best+running+shoes+for+flat+feet" }
  ],
  "suggestedResults": [
    { "title": "Nike running shoes", "url": "https://www.google.com/search?q=Nike+running+shoes" }
  ],
  "featuredSnippet": null,
  "knowledgePanel": null,
  "localPack": [],
  "aiOverview": {
    "present": true,
    "text": "The best running shoes for 2026 depend on your use case. For daily road training, cushioned neutral shoes are widely recommended, while trail runners benefit from grippier outsoles...",
    "sources": [
      {
        "url": "https://www.runnersworld.com/gear/best-running-shoes",
        "title": "The 12 Best Running Shoes of 2026",
        "sourceName": "Runner's World",
        "domain": "runnersworld.com",
        "description": null,
        "citationIndex": 1
      },
      {
        "url": "https://www.nike.com/running",
        "title": "Running Shoes",
        "sourceName": "Nike",
        "domain": "nike.com",
        "description": null,
        "citationIndex": 2
      }
    ],
    "sourceDomains": ["runnersworld.com", "nike.com"],
    "position": "above",
    "captureType": "static_html"
  },
  "aiMode": null,
  "meta": {
    "statusCode": 200,
    "resolvedUrl": "https://www.google.com/search?q=best+running+shoes+2026&num=10&gl=us&hl=en",
    "proxyUsed": true,
    "blocked": false,
    "blockReason": null,
    "htmlKvsKey": null
  }
}

Output fields

• searchQuery — request metadata: resolved term, canonical url, device, page, Google domain, countryCode, languageCode, locationUule, and scrapedAt timestamp.
• resultsTotal — Google's "About N results" count (integer, or null when Google omits it).
• organicResults[] — position, title, url, displayedUrl, snippet, domain, type (organic / product / video / news), emphasizedKeywords[], siteLinks[] ({title, url}), date, richSnippet ({rating, reviews, price}), productInfo ({price, currency, rating, reviews, availability}). When includeWebsiteContent is on, each result also gets a websiteContent object.
• ads[] — position, title, url, displayedUrl, description, extensions[].
• shopping[] — Popular Products carousel: position, title, url, price, oldPrice, discount, merchant, rating, reviews.
• peopleAlsoAsk[] — question, answer, sourceUrl, sourceTitle (see limitations on answers).
• relatedSearches[] / suggestedResults[] — {query, url} and {title, url}.
• featuredSnippet — {type, text, sourceUrl, sourceTitle} or null.
• knowledgePanel — {title, type, description, attributes{}, links[]} or null.
• localPack[] — name, rating, reviews, address, phone, category, hours.
• aiOverview — present, text, sources[] ({url, title, sourceName, domain, description, citationIndex}), sourceDomains[] (ordered), position (above / below / null), and captureType.
• aiMode — same shape as aiOverview plus queryFanOut[], url, and captureType / blockReason; null unless includeAiMode is on.
• geminiResult / perplexityResult / chatGptResult / copilotResult — optional per-query: {engine, present, skipped, blocked, blockReason, answerText, citedSources[], model}.
• meta — statusCode, resolvedUrl, proxyUsed, blocked, blockReason, htmlKvsKey.

aiOverview.captureType values: static_html (parsed from the HTTP response), rendered (Playwright), async_pending (Google deferred it behind a JS/async token), not_present, disabled, or error.

Pricing

Billed per event — you pay for what actually arrives, and platform usage (compute and proxy) is billed to your Apify account on top.

Event	Price	Charged when
Search page	$0.002	one search page is delivered, with every block you enabled
AI Overview	$0.005	the page really contained an AI Overview
Contact	$0.010	one named contact is found (enrichLeads)
Result page content	$0.002	one result page is fetched and its text extracted
Actor start	$0.010	per GB of run memory, when the run starts

So a search with no AI Overview costs $0.002, and one with an overview $0.007.

Not charged, ever:

• a search page Google refused to serve — you see the failure, you do not pay for it;
• an AI Overview that did not arrive (Google serves it inconsistently — see the limitations);
• a result page that refused the request or returned no readable text;
• AI Mode, ads, and email verification — they cannot work here and are never billed (the limitations section explains why).

Use mode: "fast" for organic-only runs when you do not need the extra blocks.

Use cases

• SEO rank tracking — monitor position and resultsTotal for your keywords across countries, languages, and devices; diff runs to detect ranking movement.
• AEO / GEO citation tracking — measure which domains Google's AI Overview actually cites (aiOverview.sourceDomains + citationIndex) for your target queries, and whether your own domain appears. Optionally compare against Gemini / Perplexity / ChatGPT answers.
• Lead generation — turn the domains that rank for a commercial query into published contacts (enrichLeads), with department and seniority filtering, exported to a dedicated leads dataset.
• Competitor & SERP monitoring — watch competitors' organic positions, ad presence, shopping listings, knowledge panels, and local pack rankings over time.

Proxy

Leave proxyConfiguration at its default. It is preconfigured for this Actor, and Google refuses most other networks on the search endpoint — a custom proxy will simply return captcha pages. The Actor warns you in the log if a non-default configuration is supplied.

Honest limitations

We would rather you know these up front than be surprised by empty fields:

• Paid ads[] are effectively always empty. Google does not serve ads to this Actor's sessions, so the block stays empty even in render mode. Organic "Popular Products" still come through in shopping[], and organic results, AI Overview, knowledge panel and local pack all work fully.
• People Also Ask answers often stay null. The questions are always captured; the answers load separately and are only available in render mode (aiOverviewMode: "render").
• Google AI Mode requires a signed-in Google account, which this Actor does not use. It is reported honestly as aiMode.captureType: "not_present" with a reason, never faked.
• AI Overview over pure HTTP is opportunistic. Google inlines the AIO inconsistently. When it does, you get fully parsed sources + domains (captureType: "static_html"); otherwise the actor honestly reports async_pending or not_present. For a reliable AIO on every query, use aiOverviewMode: "render" (Playwright — slower, more CPU/memory).
• Combining several search operators can make Google drop them. An over-constrained query (exact phrase + site: + excludeWords, say) returns too few matches, so Google quietly relaxes the restriction and serves loosely related results instead. Each operator works on its own — site: alone returns only that domain. Narrow one filter at a time.
• Organic result URLs are the site root, not the deep page. Google does not expose the destination of a result link in a resolvable form, so url is the site origin and displayedUrl keeps the breadcrumb Google shows. A link that cannot be resolved is reported as null rather than handed back as an unusable Google address. includeWebsiteContent therefore fetches the site's landing page, not the ranked page. AI Overview sources are different — their citations usually carry the full deep URL.
• The AI Overview is served opportunistically. The same query can return one on one run and none on the next; this is Google's behaviour, not a failure. captureType reports exactly what happened (static_html, rendered, async_pending, not_present).
• AI Mode (includeAiMode) is gated behind a Google login and returns ai_mode_not_generated_logged_out for anonymous sessions. It is reported honestly rather than faked.
• Shopping product links are not available. Google does not expose them, so shopping[].url is null; the title, price, old price, discount, merchant label, rating and review count all come through.
• suggestedResults[] repeats relatedSearches[] on most SERPs — Google merges the two blocks and both keys are emitted for schema parity with other SERP actors.
• Contact crawling depends on the target site. Company sites throttle repeated visits, and a site that serves only its homepage has no team page to read — the run then reports leadsCount: 0 with the crawled page count visible in the log. The crawl goes through Apify Proxy to spread the load.
• Email verification is MX-only on the Apify platform. Outbound port 25 is filtered there, so the SMTP handshake cannot run and emailStatus stays unknown with smtpBlocked: true. An MX record proves the domain accepts mail — it does not prove the address exists.
• Multi-engine answers need your own API keys. geminiSearch / perplexitySearch / chatGptSearch require your geminiApiKey / perplexityApiKey / openaiApiKey (or the matching env var); without a key the actor emits an honest skipped record. Microsoft Copilot has no public bring-your-own-key answer API, so copilotSearch always returns an unsupported record.

FAQ

Do I need a Google or SerpApi API key? No. The Actor reads public Google Search pages directly; proxying is preconfigured. You only need your own keys if you opt into the multi-engine answers (Gemini / Perplexity / ChatGPT).

How is this different from other Google SERP scrapers? The AI Overview block is parsed into real source domains with a citation index and ordered sourceDomains[], not just raw text — the data you need for AEO/GEO work and that Search Console does not expose. It also ships every SERP-feature block over the cheap HTTP path and can chain into lead enrichment.

Can I target a specific city or country? Yes. Set countryCode (gl) and languageCode (hl), pick a googleDomain, and use locationName for city-level targeting (it is encoded for you), or pass a pre-computed locationUule.

Why is the AI Overview sometimes missing or async_pending? Google serves the AI Overview inconsistently in the cheap HTTP mode. Set aiOverviewMode: "render" to run Google's JS in headless Chromium and reliably render/expand it (slower and more resource-intensive, so it is opt-in).

Legal / ToS

This actor scrapes only public Google Search output — no logins, paywalls, or personal-data harvesting beyond what Google publicly renders. Respect Google's Terms of Service and rate limits, and comply with the laws of your jurisdiction. Google has litigated against SERP scrapers; use this actor responsibly and at your own discretion.