Google Search SERP Scraper

A production-grade, ultra-fast Apify Actor that scrapes Google Search Engine Results Pages (SERPs) and returns structured data. Functions as a reliable Google SERP API for SEO rank tracking, market research, content-gap analysis, and competitive intelligence.

---

✨ Features

• ⚡ High-Speed Static Scraping: Built on Crawlee's CheerioCrawler (pure HTTP fetching with zero browser overhead) tuned for high concurrency.
• 🛡️ Hard AI Overview Exclusion: Strictly purges and excludes Google AI Overviews, AI Mode, and generative answer elements from extracted data.
• 🌐 Apify Google SERP Proxy Built-in: Utilizes Apify's dedicated GOOGLE_SERP proxy group (groups-GOOGLE_SERP), routing requests to the appropriate regional domain (google.co.uk, google.de, google.fr, etc.) with fallback to RESIDENTIAL.
• 📍 Precision Geo-Targeting (UULE): Converts free-text location names (e.g. Austin,Texas,United States) into Google's canonical uule parameter.
• 📦 Comprehensive SERP Extraction:
  • Organic Results: Position, title, destination URL, displayed URL / breadcrumbs, description/snippet, sitelinks, and rich snippets (ratings, review counts, prices, dates).
  • Paid Ads: Placement (top vs bottom), position, advertiser domain, title, ad URL, description, callout extensions, and ad sitelinks.
  • People Also Ask (PAA): Question text, answer preview, and source citation URL.
  • Related Searches: Suggested search queries.
  • Knowledge Graph / Answer Box: Title, subtitle, description, source URL, key-value attributes, and entity images.
  • Google Shopping: Product title, price, currency, merchant/seller, rating, and product URL.
• 📊 Throughput Tracking & Summaries: Emits live queries/sec and pages/sec metrics, with run summaries stored in the Key-Value Store (RESULTS-SUMMARY and OUTPUT).

---

📥 Input Parameters

Field	Type	Default	Description
queries	Array of strings	["best running shoes"]	Search keywords to run (accepts array or one query per line).
resultsPerPage	Integer (10, 20, 30, 40, 50, 100)	10	Number of results requested per page (num parameter).
maxPagesPerQuery	Integer	1	Pagination depth per query.
countryCode	String (ISO 3166-1 alpha-2)	"us"	Regional country code. Determines regional Google domain (google.de, google.co.uk, etc.) and gl param.
languageCode	String (ISO 639-1)	"en"	Search interface language (hl parameter).
location	String	""	Canonical location string (e.g. Berlin,Germany). Encoded into Google uule.
device	Enum (desktop, mobile)	"desktop"	Emulated client device and User-Agent headers.
includeOrganicResults	Boolean	true	Extract organic search results.
includePaidAds	Boolean	true	Extract sponsored top and bottom ads.
includePeopleAlsoAsk	Boolean	true	Extract People Also Ask Q&As.
includeRelatedSearches	Boolean	true	Extract related search queries.
includeKnowledgeGraph	Boolean	true	Extract Knowledge Graph and answer box details.
includeShoppingResults	Boolean	true	Extract Google Shopping product cards.
includeSiteLinks	Boolean	true	Extract sitelinks under organic results and ads.
excludeAiOverview	Boolean	true (Locked)	Hard exclusion. Discards all Google AI Overview / AI Mode blocks.
maxConcurrency	Integer	30	Maximum parallel requests (auto-scaled).
proxyConfiguration	Object	{ useApifyProxy: true, apifyProxyGroups: ["GOOGLE_SERP"] }	Apify proxy group settings.
outputFormat	Enum (dataset, dataset+csv, dataset+json)	"dataset"	Storage format for results.
debugMode	Boolean	false	Save raw SERP HTML snapshots to Key-Value Store under DEBUG-*.

---

📤 Output Structure

Data is streamed into the default Apify Dataset as each SERP page finishes loading.

Example Organic Result Item

{
  "query": "best running shoes",
  "page": 1,
  "resultType": "organic",
  "position": 1,
  "title": "The 10 Best Running Shoes of 2025, Tested and Reviewed",
  "url": "https://www.runnersworld.com/gear/a20865904/best-running-shoes/",
  "displayedUrl": "https://www.runnersworld.com > gear",
  "description": "We tested dozens of road and trail shoes to find the best models for every type of runner, budget, and foot strike.",
  "sitelinks": [
    {
      "title": "Best Cushioned Shoes",
      "url": "https://www.runnersworld.com/gear/cushioned-shoes"
    }
  ],
  "richSnippet": {
    "rating": 4.8,
    "reviewCount": 124,
    "date": "Jan 12, 2025"
  },
  "searchParameters": {
    "countryCode": "us",
    "languageCode": "en",
    "device": "desktop",
    "googleDomain": "google.com"
  },
  "timestamp": "2026-09-17T12:00:00.000Z"
}

Example Paid Ad Item

{
  "query": "running shoes",
  "page": 1,
  "resultType": "ad",
  "position": 1,
  "title": "Shop Top Running Shoes | Official Nike Store",
  "url": "https://www.nike.com/running",
  "displayedUrl": "https://www.nike.com",
  "description": "Experience comfort and speed. Browse the newest road racing and trail running shoes with free shipping.",
  "adInfo": {
    "placement": "top",
    "advertiserDomain": "nike.com",
    "callouts": ["Free Shipping on $50+", "60-Day Returns"]
  },
  "searchParameters": {
    "countryCode": "us",
    "languageCode": "en",
    "device": "desktop",
    "googleDomain": "google.com"
  },
  "timestamp": "2026-09-17T12:00:00.000Z"
}

---

🚀 How to Run via API

1. Apify API (cURL)

curl --request POST 'https://api.apify.com/v2/acts/YOUR_USERNAME~google-search-serp-scraper/runs?token=YOUR_API_TOKEN' \
  --header 'Content-Type: application/json' \
  --data '{
    "queries": ["best running shoes", "seo rank tracker"],
    "countryCode": "us",
    "languageCode": "en",
    "resultsPerPage": 10
  }'

2. Python (apify-client)

from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")

run_input = {
    "queries": ["python programming", "apify web scraping"],
    "countryCode": "gb",
    "languageCode": "en",
    "maxPagesPerQuery": 2,
    "device": "desktop"
}

run = client.actor("YOUR_USERNAME/google-search-serp-scraper").call(run_input=run_input)

# Fetch results from dataset
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(f"[{item['resultType'].upper()}] {item.get('title')} - {item.get('url')}")

3. JavaScript / Node.js (apify-client)

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_API_TOKEN' });

const run = await client.actor('YOUR_USERNAME/google-search-serp-scraper').call({
    queries: ['cybersecurity tools', 'cloud backup solutions'],
    countryCode: 'de',
    languageCode: 'de',
    resultsPerPage: 20,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(`Extracted ${items.length} SERP elements`);

---

⏰ Scheduling

To run this scraper automatically (e.g. daily for rank tracking):

1. Navigate to the Actor in Apify Console.
2. Click the Schedules tab.
3. Click Add schedule.
4. Set the Cron expression (e.g., 0 6 * * * for daily at 6:00 AM UTC).
5. Specify your search queries in the Input payload.
6. Connect the run to a Webhook or Slack/Email integration to receive automated alerts!

---

🔒 Hard AI Overview Exclusion Guarantee

This Actor is engineered to return clean, reliable SERP search results without generative interference:

• DOM nodes matching Google AI Overviews ([aria-label="AI Overview"], [data-attrid="wa:/description"], .kno-ah, and generative text containers) are automatically purged from memory before any parser runs.
• The excludeAiOverview flag is locked to true and enforced in code.
• No third-party AI answer engines (Perplexity, Copilot, ChatGPT, Gemini) are integrated.