Google SERP Scraper

Scrape Google Search results for any keyword — organic rankings, featured snippets, People Also Ask, related searches, and ads.

Google SERP Scraper is an Apify Actor built with TypeScript and Crawlee CheerioCrawler. Feed it one or more search queries and get clean, structured SERP data ready for SEO research, rank tracking, competitive analysis, and AI agent tool use. Results are volume-friendly (one dataset item per organic result) with optional per-query summary items.

Note: Google aggressively rate-limits and captcha-challenges automated traffic. For production, enable Apify Proxy (preferably residential). Local runs without a proxy often receive captcha pages — the Actor records an error dataset row instead of crashing. Unit tests cover HTML parsing with fixture SERP HTML so the pipeline stays reliable even when live Google is blocked.

What does Google SERP Scraper do?

It requests classic Google Search URLs (https://www.google.com/search?q=…&hl=…&gl=…&num=…&start=…), parses the HTML with Cheerio using multiple CSS selector fallbacks, and exports:

• Organic results — position, title, URL, displayed URL, description, query, page
• Featured snippets — answer-box style content when present
• People Also Ask — questions (+ optional snippet / URL)
• Related searches — bottom-of-page suggestions
• Ads — top/bottom text ads when detectable in HTML

Apify platform benefits: scheduling, API access, integrations, proxy rotation, monitoring, and Pay-per-event pricing.

Why use this Actor?

Use case	Benefit
SEO rank tracking	Capture organic positions per keyword over time
Competitor research	See who ranks, ad copy, and related queries
Content ideation	People Also Ask + related searches → topic clusters
AI / agent tools	Structured JSON SERP as a tool output for LLM agents
Market research	Multi-query batches with country / language localization

Features

• Multi-query batches — scrape many keywords in one run
• Pagination — maxPagesPerQuery + resultsPerPage (num / start)
• Localization — countryCode (gl) and languageCode (hl)
• One row per organic result — ideal for PPE monetization volume
• Optional summary item per query page — PAA, related, ads, featured snippet
• Deduplication — unique by query + url (tracking params stripped)
• Captcha resilience — blocked pages produce type: error rows, not hard crashes
• Retries & rate limiting — concurrency, retries, delay + jitter
• Proxy-ready — Apify Proxy configuration
• Exports — default dataset, SERP.csv, and SUMMARY.md in the key-value store

How to use Google SERP Scraper

1. Open the Actor in Apify Console (or run locally with apify run).
2. Enter one or more queries (keywords).
3. Optionally set country, language, pages per query, and concurrency.
4. Keep Apify Proxy enabled for production.
5. Run the Actor and download results as JSON, CSV, Excel, or via API.

Input

Field	Type	Required	Default	Description
queries	string[]	Yes	—	Search keywords / phrases
maxPagesPerQuery	integer	No	1	SERP pages per query (1–10)
resultsPerPage	integer	No	10	Requested results per page (num)
countryCode	string	No	us	Google gl country code
languageCode	string	No	en	Google hl language code
maxConcurrency	integer	No	2	Parallel requests (keep low)
maxRequestRetries	integer	No	3	HTTP retry count
requestDelayMs	integer	No	1500	Base delay before each request (ms)
includePeopleAlsoAsk	boolean	No	true	Extract PAA into summary
includeRelatedSearches	boolean	No	true	Extract related searches
includeAds	boolean	No	true	Extract ads when detectable
includeSummaryItems	boolean	No	true	Push type=summary items
proxyConfiguration	object	No	Apify Proxy on	Proxy settings

Example input

{
    "queries": ["best project management software", "apify alternatives"],
    "maxPagesPerQuery": 1,
    "resultsPerPage": 10,
    "countryCode": "us",
    "languageCode": "en",
    "maxConcurrency": 2,
    "maxRequestRetries": 3,
    "requestDelayMs": 1500,
    "proxyConfiguration": { "useApifyProxy": true },
    "includePeopleAlsoAsk": true,
    "includeRelatedSearches": true
}

See also: examples/input.json

Output

Organic result (type: "organic")

Field	Type	Description
type	string	Always organic
position	integer	Rank on the SERP page
title	string	Result title
url	string	Destination URL (redirects unwrapped when possible)
displayedUrl	string	Breadcrumb / cite text
description	string	Snippet
query	string	Search query
page	integer	Page number (1-based)
countryCode	string	gl used
languageCode	string	hl used
scrapedAt	string	ISO timestamp
error	string	Present only on failure rows

Summary item (type: "summary")

Includes organicCount, featuredSnippet, peopleAlsoAsk[], relatedSearches[], and ads[] for that query page.

Example organic output

{
    "type": "organic",
    "position": 1,
    "title": "Asana · Work management platform for teams",
    "url": "https://www.asana.com/",
    "displayedUrl": "https://www.asana.com",
    "description": "Asana is the work management platform teams use to stay focused on the goals, projects, and daily tasks that grow your business.",
    "query": "best project management software",
    "page": 1,
    "countryCode": "us",
    "languageCode": "en",
    "scrapedAt": "2026-07-13T12:00:00.000Z"
}

You can download the dataset in JSON, HTML, CSV, or Excel from the Apify Console Output tab or via API.

Key-value store artifacts

Key	Description
SERP.csv	CSV export of all organic results
SUMMARY.md	Markdown run summary by query
OUTPUT	JSON totals

How it works

1. Normalize input — dedupe queries, clamp pagination / concurrency, build Google URLs.
2. Fetch SERPs — CheerioCrawler with browser-like headers, optional proxy, retries, rate limits, delay + jitter.
3. Parse HTML — multi-selector fallbacks for organic cards, featured snippets, PAA, related searches, ads; unwrap /url?q= links.
4. Deduplicate — skip duplicate query+url pairs within the run.
5. Handle blocks — captcha / unusual-traffic pages push type: error and continue.
6. Export — dataset items + SERP.csv + SUMMARY.md.

Anti-block tips

• Always enable Apify Proxy in production (useApifyProxy: true).
• Prefer residential proxy groups when available.
• Keep maxConcurrency at 1–3 and requestDelayMs ≥ 1500.
• Scrape fewer pages per query if you see many captcha errors.
• Cheerio is used for speed; Google may still return captchas without strong proxies — fixture tests keep parsers verified offline.

Pricing

Pay-per-event pricing: $0.002 per dataset item (primarily each organic result, plus optional summary/error items), plus a small actor-start fee.

Run locally

cd factory/google-serp-scraper
npm install
npm test
npm run build
apify run --purge

Without a proxy, live Google pages may return captchas. Parser correctness is covered by fixture tests under test/fixtures/.

Deploy to Apify

apify login
apify push

Then publish to the Store and set monetization to Pay per event with primary event apify-default-dataset-item at $0.002.

FAQ & disclaimers

Is scraping Google legal?
Scraping publicly available search results may be restricted by Google’s Terms of Service. You are responsible for complying with applicable laws and Google’s terms. Use proxies and polite rate limits; this Actor is provided for legitimate research and automation where permitted.

Why are some runs empty / only errors?
Google served a captcha or consent wall. Enable residential proxies, lower concurrency, and increase delay.

Does this use a browser?
No — HTTP + Cheerio for speed and cost. Selector fallbacks target classic SERP HTML layouts.

Support
Open an issue on the Actor’s Issues tab in Apify Console for bugs or feature requests. Custom solutions available via the author account.

Resources

• Apify SDK for JavaScript
• Crawlee CheerioCrawler
• Pay per event monetization
• Apify Proxy