Google SERP + AI Overviews API

SerpAPI-style Google search JSON — AI Overview answer + citations + organic results + knowledge graph + top stories + local results in one response per query, at a fraction of the cost.

• Batch up to 1000 queries per run (array input, one API call) — processed with 10-way concurrency (configurable 1–30), rows stream to the dataset as they complete (a timed-out run still delivers partial results)
• AI Overview parsed into structured text_blocks (paragraphs, headings, lists, tables — in document order) + references (citation title + link + snippet) — including deferred AIOs Google generates asynchronously (resolved via Google's own /async/folsrch endpoint, no browser rendering, no third-party API)
• Citation tokens auto-resolved: AI Overview citations that arrive as /goto?url=<token> are resolved to their real destination via Google's redirect endpoint (read Location header, ~90ms each), and so are raw /goto? tokens in organic results, top stories, discussions and local results. The original token is kept as goto_url on the item for provenance. If a token genuinely cannot be resolved, the citation is marked unresolved: true and search_metadata.unresolved_citations reports the count — never a silent failure.
• Rich SERP surface: organic results, knowledge graph (facts), top stories, local results (rating/reviews), People Also Ask, discussions & forums
• Geo targeting via location (e.g. "New York, NY", "Kuala Lumpur") — Google uule encoding, same as SerpAPI
• Country (gl) + language (hl) targeting
• Pay per search — $0.005 per completed query (0.5¢ each; store badge $5.00 / 1,000 searches is the same price). Queries that return no parseable content (status Empty), return a partial page with no organic results (status Partial), or error out are not charged.
• Buyer pays platform usage on top ($0.004/query) — **$0.009 all-in per query**, ~$1.05 for 120 queries
• What you get for $0.005: the full SERP in one row — AI Overview + citations, organic results, knowledge graph, top stories, local results, People Also Ask, discussions & forums. The AI-Overview-only alternatives charge $0.003–0.015 per AI Overview alone and return nothing else.

Input

{
  "queries": ["best crm for startups", "what is retrieval augmented generation"],
  "gl": "us",
  "hl": "en",
  "maxResults": 10,
  "location": "New York, NY",
  "retryDeferredAIO": true,
  "retryDeferredMaxFetches": 2,
  "captureRawHtml": false,
  "concurrency": 10,
  "retryDeferredPoll": false,
  "resolveGotoTokens": true,
  "udm56": false,
  "mobile": false,
  "retryAiModeForClassic": false,
  "excludeAiMode": true
}

Field	Type	Default	Description
queries	array of strings	required	Search queries (max 1000 per run)
gl	string	us	Two-letter country code (us, my, uk, sg…)
hl	string	en	Two-letter language code (en, ms, zh…)
maxResults	integer	10	Max organic results per query (1–10)
location	string	(proxy IP geo)	Geo target, e.g. "New York, NY". Google uule-encoded server-side.
retryDeferredAIO	boolean	true	Resolve deferred AI Overviews (Google generates some answers asynchronously). Uses the embedded /async/folsrch template via a datacenter proxy. Extra cost only on affected queries. false = always single fetch.
retryDeferredMaxFetches	integer	2	Max total fetches per query when a deferred AI Overview shell is detected (1–5). 1 = single fetch, same as retryDeferredAIO: false.
retryThinOverview	boolean	true	Thin AI Overviews — answers that end in a dangling heading or have under 600 characters of text, usually a partly rendered serve — get one fresh SERP fetch (folsrch resolution included). The richer answer is kept; a poorer or AI Mode re-fetch is discarded. Needs retryDeferredAIO and retryDeferredMaxFetches > 1. Rows stay Success and charged. Output marks search_metadata.thin_recovered (replaced) or search_metadata.thin_recovery_attempted (original kept).
captureRawHtml	boolean	false	Save the raw SERP HTML for every query to the key-value store (key raw-<query>-<ts>.html). For building golden test corpora / debugging selector drift. Adds ~200–400KB per query to KV storage.
captureFlaggedFolsrch	boolean	false	When a row ends with truncation_detected true (truncated answer — see Truncation semantics), save the answer's source HTML — the raw folsrch fragment when resolved_by is folsrch, else the shell — to the default key-value store as folsrch-evidence-<query>-<ts>.html and point search_metadata.folsrch_evidence at the key. Best-effort diagnostic for studying exact cut points; never fails a row. Adds one KV value per truncated row.
concurrency	integer	10	Parallel queries (1–30). A/B measured: c5=34s, c10=35s, c20=65s for 20 queries — the SERP proxy pool is the bottleneck, so beyond ~10 workers latency balloons. 10 = same throughput as 5 with headroom for AIO-heavy sets.
retryDeferredPoll	boolean	false	Legacy poll re-fetch for deferred AIOs (re-fetch up to retryDeferredMaxFetches times with backing-off delay 1.5s → 3s → 6s and a fresh proxy IP per attempt, keep richest). Off by default — folsrch resolves 90%+ of deferred AIOs; the poll recovered 0/120 in the stress test. Enable only if folsrch regresses.
resolveGotoTokens	boolean	true	Resolve AI Overview citations that still carry a raw /goto?url=<token> after the resolution chain by reading the Location header of google.com/goto?url=<token> (~90ms each; ~11.5% of citations need it). Original token kept as goto_url on the reference. Failures keep unresolved: true + unresolved_citations count. Verified 93/93 (EnterRank) + 6/6 (our probe).
udm56	boolean	false	Append &udm=56 to the search URL. Default OFF after measurement (2026-09-07 A/B, 10 queries × 2 variants): udm=56 cut AIO coverage 8/10 → 4/10 and did not pin the classic surface (AI Mode share unchanged). Kept for control; leave off for Google's default routing.
mobile	boolean	false	Fetch SERP + folsrch fragment with a mobile device header set (got-scraping header generator, devices: ['mobile']). Default OFF. Measurement (2026-09-07 spike, 10 queries × 4 variants, gl=sg): mobile UA cut truncation 6/8 (75%) → 3/8 (37%), mean answer length 687 → 852 chars, fetch success 15/20 → 19/20. n=10, one market, one day — EnterRank's paired A/B (30 queries × 2, MY market) was inconclusive: zero of 29 queries were classic-with-an-overview in both arms (commercial prompt set routes to AI Mode). Do not flip the default until a classic-serving prompt set settles it.
excludeAiMode	boolean	true	Classic AI Overview only. When the answer is still AI Mode after the optional retryAiModeForClassic re-fetch, the row carries no ai_overview (null) and is marked search_metadata.ai_mode_detected: true and search_metadata.ai_mode_excluded: true, with a warning. Organic results and People Also Ask are still delivered; the row is Success and charged, the same as a query Google shows no AI Overview for. Set false to ship the AI Mode answer as before (flagged ai_mode_detected, Success, charged).
retryAiModeForClassic	boolean	false	When true, an AI Mode serve triggers one re-fetch hoping for a classic AI Overview. Default OFF after measurement (EnterRank, 2026-09-07, 101/101 answers across 4 runs): on a commercial query mix the re-fetch recovered ZERO classic serves — Google routes commercial queries to AI Mode today, so the re-fetch is a doubled SERP fetch with no recovery. Kept behind the input because routing moved once (2026-09-02: 2/2 re-fetches served classic) and can move back. When false (default), an AI Mode answer is not re-fetched: with excludeAiMode on (default) it is dropped (ai_overview: null); with excludeAiMode: false it ships as-is (Success, ai_mode_detected: true, warning set, charged).

Output

One dataset item per query:

{
  "ai_overview": {
    "text_blocks": [
      { "type": "paragraph", "snippet": "The best CRM for a startup depends on your core workflow…" },
      { "type": "heading", "snippet": "Key Comparison Overview" },
      { "type": "table", "headers": ["Feature", "GXS Bank", "UOB"], "rows": [["Base Interest Rate", "0.88% p.a.", "0.05% p.a."]] },
      { "type": "list", "list": [{ "title": "…" }] }
    ],
    "references": [
      { "title": "CRM for Startups: Free Software to Scale Your Business", "link": "https://www.hubspot.com/products/crm/startups", "snippet": "…", "index": 0 }
    ],
    "resolved_by": "folsrch"
  },
  "organic_results": [
    { "position": 1, "title": "…", "url": "https://…", "displayed_link": "…", "snippet": "…" }
  ],
  "people_also_ask": ["…", "…"],
  "search_metadata": { "status": "Success", "created_at": "…", "total_time_taken_ms": 3088, "google_url": "…" },
  "search_parameters": { "q": "best crm for startups", "gl": "us", "hl": "en", "location": "New York, NY" },
  "knowledge_graph": {
    "title": "…",
    "description": "…",
    "source": { "name": "Wikipedia", "link": "…" },
    "facts": [{ "label": "Born", "value": "…" }]
  },
  "top_stories": [ { "title": "…", "url": "…", "source": "The Economist" } ],
  "local_results": [ { "title": "…", "rating": "4.9", "reviews": 534 } ],
  "discussions_and_forums": [ { "title": "…", "url": "…", "source": "reddit" } ],
  "selector_strategy": {
    "organic_results": "primary",
    "people_also_ask": "primary",
    "knowledge_graph": "fallback",
    "top_stories": "structural",
    "local_results": "primary",
    "discussions_and_forums": "primary"
  }
}

• ai_overview.resolved_by is folsrch (deferred answer resolved via Google's async endpoint), poll (recovered via re-fetch), or absent (answer was inline in the SERP).
• Modules are null when Google returns none for that query.
• ai_overview.text_blocks types: paragraph and heading (snippet), list (list: [{ title }], consecutive bullets share one block), and table (headers: the first row, rows: the remaining rows, cell text only; citation cards are never tables). Blocks come out in the order Google renders them. Answer text is taken only from the AI Overview block; the AI Mode teaser card is never read.
• search_metadata.ai_mode_detected: true means the answer was served from the AI Mode surface (no AI Overview block around the answer). A classic AI Overview answer never sets it, even when Google's share-panel furniture is present; that furniture is stripped from the text either way.
• search_metadata.ai_mode_excluded: true (with excludeAiMode, default on) means an AI Mode answer was dropped: ai_overview is null, modules_present.ai_overview is false, status is Success and charged. Organics and PAA are delivered. Set excludeAiMode: false to get the AI Mode text instead.
• search_metadata.status is Success (content parsed), Partial (organic results missing even after re-fetch — not charged), Empty (no parseable content — not charged), or Error (fetch failed after retries — not charged).
• search_metadata.modules_present (all rows) records which modules actually parsed as booleans: organic_results, ai_overview, people_also_ask, knowledge_graph, top_stories, local_results, discussions_and_forums. No more guessing from status strings — an organic-less row that still delivered a full AI Overview is visible as { organic_results: false, ai_overview: true, ... }. On Partial rows the warning also names the modules that were present.
• search_metadata.partial_recovered is true when organic results were recovered by the partial re-fetch (charged normally).
• search_metadata.drift_evidence (present on Empty/Partial rows) is the key of the raw HTML sample saved to the actor's key-value store — grab it for instant diagnosis of selector drift.
• search_metadata.truncation_detected / truncation_evidence — see Truncation semantics.
• selector_strategy records which parser strategy fired per module: primary (current Google classes), fallback (stable anchors), or structural (class-free heuristics). primary everywhere = healthy; fallback/structural = Google rotated classes but results still parse.
• ads and related_searches are not included — the Google SERP proxy serves a server-rendered variant where those JS-only modules are absent (verified: the tads ad container arrives empty, "Sponsored" labels never appear). This keeps results clean and ad-free.
• No pagination field — each query returns page 1 results; batch more queries for broader coverage.

Truncation semantics (2026-10-13 rework)

search_metadata.truncation_detected is true when words are actually missing from the AI Overview — Google stopped mid-sentence in the fetched response. It no longer fires on a merely missing full stop (the old punctuation-majority rule flagged 55 false positives out of 67 in run YXlzbPpYtf7PvP0ze). The decision is the text rule only, four reasons:

• unclosed_bracket — an opened ( never closed ("…such as Emirates NBD").
• ends_on_comma — ends on , or Arabic ، ("…balance of AED 50,").
• ends_on_open — ends on ( or [.
• ends_on_joining_word — ends on a connective ("and", "such as", "بشرط", "مثل"…).

Inline links at the end of a sentence are NOT truncation evidence on their own. Google wraps entity names in span[jscontroller$="#Mx1qPe"] and routinely omits the full stop exactly there, so a complete sentence and a real cut are markup-identical under it (live AE fragment audit: blob counts, anchors and stream tokens all overlap; every real cut at a link was already caught by the text rule above). Instead, any list item / paragraph whose last meaningful node is such an inline link carries two metadata fields:

• ends_on_inline_link: true — Google stopped rendering at the link; often the sentence is complete, sometimes words are missing (if they are, the item also carries truncated + truncation_reason from the text rule, so a flagged "(such as Emirates NBD" item carries BOTH its reason and its sources).
• inline_link_sources: [{ url, title, source }] — the pages Google attached to that link, decoded from its TgQPHd comment blobs. The sentence tail itself is never in Google's response.

A missing full stop alone, a trailing colon (labels like "Petrol:") and a semicolon (list style) are deliberately NOT cuts. Details are per item/paragraph: truncated: true + truncation_reason on list items and paragraph blocks; a list/paragraph block with any cut carries truncation_detected: true. On the row, search_metadata.truncation_evidence lists the cuts as { tail, reason } (last 80 characters, max 10). The re-fetch triggers use the same evidence, so no re-fetch ever happens because an answer ends without a period.

Pricing

• $0.005 per completed query (pay-per-event, search-completed)
• Platform usage billed on top, paid by the user: Google SERP proxy ~$0.0025/query + compute ~$0.0015/query (amortized at batch scale)
• Deferred-AIO resolution adds a shell re-fetch + a datacenter proxy call on affected queries only — the datacenter leg is not billed as SERP usage
• All-in ≈ $0.009 per query → 120 queries ≈ $1.05
• vs the leading AI-Overview-only alternative: ~$1.81 for the same 120 queries
• vs SerpAPI (prices checked Aug 2026): $0.025/search (Starter, $25/mo, 1,000 searches), $0.015/search (Developer, $75/mo), $0.01/search (Production, $150/mo) — all require a monthly plan. Ours: ~$0.009 all-in with no subscription, no minimum → ~2.8× cheaper than their entry paid tier. (SerpAPI's premium: SOC 2, uptime SLAs, 100+ search engines.)
• Note: tiny runs (1–5 queries) pay ~$0.02–0.03/query effective due to container cold start — batch ≥25 for best economics

How we compare on the Apify Store (checked via Store API, Oct 2026)

Actor	What you get	Price per query
Ours — google-serp-ai-overviews	AI Overview + citations + organics + PAA + KG + top stories + local	$0.005
apify/google-ai-overviews-scraper (official)	AI Overview only — no organics	$0.003
igolaizola/google-search-scraper-ppe	organics only — no AI Overview	~$0.003 / 10 results
s-r/free-google-search-results-serp	organics only — no AI Overview	~$0.0035 / 10 results
johnvc/Google-AI-Overview-API	AI Overview only	$0.015
clearpath/google-ai-overview	AI Overview only	$0.015
sian.agency/google-serp-scraper	same bundle, split events (result $0.02 + PAA $0.022 + AIO $0.03)	~$0.042

The math: competitors either charge less but return a fraction of the row (AIO-only at $0.003, or organics-only without the AI Overview), or charge 4–8× more for the same complete bundle. Ours is the cheapest single-call source of the full SERP + AI Overview. No subscription, no minimum — and Empty/Partial/Error rows are never charged.

Use cases

• AEO/GEO monitoring — track whether your brand appears in AI Overviews and which sources Google cites, per market with location
• SEO rank tracking at scale
• Research / data collection for LLM training sets
• Competitive SERP intelligence

Costs & limits

• 1000 queries max per run (bump the array; batch in chunks for more)
• Each query = 1 Google SERP request through Apify's Google SERP proxy (+ up to 2 extra requests only when Google defers the AI Overview)
• 10-way concurrency (default): 1000 queries ≈ 30 min (fits the 7200s timeout); rows stream to the dataset as they complete. A/B measured: c5=34s, c10=35s, c20=65s for 20 queries — the SERP proxy pool is the bottleneck, so beyond ~10 workers latency balloons. Set concurrency lower (e.g. 5) for gentler load.
• Faster than 30 min? Split the batch into parallel runs (e.g. 2× 500 queries) and fire them concurrently — each run gets its own proxy pool share. Your plan's concurrency limits apply; rows land in separate datasets you can merge by search_parameters.q.
• 7200s timeout (2h), 4096 MB memory
• Build tags: the actor runs the stable build by default (verified, production-hardened). New builds are pushed to latest and promoted to stable only after verification — buyers are never exposed to unverified code.