Google Search Results (SERP) Scraper By Site & File Type

Google SERP Scraper — Extract Search Results & Documents as JSON

This Actor runs Google searches built from real search operators — site:, filetype:, intitle:, inurl: and exact-phrase — and returns organic results, paid results, People Also Ask items and related searches, plus a documentResults[] audit showing which operators each result actually satisfied, the file-type badge Google printed, and the real cite URL. Every response comes back as typed, normalized JSON — no HTML, no selectors, no parsing. Read on for the exact input fields, every output key the Actor writes, and how to filter for genuine operator matches.

---

What is Google Search Results (SERP) Scraper By Site & File Type?

It is an Apify Actor that turns a list of keywords into full Google search-results pages, with Google's own site:, filetype:, intitle:, inurl: and exact-phrase operators exposed as first-class input fields instead of raw query syntax. No Google account or login is required — only an Apify account to run the Actor. It is reachable as a plain HTTP endpoint or through Apify's MCP server, so an agent can call it directly.

• 🔎 Scrape organic search results — title, URL, snippet, position, sitelinks
• 🗂️ Scrape audited documents — one record per result with the file-type badge, the "N pages" line when Google prints it, and matchedOperators[] showing which operator that result really satisfied
• 💬 Scrape paid results, shopping products, People Also Ask and related searches from the same run
• 📤 Export as JSON, CSV or Excel from the Apify dataset
• 🚫 No proxy management — the Actor escalates from Apify's Google-SERP proxy to residential proxies on its own when a request is challenged

---

What data does Google Search Results (SERP) Scraper By Site & File Type collect?

Every keyword returns organic results plus an audited document table, and — when present on the page — paid results, shopping products, People Also Ask and related searches.

Data Type	Key Fields	JSON Field Names
Organic search results	title, URL, displayed URL, snippet, position	organicResults[] (title, url, displayedUrl, description, position)
Audited documents	file-type badge, page count, matched operators	documentResults[] (fileType, fileTypeSource, pageCountText, matchedOperators)
Paid results & shopping products	ad title/URL, product title, prices	paidResults[], paidProducts[] (title, url, prices)
People Also Ask	related question, link	peopleAlsoAsk[] (question, url)
Related searches	suggested query, link	relatedQueries[] (title, url)

Need more Google data?

If the page you need isn't a search-results page, the same account publishes Google Search Results Scraper (Knowledge Panel & Answer Box) for answer-box and knowledge-panel data, Google Scholar Scraper for academic search, and Google Maps Reviews Scraper for location data — each scoped to its own part of Google rather than bolted onto this one.

---

How does Google Search Results (SERP) Scraper By Site & File Type differ from the official Google API?

Google's own programmatic route to search results, the Custom Search JSON API, is closed to new customers — existing integrations must migrate off it by January 1, 2027. This Actor runs today, on any Apify account, with no Google Cloud project or search-engine registration.

Feature	Google Custom Search JSON API	Google Search Results (SERP) Scraper By Site & File Type
New sign-ups	Closed to new customers; must migrate by Jan 1, 2027	Open — run it from any Apify account today
Free quota	100 queries/day free, then $5 per 1,000 up to 10,000/day	Pay per result row returned (Apify pay-per-event); no daily query cap in the Actor
Results per query	Max 10 per request (num 1–10), capped at 100 results total	Up to 10 results per page, walks up to 100 pages per keyword
Setup	Google Cloud project, API key, and a Programmable Search Engine ID (cx)	Apify account and a keyword — no Google credential of any kind
site: / filetype: / intitle: / inurl: operators	Accepted as raw query syntax, unaudited	Exposed as dedicated input fields, and every result is checked against them (matchedOperators[])
Result freshness	Live	Live, per run

The official API is the right call if you already have an active Custom Search Engine project and stay under its query caps. This Actor is the fit once you need site:/filetype: audited at the result level, more than 100 results per query, or you can't get a new Custom Search project approved at all.

---

Why do developers and teams scrape Google search results?

For AI engineers and agent builders

organicResults[] and documentResults[] drop straight into a RAG index or an agent tool without an HTML-parsing step. An agent researching a regulatory topic can run site:europa.eu filetype:pdf, then filter documentResults[] on matchedOperators and fileType: "PDF" to keep only genuine PDF hits before it ever reaches an LLM context window — Google routinely loosens filetype: and returns HTML pages anyway, so that filter step matters.

For marketers and SEO teams

site:yourdomain.com walked page by page shows what Google has actually indexed and how each URL is displayed, using the same displayedUrl and snippet Google shows a searcher. site:competitor.com intitle:"pricing" returns only a competitor's pricing pages, with matchedOperators showing which ones Google actually honoured rather than loosely matched.

For researchers and analysts

site:*.gov or site:europa.eu combined with fileTypes: ["pdf"] surfaces published reports and filings that a normal keyword search buries under news and blog content, and documentsFound / fileTypeBreakdown tell you at a glance how many of a search's results are real documents versus HTML pages — public search-results scope only, nothing behind a login.

For developers building data products

resolvedQuery and operatorMatchSummary give a stable, auditable record of exactly what was searched and how well it was honoured, which is a workable base for a scheduled index-monitoring tool or a document-discovery pipeline — run it on an Apify schedule and pipe the dataset into your own store.

---

How to scrape Google search results (step by step)

1. Open Google Search Results (SERP) Scraper By Site & File Type on its Apify Store listing.
2. Provide keyword — one search per line. It's the only required field.
3. Set the operator fields that apply — site, fileTypes, wordsInTitle, wordsInUrl, forceExactMatch — and pick a limit for how many pages to walk.
4. Click Start and watch rows stream into the dataset per page, per keyword.
5. Open the Output tab and export the run as JSON, CSV or Excel.

What to do when Google changes its structure

The Actor is maintained, and the output schema stays the same on your end regardless of what changes in Google's markup. Every selector that reads the page — the result container, the file-type badge, the "N pages" line — is checked against a ladder of fallback selectors rather than one hard-coded class, so a single class-name change doesn't silently zero out a field.

---

What changed in Google search scraping recently?

Google has made two dated, verifiable changes that affect anyone pulling search results at scale. On September 11, 2025, Google deprecated the &num=100 parameter that let a single request return 100 results — confirmed by Google's own Danny Sullivan as an intentional move, not a bug — so any bulk pull now has to walk pages 10 results at a time instead of one large request (Search Engine Land). Separately, Google's own Custom Search JSON API — the official programmatic route — has been closed to new customers, with existing integrations required to migrate off it by January 1, 2027.

For a DIY scraper, both changes mean more requests per crawl and one less officially sanctioned path to get there. This Actor already paginates in 10-result pages by design and doesn't depend on the official API, so neither change requires any action from you — the Actor is maintained on an ongoing basis as Google's markup continues to shift.

---

⬇️ Input

Only keyword is required — every operator and localization field below is optional.

Parameter	Required	Type	Description	Example Value
keyword	Yes	string	One search per line. Operators typed here win over the fields below — a line reading site:europa.eu budget keeps its own site: and the site field is skipped for it. Default "nike".	"annual report"
site	No	string	Adds site: to the query, e.g. europa.eu. Subdomains are included. Default "".	"europa.eu"
fileTypes	No	array (select, enum: pdf, doc, docx, xls, xlsx, ppt, pptx, txt, rtf, epub, ps, xml, csv, odt, ods, odp)	Adds filetype:. Several values become filetype:pdf OR filetype:docx. Default [].	["pdf"]
wordsInTitle	No	array	Each entry becomes an intitle:"…" operator. Default [].	["annual"]
wordsInUrl	No	array	Each entry becomes an inurl:"…" operator. Default [].	["reports"]
forceExactMatch	No	boolean	Wraps the keyword line in quotes for a verbatim phrase match. Default false.	false
limit	No	string (select, enum: 10, 20, 30, 40, 50, 100, all)	How many result pages to walk. Each page returns up to 10 results and is a charged row. Default "all".	"20"
include_merged	No	boolean	Adds one extra row per keyword merging every page (searchQuery.page = "all"). Default true.	true
start	No	integer	Google start offset for the first result.	10
page	No	integer	Fetch only this page number (1 = first page).	1
country	No	string (select, ISO 3166-1 alpha-2, 249 entries)	Main country selector — sets Google gl and the default proxy region.	"US"
gl	No	string (select, same 249-entry ISO 3166-1 alpha-2 list)	Override the results country (Google gl).	"GB"
hl	No	string (select, 44 language codes)	Interface language (Google hl). The "N pages" line is only parsed in English.	"en"
lr	No	string (select, lang_xx codes matching the hl list)	Restrict results by content language (Google lr).	"lang_en"
cr	No	string (select, countryXX codes matching the country list)	Restrict results by country of origin (Google cr).	"countryUS"
tbs	No	string	Raw Google time filter, e.g. qdr:y (past year).	"qdr:y"
proxy_location	No	string (select, enum: us, ca)	Forces the proxy exit region independently of the Google country.	"us"
includeDocumentResults	No	boolean	Returns documentResults[] — the audit table. While on, Google's Web-only tab is not requested, because that tab drops the related-searches block this Actor reads. Default true.	true
onlyResultsMatchingAllOperators	No	boolean	Drops audited records that don't match every requested operator. Applies to documentResults[] only; organicResults[] is untouched. Default false.	false
onlyQueriesWithDocuments	No	boolean	Drops a keyword whose SERP carried no file-type badge on any result. The dropped keyword still gets an uncharged accounting row. Default false.	false
enrichOrganicResults	No	boolean	Replaces organicResults[].displayedUrl/.description/.emphasizedKeywords with the real printed values. Default false.	false
forceWebOnlyTab	No	boolean	Sends udm=14 on every request, reproducing the base Actor's request shape byte-for-byte. Default false.	false
dedupeResultsAcrossPages	No	boolean	Each row carries only its own page's new results instead of repeating pages 1..n. Default false.	false

Example JSON input

{
  "keyword": "annual report",
  "site": "europa.eu",
  "fileTypes": ["pdf"],
  "wordsInTitle": ["annual"],
  "wordsInUrl": [],
  "forceExactMatch": false,
  "limit": "20",
  "include_merged": true,
  "country": "US",
  "gl": "US",
  "hl": "en",
  "tbs": "qdr:y",
  "proxy_location": "us",
  "includeDocumentResults": true,
  "onlyResultsMatchingAllOperators": false,
  "onlyQueriesWithDocuments": false,
  "enrichOrganicResults": false,
  "forceWebOnlyTab": false,
  "dedupeResultsAcrossPages": false
}

Common pitfall: leaving limit at its default "all" walks up to 100 pages per keyword, and ⚠️ every page is a separate charged row — for a narrow operator search that only has one page of real hits, set limit to a small value like "10" or "20" instead of paying for 99 empty pages.

---

⬆️ Output

Results are pushed to the Apify dataset as typed, normalized JSON — one row per page, per keyword, plus an optional merged row per keyword. Export as JSON, CSV or Excel from the Output tab, or pull rows with the API/SDK.

Search result row

{
  "searchQuery": {
    "term": "annual report site:europa.eu filetype:pdf",
    "url": "https://www.google.com/search?q=annual+report+site%3Aeuropa.eu+filetype%3Apdf&gl=us&hl=en",
    "device": "DESKTOP",
    "page": 1,
    "type": "SEARCH",
    "domain": "google.com",
    "countryCode": "US",
    "languageCode": "en",
    "locationUule": null,
    "resultsPerPage": 10
  },
  "searchQueryTerm": "annual report site:europa.eu filetype:pdf",
  "url": "https://www.google.com/search?q=annual+report+site%3Aeuropa.eu+filetype%3Apdf&gl=us&hl=en",
  "hasNextPage": true,
  "serpProviderCode": "O",
  "resultsTotal": 1240,
  "relatedQueries": [
    { "title": "europa.eu annual report 2024 pdf", "url": "https://www.google.com/search?q=europa.eu+annual+report+2024+pdf" }
  ],
  "paidResults": [],
  "paidProducts": [],
  "aiOverview": null,
  "organicResults": [
    {
      "title": "Annual Report 2024 - European Union",
      "url": "https://www.srb.europa.eu/system/files/media/document/annual_report_2024.pdf",
      "displayedUrl": "www.srb.europa.eu",
      "description": "annual report site:europa.eu filetype:pdf",
      "emphasizedKeywords": ["annual report site:europa.eu filetype:pdf"],
      "siteLinks": [],
      "productInfo": {},
      "type": "organic",
      "position": 1
    }
  ],
  "suggestedResults": [
    { "title": "europa.eu annual report 2024 pdf", "url": "https://www.google.com/search?q=europa.eu+annual+report+2024+pdf", "type": "organic", "position": 1 }
  ],
  "peopleAlsoAsk": [
    { "answer": null, "question": "How many pages is the EU annual report?", "title": "How many pages is the EU annual report?", "url": "https://www.google.com/search?q=how+many+pages+eu+annual+report", "date": null }
  ],
  "customData": {
    "perplexitySearch": { "enablePerplexity": false, "searchRecency": null, "returnImages": false, "returnRelatedQuestions": false },
    "chatGptSearch": { "enableChatGpt": false },
    "maximumLeadsEnrichmentRecords": 0,
    "leadsEnrichmentDepartments": []
  },
  "htmlSnapshotUrl": null,
  "resolvedQuery": "annual report site:europa.eu filetype:pdf",
  "operatorsRequested": ["site:europa.eu", "filetype:pdf"],
  "documentResultsCount": 10,
  "documentsFound": 8,
  "fileTypeBreakdown": { "PDF": 8 },
  "operatorMatchSummary": { "site:europa.eu": 10, "filetype:pdf": 8 },
  "documentResults": [ { "...": "see the audited document record below" } ]
}

Audited document record (documentResults[] item)

{
  "position": 1,
  "title": "Annual Report 2024 - European Union",
  "url": "https://www.srb.europa.eu/system/files/media/document/annual_report_2024.pdf",
  "displayedUrl": "www.srb.europa.eu › media › document",
  "siteDomain": "www.srb.europa.eu",
  "description": "The Single Resolution Board's 2024 annual report summarises resolution planning and crisis-management activity across the Banking Union.",
  "emphasizedKeywords": ["annual", "report"],
  "fileType": "PDF",
  "fileTypeSource": "badge",
  "pageCountText": "79 pages",
  "isDocument": true,
  "matchedOperators": ["site:europa.eu", "filetype:pdf"]
}

fileTypeSource is "badge" when Google printed the file-type badge itself, and "url-extension" when no badge was on the page and the value was derived from the URL instead — the two are never mixed together silently. pageCountText is Google's own "N pages" line; Google only prints it on some results, so it is null on the rest and is never estimated.

All other row-level fields

Field	Type	Notes
hasNextPage	boolean	Whether a further page exists for this keyword
serpProviderCode	string	Constant "O" (organic)
resultsTotal	integer	Google's own "About N results" count, parsed from the page; 0 when not printed
paidResults[]	array	Same shape as organicResults[] items, with type: "paid"
paidProducts[]	array	title, displayedUrl, prices[] from shopping-style cards
peopleAlsoAsk[]	array	question, title, url; answer and date are always null — the Actor does not expand PAA answers
relatedQueries[] / suggestedResults[]	array or null	Google's real "Related searches" block; null when the page carries none, never fabricated. suggestedResults[] is the same data re-shaped with type/position
aiOverview	object or null	Only populated when an internal AI-overview lookup is configured — a path this Actor's public input schema does not expose, so on a normal Console run this is always null
customData	object	perplexitySearch, chatGptSearch, maximumLeadsEnrichmentRecords, leadsEnrichmentDepartments — legacy fields carried from the base Actor's output contract; they stay at their false/0/[] defaults on a normal Console run
htmlSnapshotUrl / html	string or null	Only present when the run was configured to persist the raw SERP HTML; absent otherwise
errorReason / errorMessage / charged	string / string / false	Present only on the uncharged accounting rows described below — never on a normal result row

⚠️ aiOverview, and every field inside customData, depend on internal configuration keys (aiMode, serpApiKey, perplexitySearch, chatGptSearch, lead-enrichment settings) that are not part of this Actor's public input schema — on a normal Console or API run through the documented keyword/site/fileTypes fields above, expect aiOverview: null and customData at its listed defaults every time.

An uncharged accounting row is written instead of a result row when a keyword is blocked after the full proxy retry ladder, or dropped by onlyQueriesWithDocuments. It carries errorReason (e.g. JS_CHALLENGE_BOOTSTRAP, CHALLENGE_PAGE, SORRY_REDIRECT, FILTERED_NO_DOCUMENT_RESULTS), errorMessage, charged: false, plus searchQueryTerm, resolvedQuery, operatorsRequested and searchQuery — but none of organicResults, paidResults, peopleAlsoAsk or the other result-only fields. To keep only billed rows in an export, filter out any item where errorReason is present (equivalently, charged !== false).

---

How does Google Search Results (SERP) Scraper By Site & File Type compare to other Google scrapers?

Feature	This Actor	Generic Google SERP scraper
Operator input	site:, filetype:, intitle:, inurl:, exact-phrase as dedicated fields	Usually only a free-text keyword field; operators have to be typed by hand into the query
Operator verification	Every result carries matchedOperators[], checked against its own URL/title/badge	Not verified — a loosened operator match looks identical to a real one
File-type audit	fileType, fileTypeSource, pageCountText per result	Not exposed
Related searches	Parses Google's real "Related searches" block; null when absent	Often fabricated from the first organic titles when the real block isn't parsed
Output format	Typed, normalized JSON, stable field names	Varies — often raw HTML or inconsistent field names

If you're building an AI agent or RAG pipeline, the output-format row is the decision-maker — parsing HTML inside an agent loop is a reliability failure mode, not a feature.

---

How many results can you scrape with Google Search Results (SERP) Scraper By Site & File Type?

Each page returns up to 10 organic results. limit controls how many pages are walked per keyword: numeric values (10–100) map to ceil(n / 10) pages, and "all" (the default) walks up to 100 pages — up to 1,000 results per keyword. start/page can target a specific offset or single page instead of walking from the top. There is no fixed cap on the number of keyword lines in one run. Every page fetched is a separate charged row, so a broad limit: "all" run on a keyword with few real matches produces mostly empty pages at cost — narrow the operator fields or lower limit for a tighter search.

---

Integrate Google Search Results (SERP) Scraper By Site & File Type and automate your workflow

Google Search Results (SERP) Scraper By Site & File Type works with any language or tool that can send an HTTP request.

REST API integration

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")

run = client.actor("<YOUR_USERNAME>/google-search-results-serp-scraper-by-site-and-file-type").call(
    run_input={
        "keyword": "annual report",
        "site": "europa.eu",
        "fileTypes": ["pdf"],
        "limit": "20",
        "includeDocumentResults": True,
    }
)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["resolvedQuery"], item.get("documentsFound"))

Works in Python, Node.js, Go, Ruby, cURL.

MCP for AI agents

This Actor is reachable through Apify's MCP server. Register it directly with https://mcp.apify.com?tools=<YOUR_USERNAME>/google-search-results-serp-scraper-by-site-and-file-type, or run it locally via npx @apify/actors-mcp-server --tools <YOUR_USERNAME>/google-search-results-serp-scraper-by-site-and-file-type. Compatible with Claude, Cursor, VS Code and any other MCP client.

Automation platforms (n8n, Make, LangChain)

In n8n, the official @apify/n8n-nodes-apify package's "Run Actor" and "Get Dataset Items" nodes point at this Actor's ID and pass the same input fields shown above. In Make, the official Apify app's "Run an Actor" module does the same, either synchronously or via a "Watch Actor Runs" trigger. For LangChain, ApifyWrapper.call_actor() from langchain_apify runs the Actor and hands its dataset rows straight to a document loader for RAG indexing.

---

Is it legal to scrape Google search results?

Yes — scraping publicly available search-results pages is generally lawful, and this Actor only returns what Google already shows any visitor: organic listings, paid results, People Also Ask and related searches. This is business and catalogue-style data (page titles, URLs, snippets, file-type badges), not personal profile data, so Google's Terms of Service and database-rights considerations apply rather than GDPR/CCPA. A search result can incidentally surface a person's name inside a page title or snippet, so consult legal counsel before storing or reusing bulk personal data extracted this way for commercial purposes.

---

Frequently asked questions

Does this Actor work without a Google account?

Yes. No Google account, cookie or login is required — the Actor fetches Google's public search-results pages through Apify's own proxy. Only an Apify account is needed to run it.

How often is the scraped data updated?

Every run fetches live from Google — there is no caching layer. Each keyword is searched fresh for that run.

What happens when a keyword is blocked or returns nothing?

The Actor retries through Apify's Google-SERP proxy, then escalates to residential proxies (three retries at each tier). If every attempt fails, it writes an uncharged accounting row with an errorReason (JS_CHALLENGE_BOOTSTRAP, CHALLENGE_PAGE, SORRY_REDIRECT, FETCH_FAILED, or similar) instead of a fabricated result row — nothing is billed for a keyword that produced no usable page.

Can I scrape gated or paywalled content this way?

No. The Actor returns only what's on Google's own public search-results page — titles, snippets and links. It never fetches or bypasses the linked pages themselves, so content behind a login or paywall on the destination site is not retrieved.

How is this Actor billed?

Pay-per-event: one row_result charge per result row actually pushed — a per-page row and, if include_merged is on, one merged row per keyword. Blocked or filtered-out keywords produce an uncharged accounting row instead, identifiable by charged: false. Exact pricing is set on the Actor's Console listing.

Does this Actor work for AI agent workflows and LLM pipelines?

Yes. It's callable as a plain HTTP endpoint by any agent framework, or through Apify's MCP server (https://mcp.apify.com) as a registered tool. Every response is typed JSON — no parsing step before passing it to an LLM.

How does this Actor handle Google's anti-bot system?

Requests start on Apify's Google-SERP proxy and escalate to a sticky residential proxy tier on any block, retrying up to three times per tier. Block detection checks for a real result container first — a page that has one is never flagged as blocked — before falling back to challenge-page and JS-bootstrap markers, so a genuine large results page is never mistaken for a block.

How does this Actor compare to other Google SERP scrapers?

Unlike a scraper that only exposes a free-text keyword field, this one turns site:, filetype:, intitle: and inurl: into dedicated inputs and audits every result against them with matchedOperators[], since Google itself doesn't always honour an operator strictly.

Does this Actor return data in a format LLMs can use directly?

Yes. Typed, normalized JSON with stable field names on every row. Pass it directly into an LLM context window, index it into a vector store, or route it through an agent tool call.

Can I use this Actor without managing proxies?

Yes. The Google-SERP-to-residential escalation is handled automatically inside the Actor — no separate proxy configuration is required to run it.

What happens when Google changes its structure or blocks the scraper?

The Actor is maintained and the output schema stays stable — field names and types don't change on your end. Every page-reading selector is checked against a ladder of fallback selectors rather than one fixed class, so a single markup change doesn't silently zero out a field.

---

Your feedback

Found a bug, or missing a field you need from a Google search result? We want to know. Use the Issues tab on this Actor's Apify Store page — active maintenance is how the schema stays stable for you.