Google Search Results Scraper (SERP) + Rank Tracker

Scrape Google search results by keyword and get back the real destination URL of every result — https://en.wikipedia.org/wiki/Web_scraping, not Google's /goto?url=CAESSgHrOzAV… redirect marker. Organic positions, related queries, AI Overview sources and People-also-ask questions arrive in one table, and a second run tells you what moved.

No browser, no API key, no external service. Pure HTTP through Apify Proxy.

Why the URL matters more than it sounds

Google today serves two different pages for the same query:

The marker in the second column is encrypted. Measured on 35 such pages: the full address could be recovered for only 3 results out of 7 — the rest exist on the page as a truncated breadcrumb (

www.geeksforgeeks.org › blogs › what-is-web-s…

Which page you get is not decided by the query. It is decided by the proxy session. Measured on 17.09.2026 over 81 captured pages:

This Actor asks for the lightweight page with a fresh session, checks what came back, and retries with another session if Google handed it markers instead of addresses. Retries are not charged to you — you pay per delivered page.

What one page costs us, and why the price is what it is

A real cloud run, 1024 MB, 6 pages over 3 queries, measured end to end: 0.00345 compute units, 0 Google-SERP-proxy requests, $0.0016 of platform usage for the whole run — that is $0.267 per 1,000 search result pages in platform cost; the Actor itself is priced at $1.99 per 1,000 rows. The reference Actor in this category charges $4.50 per 1,000 pages.

That number is only possible because of one measurement: Google accepts Apify's ordinary datacenter proxy for the lightweight page. 24 requests out of 24, across 12 different queries, came back as readable results with no challenge at all. The residential proxy, by contrast, hit Google's "unusual traffic" page on the first try, both times. The paid Google SERP proxy is kept as an automatic fallback, not as the normal path.

What you get per row

Twenty-two columns. The type column says which block a row came from.

Every row also carries the query, the page number, the start offset, country, language, device, which version of Google's page it came from, the exact search URL, and the timestamp.

Rank tracking, which is what this is really for

Turn on trackPositions and schedule the run. The Actor remembers where each URL stood for each query and returns only what moved:

• change_type — NEW, UP, DOWN, UNCHANGED or GONE
• previous_position — where it stood last time
• position_change — +6 means it climbed six places

GONE rows are the half that other scrapers leave out: a URL that dropped out of the results entirely does not appear in today's SERP, so a plain scraper simply stops mentioning it. Here it comes back as its own row with the position it used to hold.

Unchanged positions are not returned by default, so a daily run does not bill you for the same table every morning. Switch on emitUnchanged when you want the full picture anyway.

How deep it goes

Google ignores the num parameter today. Measured: num=20, num=30 and num=100 returned byte-identical pages with the same 6–10 results. Depth is therefore the only lever, and maxPagesPerQuery walks start=0, start=10, start=20 and so on.

The ceiling is Google's own and depends on the query. Measured on

web scraping

Filters that cannot silently empty your dataset

domainContains, excludeDomains and titleContains apply to organic rows. A row is never dropped because a field is empty — absence of data is not a mismatch. And if the filters do remove everything, the run says so in words rather than handing you an empty table.

Every empty result is explained by name

A successful run with an empty dataset is the worst kind of failure: it looks fine and you paid for nothing. So the run report always carries an empty_reason, and the log says the same thing in a sentence:

• no-queries — nothing was typed in.
• no-blocks-selected — every result block was switched off.
• filters-need-organic-results — a domain or title filter with organic results switched off.
• one-attempt-cannot-guarantee-links — requireResolvedLinks with maxAttempts: 1.
• charge-limit-reached — your spending limit for the run was already used up.
• google-returned-no-results — Google itself said "did not match any documents".
• filtered-out — results were collected, your filters removed all of them.
• links-not-resolved — every attempt came back with encrypted markers.
• nothing-changed — rank tracking working as asked: no position moved.
• blocked-by-google — a challenge page on every attempt.
• source-returned-nothing — the one case worth reporting: the page could not be read as search results at all.

The first four are decided by your input alone, so they are answered before a single request is made and cost you nothing.

Limitations, named honestly

• No paid results (ads). Through this data source Google returns the ad container empty: measured on best running shoes and buy laptop online, zero /aclk links and zero "Sponsored" labels on every capture. The Actor does not invent them and does not charge for them. If ads are what you need, another Actor in this category attempts them.
• No total result count. Google's own result-stats element came back as "About 0 results" through this route, so the field would be a lie. It is not in the output.
• People also ask costs extra. That block does not exist on the lightweight page at all — 0 questions across 40 captures — and the JavaScript page is not served to the datacenter proxy: without gbv=1 it answers with a 92 KB redirect notice carrying no results. So the block is fetched through Apify's paid Google SERP proxy, is off by default, and is billed as its own event.
• No Maps, Shopping, Images or News tabs. This Actor reads the main web results tab only.
• No start URLs. You give keywords; the Actor builds the Google URL. Paste a ready-made Google URL and it will not be used.
• num does nothing. Not our choice — see above.
• Roughly one result in twenty arrives with an encrypted link. Measured: 200 of 209 rows over 24 pages carried a real address. A retry with a fresh session clears it; with requireResolvedLinks on, anything left over is dropped rather than handed to you as a row with an empty url.

FAQ

Do I need a proxy? Yes, and Apify Proxy is enough. A direct request from the Apify cloud gets Google's cookie-consent page and no results. The default setting is the shared datacenter pool, which measured 24 readable pages out of 24.

Will Google block this? It challenged the residential pool immediately and the datacenter pool not once in 24 requests. If a challenge does appear, the run switches to Apify's Google SERP proxy by itself, marks the page in the report and carries on.

Why is there a retry counter in the report? Because honesty about cost is part of the product. requests_per_page tells you how many HTTP requests a delivered page actually took. In the verification run it was 1.0.

Can I search from another country? Set countryCode (Google's gl) and languageCode (hl). They change which results rank, not just the wording.

Does mobile ranking differ? Yes, and device: mobile asks Google for it with a mobile user agent.

How many results per page? Six to ten. Google decides, and the num parameter no longer changes it.

What is the rendering column? Which of Google's two pages a row came from: basic is the lightweight page with real URLs, rich is the JavaScript page, history marks a GONE row that comes from memory rather than from today's search.

Can I get only the columns I need? compactOutput gives nine: position, type, title, url, domain, query, page, change_type, position_change. excludeEmptyFields drops empty columns entirely.

How do I stop it from spending too much? Set a spending limit on the run. The Actor reads it at the start and stops at exactly that many pages instead of crawling on unpaid.

Is the output stable enough to schedule? The parser works on the structure of the page — container, link with a heading, caption, description — and not on Google's shuffling class names. The verification suite runs against 8 real captured pages, including one where Google encrypted every link, one where it found nothing, and one challenge page.

Input in one look

{
  "queries": ["best running shoes", "running shoes review"],
  "countryCode": "us",
  "languageCode": "en",
  "maxPagesPerQuery": 2,
  "trackPositions": true,
  "includePeopleAlsoAsk": false,
  "excludeDomains": ["pinterest.com"]
}

Output in one look

{
  "position": 1,
  "type": "organic",
  "title": "The 15 Best Running Shoes of 2026, Tested by Runner's World",
  "url": "https://www.runnersworld.com/gear/a19663621/best-running-shoes/",
  "displayed_link": "www.runnersworld.com › Running Gear › Running Shoes",
  "domain": "runnersworld.com",
  "description": "For new runners we recommend a shoe like the Brooks Ghost…",
  "date": "Aug 3, 2026",
  "query": "best running shoes",
  "page": 1,
  "start": 0,
  "country_code": "us",
  "language_code": "en",
  "device": "desktop",
  "rendering": "basic",
  "change_type": "UP",
  "previous_position": 4,
  "position_change": 3
}