Google Search Results Scraper

What is a Google scraper?

Our free SERP scraper crawls Google Search Results Pages (SERPs) and extracts data from those web pages in structured formats such as JSON, XML, CSV, or Excel. With this SERP Scraper API, you will be able to extract from each Google search page the following Google data:

🌱 Organic results	🛍 Paid results
📢 Product ads	❓Related queries
🙋‍♀️ People Also Ask	🏷 Prices
⭐️ Review rating and review count	🔍 Additional custom attributes

Why do you need a Google search API to scrape Google?

Since 2011, Google Search does not provide a functional official SERP API able to show how you or your competitors rank on Google. So the only way to efficiently monitor search results and ranking is to use web scraping tools.

Our free Google API gives you your own, customizable SERP scraper, a RESTful SERP API that provides real-time results optimized for structured output that you can download. Once you extract and download Google data, you can easily plug it into your data projects or integrate it with with other apps such as Zapier or Airbyte.

How can I use data scraped from Google Search?

🔸 Use it for search engine optimization (SEO) and monitor how your website performs on Google for certain queries over time.

🔹 Monitor how frequently a search term has been used on Google, and how it compares with total search volume.

🔸 Analyze display ads for a given set of keywords.

🔹 Monitor your competition in both organic and paid results.

🔸 Build a URL list for certain keywords. This is useful if, for example, you need good relevant starting points when scraping web pages containing specific phrases.

🔹 Analyze Google algorithm and identify its main trends

How do I use Google SERP Scraper?

To understand how to set up and run the Google SERP Scraper, check out our step-by-step guide on how to scrape Google Search with screenshots and examples or watch a short video tutorial ▷ on YouTube.

How much does it cost to scrape Google?

In terms of platform usage credits, it is our experience that you will get 5,000 results per month from Google Scraper using the Free plan. If you need to get more data regularly from google.com, we recommend our Starter plan which allows to get up to 1.5 million results every month. If even that is short for you, the Scale plan will get you 10 million Google results!

For more details about what platform credits are and how much of them this scraper consumes, see the platform pricing page or this video guide on ▷ how to choose the right subscription plan.

Do I need proxies to scrape Google SERPs?

Yes. To scrape SERPs effectively, you should use Apify Proxy and you need to set a sufficient limit for Google SERP queries (you can do it on your Account page). New Apify users have a free trial of Apify Proxy and Google SERPs, so you can use this scraper for free at the beginning.

However, this scraper consumes 1 SERP proxy per request, and since you have 500 SERP proxies for free, it leaves you at exactly 500 free requests within the Free plan - no matter the number of results you will get. You can make the best use of your SERP requests by maxing out results per page which is why it's set at 100 by default. You can always reduce the number of results per page by using the Results per Google page field.

Google Search data input

The scraper gives you really good control over what kind of Google Search results you'll get. You can specify the following settings:

• Query phrases or raw Google search URLs 🔍
• Country/search domain 🗺
• Language of search 🇬🇧
• Exact geolocation 📍
• Number of results per page 🔟
• Mobile or desktop version results 📱

For a complete description of all settings, see the Input tab.

How to scrape Google search results by URL or keyword

There are two ways you can scrape Google search pages: either by URL or by search term.

1. Scraping by URL will get you data from Google search results page. You'll get Google data from a copypasted Google URL, any domain. You can add as many URLs as you want.
2. Scraping by search term will also get you data from Google search results page. You can also add as many search terms as you want.

Both of these options are provided to fit.

How to scrape Google search results by URL

It's super easy to get Google Search results by URL. Just copypaste the Google URL you're interested in, and indicate the number of Google pages you want to scrape.

Here's its equivalent in JSON:

{
  "countryCode": "cz",
  "csvFriendlyOutput": false,
  "customDataFunction": "async ({ input, $, request, response, html }) => {\\n  return {\\n    pageTitle: $('title').text(),\\n  };\\n};",
  "includeUnfilteredResults": false,
  "languageCode": "en",
  "maxPagesPerQuery": 2,
  "mobileResults": false,
  "queries": "<https://www.google.com/search?q=hotels+in+seattle&rlz=1C5CHFA_enCZ962CZ963&oq=hotels+in+seattle&aqs=chrome..69i57j0i22i30l9.2554j0j7&sourceid=chrome&ie=UTF-8>",
  "resultsPerPage": 100,
  "saveHtml": false,
  "saveHtmlToKeyValueStore": false,
  "maxConcurrency": 10
}

How to scrape Google by search term

It is also easy to get Google search data by search term. Just enter the search term, number of Google pages to scrape. With this option, you also can:

• scrape by just one search term (one keyword)
• scrape by multiple keywords in parallel by adding more search terms
• indicate how many results you want to see per each Google page (10-100)
• indicate the country of search (domain), language, and UULE location parameter

Here's its equivalent in JSON:

{
  "countryCode": "us",
  "csvFriendlyOutput": false,
  "customDataFunction": "async ({ input, $, request, response, html }) => {\\n  return {\\n    pageTitle: $('title').text(),\\n  };\\n};",
  "includeUnfilteredResults": false,
  "languageCode": "en",
  "maxPagesPerQuery": 2,
  "mobileResults": false,
  "queries": "hotels in Seattle",
  "resultsPerPage": 100,
  "saveHtml": false,
  "saveHtmlToKeyValueStore": false,
  "maxConcurrency": 10
}

Google Search data output

The scraper stores its result in the default dataset associated with the scraper run, from which you can export it to various formats, such as JSON, XML, CSV, or Excel.

You can download the results directly from the platform with a button or from the Get dataset items API endpoint:

<https://api.apify.com/v2/datasets/[DATASET_ID]/items?format=[FORMAT]>

where [DATASET_ID] is the ID of the dataset and [FORMAT]can be csv, html, xlsx, xml, rss or json.

Output example (by search term)

For each Google Search results page, the dataset will contain a single record, which in JSON format looks as follows. Bear in mind that some fields have example values.

[{
  "searchQuery": {
    "term": "hotels in seattle",
    "device": "DESKTOP",
    "page": 1,
    "type": "SEARCH",
    "domain": "google.com",
    "countryCode": "US",
    "languageCode": null,
    "locationUule": null,
    "resultsPerPage": 10
  },
  "type": "paid",
  "adPosition": 1,
  "title": "Hotels In Seattle - Hotels in Seattle, WA",
  "url": "<https://www.booking.com/city/us/seattle.html>",
  "displayedUrl": "<http://www.booking.com/>",
  "description": "Choose from a wide range of properties which Booking.com offers. Search now! Book your Hotel in Seattle now. Quick, Easy Booking. No Reservation Costs. Save 10% with Genius. Hotels. No Booking Fees. Apartments. Motels. 24/7 Customer Service. Hostels. Villas.",
  "emphasizedKeywords": "Hotel in Seattle | Hotels"
},
{
  "searchQuery": {
    "term": "hotels in seattle",
    "device": "DESKTOP",
    "page": 1,
    "type": "SEARCH",
    "domain": "google.com",
    "countryCode": "US",
    "languageCode": null,
    "locationUule": null,
    "resultsPerPage": 10
  },
  "type": "paid",
  "adPosition": 2,
  "title": "Hotels In Seattle - Hotel Best Price Guaranteed - agoda.com",
  "url": "<https://www.agoda.com/city/seattle-wa-us.html>",
  "displayedUrl": "<https://www.agoda.com/>",
  "description": "Book Now & Save Big at Agoda®! Always The Lowest Price Guarantee. We’re Available Live 24 7 to Help. Don’t Get Stranded When Traveling! 24/7 Customer Support. 100% Real Reviews. Instant Confirmation. Easy & Fast Booking. Secure Payments. Up To 80% Off.",
  "emphasizedKeywords": ""
},
{
  "searchQuery": {
    "term": "hotels in seattle",
    "device": "DESKTOP",
    "page": 1,
    "type": "SEARCH",
    "domain": "google.com",
    "countryCode": "US",
    "languageCode": null,
    "locationUule": null,
    "resultsPerPage": 10
  },
  "type": "organic",
  "position": 1,
  "title": "THE 10 BEST Hotels in Seattle, WA for 2022 (from $79)",
  "url": "<https://www.tripadvisor.com/Hotels-g60878-Seattle_Washington-Hotels.html>",
  "displayedUrl": "<https://www.tripadvisor.com> › ... › Seattle",
  "description": "1. Hyatt Regency Seattle. Show prices · 2. Inn at the Market. Show prices · 3. Embassy Suites by Hilton Seattle Downtown Pioneer Square. Show prices · 4.",
  "emphasizedKeywords": "Seattle | Seattle",
  "productInfo": {},
  "personalInfo": {
    "name": "THE 10 BEST Hotels in Seattle, WA for 2022 (from $79)",
    "location": "Seattle",
    "cleanDescription": "Seattle",
    "rawText": "Seattle"
  }
},
{
  "searchQuery": {
    "term": "hotels in seattle",
    "device": "DESKTOP",
    "page": 1,
    "type": "SEARCH",
    "domain": "google.com",
    "countryCode": "US",
    "languageCode": null,
    "locationUule": null,
    "resultsPerPage": 10
  },
  "type": "organic",
  "position": 2,
  "title": "Top Hotels in Seattle, WA - Cancel FREE on most hotels",
  "url": "<https://www.hotels.com/de1481165/hotels-seattle-washington/>",
  "displayedUrl": "<https://www.hotels.com> › ... › Hotels in Washington",
  "description": "Green Tortoise Hostel Seattle · Hyatt Regency Seattle · Mediterranean Inn · citizenM Seattle Pioneer Square · The Paramount Hotel · Sheraton Grand Seattle · The ...",
  "emphasizedKeywords": "Seattle | Seattle | Seattle | Hotel | Seattle",
  "productInfo": {}
},
{
  "searchQuery": {
    "term": "hotels in seattle",
    "device": "DESKTOP",
    "page": 1,
    "type": "SEARCH",
    "domain": "google.com",
    "countryCode": "US",
    "languageCode": null,
    "locationUule": null,
    "resultsPerPage": 10
  },
  "type": "organic",
  "position": 3,
  "title": "The 10 Best Seattle Hotels (From $75) - Booking.com",
  "url": "<https://www.booking.com/city/us/seattle.html>",
  "displayedUrl": "<https://www.booking.com> › city › seattle",
  "description": "Budget hotels in Seattle ; Ballard Studio. Hotel in Seattle · From $55 per night ; Sonder The Boylston. Hotel in Capitol Hill, Seattle · From $197 per night.",
  "emphasizedKeywords": "hotels in Seattle | Hotel in Seattle",
  "productInfo": {},
  "personalInfo": {
    "name": "The 10 Best Seattle Hotels (From $75)",
    "location": "hotels in Seattle",
    "cleanDescription": "Hotel in Seattle",
    "rawText": "hotels in Seattle"
  }
},
{
  "searchQuery": {
    "term": "hotels in seattle",
    "device": "DESKTOP",
    "page": 1,
    "type": "SEARCH",
    "domain": "google.com",
    "countryCode": "US",
    "languageCode": null,
    "locationUule": null,
    "resultsPerPage": 10
  },
  "type": "organic",
  "position": 4,
  "title": "16 Best Hotels in Seattle. Hotels from $39/night - KAYAK",
  "url": "<https://www.kayak.com/Seattle-Hotels.7054.hotel.ksp>",
  "displayedUrl": "<https://www.kayak.com> › ... › Hotels in West Coast",
  "description": "Most recommended Seattle hotels ; Ballard Inn. 7.7 Good · $180+ ; Hampton Inn & Suites Seattle-Downtown. 8 Excellent · $189+ ; MarQueen Hotel. 8.2 Excellent · $207+.",
  "emphasizedKeywords": "Seattle hotels | Seattle | Hotel",
  "productInfo": {}
},
{
  "searchQuery": {
    "term": "hotels in seattle",
    "device": "DESKTOP",
    "page": 1,
    "type": "SEARCH",
    "domain": "google.com",
    "countryCode": "US",
    "languageCode": null,
    "locationUule": null,
    "resultsPerPage": 10
  },
  "type": "organic",
  "position": 5,
  "title": "Top Hotels in Seattle, WA from $99 - Expedia",
  "url": "<https://www.expedia.com/Seattle-Hotels.d178307.Travel-Guide-Hotels>",
  "displayedUrl": "<https://www.expedia.com> › ... › Washington",
  "description": "Recent reviews of Seattle hotels. expedia verified reviews · Coast Gateway Hotel · Country Inn & Suites by Radisson, Seattle-Tacoma International Airport, WA.",
  "emphasizedKeywords": "Seattle hotels | Hotel | Seattle",
  "productInfo": {}
},
{
  "searchQuery": {
    "term": "hotels in seattle",
    "device": "DESKTOP",
    "page": 1,
    "type": "SEARCH",
    "domain": "google.com",
    "countryCode": "US",
    "languageCode": null,
    "locationUule": null,
    "resultsPerPage": 10
  },
  "type": "organic",
  "position": 6,
  "title": "Seattle Hotels from $81 - Hotel Deals - Travelocity",
  "url": "<https://www.travelocity.com/Seattle-Hotels.d178307.Travel-Guide-Hotels>",
  "displayedUrl": "<https://www.travelocity.com> › ... › Washington",
  "description": "Most frequently booked Seattle hotels · Hyatt Regency Seattle · Mediterranean Inn · citizenM Seattle Pioneer Square · The Paramount Hotel · The Belltown Inn.",
  "emphasizedKeywords": "Seattle hotels | Seattle | Seattle | Hotel",
  "productInfo": {}
},
{
  "searchQuery": {
    "term": "hotels in seattle",
    "device": "DESKTOP",
    "page": 1,
    "type": "SEARCH",
    "domain": "google.com",
    "countryCode": "US",
    "languageCode": null,
    "locationUule": null,
    "resultsPerPage": 10
  },
  "type": "organic",
  "position": 7,
  "title": "11 Best Hotels in Seattle (WA), United States - Agoda",
  "url": "<https://www.agoda.com/city/seattle-wa-us.html>",
  "displayedUrl": "<https://www.agoda.com> › ... › Washington Hotels",
  "description": "10 best hotels in Seattle (WA) ; Belltown Inn. Belltown,Seattle (WA) - View on map · Pets allowed ; Motel 6-Seattle, WA - Sea-Tac Airport South. SeaTac Int.",
  "emphasizedKeywords": "hotels in Seattle",
  "productInfo": {}
},
]
    ...

How to get one search result per row

We already have an easy CSV friendly output toggle button just for this case, but if you need to do it programmatically, here's your instruction.

Built-in approach

If you're only interested in Google Search results and want to get one organic or paid result per row on output, simply set true for csvFriendlyOutput input field. This option is switched off by default as it excludes all additional fields apart from searchQuery, organicResults and paidResults to preserve CSV-friendly format. It also removes siteLinks array from both organic and paid results and stringifies emphasizedKeywords array. The resulting dataset stores an array of organic and paid results.

An organic result is represented using the following format:

{
  "searchQuery": {
    "term": "laptop",
    "device": "DESKTOP",
    "page": 1,
    "type": "SEARCH",
    "domain": "google.com",
    "countryCode": "US",
    "languageCode": "en",
    "locationUule": null,
    "resultsPerPage": 10
  },
  "type": "organic",
  "position": 1,
  "title": "Laptops & Notebook Computers - Best Buy",
  "url": "<https://www.bestbuy.com/site/computers-pcs/laptop-computers/abcat0502000.c?id=abcat0502000>",
  "displayedUrl": "<https://www.bestbuy.com> › Computers & Tablets",
  "description": "Shop Best Buy for laptops. Work & play from anywhere with a notebook computer. We can help you find the best laptop for your specific needs in store and online.",
  "emphasizedKeywords": "laptops | laptop",
  "productInfo": {}
}

Compared to a paid result example:

{
  "searchQuery": {
    "term": "laptop",
    "device": "DESKTOP",
    "page": 7,
    "type": "SEARCH",
    "domain": "google.com",
    "countryCode": "US",
    "languageCode": "en",
    "locationUule": null,
    "resultsPerPage": 10
  },
  "type": "paid",
  "adPosition": 1,
  "title": "Affordable Laptops For Sale - Buy A Discounted Laptops",
  "url": "<https://rebornlaptops.com/category/laptops-and-computers>",
  "displayedUrl": "<https://www.rebornlaptops.com/laptops>",
  "description": "laptop hplaptop cheaplaptop applelaptop amazongaming laptoplaptop dealslaptops walmartlaptop best buy",
  "emphasizedKeywords": ""
}

Note the difference in type field value and position vs adPosition fields in a paid result format. Paid result position is calculated separately from the organic results and it's saved under adPosition field instead of position field.

Alternative approach via an API call

You can also pass query parameters fields=searchQuery,organicResults and unwind=organicResults to the API endpoint URL:

<https://api.apify.com/v2/datasets/[DATASET_ID]/items?format=[FORMAT]&fields=searchQuery,organicResults&unwind=organicResults>

The API will return a result like this (in JSON format):

[
  {
    "searchQuery": {
      "term": "Restaurants in Prague",
      "page": 1,
      // ...
    },
    "title": "THE 10 BEST Restaurants in Prague 2019 - TripAdvisor",
    "url": "<https://www.tripadvisor.com/Restaurants-g274707-Prague_Bohemia.html>",
    "displayedUrl": "<https://www.tripadvisor.com/Restaurants-g274707-Prague_Bohemia.html>",
    "description": "Best Dining in Prague, Bohemia: See 617486 TripAdvisor traveler reviews of 6232 Prague restaurants and search by cuisine, price, location, and more.",
    "siteLinks": []
  },
  {
    "searchQuery": {
      "term": "Restaurants in Prague",
      "page": 1,
      // ...
    },
    "title": "The 11 Best Restaurants in Prague | Elite Traveler",
    "url": "<https://www.elitetraveler.com/finest-dining/restaurant-guide/the-11-best-restaurants-in-prague>",
    "displayedUrl": "<https://www.elitetraveler.com/finest-dining/restaurant.../the-11-best-restaurants-in-prag>...",
    "description": "Jan 16, 2018 - With the regional fare certainly a highlight of dining in Prague, a great number of superb international eateries have touched down to become ...",
    "siteLinks": []
  },
  // ...
]

When using a tabular format such as csv or xls, you'll get a table where each row contains just one organic result. For more details about exporting and formatting the dataset records, please see the documentation for the Get dataset items API endpoint.

Tips and tricks

• You can scrape a lot of results more efficiently with a single query by setting Results per Google page to 100, so that you get 100 results per 1 page instead of crawling 10 pages, each with 10 results.
• If you are not sure that the results are complete and of good quality, each run stores the full HTML page to the default Key-Value Store. You can view it by clicking on it and comparing the results. Our team is constantly monitoring the quality but we are happy for any reports.

Disclaimers

How do I scrape millions of results from Google?

Please note that, although Google always shows that it has found millions of results, Google will never display more than a few hundred results for a single search query 🤥 If your goal is to get as many results as possible, try creating many similar queries and combine different parameters and locations.

How do I scrape Google ads?

Despite the fact that scraping Google is the best shot we've got at seeing objective Google results, displayed paid results are still heavily dependent on your location and browsing history. It’s also up to Google to choose which ads to show to which user (hello, cookies 🍪). It might be underwhelming to see less results than you'd expected. Most probably, Google's precise algorithm is the reason for it.

Is it legal to scrape Google search results?

Web scraping is legal if you are extracting publicly available data, but you should respect boundaries such as personal data and intellectual property regulations. Rule of thumb is: you should only scrape personal data if you have a legitimate reason to do so, factoring in Google's Terms of Use as well. If you're unsure whether your reason is legitimate, consult your lawyers. We also recommend that you read our blog post on the subject: is web scraping legal?

Can I integrate Google Scraper with other apps?

Last but not least, Google Search Results Scraper can be connected with almost any cloud service or web app thanks to integrations on the Apify platform. You can integrate with Make, Zapier, Slack, Airbyte, GitHub, Google Sheets, Google Drive, and more. Or you can use webhooks to carry out an action whenever an event occurs, e.g. get a notification whenever Google Search Results Scraper successfully finishes a run.

Can I use Google Scraper with the API?

The Apify API gives you programmatic access to the Apify platform. The API is organized around RESTful HTTP endpoints that enable you to manage, schedule, and run Apify Actors. The API also lets you access any datasets, monitor actor performance, fetch results, create and update versions, and more.

To access the API using Node.js, use the apify-client NPM package. To access the API using Python, use the apify-client PyPi package.

Check out the Apify API reference docs for full details or click on the API tab for code examples.

What are other tools for scraping Google services?

▶️ Google Play Reviews Scraper	📍 Google Maps Scraper
🔍 Google Trending Searches	🛍 Google Shopping Scraper
📈 Google Trends Scraper	⭐️ Google Maps Reviews Scraper
📩📍 Google Maps with Contact Details	🤟 Google Datasets Translator

Resources on how to scrape Google Search Pages

1. Is web scraping legal? - blogpost on ethical scraping.
2. Platform pricing page with pricing specifications.
3. Video guide ▷ on how to choose the right subscription plan.
4. Step-by-step guide on how to use Google SERP scraper.
5. Video tutorial ▷ on how to use Google SERP scraper.
6. Input tab with all the technical parameters of this scraper.
7. A list of other Google-related scrapers.