Actor picture

Easy Google Search Scraper

hooli/easy-google-scraper

Make collecting data from Google easy with Easy Google Scraper. Extract organic and paid Google search engine results pages, then download your data as HTML table, JSON, CSV, Excel, or XML.

No credit card required

Author's avatarHooli
  • Modified
  • Users276
  • Runs1,742
Actor picture
Easy Google Search Scraper

.actor/actor.json

{
    "actorSpecification": 1,
    "name": "easy-google-search-scraper",
    "title": "Easy Google Search Scraper",
    "description": "Scrapes results from Google",
    "version": "0.0.1",
    "storages": {
        "dataset": {
            "actorSpecification": 1,
            "title": "Easy Google Search Scraper",
            "description": "Too see all scraped properties, export the whole dataset or select All fields instead of Overview",
            "views": {
                "overview": {
                    "title": "Overview",
                    "description": "",
                    "transformation": {
                        "fields": [
                            "searchQuery",
                            "type",
                            "title",
                            "url",
                            "description",
                            "emphasizedKeywords"
                        ]
                    },
                    "display": {
                        "component": "table",
                        "columns": [
                            {
                                "label": "Title",
                                "format": "text",
                                "field": "title"
                            },
                            {
                                "label": "Description",
                                "format": "text",
                                "field": "description"
                            },
                            {
                                "label": "emphasizedKeywords",
                                "format": "text",
                                "field": "emphasizedKeywords"
                            },
                            {
                                "label": "URL",
                                "format": "link",
                                "field": "url"
                            },
                            {
                                "label": "Type (organic/paid)",
                                "format": "text",
                                "field": "type"
                            },
                            {
                                "label": "Search term",
                                "format": "text",
                                "field": "searchQuery.term"
                            },
                            {
                                "label": "Page number",
                                "format": "number",
                                "field": "searchQuery.page"
                            },
                            {
                                "label": "Country code",
                                "format": "text",
                                "field": "searchQuery.countryCode"
                            }
                        ]
                    }
                }
            }
        }
    }
}

Dockerfile

# First, specify the base Docker image. You can read more about
# the available images at https://sdk.apify.com/docs/guides/docker-images
# You can also use any other image from Docker Hub.
FROM apify/actor-node-puppeteer-chrome:16

# Second, copy just package.json and package-lock.json since those are the only
# files that affect "npm install" in the next step, to speed up the build.
COPY package*.json ./

# Install NPM packages, skip optional and development dependencies to
# keep the image small. Avoid logging too much and print the dependency
# tree for debugging
RUN npm --quiet set progress=false \
 && npm install --only=prod --no-optional \
 && echo "Installed NPM packages:" \
 && (npm list --all || true) \
 && echo "Node.js version:" \
 && node --version \
 && echo "NPM version:" \
 && npm --version

# Next, copy the remaining files and directories with the source code.
# Since we do this after NPM install, quick build will be really fast
# for most source file changes.
COPY . ./

ENV APIFY_DISABLE_OUTDATED_WARNING 1
ENV npm_config_loglevel=silent

# Optionally, specify how to launch the source code of your actor.
# By default, Apify's base Docker images define the CMD instruction
# that runs the Node.js source code using the command specified
# in the "scripts.start" section of the package.json file.
# In short, the instruction looks something like this:
#
# CMD npm start

INPUT_SCHEMA.json

This file is 506 lines long. Only the first 50 are shown. Show all

{
  "title": "Google Search Scraper input",
  "description": "Specify Google Search queries.",
  "type": "object",
  "schemaVersion": 1,
  "properties": {
      "search": {
          "title": "Search",
          "type": "array",
          "description": "Google Search queries (e.g. <code>food in NYC</code>)",
          "prefill": ["Hotels in NYC"],
          "editor": "stringList"
      },
      "countryCode": {
          "title": "Country",
          "type": "string",
          "description": "Country determines the IP address of the proxy used for the query and the Google Search domain (e.g. <code>google.es</code> for Spain). The values must be lower-cased <a href=\"https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2\" target=\"_blank\">ISO 3166</a> country codes supported by Google. By default, the actor uses United States (<code>google.com</code>).<br><br>This setting only applies to <strong>Search queries</strong>, but not to <strong>URLs</strong>.",
          "prefill": "",
          "default": "",
          "editor": "select",
          "enum": [
              "",
              "af",
              "al",
              "dz",
              "as",
              "ad",
              "ao",
              "ai",
              "aq",
              "ag",
              "ar",
              "am",
              "aw",
              "au",
              "at",
              "az",
              "bs",
              "bh",
              "bd",
              "bb",
              "by",
              "be",
              "bz",
              "bj",
              "bm",
              "bt",
              "bo",
              "ba",
              "bw",

README.md

## What does Easy Google scraper do?
Easy Google Scraper enables you to extract data from Google search engine results pages (SERPs).

Easy Google Scraper can scrape:
-  organic results
-  paid results based on your search term

## Why scrape Google?
Google is a great source of data for virtually any business or [industry](https://apify.com/industries)

Here are just some of the ways you could use that data:
-  monitor competition
-  research your position

## How to scrape Google.com
It's easy to [scrape Google.com](https://google.com/) with Easy Google Scraper. Just follow these few steps and you'll get your data in a few minutes.

1. Click on *Try for free*.
2. Enter the keywords or search terms you want to scrape. Or enter a URL to start scraping.
3. Click on *Start*.
4. When Easy Google Scraper has finished its run, you can preview or download your data from the Dataset tab.

## How much will it cost to scrape Google?
Apify gives you $5 of credit every month with the Apify **Free plan**. You can get 5,000 results per month from Easy Google Scraper for that, so those 5,000 results will be completely free!

But if you need to get more data regularly from Google.com, you should get an Apify subscription. We recommend our $49 per month **Personal plan**. With this plan, you can get up to 1.5 million results every month.

If that is not enough for you, the **Team plan** for $499 will get you 10 million results!

## Results
Example results of searching web scraping on Google: ![Easy Google Scraper dataset](https://apifyusercontent.com/23570fbcea9fd3002b6bbcce7e4bc355cd35d7a0/68747470733a2f2f692e696d6775722e636f6d2f4f7275544642302e706e67) 

## Need something more advanced?
Try our more advanced [Google Search Results Scraper](https://apify.com/apify/google-search-scraper) if you need a greater volume or variety of data. 

Let us know if you need a [custom Google scraping solution](https://apify.com/custom-solutions).

## Tips for scraping Google.com
-  You will typically not get more than 300 results for one keyword (usually even less). If you want more, you have to use different combinations of keywords, e.g., New York, New York City, New York life, etc.
-  Don't expect to get the same results as you see in the browser; Google personalizes content for you every step of the way...

## Is it legal to scrape Google.com?
Note that personal data is protected by GDPR in the European Union and by other regulations around the world. You should not scrape personal data unless you have a legitimate reason to do so. If you're unsure whether your reason is legitimate, consult your lawyers. We also recommend that you read our blog post: [is web scraping legal?](https://blog.apify.com/is-web-scraping-legal/)

main.js

import Apify from 'apify';

Apify.main(async () => {
    const input = await Apify.getInput();
    const queries = input.search.join('\n')

    await Apify.metamorph('apify/google-search-scraper', {
        ...input,
        queries,
        resultsPerPage: 100,
        maxPagesPerQuery: 5,
        csvFriendlyOutput: true
    });
});

package.json

{
    "name": "easy-google-scraper",
    "version": "0.0.1",
    "type": "module",
    "dependencies": {
        "apify": "^2.1.0"
    },
    "scripts": {
        "start": "node main.js"
    }
}