Search Engine Scraper

Description

This Apify Actor scrapes search results from Google based on a provided query and customizable parameters. It supports various search types (web, images, news, etc.) and allows filtering by language, region, date, and more. The Actor returns structured data including links, titles, text snippets, and other relevant information.

Input Schema

The Actor accepts the following input parameters to configure the scraping process:

Field	Type	Description	Default Value	Required
query	String	The search query to look for	python scraping	Yes
search_type	String	Type of search (e.g., 'web', 'images', 'news')	web	No
results_per_page	Integer	Number of results per page	10	No
pages	Integer	How many pages to crawl from search results	1	No
start_page	Integer	Page number to start crawling from	1	No
unique_urls	Boolean	Return only unique URLs	false	No
unique_domains	Boolean	Return only unique domains	false	No
language	String	Restrict results to a specific language (e.g., 'english')	-	No
region	String	Restrict results to a specific region (e.g., 'US')	-	No
last_update	String	Restrict results to a certain update period (e.g., 'day', 'week')	-	No
site	String	Restrict results to a specific site	-	No
custom_url	String	Custom Google search URL with parameters (for advanced users)	-	No
all_words	String	All these words must appear in results	-	No
exact_phrase	String	Exact phrase to appear in results	-	No
any_words	String	Any of these words can appear in results	-	No
none_words	String	None of these words should appear in results	-	No
numbers_from	Integer	Minimum number value to appear in results (e.g., for price lookup)	-	No
numbers_to	Integer	Maximum number value to appear in results (e.g., for price lookup)	-	No
terms_appearing	String	Where terms should appear (e.g., 'anywhere', 'title', 'url')	-	No
file_type	String	Restrict results to a specific file type (e.g., 'pdf')	-	No
safe_search	Boolean	Enable safe search filtering	true	No
from_date	String	Restrict results from this date (YYYY-MM-DD)	-	No
to_date	String	Restrict results up to this date (YYYY-MM-DD)	-	No
custom_tbs	String	Custom Google TBS parameter (for advanced users)	-	No
delay	String	Min and max delay between requests (e.g., '0.6,0.9')	0.6,0.9	No
proxy_configuration	Object	Proxy settings for the Actor run	{useApifyProxy: true, apifyProxyGroups: ["RESIDENTIAL"], apifyProxyCountry: "US"}	No

Example Input

{
  "delay": "0.6,0.9",
  "query": "python",
  "safe_search": true,
  "search_type": "images",
  "unique_domains": false,
  "unique_urls": false,
  "results_per_page": 10,
  "pages": 1,
  "start_page": 1,
  "proxy_configuration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"],
    "apifyProxyCountry": "US"
  }
}

Output Schema

The Actor outputs data in the following format:

Field	Type	Description
host	String	Hostname of the result URL
link	String	URL of the search result
image	String	Image URL (if applicable)
title	String	Title of the search result
text	String	Text snippet from the search result
engine	String	Search engine used (e.g., Google)
page	Number	Page number of the result

Example Output

[
  {
    "host": "example.com",
    "link": "https://example.com/python-guide",
    "image": null,
    "title": "Python Programming Guide",
    "text": "A comprehensive guide to Python programming for beginners and experts.",
    "engine": "Google",
    "page": 1
  },
  {
    "host": "docs.python.org",
    "link": "https://docs.python.org/3/tutorial/",
    "image": null,
    "title": "Python Tutorial",
    "text": "Official Python tutorial covering the basics and advanced topics.",
    "engine": "Google",
    "page": 1
  }
]

Supported Regions

The Actor supports restricting search results to the following regions:

Region	Region	Region	Region
Afghanistan	Albania	Algeria	Andorra
Argentina	Australia	Austria	Belgium
Brazil	Canada	Chile	China
Colombia	Czech Republic	Denmark	Egypt
Finland	France	Germany	Greece
Hong Kong	India	Indonesia	Ireland
Israel	Italy	Japan	Kenya
Malaysia	Mexico	Netherlands	New Zealand
Nigeria	Norway	Pakistan	Philippines
Poland	Portugal	Romania	Russia
Saudi Arabia	Singapore	South Africa	South Korea
Spain	Sweden	Switzerland	Taiwan
Thailand	Turkey	Ukraine	United Kingdom
United States	Vietnam

Supported Languages

The Actor supports restricting search results to the following languages:

Language	Language	Language	Language
Arabic	Chinese (Simplified)	Chinese (Traditional)	Czech
Danish	Dutch	English	Estonian
Finnish	French	German	Greek
Hebrew	Hindi	Hungarian	Icelandic
Indonesian	Italian	Japanese	Korean
Latvian	Lithuanian	Norwegian	Persian
Polish	Portuguese	Romanian	Russian
Spanish	Swedish	Thai	Turkish
Ukrainian	Vietnamese

Using Custom TBS Parameter

The custom_tbs parameter allows advanced users to specify custom Google Time-Based Search (TBS) parameters to fine-tune search results. TBS parameters are used in Google search URLs to filter results by time, type, or other criteria. For example, tbs=qdr:h restricts results to the past hour.

How to Use custom_tbs

• Format: Provide the TBS parameter as it appears in a Google search URL (e.g., qdr:w for results from the past week).
• Examples:
  • tbs=qdr:d: Results from the past day.
  • tbs=qdr:m: Results from the past month.
  • tbs=li:1: Verbatim search (exact match without Google's query expansion).
  • tbs=iar:i: Filter for image search results.
• Documentation: For a comprehensive list of TBS parameters, refer to the Scrapeless Google Search API documentation.
• Note: Use custom_tbs cautiously, as invalid or unsupported parameters may result in errors or unexpected behavior. Ensure compatibility with other input parameters like last_update, as custom_tbs may override them.

Example with custom_tbs

{
  "query": "python tutorial",
  "search_type": "web",
  "custom_tbs": "qdr:y",
  "proxy_configuration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"],
    "apifyProxyCountry": "US"
  }
}

This configuration restricts results to those indexed in the past year.

Advanced Usage Tips

• Custom URL: The custom_url parameter allows advanced users to provide a complete Google search URL with custom parameters. This overrides most other filters (e.g., language, region, last_update). Ensure the URL is valid and follows the pattern ^(http|https)://[\w\-\.]+([:\d]+)?([\w\-\._~:/?#\[\]@!$&'()*+,;=])*$.
  • Example: "custom_url": "https://www.google.com/search?q=python+tutorial&hl=en&tbs=qdr:m"
• Proxy Configuration: Use Apify Proxy for reliable scraping, or configure custom proxies for specific needs. The default configuration uses residential proxies in the US for better success rates.
• Rate Limiting: Adjust the delay parameter (e.g., "0.6,0.9") to control the time between requests, reducing the risk of being blocked by Google.
• Combining Filters: Combine all_words, exact_phrase, any_words, and none_words for precise query construction. For example:
  • "all_words": "python tutorial", "exact_phrase": "data science" ensures results contain both "python tutorial" and the exact phrase "data science".
• File Type Filtering: Use file_type to scrape specific document types (e.g., "pdf") for targeted results like whitepapers or reports.
• Safe Search: Disable safe_search (false) for less restrictive results, but be cautious of potentially sensitive content.

For further assistance or to report issues, contact the Apify support team or refer to the Apify documentation.