Fast Google Scraper

2024-08-22 (0.0.128)

Fixes

• Renable exluding results based on :site filter parameter as there were several reports when Google didn't respect this filter.

2024-06-14 (0.0.122)

Fixes

• Disable excluding results based on :site filter parameter. The actor was checking whether all results from Google match the site: filter and excluded those that didn't match. This feature was buggy so we decided to disable it.

2024-06-03 (0.0.121)

Fixes

• Don't distinguish http vs https when filtering results with the :site clause.

Features

• Add includeIcons to input to extract result icons in base64 format. The image data is relatively large so we decided not to include it by default.

2024-04-23 (0.0.119)

Fixes

• Fixed extracting personalInfo from linkedin sites

2024-04-08 (0.0.118)

Fixes

• Fixed personalInfo extraction in mobile results
• Fixed relatedQueries extraction in mobile results
• Store the results suggested by Google to suggestedResults instead of organicResults

2024-02-12 (0.0.113)

Improvements:

• Actor continues scraping the next page if number of organic and paid results is greater than 80% of resultsPerPage.

2023-12-23 (0.0.109)

Fixes

• Improve handling of potentially non-matching site: filters. We check for a correct format and track any inconsistencies if there would be any. Users often use the filter in wrong format (e.g. -site: or multiple site: which doesn't work)

2023-12-23 (0.0.108)

Fixes

• Fix problem that sometimes not all relatedQueries were extracted

2023-10-30 (0.0.106)

Fixes:

• Fixed extraction of companyName field

2023-09-25 (0.0.103)

Fixes:

• Updated css selector for result's description

2023-08-21 (0.0.100)

Fixes:

• Fixed issue with pagination, where some pages Google returned didn't have the next page (depending on IP used). The actor will now visit every page it ever saw in pagination which fixes this issue

2023-08-11 (0.0.99)

Fixes:

• Updated mobile selectors

2023-07-08

Fixes:

• Retry requests where some result domains don't correctly match the site: restriction. This happened very rarely and we are still investigating why Google would provide such results.

2023-06-26 (0.0.84)

Fixes:

• Correctly parse new result type with one full result-like site link

2023-05-31 (0.0.82)

Features

• Added output schema formats to download organic and paid results separately with one result per row format. For API access, add &view=paid_results or &view=organic_results to the URL.
• Added type: organic or paid and position or adPosition to all organic and paid results for consistency.

Changes

• Deprecated input field csvFriendlyOutput, please migrate to using the new output schema formats. We will keep supporting csvFriendlyOutput for some time.

2023-05-29 (0.0.81)

Fixes

• Replaces \t characters in queries with spaces. Previously, they were briefly ignored which broke some queries.

2023-04-06

Fixes

• Fixed personal info for LinkedIn results

2023-03-7

Changes

• Updated to version 3

Fixes

• Updated ads URLs selectors for desktop and mobile

2023-02-15

Fixes

• Updated relatedQueries selectors to work with the new layout

2023-01-20

Fixes

• Log proper error is customDataFunction throws an error
• Log an error if a request fails all retries

Changes

• If customDataFunction throws an error, normal output is still returned, instead of crashing

2023-01-19

Fixes (new layout)

• Fixed organic desktop results
• Fixed organic mobile missing title and displayUrl
• Fixed organic mobile missing top results
• Deduplicate results

2022-11-17

• Fixed mobile extractor, paidResults, siteLinks, and description

2022-11-16

• Fixed mobile extractor paidResults, paidProducts, organicResults

2022-08-24

• Fixed organic results sometimes missing top result and twitter based results

2022-05-03

• Added personalInfo to output containing name, location, jobTitle, companyName, cleanDescription. These are available mainly for LinkedIn results.

2022-01-31

• Implemented csvFriendlyOutput option
• Handled missing www. on domains

2022-01-21

• Fixed organic results for desktop (new layout)

2021-02-25

• Hotfixed relatedQueries and peopleAlsoAsk in the new layout

2021-01-19

• Fixed extractor to work on new Google layout
• Added parsing emphasized text

2020-11-19

• Fixed new layout for mobile paid ads

2020-10-01

• Hotfix after Google changed the page layout for desktop
• Fixed main selector for paid results
• Fixed description for paid results
• Fixed description, url and siteLinks for organic results

2020-07-15 (beta)

• Fixed site links being present in a title of paid results
• Better explanation about max limit of results
• Added Apify to customDataFunction
• Added includeUnfilteredResults option
• Parsing of product info - rating, number of reviews and price
• Fixed unmatched pseudo-class :first error on desktop paid products