Google Search Results Scraper avatar
Google Search Results Scraper

Pricing

$3.50 / 1,000 results

Go to Store
Google Search Results Scraper

Google Search Results Scraper

Developed by

Apify

Apify

Maintained by Apify

Scrape Google Search Engine Results Pages (SERPs). Select the country or language and extract organic and paid results, AI overviews, ads, queries, People Also Ask, prices, reviews, like a Google SERP API. Export scraped data, run the scraper via API, schedule runs, or integrate with other tools.

4.8 (24)

Pricing

$3.50 / 1,000 results

520

Total users

60K

Monthly users

3.2K

Runs succeeded

>99%

Issues response

12 days

Last modified

12 days ago

XR

AI Overviews are sometimes truncated after the first paragraph and/or before the list content

Closed

xlsa-rbm opened this issue
2 months ago

google-search-scraper does return AI overview data, but oftentimes it is truncating the data after the first paragraph or two. AIOs often have a lead in paragraph and then some text that ends in a colon, like this:

A brief summary of the search results for the query.

Here's some more information about topic of query:

Multi-section list goes here

I'm observing that the scraper often truncates the AIO either after the first paragraph or after the second lead-in paragraph, omitting the list content.

Observations: AIO content is basically a bunch of child divs inside a container div. Each of the first 2 paras is in a separate div, then each heading/subheading and list section is also in a div.

At least in some cases Google injects a <style> tag after the first two paragraph-containing divs (so right after the "Here's some more information about topic of query:" section.) Perhaps something in the markup extraction is breaking on this? Unfortunately, the HTML the actor saves to KV doesn't contain the rendered AIO output, so I can't verify that it is there in the DOM/markup the actor sees.

aja_bar avatar

Hi, thank your for the report and apologies for the delayed response. We're looking into it and will get back to you.