GPT Scraper avatar
GPT Scraper

Pricing

$9.00 / 1,000 pages

Go to Store
GPT Scraper

GPT Scraper

Developed by

Jakub Drobník

Jakub Drobník

Maintained by Apify

Extract data from any website and feed it into GPT via the OpenAI API. Use ChatGPT to proofread content, analyze sentiment, summarize reviews, extract contact details, and much more.

4.4 (7)

Pricing

$9.00 / 1,000 pages

101

Total users

5.9K

Monthly users

145

Runs succeeded

>99%

Last modified

5 months ago

BL

Scraper Fails with Google Play Store Urls

Closed

blkbox opened this issue
9 months ago

Error logs

2024-09-10T11:49:34.199Z ACTOR: Pulling Docker image of build nXM2iV00mWsXHDDu6 from repository.
2024-09-10T11:49:34.297Z ACTOR: Creating Docker container.
2024-09-10T11:49:35.463Z ACTOR: Starting Docker container.
2024-09-10T11:49:36.674Z Starting X virtual framebuffer using: Xvfb :99 -ac -screen 0 1920x1080x24+32 -nolisten tcp
2024-09-10T11:49:36.676Z Executing main command
2024-09-10T11:49:39.630Z INFO System info {"apifyVersion":"3.1.16","apifyClientVersion":"2.9.3","crawleeVersion":"3.8.1","osType":"Linux","nodeVersion":"v18.20.4"}
2024-09-10T11:49:39.773Z INFO Max pages per crawl: 1
2024-09-10T11:49:40.462Z INFO Configuration completed. Starting the crawl.
2024-09-10T11:49:40.549Z INFO PlaywrightCrawler: Starting the crawler.
2024-09-10T11:49:45.697Z INFO Opening https://play.google.com/store/apps/details?id=com.hardrockdigital.client&hl=en_US&gl=US...
2024-09-10T11:49:46.292Z WARN PlaywrightCrawler: Reclaiming failed request back to the list or queue. page.evaluate: TypeError: Failed to execute 'parseFromString' on 'DOMParser': This document requires 'TrustedHTML' assignment.
2024-09-10T11:49:46.295Z at eval (eval at evaluate (:226:30), <anonymous>:2:37)
2024-09-10T11:49:46.298Z at UtilityScript.evaluate (<anonymous>:228:17)
2024-09-10T11:49:46.300Z at UtilityScript.<anonymous> (<anonymous>:1:44)
2024-09-10T11:49:46.303Z at shrinkHtml (/home/myuser/dist/processors.js:10:33) {"id":"u95wgQVoq9JiBtG","url":"https://play.g... [trimmed]
BL

blkbox

9 months ago

Input used

{
"startUrls": [
{
"url": "https://play.google.com/store/apps/details?id=com.hardrockdigital.client&hl=en_US&gl=US"
}
],
"maxPagesPerCrawl": 1,
"maxCrawlingDepth": 1,
"linkSelector": "a[href]",
"instructions": " \"I have a website link that provides information about a specific brand. \"\n \"Please visit the link and extract detailed information about the brand.\n\"\n \"Include the following details in your response:\n\n\"\n \"Brand Name: The official name of the brand.\n\"\n \"Overview: A brief description of what the brand is about, including its mission, vision, and values.\n\"\n \"Products/Services: A list of main products or services offered by the brand.\n\"\n \"Category of products: Provide only the name of categories of product\"\n \"Unique selling points:\n\"\n \"Target Audience: The primary demographic or market segment that the brand caters to.\n\"\n \"Contact Information: How to get in touch with the brand, including customer service, social media handles, and physical addresses if available.\n\"\n \"Website Link: The original website link provided for reference.\n\"\n \"Social Media Handles:\n\n\"\n Provide the output in JSON format with keys representing above points and values representing the details.\n ",
"extractor": {
"type": "gpt",
"model": "gpt-4",
"maxTokens": 4000
},
... [trimmed]
lukas.prusa avatar

Hi, thanks for opening this issue!

I can see that you've found the original issue from a different user for the same website here - https://console.apify.com/actors/paOtbjvyUiNsr1Qms/issues/puLBJmEVQSvgzuXw7

Seeing more users having the same problem, we will prioritize this issue more :)

I will keep you updated here, thanks!

BL

blkbox

9 months ago

Hey Lukas,

Thanks for the response, can we do anything to put this on top of your priority list ?

Context: We are a bootstrapped startup just launched this product, expecting to see scrapings around 4-5K pages per day.

This a blocker for our business, we will be forced to look for alternatives if this doesn't work.

PS: We love your tool it has unlocked key capabilities for us. Would be happy to setup a chat with our engineering team, if that can help resolve the issue quicker.

BL

blkbox

9 months ago

I tried scraping the same url, with apify/playwright-scraper , seems to be working there.

lukas.prusa avatar

Thanks for your insight, we will try to finish it this week :)

It's most likely just some very stupid bug in our backend that's somehow magically clashing with this very specific website, so it's just incredibly annoying to catch.

BL

blkbox

9 months ago

Thank you looking forward to the fix. :)

Happy debugging, a rubber duck might help ;)

lukas.prusa avatar

Hi again, thanks for the rubber duck, it helped us a ton :) I'm happy to inform you that we've just updated the scraper with the fix!

Try it out and let me know how it works now. Thanks and happy scraping!

BL

blkbox

8 months ago

Hey Lukas,

I can confirm we are able to scrape the play store urls now.

Thank you so much for the fix. :)