3 days trial then $30.00/month - No credit card required now
Google Shopping Insights
3 days trial then $30.00/month - No credit card required now
Unlock valuable insights from Google Shopping with our Data Extractor. Get reviews, descriptions, prices, merchant details, and affiliation links. Export data in JSON, XML, CSV, Excel, and HTML formats with no limits!
Actor stopped after first request
Closed
The actor is stopping after the first request. It works only when I resurrect the whole run.
exquisite_tab
tugkan
Hey Šimon Kubita,
Unfortunately, we could not reproduce the issue. Can you please share your run ID with us? The problem might happened due to the Apify's own infrastructure.
Best
exquisite_tab
Hello, thx for fixing it. This issue appears to me with every run.
examples: Q4XsFwPyiXukiB96T xL4cJacLsXsBfNKoh Epvqocp5y15ZhUItu
example with multiple resurrections: SGmiGdTM71QmwFISa
exquisite_tab
One fresh run: EC5rpGZAANgWsNqUV
tugkan
Hey Šimon Kubita,
Everything seems working properly on our side. Did you try to increase the Max Items (maxItems) attribute to retrieve more results? Or remove it completely?
Best
exquisite_tab
I dont need more results. I need to Actor try every input (search query) and dont stop after first. Example EC5rpGZAANgWsNqUV has 4 inputs with 3 max items parameter but has only 3 results overall. In the header of the run is "Requests 1 of 5 handled".
exquisite_tab
I need 4x3 results = 12 results.
exquisite_tab
In the result data is only first search query.
exquisite_tab
Can you reopen this issue, please? This is not fixed.
tugkan
Hey Šimon Kubita,
Unfortunately, the maxItems attribute that we have in all of our public actors is not working by this logic. It is there to tweak the total number of items. If you want to have 3x4 = 12 results, my suggestion for you is to create 4 different tasks and proceed forward with it.
Please let me know if there is anything else we can do for you. Best
exquisite_tab
What? How works input "search queries"? If I have 4 input queries (search query) with MaxItems 3 how many results are correct?
exquisite_tab
This example
exquisite_tab
I expect 12 results - 4 queries with 3 items per query. 12 results in the result set.
tugkan
If maxItems have been provided, the total number of results will be equal to maxItems. For example; if you put 5 on the maxItems field, you'll only get 5 results in total.
On the other hand, search queries are there for running the public actor within multiple fields. As mentioned above, the best approach for this kind of case is to create 4 different tasks, put only 1 query, and add 3 as maxItems.
Best.
exquisite_tab
I need to search 15k this queries. You are saying that i have to do 15k different tasks? Im convinced that this is not how this crawler was designed with input fields.
tugkan
If you want to run the public actor programmatically and use it with 15K different queries, we can integrate a new field such as maxItemsPerQuery. Do you think this would resolve your problem?
exquisite_tab
maxItemsPerQuery is what maxItems was originally designed for I mean. Yes, this would fix this problem.
maxItems how you described would be used only with one search query I mean. And with one query you dont need it, because you have maxItemsPerQuery :)
exquisite_tab
I tried to delete maxItems input as you described and give 4 search queries. But this run crashed and now it's repeating request permanently. It won't stop. For now, has tried 33 requests and continues.
6UeLqgwbaR4o17v7d
exquisite_tab
2023-02-13T18:19:45.479Z ACTOR: Pulling Docker image from repository.
2023-02-13T18:19:58.074Z ACTOR: Creating Docker container.
2023-02-13T18:20:02.533Z ACTOR: Starting Docker container.
2023-02-13T18:20:05.212Z INFO System info {"apifyVersion":"3.1.1","apifyClientVersion":"2.6.2","crawleeVersion":"3.2.2","osType":"Linux","nodeVersion":"v16.19.0"}
2023-02-13T18:20:05.223Z INFO PHASE -- STARTING ACTOR.
2023-02-13T18:20:05.476Z INFO PHASE -- SETTING UP CRAWLER.
2023-02-13T18:20:05.690Z INFO CRAWLER STARTED.
2023-02-13T18:20:07.596Z INFO PuppeteerCrawler: Starting the crawl
2023-02-13T18:20:20.081Z INFO CRAWLER: -- Found 60 items from http://www.google.cz/search?q=JOOP%20JUMP&tbm=shop&tbs=vw:l&start=0&gl=cz
2023-02-13T18:20:30.273Z INFO CRAWLER: -- Found 6 items from http://www.google.cz/search?q=LAMBRETTA%20PRIVATO&tbm=shop&tbs=vw:l&start=0&gl=cz
2023-02-13T18:20:38.230Z INFO CRAWLER: -- Found 20 items from http://www.google.cz/search?q=JOOP%20JUMP&tbm=shop&tbs=vw:l&start=60&gl=CZ
2023-02-13T18:20:39.506Z INFO CRAWLER: -- Found 60 items from http://www.google.cz/search?q=LALIQUE%20SOLEIL&tbm=shop&tbs=vw:l&start=0&gl=cz
2023-02-13T18:21:07.597Z INFO Statistics: PuppeteerCrawler request statistics: {"requestAvgFailedDurationMillis":null,"requestAvgFinishedDurationMillis":13842,"requestsFinishedPerMinute":4,"requestsFailedPerMinute":0,"requestTotalDurationMillis":55366,"requestsTotal":4,"crawlerRuntimeMillis":62016,"retryHistogram":[4]}
2023-02-13T18:21:07.601Z INFO PuppeteerCrawler:AutoscaledPool: state {"currentConcurrency":4,"desiredConcurrency":6,"systemStatus":{"isSystemIdle":true,"memInfo":{"isOverloaded":false,"limitRatio":0.2,"actualRatio":0},"eventLoopInfo":{"isOverloaded":false,"limitRatio":0.6,"actualRatio":0},"cpuInfo":{"isOverloaded":false,"limitRatio":0.4,"actualRatio":0},"clientInfo":{"isOverloaded":false,"limitRatio":0.3,"actualRatio":0}}}
2023-02-13T18:21:14.554Z WARN PuppeteerCrawler: Reclaiming failed request back to the list or queue. Navigation timed out after 60 seconds. {"id":"6q1AmWmi9wQUFq4","url":"http://www.google.cz/search?q=JEAN%20LOUES&tbm=shop&tbs=vw:l&start=0&gl=cz","retryCount":1}
2023-02-13T18:21:18.251Z WARN PuppeteerCrawler: Reclaiming failed request back to the list or queue. Waiting for selector div.sh-pr__product-results failed: Waiting failed: 30000ms exceeded
2023-02-13T18:21:18.252Z {"id":"KAv7Fk3aEsfu4gJ","url":"http://www.google.cz/search?q=LAMBRETTA%20PRIVATO&tbm=shop&tbs=vw:l&start=6&gl=CZ","retryCount":1}
2023-02-13T18:21:22.541Z WARN PuppeteerCrawler: Reclaiming failed request back to the list or queue. Waiting for selector div.sh-pr__product-results failed: Waiting failed: 30000ms exceeded
2023-02-13T18:21:22.542Z {"id":"3WKpHGG5js8lvho","url":"http://www.google.cz/search?q=JOOP%20JUMP&tbm=shop&tbs=vw:l&start=80&gl=CZ","retryCount":1}
2023-02-13T18:21:37.356Z INFO CRAWLER: -- Found 60 items from http://www.google.cz/search?q=JEAN%20LOUES&tbm=shop&tbs=vw:l&start=0&gl=cz
2023-02-13T18:21:39.802Z WARN PuppeteerCrawler: Reclaiming failed request back to the list or queue. Navigation timeout of 60000 ms exceeded
2023-02-13T18:21:39.803Z {"id":"VjBD7tUrMGcr6mb","url":"http://www.google.cz/search?q=LALIQUE%20SOLEIL&tbm=shop&tbs=vw:l&start=60&gl=CZ","retryCount":1}
2023-02-13T18:22:03.263Z WARN PuppeteerCrawler: Reclaiming failed request back to the list or queue. Waiting for selector div.sh-pr__product-results failed: Waiting failed: 30000ms exceeded
2023-02-13T18:22:03.264Z {"id":"KAv7Fk3aEsfu4gJ","url":"http://www.google.cz/search?q=LAMBRETTA%20PRIVATO&tbm=shop&tbs=vw:l&start=6&gl=CZ","retryCount":2}
2023-02-13T18:22:07.597Z INFO Statistics: PuppeteerCrawler request statistics: {"requestAvgFailedDurationMillis":null,"requestAvgFinishedDurationMillis":14977,"requestsFinishedPerMinute":2,"requestsFailedPerMinute":0,"requestTotalDurationMillis":74884,"requestsTotal":5,"crawlerRuntimeMillis":122016,"retryHistogram":[4,1]}
2023-02-13T18:22:07.605Z INFO PuppeteerCrawler:AutoscaledPool: state {"currentConcurrency":4,"desiredConcurrency":6,"systemStatus":{"isSystemIdle":true,"memInfo":{"isOverloaded":false,"limitRatio":0.2,"actualRatio":0},"eventLoopInfo":{"isOverloaded":false,"limitRatio":0.6,"actualRatio":0},"cpuInfo":{"isOverloaded":false,"limitRatio":0.4,"actualRatio":0},"clientInfo":{"isOverloaded":false,"limitRatio":0.3,"actualRatio":0}}}
2023-02-13T18:22:14.084Z WARN PuppeteerCrawler: Reclaiming failed request back to the list or queue. Waiting for selector div.sh-pr__product-results failed: Waiting failed: 30000ms exceeded
2023-02-13T18:22:14.085Z {"id":"3WKpHGG5js8lvho","url":"http://www.google.cz/search?q=JOOP%20JUMP&tbm=shop&tbs=vw:l&start=80&gl=CZ","retryCount":2}
2023-02-13T18:22:17.509Z INFO CRAWLER: -- Found 1 items from http://www.google.cz/search?q=LAMBRETTA%20PRIVATO&tbm=shop&tbs=vw:l&start=6&gl=CZ
2023-02-13T18:22:33.527Z WARN PuppeteerCrawler: Reclaiming failed request back to the list or queue. Waiting for selector div.sh-pr__product-results failed: Waiting failed: 30000ms exceeded
2023-02-13T18:22:33.529Z {"id":"VjBD7tUrMGcr6mb","url":"http://www.google.cz/search?q=LALIQUE%20SOLEIL&tbm=shop&tbs=vw:l&start=60&gl=CZ","retryCount":2}
2023-02-13T18:22:42.843Z WARN PuppeteerCrawler: Reclaiming failed request back to the list or queue. Navigation timed out after 60 seconds. {"id":"2U47mZMapToLdxr","url":"http://www.google.cz/search?q=JEAN%20LOUES&tbm=shop&tbs=vw:l&start=60&gl=CZ","retryCount":1}
2023-02-13T18:22:58.149Z WARN PuppeteerCrawler: Reclaiming failed request back to the list or queue. Waiting for selector div.sh-pr__product-results failed: Waiting failed: 30000ms exceeded
2023-02-13T18:22:58.150Z {"id":"qwupy09MUf555UK","url":"http://www.google.cz/search?q=LAMBRETTA%20PRIVATO&tbm=shop&tbs=vw:l&start=7&gl=CZ","retryCount":1}
2023-02-13T18:22:58.872Z WARN PuppeteerCrawler: Reclaiming failed request back to the list or queue. Waiting for selector div.sh-pr__product-results failed: Waiting failed: 30000ms exceeded
2023-02-13T18:22:58.873Z {"id":"3WKpHGG5js8lvho","url":"http://www.google.cz/search?q=JOOP%20JUMP&tbm=shop&tbs=vw:l&start=80&gl=CZ","retryCount":3}
2023-02-13T18:23:07.597Z INFO Statistics: PuppeteerCrawler request statistics: {"requestAvgFailedDurationMillis":null,"requestAvgFinishedDurationMillis":14321,"requestsFinishedPerMinute":2,"requestsFailedPerMinute":0,"requestTotalDurationMillis":85928,"requestsTotal":6,"crawlerRuntimeMillis":182016,"retryHistogram":[4,1,1]}
2023-02-13T18:23:07.662Z INFO PuppeteerCrawler:AutoscaledPool: state {"currentConcurrency":4,"desiredConcurrency":6,"systemStatus":{"isSystemIdle":true,"memInfo":{"isOverloaded":false,"limitRatio":0.2,"actualRatio":0},"eventLoopInfo":{"isOverloaded":false,"limitRatio":0.6,"actualRatio":0},"cpuInfo":{"isOverloaded":false,"limitRatio":0.4,"actualRatio":0},"clientInfo":{"isOverloaded":false,"limitRatio":0.3,"actualRatio":0}}}
2023-02-13T18:23:10.666Z INFO CRAWLER: -- Found 20 items from http://www.google.cz/search?q=JEAN%20LOUES&tbm=shop&tbs=vw:l&start=60&gl=CZ
2023-02-13T18:23:28.386Z INFO CRAWLER: -- Found 20 items from http://www.google.cz/search?q=JEAN%20LOUES&tbm=shop&tbs=vw:l&start=80&gl=CZ
2023-02-13T18:23:40.145Z WARN PuppeteerCrawler: Reclaiming failed request back to the list or queue. Waiting for selector div.sh-pr__product-results failed: Waiting failed: 30000ms exceeded
2023-02-13T18:23:40.146Z {"id":"qwupy09MUf555UK","url":"http://www.google.cz/search?q=LAMBRETTA%20PRIVATO&tbm=shop&tbs=vw:l&start=7&gl=CZ","retryCount":2}
2023-02-13T18:23:43.688Z ERROR PuppeteerCrawler: Request failed and reached maximum retries. TimeoutError: Waiting for selector div.sh-pr__product-results failed: Waiting failed: 30000ms exceeded
2023-02-13T18:23:43.689Z at Timeout.
exquisite_tab
The searchKeyword is empty in the result set in most of them.
tugkan
Hey Šimon Kubita,
We've integrated the maxItemsPerQuery field.
Best
exquisite_tab
Hi, Nice! Thanks. But i tried it - pjhAfer3q1neKNMfc and have same trouble as my last comment. Pls can you check it?
exquisite_tab
Inputs - maxPerQuery:3,queries:4,maxItems:null Results:600+
tugkan
Hey Šimon Kubita,
The key must be maxItemsPerQuery. We updated the actor again for the broken key. If you are using the actor programmatically, you should use it in this way.
About the issue you've been facing, the problem should be fixed just now. Can you please try it again?
Best
exquisite_tab
Great! It works fine on test inputs. I will try it in production tomorrow. Many thanks!
epctex- 33 monthly users
- 99.6% runs succeeded
- 0.0 days response time
- Created in Mar 2020
- Modified about 21 hours ago
