3 days trial then $30.00/month - No credit card required now
Walmart Data Extractor
3 days trial then $30.00/month - No credit card required now
Access a vast array of product data from Walmart with our Walmart Scraper. Extract images, brand details, prices, variations, and more. Customize your search with filters, categories, and lists to gather the information you need.
less products
Closed
Hello. I'm trying this solution but I get less products. That category should be around 3.4k products but only got about 980 in total results. Is it because of the trial limitation? That a category https://www.walmart.com/browse/toys/card-games/4171_4191_163842
confidential_sled
Finished! Total 1011 requests: 1009 succeeded, 2 failed. 983 2024-02-05 08:12:54 2024-02-05 08:48:58
confidential_sled
I also looking on the log and I see like
Fetching list: https://www.walmart.com/browse/toys/card-games/4171_4191_163842?page=20 with page: 1 2024-02-05T06:40:38.104Z INFO CRAWLER: -- Found 50 products on https://www.walmart.com/browse/toys/card-games/4171_4191_163842?page=1 Fetching list: https://www.walmart.com/browse/toys/card-games/4171_4191_163842?page=20 with page: 2 2024-02-05T06:40:38.104Z INFO CRAWLER: -- Found 50 products on https://www.walmart.com/browse/toys/card-games/4171_4191_163842?page=2
I can see all 50 pages! So it really do the job for all pages. But it did not extract 50 products... it like extracts 25-30 on every page, so the total is less in 30%
confidential_sled
This is a run https://console.apify.com/actors/dKylGAj0fF0pCjGeW/runs/zdKYmwfjKNxhuz864#output
Thank you
Hey there!
Thank you very much for reaching out, using our actor, and letting us know about your problem. I just investigated the problem you had. Seems like everything worked pretty well but there are a couple of things that need clarification.
- If you meant "Trial limitation" by Apify Subscription, no. We are providing the full actor capabilities during the trial period. We do not limit our users in any way in this period so that they can experience all the capabilities.
- But there is a limitation for sure. Many big retail websites around the web use a limitation technique called Soft Limiting. While you are viewing a list page, you see a number (or around a number) of products. But while you are paginating over them, you cannot reach them all. For example, the link you shared shows 1000+ products in it but Walmart provides only ~900 items in total. Therefore, you cannot fetch all of them by checking a single page.
How to pass this limitation? It is pretty simple actually. You have to split your pages. Either by category, brand, price, or something else. Splitting the listing page into multiple chunks so that you will have less than the soft limitation per each split page. Unfortunately, the actor doesn't support this splitting functionality by default by it have the capability of getting a start URL. Once you split them, you can pass these URLs into the actor and let it run.
I hope this helps. Please let us know if you need any further assistance. Best
epctex- 22 monthly users
- 98.7% runs succeeded
- 0.0 days response time
- Created in Apr 2021
- Modified about 11 hours ago
