Walmart Data Extractor avatar
Walmart Data Extractor
Try for free

3 days trial then $30.00/month - No credit card required now

View all Actors
Walmart Data Extractor

Walmart Data Extractor

epctex/walmart-scraper
Try for free

3 days trial then $30.00/month - No credit card required now

Access a vast array of product data from Walmart with our Walmart Scraper. Extract images, brand details, prices, variations, and more. Customize your search with filters, categories, and lists to gather the information you need.

CS

less products

Closed

confidential_sled opened this issue
5 months ago

Hello. I'm trying this solution but I get less products. That category should be around 3.4k products but only got about 980 in total results. Is it because of the trial limitation? That a category https://www.walmart.com/browse/toys/card-games/4171_4191_163842

CS

confidential_sled

5 months ago

Finished! Total 1011 requests: 1009 succeeded, 2 failed. 983 2024-02-05 08:12:54 2024-02-05 08:48:58

CS

confidential_sled

5 months ago

I also looking on the log and I see like

Fetching list: https://www.walmart.com/browse/toys/card-games/4171_4191_163842?page=20 with page: 1 2024-02-05T06:40:38.104Z INFO CRAWLER: -- Found 50 products on https://www.walmart.com/browse/toys/card-games/4171_4191_163842?page=1 Fetching list: https://www.walmart.com/browse/toys/card-games/4171_4191_163842?page=20 with page: 2 2024-02-05T06:40:38.104Z INFO CRAWLER: -- Found 50 products on https://www.walmart.com/browse/toys/card-games/4171_4191_163842?page=2

I can see all 50 pages! So it really do the job for all pages. But it did not extract 50 products... it like extracts 25-30 on every page, so the total is less in 30%

CS

confidential_sled

5 months ago
epctex avatar

epctex (epctex)

5 months ago

Hey there!

Thank you very much for reaching out, using our actor, and letting us know about your problem. I just investigated the problem you had. Seems like everything worked pretty well but there are a couple of things that need clarification.

  • If you meant "Trial limitation" by Apify Subscription, no. We are providing the full actor capabilities during the trial period. We do not limit our users in any way in this period so that they can experience all the capabilities.
  • But there is a limitation for sure. Many big retail websites around the web use a limitation technique called Soft Limiting. While you are viewing a list page, you see a number (or around a number) of products. But while you are paginating over them, you cannot reach them all. For example, the link you shared shows 1000+ products in it but Walmart provides only ~900 items in total. Therefore, you cannot fetch all of them by checking a single page.

How to pass this limitation? It is pretty simple actually. You have to split your pages. Either by category, brand, price, or something else. Splitting the listing page into multiple chunks so that you will have less than the soft limitation per each split page. Unfortunately, the actor doesn't support this splitting functionality by default by it have the capability of getting a start URL. Once you split them, you can pass these URLs into the actor and let it run.

I hope this helps. Please let us know if you need any further assistance. Best

Developer
Maintained by Community
Actor metrics
  • 22 monthly users
  • 5 stars
  • 100.0% runs succeeded
  • Created in Apr 2021
  • Modified about 9 hours ago
Categories