Web Scraper avatar
Web Scraper
Try for free

No credit card required

View all Actors
Web Scraper

Web Scraper

apify/web-scraper
Try for free

No credit card required

Crawls arbitrary websites using the Chrome browser and extracts data from pages using a provided JavaScript code. The actor supports both recursive crawling and lists of URLs and automatically manages concurrency for maximum performance. This is Apify's basic tool for web crawling and scraping.

User avatar

unable to get data from script even though works on Chome Console

Closed

musty_yautia opened this issue
3 months ago

url: https://singmalls.app/en/malls/triple-one-somerset/merchants/triple-one-somerset-allu

Problem: I looked at your tutorial and used that to create the below code. Unable to get a) website and b) description fields from the script, even though it works when using the Chrome Console. Also unable to get the images from the url, but i am still trying to figure this out.

Code is below: async function pageFunction(context) {

1const $ = context.jQuery;
2const storeName = $('h2').first().text();
3const store_number = $('p').first().text();
4const store_location = $("[class^='LocationDetails']").text();
5const store_contact = $("[class^='tel']").text();
6const store_openinghrs = $("[class^='MerchantOpeningHours_container']").text();
7const store_desc = $("[class^='MerchantDetail_descriptionContainer__']").text();
8const store_website = $("[class^='MerchantDataItem_websiteInset']").text();
9const googleMapsLink = $("a[href^='https://maps.google.com']");
10const googleMapsUrl = googleMapsLink.attr('href');
11
12// Select the slider wrapper
13const slider = $("[class^='MerchantDetail_hero__']");
14
15// Initialize an array to store the URLs
16const imageUrls = [];
17
18// Find all img tags within the slider and extract their src attributes
19slider.find('img').each(function() {
20    const imageUrl = $(this).attr('src');
21    imageUrls.push(imageUrl);
22});
23
24return {
25    url: context.request.url,
26    storeName,
27    store_number,
28    store_location,
29    store_contact,
30    store_openinghrs,
31    store_desc,
32    store_website,
33    googleMapsUrl,
34    imageUrls
35};

}

User avatar

Hello and thank you for your interest in this Actor!

You almost got it - the only problem is that for the missing items, you're using mobile selectors (perhaps because the open Chrome Console shrank your viewport, forcing the website to render the mobile version). Replace the store_desc and store_website definitions with the following lines:

1const store_desc = $("[class^=MerchantDetailDesktop_description]").text();
2const store_website = $("[class^=MerchantDataItemDesktop_website] > a").attr('href');

For the image URLs, you can use the following code:

1const image_urls = $(".slider img").map(function() {
2        return $( this ).attr('src');
3    }).get();

Did this solve your problem? I'll close this issue now, but feel free to ask any additional questions if you have some. Happy scraping!

Developer
Maintained by Apify
Actor metrics
  • 3.4k monthly users
  • 99.9% runs succeeded
  • 3.2 days response time
  • Created in Mar 2019
  • Modified about 2 months ago