Web Scraper avatar
Web Scraper

Pricing

Pay per usage

Go to Store
Web Scraper

Web Scraper

Developed by

Apify

Apify

Maintained by Apify

Crawls arbitrary websites using a web browser and extracts structured data from web pages using a provided JavaScript function. The Actor supports both recursive crawling and lists of URLs, and automatically manages concurrency for maximum performance.

4.5 (23)

Pricing

Pay per usage

917

Total users

90K

Monthly users

5K

Runs succeeded

>99%

Issues response

7.8 days

Last modified

2 months ago

DE

Unsure of how to get what I'm looking for

Closed

dej opened this issue
a month ago

I'm looking for how I can xtract particluar elements from the webpage. I'd like to extract the email from the following element "yang@ee.ucla.edu" And also the name from this element "C.K. Ken Yang" But I'm unsure what exactly to include to get these values

jindrich.bar avatar

Hello, and thank you for your interest in this Actor.

Your page function has syntax errors - the Actor cannot parse it correctly, since it's not valid JavaScript. You can use the injected jQuery library to collect data from the web page like this:

async function pageFunction(context) {
const $ = context.jQuery;
const results = $('#sp-resulthere .card').map(function() {
const name = $(this).find('.people-title').text().trim();
const email = $(this).find('.mailto-link').attr('href').slice("mailto:".length);
return { name, email };
}).get();
return results;
}

To scrape the data correctly, you also need to make the Actor wait for networkidle0 event, not networkidle2. This is under Performance and limits > Navigation waits until. networkidle2 fires when there are at most 2 pending requests - on this page, it means that the actual data hasn't loaded yet.

See my fixed run here: https://console.apify.com/view/runs/odPkeMsv0IHDM3yPW

I'll close this issue now, but feel free to ask additional questions if you have any. Cheers!