Web Scraper
No credit card required
Web Scraper
No credit card required
Crawls arbitrary websites using the Chrome browser and extracts data from pages using JavaScript code. The Actor supports both recursive crawling and lists of URLs and automatically manages concurrency for maximum performance. This is Apify's basic tool for web crawling and scraping.
Do you want to learn more about this Actor?
Get a demoperformace issue
Closed
I only crawl a page for about 30 seconds to 1 minute, is there any way to improve performance.
potent_fox
i'm waiting....
potent_fox
some execute results
Hi potent_fox,
Thank you for your patience. To improve your scraping performance, here are a few suggestions:
- Scrape Multiple URLs Together: Scraping multiple URLs in a single crawl session can give you a better time per URL. Currently, your Actor spends a lot of time in the setup phase as you run each URL in a separate run.
- Increase Resources: Adding more memory or CPU can help speed up the crawling process. Note that this should be more noticeable once you put more URLs together in one run.
- Use Cheerio Crawler: If possible, scrape the page without a browser using tools like Cheerio Crawler. This is up to 20x more efficient than the Web Scraper.
- Explore Original APIs: For optimal performance, consider scraping directly from the original API. More details can be found here.
Feel free to try these suggestions, and let us know if you encounter any further issues.
We'll be closing this issue now. Cheers!
ApifyActor Metrics
2.6k monthly users
-
222 stars
>99% runs succeeded
45 days response time
Created in Mar 2019
Modified 3 months ago
