Web Scraper
Try for free
No credit card required
View all Actors
Web Scraper
apify/web-scraper
Try for free
No credit card required
Crawls arbitrary websites using the Chrome browser and extracts data from pages using JavaScript code. The Actor supports both recursive crawling and lists of URLs and automatically manages concurrency for maximum performance. This is Apify's basic tool for web crawling and scraping.
Do you want to learn more about this Actor?
Get a demoperformace issue
Closed
potent_fox opened this issue
2 months ago
I only crawl a page for about 30 seconds to 1 minute, is there any way to improve performance.
potent_fox
2 months ago
i'm waiting....
potent_fox
2 months ago
some execute results
2 months ago
Hi potent_fox,
Thank you for your patience. To improve your scraping performance, here are a few suggestions:
- Scrape Multiple URLs Together: Scraping multiple URLs in a single crawl session can give you a better time per URL. Currently, your Actor spends a lot of time in the setup phase as you run each URL in a separate run.
- Increase Resources: Adding more memory or CPU can help speed up the crawling process. Note that this should be more noticeable once you put more URLs together in one run.
- Use Cheerio Crawler: If possible, scrape the page without a browser using tools like Cheerio Crawler. This is up to 20x more efficient than the Web Scraper.
- Explore Original APIs: For optimal performance, consider scraping directly from the original API. More details can be found here.
Feel free to try these suggestions, and let us know if you encounter any further issues.
We'll be closing this issue now. Cheers!
Developer
Apify Maintained by Apify
Actor metrics
- 2.6k monthly users
- 155 stars
- 99.9% runs succeeded
- 6.6 days response time
- Created in Mar 2019
- Modified 23 days ago
Categories
