Web Scraper avatar
Web Scraper
Try for free

No credit card required

View all Actors
Web Scraper

Web Scraper

apify/web-scraper
Try for free

No credit card required

Crawls arbitrary websites using the Chrome browser and extracts data from pages using a provided JavaScript code. The actor supports both recursive crawling and lists of URLs and automatically manages concurrency for maximum performance. This is Apify's basic tool for web crawling and scraping.

User avatar

fail to crawl ( worked before)

Closed

oblanceolate_epoch opened this issue
2 months ago
User avatar

oblanceolate_epoch

2 months ago

happy to get an update

User avatar

Hello and thank you for your interest in this Actor (and sorry for the wait)!

I tried running the Actor with the same input under my account - just with Datacenter proxies instead of Residentials - and it worked just fine. While Residential proxies should be in general more reliable, they are also more costly and noticeably slower (this is balanced by their good score with antibot protection services). Switching to Datacenter proxies gives you a faster connection to the target server with still a reasonably good (i.e. low) bot score.

Check out my example run at https://console.apify.com/view/runs/hGShY3iKdDgLIsxvr and feel free to reuse the input from there (the only difference is the proxy setup, though).

I'll close this issue now, but feel free to ask more questions regarding this problem, if you have any. Thanks again!

Developer
Maintained by Apify
Actor metrics
  • 3.4k monthly users
  • 99.9% runs succeeded
  • 3.2 days response time
  • Created in Mar 2019
  • Modified about 2 months ago