Web Scraper
Pricing
Pay per usage
Web Scraper
Crawls arbitrary websites using the Chrome browser and extracts structured data from web pages using a provided JavaScript function. The Actor supports both recursive crawling and lists of URLs, and automatically manages concurrency for maximum performance.
4.5 (22)
Pricing
Pay per usage
564
Monthly users
3.5k
Runs succeeded
>99%
Response time
10 days
Last modified
2 months ago
Crawler goes off domain
What is the setting that tells the scraper to not leave the original domain? For example if I scrape a site example.com it finds a social link and then its scraping facebook.com/link but I want it to stay on example.com
Hello and thank you for your interest in this Actor!
This is the default behavior of Web Scraper, i.e., by default; it only follows the links targeting the same domain as at least one of the start URLs.
See my example run on my personal blog - while I have links to other websites (my GitHub profile, LinkedIn, or Apify homepage), the Actor doesn't visit these. To change this, you can use the Include globs input option - using this, you can set custom URL patterns to crawl.
I'll close this issue now, but feel free to ask additional questions if you have any. Cheers! (and sorry for the wait).
Pricing
Pricing model
Pay per usageThis Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage.
