No credit card required
Web Scraper
No credit card required
Crawls arbitrary websites using the Chrome browser and extracts data from pages using a provided JavaScript code. The actor supports both recursive crawling and lists of URLs and automatically manages concurrency for maximum performance. This is Apify's basic tool for web crawling and scraping.
how do i scrape with mobile or specific UA
Closed
hi the pages i scrape see linux OS os the os. how can i override that and scrape with a specific UA?
Hello and thank you for your interest in this Actor!
All our Actors run as Linux-based Docker containers. While we do randomize the user-agent strings, we found out that it's better not to lie about your operating system - modern bot protection services can figure it out by querying the list of installed fonts or by some advanced canvas-based fingerprinting techniques. Because of this, we don't allow changing the user-agent string "by hand" in this Actor.
If you need to randomize your user-agent string, you can create your our Actor - see our Crawlee + Puppeteer + Chrome Typescript Actor template, which is basically a "programmatic" version of this Actor. There, you can modify the browser fingerprints by passing the options to the underlying BrowserPool instance - see example in our Crawlee documentation.
I'll close this issue now, but feel free to ask questions - be it about custom Actor development, or this one specific Actor. Cheers!
Apify- 3.7k monthly users
- 98.8% runs succeeded
- 3.6 days response time
- Created in Mar 2019
- Modified about 1 month ago
