Web Scraper
Pricing
Pay per usage
Web Scraper
Crawls arbitrary websites using a web browser and extracts structured data from web pages using a provided JavaScript function. The Actor supports both recursive crawling and lists of URLs, and automatically manages concurrency for maximum performance.
4.5 (23)
Pricing
Pay per usage
864
Total users
88K
Monthly users
4.6K
Runs succeeded
>99%
Issues response
9.6 days
Last modified
a month ago
how do i scrape with mobile or specific UA
Closed
hi the pages i scrape see linux OS os the os. how can i override that and scrape with a specific UA?
Hello and thank you for your interest in this Actor!
All our Actors run as Linux-based Docker containers. While we do randomize the user-agent strings, we found out that it's better not to lie about your operating system - modern bot protection services can figure it out by querying the list of installed fonts or by some advanced canvas-based fingerprinting techniques. Because of this, we don't allow changing the user-agent string "by hand" in this Actor.
If you need to randomize your user-agent string, you can create your our Actor - see our Crawlee + Puppeteer + Chrome Typescript Actor template, which is basically a "programmatic" version of this Actor. There, you can modify the browser fingerprints by passing the options to the underlying BrowserPool instance - see example in our Crawlee documentation.
I'll close this issue now, but feel free to ask questions - be it about custom Actor development, or this one specific Actor. Cheers!
