Web Scraper avatar
Web Scraper
Try for free

No credit card required

View all Actors
Web Scraper

Web Scraper

apify/web-scraper
Try for free

No credit card required

Crawls arbitrary websites using the Chrome browser and extracts data from pages using a provided JavaScript code. The actor supports both recursive crawling and lists of URLs and automatically manages concurrency for maximum performance. This is Apify's basic tool for web crawling and scraping.

User avatar

how do i scrape with mobile or specific UA

Closed

oblanceolate_epoch opened this issue
2 months ago

hi the pages i scrape see linux OS os the os. how can i override that and scrape with a specific UA?

User avatar

Hello and thank you for your interest in this Actor!

All our Actors run as Linux-based Docker containers. While we do randomize the user-agent strings, we found out that it's better not to lie about your operating system - modern bot protection services can figure it out by querying the list of installed fonts or by some advanced canvas-based fingerprinting techniques. Because of this, we don't allow changing the user-agent string "by hand" in this Actor.

If you need to randomize your user-agent string, you can create your our Actor - see our Crawlee + Puppeteer + Chrome Typescript Actor template, which is basically a "programmatic" version of this Actor. There, you can modify the browser fingerprints by passing the options to the underlying BrowserPool instance - see example in our Crawlee documentation.

I'll close this issue now, but feel free to ask questions - be it about custom Actor development, or this one specific Actor. Cheers!

Developer
Maintained by Apify
Actor metrics
  • 3.4k monthly users
  • 99.9% runs succeeded
  • 3.2 days response time
  • Created in Mar 2019
  • Modified about 2 months ago