Pricing

Pay per usage

Try for free

Go to Apify Store

Puppeteer Scraper

Try for free

Crawls websites with the headless Chrome and Puppeteer library using a provided server-side Node.js code. This crawler is an alternative to apify/web-scraper that gives you finer control over the process. Supports both recursive crawling and list of URLs. Supports login to website.

Pricing

Pay per usage

Rating

5.0

(18)

Developer

Apify

Actor stats

282

Bookmarked

12K

Total users

658

Monthly active users

39 days

Issues response

22 days ago

Last modified

Categories

Developer tools

Open source

Start URLs

startUrls

Required

URLs to start with

Type:array

Glob Patterns

globs

Optional

Glob patterns to match links in the page that you want to enqueue. Combine with Link selector to tell the scraper where to find links. Omitting the Glob patterns will cause the scraper to enqueue all links matched by the Link selector.

Type:array

Default:

[]

Pseudo-URLs

pseudoUrls

Optional

Pseudo-URLs to match links in the page that you want to enqueue. Combine with Link selector to tell the scraper where to find links. Omitting the Pseudo-URLs will cause the scraper to enqueue all links matched by the Link selector.

Type:array

Default:

[]

Exclude Glob Patterns

excludes

Optional

Glob patterns to match links in the page that you want to exclude from being enqueued.

Type:array

Default:

[]

Link selector

linkSelector

Optional

CSS selector matching elements with 'href' attributes that should be enqueued. To enqueue urls from

tags, you would enter div.my-class. Leave empty to ignore all links.

Type:string

Clickable elements selector

clickableElementsSelector

Optional

For pages where simple 'href' links are not available, this attribute allows you to specify a CSS selector matching elements that the scraper will mouse click after the page function finishes. Any triggered requests, navigations or open tabs will be intercepted and the target URLs will be filtered using Pseudo URLs and/or Glob patterns and subsequently added to the request queue. Leave empty to prevent the scraper from clicking in the page. Using this setting will have a performance impact.

Type:string

Keep URL fragments

keepUrlFragments

Optional

URL fragments (the parts of URL after a #) are not considered when the scraper determines whether a URL has already been visited. This means that when adding URLs such as https://example.com/#foo and https://example.com/#bar, only the first will be visited. Turn this option on to tell the scraper to visit both.

Type:boolean

Default:false

Respect the robots.txt file

respectRobotsTxtFile

Optional

If enabled, the crawler will consult the robots.txt file for the target website before crawling each page. At the moment, the crawler does not use any specific user agent identifier. The crawl-delay directive is also not supported yet.

Type:boolean

Default:false

Page function

pageFunction

Required

Function executed for each request

Type:string

Proxy configuration

proxyConfiguration

Required

Specifies proxy servers that will be used by the scraper in order to hide its origin.

For details, see Proxy configuration in README.

Type:object

Default:

{
  "useApifyProxy": true
}

Proxy rotation

proxyRotation

Optional

This property indicates the strategy of proxy rotation and can only be used in conjunction with Apify Proxy. The recommended setting automatically picks the best proxies from your available pool and rotates them evenly, discarding proxies that become blocked or unresponsive. If this strategy does not work for you for any reason, you may configure the scraper to either use a new proxy for each request, or to use one proxy as long as possible, until the proxy fails. IMPORTANT: This setting will only use your available Apify Proxy pool, so if you don't have enough proxies for a given task, no rotation setting will produce satisfactory results.

Type:string

Default:RECOMMENDED

Options:

RECOMMENDEDPER_REQUESTUNTIL_FAILURE

Session pool name

sessionPoolName

Optional

Use only english alphanumeric characters dashes and underscores. A session is a representation of a user. It has it's own IP and cookies which are then used together to emulate a real user. Usage of the sessions is controlled by the Proxy rotation option. By providing a session pool name, you enable sharing of those sessions across multiple Actor runs. This is very useful when you need specific cookies for accessing the websites or when a lot of your proxies are already blocked. Instead of trying randomly, a list of working sessions will be saved and a new Actor run can reuse those sessions. Note that the IP lock on sessions expires after 24 hours, unless the session is used again in that window.

Type:string

Min. length:3

Max. length:200

Initial cookies

initialCookies

Optional

The provided cookies will be pre-set to all pages the scraper opens.

Type:array

Default:

[]

Use Chrome

useChrome

Optional

The scraper will use a real Chrome browser instead of a Chromium masking as Chrome. Using this option may help with bypassing certain anti-scraping protections, but risks that the scraper will be unstable or not work at all.

Type:boolean

Default:false

Run browsers in headless mode

headless

Optional

By default, browsers run in headless mode. You can toggle this off to run them in headful mode, which can help with certain rare anti-scraping protections but is slower and more costly.

Type:boolean

Default:true

Ignore SSL errors

ignoreSslErrors

Optional

Scraper will ignore SSL certificate errors.

Type:boolean

Default:false

Ignore CORS and CSP

ignoreCorsAndCsp

Optional

Scraper will ignore CSP (content security policy) and CORS (cross origin resource sharing) settings of visited pages and requested domains. This enables you to freely use XHR/Fetch to make HTTP requests from the scraper.

Type:boolean

Default:false

Download media

downloadMedia

Optional

Scraper will download media such as images, fonts, videos and sounds. Disabling this may speed up the scrape, but certain websites could stop working correctly.

Type:boolean

Default:true

Download CSS

downloadCss

Optional

Scraper will download CSS stylesheets. Disabling this may speed up the scrape, but certain websites could stop working correctly.

Type:boolean

Default:true

Max request retries

maxRequestRetries

Optional

Maximum number of times the request for the page will be retried in case of an error. Setting it to 0 means that the request will be attempted once and will not be retried if it fails.

Type:integer

Minimum:0

Default:3

Max pages per run

maxPagesPerCrawl

Optional

Maximum number of pages that the scraper will open. 0 means unlimited.

Type:integer

Minimum:0

Default:0

Max result records

maxResultsPerCrawl

Optional

Maximum number of results that will be saved to dataset. The scraper will terminate afterwards. 0 means unlimited.

Type:integer

Minimum:0

Default:0

Max crawling depth

maxCrawlingDepth

Optional

Defines how many links away from the StartURLs will the scraper descend. 0 means unlimited.

Type:integer

Minimum:0

Default:0

Max concurrency

maxConcurrency

Optional

Defines how many pages can be processed by the scraper in parallel. The scraper automatically increases and decreases concurrency based on available system resources. Use this option to set a hard limit.

Type:integer

Minimum:1

Default:50

Page load timeout

pageLoadTimeoutSecs

Optional

Maximum time the scraper will allow a web page to load in seconds.

Type:integer

Minimum:1

Default:60

Page function timeout

pageFunctionTimeoutSecs

Optional

Maximum time the scraper will wait for the page function to execute in seconds.

Type:integer

Minimum:1

Default:60

Navigation wait until

waitUntil

Optional

The scraper will wait until the selected events are triggered in the page before executing the page function. Available events are domcontentloaded, load, networkidle2 and networkidle0. See Puppeteer docs.

Type:array

Default:

[
  "networkidle2"
]

Pre-navigation hooks

preNavigationHooks

Optional

Async functions that are sequentially evaluated before the navigation. Good for setting additional cookies or browser properties before navigation. The function accepts two parameters, crawlingContext and gotoOptions, which are passed to the page.goto() function the crawler calls to navigate.

Type:string

Post-navigation hooks

postNavigationHooks

Optional

Async functions that are sequentially evaluated after the navigation. Good for checking if the navigation was successful. The function accepts crawlingContext as the only parameter.

Type:string

Dismiss cookie modals

closeCookieModals

Optional

Using the I don't care about cookies browser extension. When on, the crawler will automatically try to dismiss cookie consent modals. This can be useful when crawling European websites that show cookie consent modals.

Type:boolean

Default:false

Maximum scrolling distance in pixels

maxScrollHeightPixels

Optional

The crawler will scroll down the page until all content is loaded or the maximum scrolling distance is reached. Setting this to 0 disables scrolling altogether.

Type:integer

Default:5000

Debug log

debugLog

Optional

Debug messages will be included in the log. Use context.log.debug('message') to log your own debug messages.

Type:boolean

Default:false

Browser log

browserLog

Optional

Console messages from the Browser will be included in the log. This may result in the log being flooded by error messages, warnings and other messages of little value, especially with high concurrency.

Type:boolean

Default:false

Custom data

customData

Optional

This object will be available on pageFunction's context as customData.

Type:object

Default:

{}

Dataset name

datasetName

Optional

Name or ID of the dataset that will be used for storing results. If left empty, the default dataset of the run will be used.

Type:string

Key-value store name

keyValueStoreName

Optional

Name or ID of the key-value store that will be used for storing records. If left empty, the default key-value store of the run will be used.

Type:string

Request queue name

requestQueueName

Optional

Name of the request queue that will be used for storing requests. If left empty, the default request queue of the run will be used.

Type:string

Playwright Scraper

apify/playwright-scraper

Crawls websites with the headless Chromium, Chrome, or Firefox browser and Playwright library using a provided server-side Node.js code. Supports both recursive crawling and a list of URLs. Supports login to a website.

Apify

5.5K

2.4

Cheerio Scraper

apify/cheerio-scraper

Crawls websites using raw HTTP requests, parses the HTML with the Cheerio library, and extracts data from the pages using a Node.js code. Supports both recursive crawling and lists of URLs. This actor is a high-performance alternative to apify/web-scraper for websites that do not require JavaScript.

Apify

14K

5.0

Web Scraper

apify/web-scraper

Crawls arbitrary websites using a web browser and extracts structured data from web pages using a provided JavaScript function. The Actor supports both recursive crawling and lists of URLs, and automatically manages concurrency for maximum performance.

Apify

107K

4.7

bcv-tasa-oficial

grupoaceivzla/bcv-tasa-oficial

Grupo ACEI

BeautifulSoup Scraper

apify/beautifulsoup-scraper

Crawls websites using raw HTTP requests. It parses the HTML with the BeautifulSoup library and extracts data from the pages using Python code. Supports both recursive crawling and lists of URLs. This Actor is a Python alternative to Cheerio Scraper.

Apify

970

4.2

Camoufox Scraper

apify/camoufox-scraper

Crawls websites with stealthy Camoufox browser and Playwright library using a provided server-side Node.js code. Supports both recursive crawling and a list of URLs. Supports login to a website.

Apify

177

Cloudflare Web Scraper

ecomscrape/cloudflare-web-scraper

Advanced web scraper designed to extract data from Cloudflare-protected websites with CAPTCHA bypass, proxy rotation, and JavaScript execution capabilities.

ecomscrape

605

3.3

Website Screenshot Generator

apify/screenshot-url

Create a screenshot of a website based on a specified URL. The screenshot is stored as the output in a key-value store. It can be used to monitor web changes regularly after setting up the scheduler.

Apify

4.5K

4.4

Vanilla JS Scraper

mstephen190/vanilla-js-scraper

Scrape the web using familiar JavaScript methods! Crawls websites using raw HTTP requests, parses the HTML with the JSDOM package, and extracts data from the pages using Node.js code. Supports both recursive crawling and lists of URLs. This actor is a non jQuery alternative to CheerioScraper.

Matthias Stephens

515

Example Puppeteer

apify/example-puppeteer

Example showing how to use headless Chromium with Puppeteer to open a web page, determine its dimensions, save a screenshot, and print the page to PDF. This actor must use images with Puppeteer (Node.js 8 + Puppeteer on Debian).

Apify

299

4.6