Actor picture

Web Scraper

apify/web-scraper

Crawls arbitrary websites using the Chrome browser and extracts data from pages using a provided JavaScript code. The actor supports both recursive crawling and lists of URLs and automatically manages concurrency for maximum performance. This is Apify's basic tool for web crawling and scraping.

The following table shows specification of the actor INPUT fields as defined by its input schema. These fields can be entered either manually in the app, or provided in a JSON object when running the actor using the API. Read more in docs.

Run mode

This property indicates the scraper's mode of operation. In DEVELOPMENT mode, the scraper ignores page timeouts, opens pages one by one and enables debugging via Chrome DevTools. Open the live view tab or the container URL to access the debugger. Further debugging options can be configured in the Advanced configuration section. PRODUCTION mode disables debugging and enables timeouts and concurrency.

For details, see Run mode in README.

Optional

Type: string

Options
  • "PRODUCTION"
  • "DEVELOPMENT"

Start URLs

A static list of URLs to scrape. To be able to add new URLs on the fly, enable the Use request queue option.

For details, see Start URLs in README.

Required

Type: array

Use request queue

If enabled, the scraper will support adding new URLs to scrape on the fly, either using the Link selector and Pseudo-URLs options or by calling context.enqueueRequest() inside Page function. Use of the request queue has some overheads, so only enable this option if you need to add URLs dynamically.

Optional

Type: boolean

URL #fragments identify unique pages

Indicates that URL fragments (e.g. http://example.com#fragment) should be included when checking whether a URL has already been visited or not. Typically, URL fragments are used for page navigation only and therefore they should be ignored, as they don't identify separate pages. However, some single-page websites use URL fragments to display different pages; in such a case, this option should be enabled.

Optional

Type: boolean

Link selector

A CSS selector saying which links on the page (<a> elements with href attribute) shall be followed and added to the request queue. This setting only applies if Use request queue is enabled. To filter the links added to the queue, use the Pseudo-URLs setting.

If Link selector is empty, the page links are ignored.

For details, see Link selector in README.

Optional

Type: string

Pseudo-URLs

Specifies what kind of URLs found by Link selector should be added to the request queue. A pseudo-URL is a URL with regular expressions enclosed in [] brackets, e.g. http://www.example.com/[.*]. This setting only applies if the Use request queue option is enabled.

If Pseudo-URLs are omitted, the actor enqueues all links matched by the Link selector.

For details, see Pseudo-URLs in README.

Optional

Type: array

Page function

JavaScript (ES6) function that is executed in the context of every page loaded in the Chrome browser. Use it to scrape data from the page, perform actions or add new URLs to the request queue.

For details, see Page function in README.

Required

Type: string

Inject jQuery

If enabled, the scraper will inject the jQuery library into every web page loaded, before Page function is invoked. Note that the jQuery object ($) will not be registered into global namespace in order to avoid conflicts with libraries used by the web page. It can only be accessed through context.jQuery in Page function.

Optional

Type: boolean

Inject Underscore.js

If enabled, the scraper will inject the Underscore.js library into every web page loaded, before Page function is invoked. Note that the Underscore.js object (_) will not be registered into global namespace in order to avoid conflicts with libraries used by the web page. It can only be accessed through context.underscoreJs in Page function.

Optional

Type: boolean

Proxy configuration

Specifies proxy servers that will be used by the scraper in order to hide its origin.

For details, see Proxy configuration in README.

Optional

Type: object

Proxy rotation

This property indicates the strategy of proxy rotation and can only be used in conjunction with Apify Proxy. The recommended setting automatically picks the best proxies from your available pool and rotates them evenly, discarding proxies that become blocked or unresponsive. If this strategy does not work for you for any reason, you may configure the scraper to either use a new proxy for each request, or to use one proxy as long as possible, until the proxy fails. IMPORTANT: This setting will only use your available Apify Proxy pool, so if you don't have enough proxies for a given task, no rotation setting will produce satisfactory results.

Optional

Type: string

Options
  • Use recommended settings: "RECOMMENDED"
  • Rotate proxy after each request: "PER_REQUEST"
  • Use one proxy until failure: "UNTIL_FAILURE"

Session pool name

Use only english alphanumeric characters dashes and underscores. A session is a representation of a user. It has it's own IP and cookies which are then used together to emulate a real user. Usage of the sessions is controlled by the Proxy rotation option. By providing a session pool name, you enable sharing of those sessions across multiple actor runs. This is very useful when you need specific cookies for accessing the websites or when a lot of your proxies are already blocked. Instead of trying randomly, a list of working sessions will be saved and a new actor run can reuse those sessions. Note that the IP lock on sessions expires after 24 hours, unless the session is used again in that window.

Optional

Type: string

Initial cookies

A JSON array with cookies that will be set to every Chrome browser tab opened before loading the page, in the format accepted by Puppeteer's Page.setCookie() function. This option is useful for transferring a logged-in session from an external web browser. For details how to do this, read this help article.

Optional

Type: array

Use Chrome

If enabled, the scraper will use a real Chrome browser instead of Chromium bundled with Puppeteer. This option may help bypass certain anti-scraping protections, but might make the scraper unstable. Use at your own risk 馃檪

Optional

Type: boolean

Use stealth mode

If enabled, the scraper will apply various browser emulation techniques to match a real user's browser as closely as possible, in order to bypass around certain anti-scraping protections. This feature works best in conjunction with the Use Chrome option, but it also carries a risk of making the scraper unstable.

Optional

Type: boolean

Ignore SSL errors

If enabled, the scraper will ignore SSL/TLS certificate errors. Use at your own risk.

Optional

Type: boolean

Ignore CORS and CSP

If enabled, the scraper will ignore Content Security Policy (CSP) and Cross-Origin Resource Sharing (CORS) settings of visited pages and requested domains. This enables you to freely use XHR/Fetch to make HTTP requests from Page function.

Optional

Type: boolean

Download media files

If enabled, the scraper will download media such as images, fonts, videos and sound files, as usual. Disabling this option might speed up the scrape, but certain websites could stop working correctly.

Optional

Type: boolean

Download CSS files

If enabled, the scraper will download CSS files with stylesheets, as usual. Disabling this option may speed up the scrape, but certain websites could stop working correctly, and the live view will not look as cool.

Optional

Type: boolean

Max page retries

The maximum number of times the scraper will retry to load each web page on error, in case of a page load error or an exception thrown by Page function.

If set to 0, the page will be considered failed right after the first error.

Optional

Type: integer

minimum: 0

Max pages per run

The maximum number of pages that the scraper will load. The scraper will stop when this limit is reached. It's always a good idea to set this limit in order to prevent excess platform usage for misconfigured scrapers. Note that the actual number of pages loaded might be slightly higher than this value.

If set to 0, there is no limit.

Optional

Type: integer

minimum: 0

Max result records

The maximum number of records that will be saved to the resulting dataset. The scraper will stop when this limit is reached.

If set to 0, there is no limit.

Optional

Type: integer

minimum: 0

Max crawling depth

Specifies how many links away from Start URLs the scraper will descend. This value is a safeguard against infinite crawling depths for misconfigured scrapers. Note that pages added using context.enqueuePage() in Page function are not subject to the maximum depth constraint.

If set to 0, there is no limit.

Optional

Type: integer

minimum: 0

Max concurrency

Specified the maximum number of pages that can be processed by the scraper in parallel. The scraper automatically increases and decreases concurrency based on available system resources. This option enables you to set an upper limit, for example to reduce the load on a target web server.

Optional

Type: integer

minimum: 1

Page load timeout

The maximum amount of time the scraper will wait for a web page to load, in seconds. If the web page does not load in this timeframe, it is considered to have failed and will be retried (subject to Max page retries), similarly as with other page load errors.

Optional

Type: integer

minimum: 1

maximum: 360

Page function timeout

The maximum amount of time the scraper will wait for Page function to execute, in seconds. It's a good idea to set this limit, to ensure that unexpected behavior in page function will not get the scraper stuck.

Optional

Type: integer

minimum: 1

maximum: 360

Navigation waits until

Contains a JSON array with names of page events to wait, before considering a web page fully loaded. The scraper will wait until all of the events are triggered in the web page before executing Page function. Available events are domcontentloaded, load, networkidle2 and networkidle0.

For details, see waitUntil option in Puppeteer's Page.goto() function documentation.

Optional

Type: array

Insert breakpoint

This property has no effect if Run mode is set to PRODUCTION. When set to DEVELOPMENT it inserts a breakpoint at the selected location in every page the scraper visits. Execution of code stops at the breakpoint until manually resumed in the DevTools window accessible via Live View tab or Container URL. Additional breakpoints can be added by adding debugger; statements within your Page function.

See Run mode in README for details.

Optional

Type: string

Options
  • Nowhere. Break only on debugger; statements: "NONE"
  • Before navigation to URL: "BEFORE_GOTO"
  • Before Page function invocation: "BEFORE_PAGE_FUNCTION"
  • After Page function invocation: "AFTER_PAGE_FUNCTION"

Enable debug log

If enabled, the actor log will include debug messages. Beware that this can be quite verbose. Use context.log.debug('message') to log your own debug messages from Page function.

Optional

Type: boolean

Enable browser log

If enabled, the actor log will include console messages produced by JavaScript executed by the web pages (e.g. using console.log()). Beware that this may result in the log being flooded by error messages, warnings and other messages of little value, especially with high concurrency.

Optional

Type: boolean

Custom data

A custom JSON object that is passed to Page function as context.customData. This setting is useful when invoking the scraper via API, in order to pass some arbitrary parameters to your code.

Optional

Type: object

  • Modified
  • Last run
  • Used2430302 times
  • Used by10125 users
Categories