Pricing

Pay per usage

Try for free

Go to Apify Store

Cheerio Scraper

Try for free

Crawls websites using raw HTTP requests, parses the HTML with the Cheerio library, and extracts data from the pages using a Node.js code. Supports both recursive crawling and lists of URLs. This actor is a high-performance alternative to apify/web-scraper for websites that do not require JavaScript.

Pricing

Pay per usage

Rating

4.7

(38)

Developer

Apify

Actor stats

313

Bookmarked

25K

Total users

1.2K

Monthly active users

19 days ago

Last modified

Categories

Developer tools

Open source

Start URLs

startUrls

Required

A static list of URLs to scrape.

For details, see the Start URLs section in the README.

Type:array

URL #fragments identify unique pages

keepUrlFragments

Optional

Indicates that URL fragments (e.g. http://example.com#fragment) should be included when checking whether a URL has already been visited or not. Typically, URL fragments are used for page navigation only and therefore they should be ignored, as they don't identify separate pages. However, some single-page websites use URL fragments to display different pages; in such cases, this option should be enabled.

Type:boolean

Default:false

Respect the robots.txt file

respectRobotsTxtFile

Optional

If enabled, the crawler will consult the robots.txt file for the target website before crawling each page. At the moment, the crawler does not use any specific user agent identifier. The crawl-delay directive is also not supported yet.

Type:boolean

Default:false

Glob Patterns

globs

Optional

Glob patterns to match links in the page that you want to enqueue. Combine with Link selector to tell the scraper where to find links. Omitting the Glob patterns will cause the scraper to enqueue all links matched by the Link selector.

Type:array

Default:

[]

Pseudo-URLs

pseudoUrls

Optional

Specifies what kind of URLs found by the Link selector should be added to the request queue. A pseudo-URL is a URL with regular expressions enclosed in [] brackets, e.g. http://www.example.com/[.*].

If Pseudo-URLs are omitted, the Actor enqueues all links matched by the Link selector.

For details, see Pseudo-URLs in README.

Type:array

Default:

[]

Exclude Glob Patterns

excludes

Optional

Glob patterns to match links in the page that you want to exclude from being enqueued.

Type:array

Default:

[]

Link selector

linkSelector

Optional

A CSS selector stating which links on the page (<a> elements with href attribute) shall be followed and added to the request queue. To filter the links added to the queue, use the Pseudo-URLs and/or Glob patterns field.

If the Link selector is empty, the page links are ignored.

For details, see the Link selector in README.

Type:string

Page function

pageFunction

Required

A JavaScript function that is executed for every page loaded server-side in Node.js 12. Use it to scrape data from the page, perform actions or add new URLs to the request queue.

For details, see Page function in README.

Type:string

Proxy configuration

proxyConfiguration

Required

Specifies proxy servers that will be used by the scraper in order to hide its origin.

For details, see Proxy configuration in README.

Type:object

Default:

{
  "useApifyProxy": true
}

Proxy rotation

proxyRotation

Optional

This property indicates the strategy of proxy rotation and can only be used in conjunction with Apify Proxy. The recommended setting automatically picks the best proxies from your available pool and rotates them evenly, discarding proxies that become blocked or unresponsive. If this strategy does not work for you for any reason, you may configure the scraper to either use a new proxy for each request, or to use one proxy as long as possible, until the proxy fails. IMPORTANT: This setting will only use your available Apify Proxy pool, so if you don't have enough proxies for a given task, no rotation setting will produce satisfactory results.

Type:string

Default:RECOMMENDED

Options:

RECOMMENDEDPER_REQUESTUNTIL_FAILURE

Session pool name

sessionPoolName

Optional

Use only english alphanumeric characters dashes and underscores. A session is a representation of a user. It has it's own IP and cookies which are then used together to emulate a real user. Usage of the sessions is controlled by the Proxy rotation option. By providing a session pool name, you enable sharing of those sessions across multiple Actor runs. This is very useful when you need specific cookies for accessing the websites or when a lot of your proxies are already blocked. Instead of trying randomly, a list of working sessions will be saved and a new Actor run can reuse those sessions. Note that the IP lock on sessions expires after 24 hours, unless the session is used again in that window.

Type:string

Min. length:3

Max. length:200

Initial cookies

initialCookies

Optional

A JSON array with cookies that will be send with every HTTP request made by the Cheerio Scraper, in the format accepted by the tough-cookie NPM package. This option is useful for transferring a logged-in session from an external web browser. For details how to do this, read this help article.

Type:array

Additional MIME types

additionalMimeTypes

Optional

A JSON array specifying additional MIME content types of web pages to support. By default, Cheerio Scraper supports the text/html and application/xhtml+xml content types, and skips all other resources. For details, see Content types in README.

Type:array

Default:

[]

Suggest response encoding

suggestResponseEncoding

Optional

The scraper automatically determines response encoding from the response headers. If the headers are invalid or information is missing, malformed responses may be produced. Use the Suggest response encoding option to provide a fall-back encoding to the Scraper for cases where it could not be determined.

Type:string

Force response encoding

forceResponseEncoding

Optional

If enabled, the suggested response encoding will be used even if a valid response encoding is provided by the target website. Use this only when you've inspected the responses thoroughly and are sure that they are the ones doing it wrong.

Type:boolean

Default:false

Ignore SSL errors

ignoreSslErrors

Optional

If enabled, the scraper will ignore SSL/TLS certificate errors. Use at your own risk.

Type:boolean

Default:false

Pre-navigation hooks

preNavigationHooks

Optional

Async functions that are sequentially evaluated before the navigation. Good for setting additional cookies or browser properties before navigation. The function accepts two parameters, crawlingContext and requestAsBrowserOptions, which are passed to the requestAsBrowser() function the crawler calls to navigate.

Type:string

Post-navigation hooks

postNavigationHooks

Optional

Async functions that are sequentially evaluated after the navigation. Good for checking if the navigation was successful. The function accepts crawlingContext as the only parameter.

Type:string

Max request retries

maxRequestRetries

Optional

The maximum number of times the scraper will retry to load each web page on error, in case of a page load error or an exception thrown by the Page function.

If set to 0, the page will be considered failed right after the first error.

Type:integer

Minimum:0

Default:3

Max pages per run

maxPagesPerCrawl

Optional

The maximum number of pages that the scraper will load. The scraper will stop when this limit is reached. It is always a good idea to set this limit in order to prevent excess platform usage for misconfigured scrapers. Note that the actual number of pages loaded might be slightly higher than this value.

If set to 0, there is no limit.

Type:integer

Minimum:0

Default:0

Max result records

maxResultsPerCrawl

Optional

The maximum number of records that will be saved to the resulting dataset. The scraper will stop when this limit is reached.

If set to 0, there is no limit.

Type:integer

Minimum:0

Default:0

Max crawling depth

maxCrawlingDepth

Optional

Specifies how many links away from the Start URLs the scraper will descend. This value is a safeguard against infinite crawling depths for misconfigured scrapers. Note that pages added using context.enqueuePage() in Page function are not subject to the maximum depth constraint.

If set to 0, there is no limit.

Type:integer

Minimum:0

Default:0

Max concurrency

maxConcurrency

Optional

Specifies the maximum number of pages that can be processed by the scraper in parallel. The scraper automatically increases and decreases concurrency based on available system resources. This option enables you to set an upper limit, for example to reduce the load on a target web server.

Type:integer

Minimum:1

Default:50

Page load timeout

pageLoadTimeoutSecs

Optional

The maximum amount of time the scraper will wait for a web page to load, in seconds. If the web page does not load in this timeframe, it is considered to have failed and will be retried (subject to Max page retries), similarly as with other page load errors.

Type:integer

Minimum:1

Default:60

Page function timeout

pageFunctionTimeoutSecs

Optional

The maximum amount of time the scraper will wait for the Page function to execute, in seconds. It is always a good idea to set this limit, to ensure that unexpected behavior in page function will not get the scraper stuck.

Type:integer

Minimum:1

Default:60

Enable debug log

debugLog

Optional

If enabled, the Actor log will include debug messages. Beware that this can be quite verbose. Use context.log.debug('message') to log your own debug messages from the Page function.

Type:boolean

Default:false

Custom data

customData

Optional

A custom JSON object that is passed to the Page function as context.customData. This setting is useful when invoking the scraper via API, in order to pass some arbitrary parameters to your code.

Type:object

Default:

{}

Dataset name

datasetName

Optional

Name or ID of the dataset that will be used for storing results. If left empty, the default dataset of the run will be used.

Type:string

Key-value store name

keyValueStoreName

Optional

Name or ID of the key-value store that will be used for storing records. If left empty, the default key-value store of the run will be used.

Type:string

Request queue name

requestQueueName

Optional

Name of the request queue that will be used for storing requests. If left empty, the default request queue of the run will be used.

Type:string

Playwright Scraper

apify/playwright-scraper

Crawls websites with the headless Chromium, Chrome, or Firefox browser and Playwright library using a provided server-side Node.js code. Supports both recursive crawling and a list of URLs. Supports login to a website.

Apify

18K

3.8

Sitemap Extractor

apify/sitemap-extractor

This Apify Actor extracts all URLs from a website's sitemaps and checks their status codes via lightweight HTTP requests. It provides a clean list of valid links, acting as an ideal pre-processor to ensure your larger crawling projects target only active URLs.

Apify

344

3.2

Coursera Scraper

antonionduarte/coursera-scraper

Extract Coursera course data for market research, competitor analysis, and educational insights. This scraper fetches titles, ratings, partners, and skills, empowering you to analyze the online education landscape.

António

247

4.5

Web Scraper

apify/web-scraper

Crawls arbitrary websites using a web browser and extracts structured data from web pages using a provided JavaScript function. The Actor supports both recursive crawling and lists of URLs, and automatically manages concurrency for maximum performance.

Apify

125K

4.7

Camoufox Scraper

apify/camoufox-scraper

Crawls websites with stealthy Camoufox browser and Playwright library using a provided server-side Node.js code. Supports both recursive crawling and a list of URLs. Supports login to a website.

Apify

455

4.7

BeautifulSoup Scraper

apify/beautifulsoup-scraper

Crawls websites using raw HTTP requests. It parses the HTML with the BeautifulSoup library and extracts data from the pages using Python code. Supports both recursive crawling and lists of URLs. This Actor is a Python alternative to Cheerio Scraper.

Apify

1.1K

4.7

Ultimate News API

glitch_404/Ultimate-News-Scraper

Scrape up to 10000 news articles from over 4500 news sources in less than 20 minutes, news from over 20 categories, e.g., Crypto news, World News, Latest News, Celebrities, and a lot more. You can find news on websites such as Fox News, BBC News, CNN, and Cryptocurrency-Related News Sources.

Yousif Wael

260

1.0

Vanilla JS Scraper

mstephen190/vanilla-js-scraper

Scrape the web using familiar JavaScript methods! Crawls websites using raw HTTP requests, parses the HTML with the JSDOM package, and extracts data from the pages using Node.js code. Supports both recursive crawling and lists of URLs. This actor is a non jQuery alternative to CheerioScraper.