Pages Checker avatar
Pages Checker

Deprecated

Pricing

Pay per usage

Go to Store
Pages Checker

Pages Checker

Deprecated

Developed by

Josef Válek

Josef Válek

Maintained by Community

Using Cheerio & Crawlee, the actor checks if given pages satisfy required conditions and fails if not.

0.0 (0)

Pricing

Pay per usage

3

Total users

2

Monthly users

1

Runs succeeded

>99%

Last modified

2 years ago

The actor runs a specific check against given list of urls. The check is provided as a JS function, that gets cheerio parsed contents of the respective url.

Based on Cheerio scraper, so it can't run the website's JavaScript or pass JS anti-scraping challenges.

The dataset contains results of check (boolean) for individual urls.

Actor always processes all provided urls, fails if at least one of the check function calls returned false. Should succeed otherwise.

Use Case

Only use-case at this point is to check if our staging applications are not indexed by search engines, meaning that they have

<meta name="robots" content="noindex, nofollow">

To check that, following check function is used

async ({ $ }) => {
const metaRobots = $('meta[name="robots"]').attr('content');
return metaRobots.includes('noindex') && metaRobots.includes('nofollow');
}