
Pages Checker
Deprecated
Pricing
Pay per usage

Pages Checker
Deprecated
Using Cheerio & Crawlee, the actor checks if given pages satisfy required conditions and fails if not.
0.0 (0)
Pricing
Pay per usage
3
Total users
2
Monthly users
1
Runs succeeded
>99%
Last modified
2 years ago
The actor runs a specific check against given list of urls. The check is provided as a JS function, that gets cheerio parsed contents of the respective url.
Based on Cheerio scraper, so it can't run the website's JavaScript or pass JS anti-scraping challenges.
The dataset contains results of check (boolean) for individual urls.
Actor always processes all provided urls, fails if at least one of the check function calls returned false. Should succeed otherwise.
Use Case
Only use-case at this point is to check if our staging applications are not indexed by search engines, meaning that they have
<meta name="robots" content="noindex, nofollow">
To check that, following check function is used
async ({ $ }) => {const metaRobots = $('meta[name="robots"]').attr('content');return metaRobots.includes('noindex') && metaRobots.includes('nofollow');}