Robots.txt Validator

Pricing

Pay per usage

Try for free

Go to Apify Store

Robots.txt Validator

Try for free

List of website base URLs whose robots.txt files will be validated

Pricing

Pay per usage

Rating

5.0

(1)

Developer

riya rawat

Maintained by Community

Actor stats

Bookmarked

Total users

Monthly active users

2 days ago

Last modified

JavaScript Website Scraper (Crawlee + CheerioCrawler)

A lightweight and efficient Apify Actor for scraping static and semi-dynamic websites using Crawlee with CheerioCrawler. This Actor extracts page titles and URLs from provided start pages and stores them in an Apify dataset.

Designed to be fast, easy to customize, and fully compliant with Apify Actor Store rules and the Apify $1 Million Challenge.

Key Features

Fast HTML parsing using Cheerio (no browser required)
Built on Crawlee’s CheerioCrawler
Supports Apify Proxy rotation to reduce blocking
Clearly defined input and output
Stores structured data in Apify Datasets
Easy to extend for custom scraping use cases

Quick Start

Run the Actor locally:

$apify run

Redirect Chain Analyzer

predictable_function/my-actor-2

List of URLs whose HTTP redirect chains will be analyzed

riya rawat

5.0

Sitemap Health Validator

predictable_function/my-actor

Validates sitemap.xml files and checks health of listed URLs

riya rawat

5.0

Cookie Consent Checker

pillowy_travel/cookie-consent-checker

Checks whether a website displays a cookie consent banner and basic GDPR compliance signals.

SAHIL KUMAR

Website Uptime & Response Monitor

predictable_function/website-uptime-response-monitor

Monitors website uptime and response time for given URLs.

riya rawat

5.0

Page Speed Basic Analyzer

predictable_function/my-actor-1

List of web pages to analyze for basic page speed indicators

riya rawat

5.0

Broken Link Finder

pillowy_travel/broken-link-finder

Finds and analyzes broken links on given web pages

SAHIL KUMAR

Security Headers Checker

pillowy_travel/security-headers-checker

Analyze HTTP security headers of websites and generate a security score. Detect missing headers like CSP, HSTS, X-Frame-Options, and more. Perfect for web security audits, vulnerability checks, learning, and automated monitoring.

SAHIL KUMAR

Website Change Notification API

gr_59017/website-change-notification-api

Monitors websites for content changes using Cheerio. Compares current and previous page versions and notifies when changes are detected. Ideal for tracking notices, updates, pricing changes, and announcements.

Gautam Rana

5.0

Website Title & Heading Quality Checker

gr_59017/website-title-heading-quality-checker

Analyzes website title tags and heading structure (H1–H6) to evaluate SEO quality, content hierarchy, and best practices. Detects issues like missing or multiple H1s, improper heading order, and suboptimal title length, and returns a quality score with suggestions.

Gautam Rana

5.0

SEO Meta Description Checker

gr_59017/seo-meta-description-checker

Checks whether a website has an SEO-friendly title and meta description. Validates their presence, length, and quality, then generates a simple SEO score to help identify on-page optimization issues quickly.

Gautam Rana

5.0