Robots.txt Monitor

Pricing

Pay per event

Robots.txt Monitor

Stateful robots.txt monitoring with baseline awareness and severity-classified alerts. Detects meaningful policy changes over time — not noisy diffs.

Pricing

Pay per event

Rating

0.0

(0)

Developer

Datawinder

Maintained by Community

Actor stats

Bookmarked

Total users

Monthly active users

5 days ago

Last modified

Robots.txt Validator

predictable_function/my-actor-3

List of website base URLs whose robots.txt files will be validated

riya rawat

5.0

Robots.txt Parser

tropical_quince/robots-txt-parser

Parse and analyze robots.txt files. Check crawl rules, sitemaps, and bot permissions for any website.

Donny Nguyen

Robots.txt Checker - CMS-Aware Analysis with AI Recommendations

alizarin_refrigerator-owner/robots-txt-checker

The Robots.txt Checker provides comprehensive analysis of your robots.txt file: Syntax Validation CMS Detection - Identify WordPress, Shopify, Drupal,& 6+ other CMS platforms Best Practice Check Companion File Checks - sitemap.xml, llms.txt, security.txt AI Recommendations - CMS-specific suggestions

John Rippy

Robots.txt Validator - Check Rules, Sitemaps & Crawl Directives

scrappy_garden/robots-txt-validator

Validate robots.txt for one or more websites: fetches /robots.txt per host, parses directive groups (User-agent/Allow/Disallow/Crawl-delay/Sitemap), reports common errors and warnings, and can test URLs against the chosen User-Agent.

Bikram Adhikari

Ai Visibility Suite - Dark Visitors Alternative

alizarin_refrigerator-owner/ai-visibility-suite---dark-visitors-alternative

Comprehensive AI bot monitoring, robots.txt analysis, LLMs.txt generation & AI shopping optimization. Monitor AI crawlers visits, check AI compliance, generate AI-friendly configurations, and optimize for AI shopping agents. AI Bot Directory Robots.txt LLMs.txt AI Shopping Competitor AI Audit

John Rippy

Fast Sitemap Generator

eunit/sitemap-generator

Boost SEO with this automatic Sitemap Generator. Crawl any site to create XML, HTML, & TXT sitemaps. Supports custom depth, regex filters, & robots.txt. Compatible with Google Search Console.

Emmanuel Uchenna

5.0

Llms Txt Generator

francis.businessfm/llms-txt-generator

Francis Marzyński

AI Readiness Checker - Website Scanner

alizarin_refrigerator-owner/ai-readiness-checker

Analyze any website for AI optimization readiness. Check robots.txt, llms.txt, structured data, meta tags & content quality. Get actionable recommendations to improve AI crawler accessibility.

John Rippy

universal-structured-web-extractor-base

pierremrd29/universal-structured-web-extractor-base

Autonomous web change detection Actor for AI agents. Monitors URLs, detects meaningful content changes, computes diffs, and outputs structured signals for automation, alerts, and decision-making workflows.

Pierre Morandeau

competitor-pricing-monitor

wholesome_quadrature/competitor-pricing-monitor

Monitors competitor pricing pages and emits events only when meaningful price changes occur. Designed for cost-efficient, stateful monitoring using Apify Tasks.