Pricing

Pay per usage

Try for free

Go to Apify Store

Robots.txt Validator

Try for free

List of website base URLs whose robots.txt files will be validated

Pricing

Pay per usage

Rating

5.0

(1)

Developer

riya rawat

Actor stats

Bookmarked

Total users

Monthly active users

8 months ago

Last modified

5.0

/ 5

1 review

Robots.txt Analyzer

mahogany_songbird/robots-txt-analyzer

Read robots.txt disallow rules and sitemap declarations.

Britton Furness

Website Tech Audit Scraper: robots.txt, Sitemaps & Meta

arman-bd/website-tech-audit-scraper

Audit any site's public technical files: robots.txt rules, sitemap discovery and URL counts, ads.txt sellers, security.txt contacts, meta tags, structured data and the detected tech stack with evidence.

Arman Hossain

Robots.txt Auditor & Sitemap Finder

andok/robotstxt-auditor

Scan robots.txt files in bulk to extract sitemap URLs and verify crawler directives for technical SEO compliance.

Andok

Robots.txt & Sitemap Discovery API

213x/robots-sitemap-discovery

Discover robots.txt rules, sitemap URLs, crawl delays, and basic crawlability signals for domains or website URLs.

Robots.txt & Security.txt Compliance Auditor

devilscrapes/robots-txt-security-txt-auditor

Keyless, plain-HTTP compliance auditor. Given a list of sites, it fetches and parses robots.txt, security.txt (RFC 9116), ads.txt, and humans.txt into one structured audit row per site — AI-crawler posture, security contacts/expiry, and ad-tech supply-chain record counts. No API key needed.

DevilScrapes

Robots.txt Tester & Validator

gifted_wagon/robots-txt-evidence-auditor

Parse robots.txt rules into normalized policy evidence, sitemap references, tested paths, and truthful failures.

Michael Olmos

Robots.txt & Sitemap Analyzer 🕷️

perryay/robots-txt-sitemap-analyzer

Fetch, parse, and analyze robots.txt and sitemap.xml for any domain. Extract crawl directives, test URL compliance against robots.txt rules, and discover all URLs from sitemaps including nested sitemap indexes. Supports batch analysis with structured JSON output.

Perry AY

Robots.txt Validator - Check Rules, Sitemaps & Crawl Directives

scrappy_garden/robots-txt-validator

Validate robots.txt for one or more websites: fetches /robots.txt per host, parses directive groups (User-agent/Allow/Disallow/Crawl-delay/Sitemap), reports common errors and warnings, and can test URLs against the chosen User-Agent.

Bikram Adhikari

Robots.txt Analyzer

sootesting/robots-txt-analyzer

Analyze robots.txt for any list of domains — crawl rules per user-agent, declared sitemaps, crawl-delay, and more. Outputs structured JSON. Pay-per-event.