Pricing

Pay per usage

Go to Apify Store

Robots.txt Analyzer

Try for free

Read robots.txt disallow rules and sitemap declarations.

Pricing

Pay per usage

Rating

0.0

(0)

Developer

Britton Furness

Actor stats

Bookmarked

Total users

Monthly active users

3 days ago

Last modified

Input

domains — hostnames or URLs (required)
requestTimeoutSecs — HTTP timeout
includeRawText — include full robots body in output

Output

One row per domain with groupsJson, sitemaps, ruleCount, and optional rawRobotsTxt.

Deploy

$python3 /root/apify-actors/fleet_deploy.py robots-txt-analyzer create

robots.txt Analyzer

eliai/robots-txt-analyzer

Anthony Snider

Robots.txt Auditor

junipr/robots-txt-auditor

Fetch and audit robots.txt syntax, user-agent rules, blocked paths, sitemap declarations, and crawl risks.

junipr

Robots Sitemap Analyzer - SEO Crawl Rules

benthepythondev/robots-sitemap-analyzer

Analyze robots.txt files and discover sitemap URLs, user-agent groups, allow rules, disallow rules and crawl-delay directives.

ben

Robots.txt Analyzer

sootesting/robots-txt-analyzer

Analyze robots.txt for any list of domains — crawl rules per user-agent, declared sitemaps, crawl-delay, and indexing red flags (e.g. 'Disallow: /' blocking the whole site). One clean report per domain. Pay-per-event: $0.01 per batch of up to 200 domains.

soot

Robots.txt & Sitemap Discovery API

213x/robots-sitemap-discovery

Discover robots.txt rules, sitemap URLs, crawl delays, and basic crawlability signals for domains or website URLs.

Robots.txt Validator - Check Rules, Sitemaps & Crawl Directives

scrappy_garden/robots-txt-validator

Validate robots.txt for one or more websites: fetches /robots.txt per host, parses directive groups (User-agent/Allow/Disallow/Crawl-delay/Sitemap), reports common errors and warnings, and can test URLs against the chosen User-Agent.

Bikram Adhikari

Robots Txt Analyzer

zerobreak/robots-txt-analyzer

Robots txt analyzer that fetches and parses crawl rules from any website in bulk, so SEO teams and developers can audit blocked paths, user agents, and sitemap locations across hundreds of domains without manual work.

ZeroBreak

Robots.txt Auditor & Sitemap Finder

andok/robotstxt-auditor

Scan robots.txt files in bulk to extract sitemap URLs and verify crawler directives for technical SEO compliance.

Andok

Sitemap, robots.txt & RSS Change Monitor

enfex/source-change-monitor

Monitor public robots.txt, sitemap XML, and RSS/Atom source changes for SEO and content intelligence without browser automation.

Marcel K

Robots.txt Checker & Parser - Crawl Rules API

pink_comic/robots-txt-validator

Check, parse, and validate robots.txt files in bulk. Extract crawl rules, sitemaps, crawl-delay, blocked paths, and per-user-agent allow/disallow results for SEO audits and crawler compliance.