robots.txt & Sitemap Analyzer: Crawlability Check
Pricing
from $12.75 / 1,000 domain analyzeds
robots.txt & Sitemap Analyzer: Crawlability Check
Give it domains. For each it parses robots.txt (user-agents, allow/disallow, crawl-delay, declared sitemaps, blocks-all) and the XML sitemap(s) (URL counts, nested sitemaps, lastmod coverage), and flags crawlability issues - with a signal and findings. Keyless, read-only.
Pricing
from $12.75 / 1,000 domain analyzeds
Rating
0.0
(0)
Developer
Centor Prat
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
5 hours ago
Last modified
Categories
Share
Make sure search engines can actually crawl and index your site. Give it domains; for each it parses
the robots.txt (user-agents, allow/disallow rules, crawl-delay, declared sitemaps, and whether it
blocks all crawlers) and the XML sitemap(s) (URL counts, nested sitemaps, lastmod coverage), and
flags crawlability issues — with a clear signal. Keyless, agent-callable.
🤖 Read-only. It fetches
/robots.txtand the XML sitemap(s). Recommendations, not a verdict.
📋 What each report contains
| Field | Meaning |
|---|---|
signal | healthy / ok-with-recommendations / needs-attention |
robotsTxt | found, group count, user-agents, total disallow rules, declared sitemaps, blocksAllCrawlers, crawl-delay |
sitemaps | each sitemap: type (urlset/sitemapindex), URL count or child-sitemap count, lastmod coverage, validity |
findings | e.g. blocks-all-crawlers, no sitemap, sitemap not declared in robots, >50k URLs, no lastmod |
📥 Input
📤 Example output (real run, trimmed — apple.com)
💰 Pricing (pay-per-event)
- Domain analyzed — $0.015 per domain.
💡 Use cases
- SEO / technical SEO: confirm nothing is accidentally blocking crawlers and your sitemap is valid and declared.
- Migrations / launches: catch a stray
Disallow: /or a missing sitemap before it deindexes you. - Agencies: audit crawlability across a portfolio of client domains.
- Agents: callable as an MCP tool — give it domains, get back a crawlability report.
🔑 Keyless & agent-callable
No API keys, no accounts, no paid child Actors — just read-only fetches of robots.txt and the sitemap.
🚀 Ready-made tasks (one click)
🤖 For AI agents (MCP)
Loads as centorprat--robots-sitemap-analyzer. Claude Code:
