robots.txt & Sitemap Analyzer: Crawlability Check
Pricing
from $12.75 / 1,000 domain analyzeds
Go to Apify Store

robots.txt & Sitemap Analyzer: Crawlability Check
Give it domains. For each it parses robots.txt (user-agents, allow/disallow, crawl-delay, declared sitemaps, blocks-all) and the XML sitemap(s) (URL counts, nested sitemaps, lastmod coverage), and flags crawlability issues - with a signal and findings. Keyless, read-only.
Pricing
from $12.75 / 1,000 domain analyzeds
Rating
0.0
(0)
Developer
Centor Prat
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
6 hours ago
Last modified
Categories
Share