Robots.txt & Sitemap Analyzer 🕷️ avatar

Robots.txt & Sitemap Analyzer 🕷️

Pricing

from $0.005 / actor start

Go to Apify Store
Robots.txt & Sitemap Analyzer 🕷️

Robots.txt & Sitemap Analyzer 🕷️

Fetch, parse, and analyze robots.txt and sitemap.xml for any domain. Extract crawl directives, test URL compliance against robots.txt rules, and discover all URLs from sitemaps including nested sitemap indexes. Supports batch analysis with structured JSON output.

Pricing

from $0.005 / actor start

Rating

0.0

(0)

Developer

Perry AY

Perry AY

Maintained by Community

Actor stats

0

Bookmarked

3

Total users

1

Monthly active users

a month ago

Last modified

Categories

Security

Limited permissions

Share