Robots & Sitemap Analyzer

Robots.txt & Sitemap Analyzer 🕷️

perryay/robots-txt-sitemap-analyzer

Fetch, parse, and analyze robots.txt and sitemap.xml for any domain. Extract crawl directives, test URL compliance against robots.txt rules, and discover all URLs from sitemaps including nested sitemap indexes. Supports batch analysis with structured JSON output.

Discover robots.txt rules, sitemap URLs, crawl delays, and basic crawlability signals for domains or website URLs.

Checks robots.txt and XML sitemaps for a site: crawlability issues, sitemap validity, and broken sitemap URLs. Fast, no browser needed. Need every check? See Full Site Health Suite for all 9 checks in one job.

Read robots.txt disallow rules and sitemap declarations.

Monitor sitemap.xml and robots.txt for URL inventory changes and return new, changed, or removed URLs in normalized JSON.

Sitemap, robots.txt & RSS Change Monitor

enfex/source-change-monitor

Monitor public robots.txt, sitemap XML, and RSS/Atom source changes for SEO and content intelligence without browser automation.

Audit a website's robots.txt directives and sitemap discovery in one fast API call. Detect blocked paths, missing sitemaps, and crawlability issues. Built for SEO agencies, technical SEO consultants, and site migration QA teams.

Robots.txt Analyzer

sootesting/robots-txt-analyzer

Analyze robots.txt for any list of domains — crawl rules per user-agent, declared sitemaps, crawl-delay, and more. Outputs structured JSON. Pay-per-event.

Robots.txt & Sitemap Extractor by Domain

technicaldost/robots-sitemap-extractor

Fetch and parse robots.txt and XML sitemaps for any domain. Extract allowed and disallowed paths, sitemap URLs and every listed page. Great for SEO audits and crawl planning. JSON output.

Extract and parse XML sitemaps from any website. Discovers sitemaps via robots.txt and common paths. Outputs structured URL data with metadata.