Sitemap & Robots.txt Auditor

Robots & Sitemap Analyzer

ib4ngz/robots-sitemap-analyzer

Analyze robots.txt and XML sitemaps into normalized crawlability intelligence.

Audit a website's robots.txt directives and sitemap discovery in one fast API call. Detect blocked paths, missing sitemaps, and crawlability issues. Built for SEO agencies, technical SEO consultants, and site migration QA teams.

Discover robots.txt rules, sitemap URLs, crawl delays, and basic crawlability signals for domains or website URLs.

Robots.txt & Sitemap Analyzer 🕷️

perryay/robots-txt-sitemap-analyzer

Fetch, parse, and analyze robots.txt and sitemap.xml for any domain. Extract crawl directives, test URL compliance against robots.txt rules, and discover all URLs from sitemaps including nested sitemap indexes. Supports batch analysis with structured JSON output.

Discover every URL a website's own XML sitemap declares, starting from robots.txt's Sitemap: directive and following sitemap indexes automatically. No key, no page crawling — charged only when a working sitemap is found.

Sitemap Checker & Broken Link Finder

aiqlabs/sitemap-checker

Find every sitemap a site publishes, expand sitemap indexes, and audit each URL: broken links, redirect chains, URLs blocked by the site's own robots.txt, XML errors, duplicates and invalid lastmod. One row per URL. Respects robots.txt.

Discover every URL a site exposes via its public sitemap chain. Reads robots.txt, follows Sitemap declarations, recursively descends sitemap-index files, extracts URLs with lastmod, changefreq, priority.

Robots.txt Auditor & Sitemap Finder

andok/robotstxt-auditor

Scan robots.txt files in bulk to extract sitemap URLs and verify crawler directives for technical SEO compliance.

Discover robots.txt sitemap entries and classify public sitemap URLs by page type.

Sitemap URL Extractor: XML Sitemap Checker, Auditor & SEO Audit

scrapevanta/sitemap-url-extractor

Extracts every URL from a sitemap or sitemap index, auto-discovers sitemaps from robots.txt, handles gzipped sitemaps, filters by URL pattern and last-modified date, and reports site structure and content freshness analytics.