Robots-Noindex Conflict Checker

Noindex Directive Validator

zerobreak/noindex-directive-validator

Noindex checker that scans URLs for meta robots and X-Robots-Tag headers, so SEO teams can find pages accidentally blocked from indexing before they drop out of search results.

Audit pages for noindex, X-Robots-Tag, canonicals, titles, meta descriptions and H1 signals.

Read robots.txt disallow rules and sitemap declarations.

Audit public robots.txt, sitemap declarations, homepage robots directives, and crawler allow/block signals for SEO and AI visibility checks.

List of website base URLs whose robots.txt files will be validated

SEO Audit: On-Page, Redirects & AI Crawler Access

aiqlabs/seo-audit-tool

Audits pages for on-page and technical SEO, and adds four checks general auditors skip: AI crawler access in robots.txt, llms.txt, redirects that drop the path, and noindex sent via X-Robots-Tag.

Store/Crawler Surface Preflight

checksmithcats/store-crawler-surface-preflight

Cross-check public robots.txt, sitemap, noindex, llms.txt, policy links, and contact signals from a storefront URL. Reads only public URLs; no account access.

AI Crawler Policy Drift Monitor

checksmithcats/ai-crawler-policy-drift-monitor

Tracks changes in public robots.txt, noindex, sitemap, and llms.txt signals against a stored baseline. Public crawler-facing signals only.

Indexability Audit

zerobreak/indexability-audit

Indexability audit tool that checks robots.txt, meta robots tags, X-Robots-Tag headers, and canonical URLs for any list of pages, so SEO teams know which ones Google can actually crawl and index.