Pricing

$11.00 / 1,000 robots.txt ai policy rows

robots.txt Parser & AI Crawler Block Checker

Audit robots.txt files across thousands of websites to detect specific crawl policies, disallowed paths, and user-agents for GPTBot and ClaudeBot.

Pricing

$11.00 / 1,000 robots.txt ai policy rows

Rating

0.0

(0)

Developer

naoki anzai

Actor stats

Bookmarked

Total users

Monthly active users

7 days ago

Last modified

AI Crawler Access Checker

q_services/ai-crawler-access-checker

Audit robots.txt policies for GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot and other AI crawlers.

Q Services

AI Crawler Robots.txt Audit & llms.txt Checker

davidbenittah/ai-crawler-robots-audit

Check robots.txt and llms.txt on up to 50 sites and see which AI crawlers are blocked: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, GPTBot and ClaudeBot. Returns a 0-100 AI search access score, the rule matched per bot, and a paste-ready robots.txt fix.

David

Robots.txt Auditor

junipr/robots-txt-auditor

Fetch and audit robots.txt syntax, user-agent rules, blocked paths, sitemap declarations, and crawl risks.

junipr

Robots.txt Checker & Parser - Crawl Rules API

pink_comic/robots-txt-validator

Check, parse, and validate robots.txt files in bulk. Extract crawl rules, sitemaps, crawl-delay, blocked paths, and per-user-agent allow/disallow results for SEO audits and crawler compliance.

Ava Torres

Robots.txt & Sitemap Extractor by Domain

technicaldost/robots-sitemap-extractor

Fetch and parse robots.txt and XML sitemaps for any domain. Extract allowed and disallowed paths, sitemap URLs and every listed page. Great for SEO audits and crawl planning. JSON output.

Technical Dost Solutions

Robots Indexability Audit

toronto_777/robots-indexability-audit

Audit public robots.txt, sitemap declarations, homepage robots directives, and crawler allow/block signals for SEO and AI visibility checks.

Steven Feng

AI Crawler Policy Audit API | 🤖 $1/1K

webdata_labs/ai-crawler-policy-audit-api

[💵 $1.00 / 1K] Audit robots.txt and llms.txt to see whether GPTBot, ClaudeBot, PerplexityBot, Google-Extended and other AI crawlers are allowed.

WebData Labs

AI Crawler Access Auditor

kingii98/ai-crawler-access-auditor

Deterministically audit which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, and more) a site's robots.txt allows or blocks, plus llms.txt and sitemap declarations.

kingii98

Robots Txt Analyzer

zerobreak/robots-txt-analyzer

Robots txt analyzer that fetches and parses crawl rules from any website in bulk, so SEO teams and developers can audit blocked paths, user agents, and sitemap locations across hundreds of domains without manual work.