Pricing

from $5.00 / 1,000 site checkeds

Try for free

Go to Apify Store

AI crawler access checker — GPTBot ClaudeBot robots

Try for free

Check whether major AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, OAI-SearchBot, …) are allowed by robots.txt — and flag likely WAF/CDN blocks on sample URLs.

Pricing

from $5.00 / 1,000 site checkeds

Rating

0.0

(0)

Developer

Viktor Kondas

Actor stats

Bookmarked

Total users

Monthly active users

a day ago

Last modified

Check ChatGPT search, user-fetch and ads bot access

Monitor robots.txt AI crawler policy changes weekly

Benchmark AI crawler opt-out policies across sites

Audit AI search crawler access for GEO clients

Check if GPTBot and ClaudeBot can crawl your site

AI Crawler Checker – robots.txt Rules for GPTBot & ClaudeBot

locomint/ai-crawler-checker

AI crawler checker: read each website's robots.txt and see whether GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and 13 other AI crawlers are allowed or blocked, with the line that decided each verdict, plus Content-Signal and llms.txt.

Locomint

AI Crawler Access Audit | Robots.txt & User-Agent Checks

a.klimm67/ai-crawler-reality-check

Compare robots.txt policy with live HTTPS responses for 11 named User-Agent strings, including GPTBot and ClaudeBot. Review possible blocks, llms.txt, and page extractability. Tests run from Apify, not verified crawler IPs; results cannot prove actual crawler access.

Andrii Klymenko

AI Readiness Checker - Website Scanner

alizarin_refrigerator-owner/ai-readiness-checker

Analyze any website for AI optimization readiness. Check robots.txt, llms.txt, structured data, meta tags & content quality. Get actionable recommendations to improve AI crawler accessibility.

The Howlers

AI Crawler Checker - robots.txt Checker for GPTBot & AI Bots

tidytools/ai-crawler-access-checker

Check if GPTBot, ClaudeBot, PerplexityBot and 26 more AI crawlers are blocked on your site: bulk robots.txt check of up to 10,000 domains, AI search score, fix snippet, Content Signals, change alerts. $2/1k sites.

Yukai Lin

AI Crawler Checker: GPTBot, ClaudeBot & Cloudflare Blocks

sefran1/ai-crawler-access-audit

Check if ChatGPT, Claude, Perplexity and Gemini crawlers can read your site: robots.txt for 19 AI bots, Cloudflare/WAF blocks, noindex, JavaScript-only pages, llms.txt. GEO audit with fixes. $5 per 1,000 pages.

Iberia Channel Desk

AI Crawler Access Checker

q_services/ai-crawler-access-checker

Audit robots.txt policies for GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot and other AI crawlers.

Q Services

AI Crawler Access Auditor

kingii98/ai-crawler-access-auditor

Audit which AI crawlers can access a site via robots.txt, llms.txt, and sitemap signals for GPTBot, ClaudeBot, PerplexityBot, and more.

kingii98

AI Crawler Access and llms.txt Visibility Panel

lavender_whistle/ai-visibility-panel

Track which AI crawlers each website blocks or allows (GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, Bytespider and more) plus whether it adopted the llms.txt standard. One clean row per domain from public robots.txt and llms.txt. Zero-config default run, no login, no PII.

Cathal Judge

AI-Ready Website Crawler

optimus-fulcria/ai-ready-website-crawler

Crawl websites and convert to clean markdown for AI/RAG, LLM fine-tuning, and document pipelines.

Fulcria Labs

AI Readability Checker: GPTBot, ClaudeBot & llms.txt Audit

eliai/ai-readability-checker

Audits whether AI crawlers can read a site from its URL: robots.txt rules for GPTBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot, llms.txt, the homepage as each bot sees it. Returns a 0-110 score, an A-D grade and per-check findings. For GEO and SEO specialists and agencies.