Pricing

from $5.00 / 1,000 site checkeds

Try for free

Go to Apify Store

AI crawler access checker — GPTBot ClaudeBot robots

Try for free

Check whether major AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, OAI-SearchBot, …) are allowed by robots.txt — and flag likely WAF/CDN blocks on sample URLs.

Pricing

from $5.00 / 1,000 site checkeds

Rating

0.0

(0)

Developer

Viktor Kondas

Actor stats

Bookmarked

Total users

Monthly active users

a day ago

Last modified

Categories

SEO tools

Features

You can access the AI crawler access checker — GPTBot ClaudeBot robots programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1from apify_client import ApifyClient
2
3# Initialize the ApifyClient with your Apify API token
4# Replace '<YOUR_API_TOKEN>' with your token.
5client = ApifyClient("<YOUR_API_TOKEN>")
6
7# Prepare the Actor input
8run_input = {
9    "sites": [
10        "anthropic.com",
11        "github.com",
12    ],
13    "bots": [
14        "GPTBot",
15        "OAI-SearchBot",
16        "ChatGPT-User",
17        "OAI-AdsBot",
18        "ClaudeBot",
19        "Claude-SearchBot",
20        "Claude-User",
21        "PerplexityBot",
22        "Perplexity-User",
23        "Google-Extended",
24        "Applebot-Extended",
25        "CCBot",
26        "meta-externalagent",
27        "meta-externalfetcher",
28    ],
29    "samplePaths": ["/"],
30    "maxSites": 10,
31}
32
33# Run the Actor and wait for it to finish
34run = client.actor("kondasviktor/ai-crawler-access-checker").call(run_input=run_input)
35
36# Fetch and print Actor results from the run's dataset (if there are any)
37print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
38for item in client.dataset(run.default_dataset_id).iterate_items():
39    print(item)
40
41# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

AI crawler access checker — GPTBot, ClaudeBot API in Python

The Apify API client for Python is the official library that allows you to use AI crawler access checker — GPTBot ClaudeBot robots API in Python, providing convenience functions and automatic retries on errors.

Install the apify-client

$pip install apify-client

Other API clients include:

AI crawler access checker — GPTBot ClaudeBot robots API in JavaScript

AI crawler access checker — GPTBot ClaudeBot robots API through CLI

AI crawler access checker — GPTBot ClaudeBot robots OpenAPI definition

AI crawler access checker — GPTBot ClaudeBot robots API

AI Crawler Checker – robots.txt Rules for GPTBot & ClaudeBot

locomint/ai-crawler-checker

AI crawler checker: read each website's robots.txt and see whether GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and 13 other AI crawlers are allowed or blocked, with the line that decided each verdict, plus Content-Signal and llms.txt.

Locomint

AI Crawler Access Audit | Robots.txt & User-Agent Checks

a.klimm67/ai-crawler-reality-check

Compare robots.txt policy with live HTTPS responses for 11 named User-Agent strings, including GPTBot and ClaudeBot. Review possible blocks, llms.txt, and page extractability. Tests run from Apify, not verified crawler IPs; results cannot prove actual crawler access.

Andrii Klymenko

AI Readiness Checker - Website Scanner

alizarin_refrigerator-owner/ai-readiness-checker

Analyze any website for AI optimization readiness. Check robots.txt, llms.txt, structured data, meta tags & content quality. Get actionable recommendations to improve AI crawler accessibility.

The Howlers

AI Crawler Checker - robots.txt Checker for GPTBot & AI Bots

tidytools/ai-crawler-access-checker

Check if GPTBot, ClaudeBot, PerplexityBot and 26 more AI crawlers are blocked on your site: bulk robots.txt check of up to 10,000 domains, AI search score, fix snippet, Content Signals, change alerts. $2/1k sites.

Yukai Lin

AI Crawler Checker: GPTBot, ClaudeBot & Cloudflare Blocks

sefran1/ai-crawler-access-audit

Check if ChatGPT, Claude, Perplexity and Gemini crawlers can read your site: robots.txt for 19 AI bots, Cloudflare/WAF blocks, noindex, JavaScript-only pages, llms.txt. GEO audit with fixes. $5 per 1,000 pages.

Iberia Channel Desk

AI Crawler Access Checker

q_services/ai-crawler-access-checker

Audit robots.txt policies for GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot and other AI crawlers.

Q Services

AI Crawler Access Auditor

kingii98/ai-crawler-access-auditor

Audit which AI crawlers can access a site via robots.txt, llms.txt, and sitemap signals for GPTBot, ClaudeBot, PerplexityBot, and more.

kingii98

AI Crawler Access and llms.txt Visibility Panel

lavender_whistle/ai-visibility-panel

Track which AI crawlers each website blocks or allows (GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, Bytespider and more) plus whether it adopted the llms.txt standard. One clean row per domain from public robots.txt and llms.txt. Zero-config default run, no login, no PII.

Cathal Judge

AI-Ready Website Crawler

optimus-fulcria/ai-ready-website-crawler

Crawl websites and convert to clean markdown for AI/RAG, LLM fine-tuning, and document pipelines.

Fulcria Labs

AI Readability Checker: GPTBot, ClaudeBot & llms.txt Audit

eliai/ai-readability-checker

Audits whether AI crawlers can read a site from its URL: robots.txt rules for GPTBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot, llms.txt, the homepage as each bot sees it. Returns a 0-110 score, an A-D grade and per-check findings. For GEO and SEO specialists and agencies.