Pricing

from $0.40 / 1,000 result delivereds

Try for free

Go to Apify Store

AI Crawler Checker – robots.txt Rules for GPTBot & ClaudeBot

Try for free

AI crawler checker: read each website's robots.txt and see whether GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and 13 other AI crawlers are allowed or blocked, with the line that decided each verdict, plus Content-Signal and llms.txt.

Pricing

from $0.40 / 1,000 result delivereds

Rating

0.0

(0)

Developer

Locomint

Actor stats

Bookmarked

Total users

Monthly active users

a day ago

Last modified

Categories

SEO tools

Developer tools

You can access the AI Crawler Checker – robots.txt Rules for GPTBot & ClaudeBot programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1from apify_client import ApifyClient
2
3# Initialize the ApifyClient with your Apify API token
4# Replace '<YOUR_API_TOKEN>' with your token.
5client = ApifyClient("<YOUR_API_TOKEN>")
6
7# Prepare the Actor input
8run_input = { "domains": [
9        "nytimes.com",
10        "vercel.com",
11        "python.org",
12    ] }
13
14# Run the Actor and wait for it to finish
15run = client.actor("locomint/ai-crawler-checker").call(run_input=run_input)
16
17# Fetch and print Actor results from the run's dataset (if there are any)
18print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
19for item in client.dataset(run.default_dataset_id).iterate_items():
20    print(item)
21
22# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

AI Crawler Checker – Which AI Bots a Website Blocks API in Python

The Apify API client for Python is the official library that allows you to use AI Crawler Checker – robots.txt Rules for GPTBot & ClaudeBot API in Python, providing convenience functions and automatic retries on errors.

Install the apify-client

$pip install apify-client

Other API clients include:

AI Crawler Checker – robots.txt Rules for GPTBot & ClaudeBot API in JavaScript

AI Crawler Checker – robots.txt Rules for GPTBot & ClaudeBot API through CLI

AI Crawler Checker – robots.txt Rules for GPTBot & ClaudeBot OpenAPI definition

AI Crawler Checker – robots.txt Rules for GPTBot & ClaudeBot API

AI Crawler Access Checker

bmiller1009/ai-crawler-access-checker

Check which known AI crawlers a domain's robots.txt policy allows or blocks.

Sentinel Signal

AI Crawler Policy Audit — robots.txt, llms.txt, noai

multiplex/ai-crawl-policy

See exactly which AI crawlers a site allows. Per domain it reads robots.txt for 25 named AI agents (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot…), plus ai.txt, llms.txt, noai meta, X-Robots-Tag and tdmrep.json, and quotes the rule behind every answer. Alerts on policy changes.

Daniel James

AI Crawler Access Checker

zinin/ai-crawler-access-checker

Audit up to 100 sites for 16 AI crawler policies. Get search vs training verdicts, robots.txt and llms.txt evidence, confidence, unknown states, priorities, and next actions for GEO/AEO reviews.

Tim Zinin

5.0

(3)

AI SEO Audit: robots.txt, llms.txt & 26 AI Crawlers

aiqlabs/seo-ai-visibility-auditor

Crawls a site from one starting URL and reports how visible it is to AI: which AI crawlers robots.txt admits, whether llms.txt exists, and what structured data and metadata each page offers. Give it a domain to crawl - for a fixed list of pages, use SEO Audit Tool instead.

Ai-Q Labs

Public AI Crawler Policy Signal Agent

jacksu/public-ai-crawler-policy-signal-agent

Analyze public robots.txt and llms.txt files for AI crawler allow/block policy evidence, LLM guidance files, stable hashes, and useful-result pricing.

jack su

llms.txt Auditor & AI Crawler Policy Checker

zinin/llms-txt-auditor

Audit public llms.txt, llms-full.txt, and root robots.txt rules for nine named AI crawlers. Get evidence-backed presence, policy, confidence, gaps, and review actions for each complete domain audit.

Tim Zinin

AI Readability Checker — Can ChatGPT & Claude Read Your Site?

eliai/ai-readability-checker

Audit a site the way AI crawlers see it: robots.txt rules per bot (GPTBot, ClaudeBot, PerplexityBot), real per-bot fetches, and how much text exists before JavaScript runs. Weighted from crawler logs, not blog advice. $0.05 per audit.

Broke to Built

Robots AI Crawler Policy Checker

seeb/robots-ai-crawler-policy-checker

Analyze robots.txt and crawler policy text for AI bot access rules, disallow paths, sitemap references, and blocked model-training crawlers.

Techionik

robots.txt Parser & AI Crawler Block Checker

taroyamada/robotstxt-ai-checker

Check if sites block GPTBot/ClaudeBot. Paste domains and Start. CSV/JSON.

naoki anzai

GEO & llms.txt Readiness Auditor

gp005/geo-llms-auditor

Audit which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Googlebot, and 30+ more) your site allows or blocks, validate your llms.txt, and score AI-citability — with an exact prioritized fix list. Non-scraper, reads public well-known files.