Pricing

from $2.00 / 1,000 domains

Try for free

Go to Apify Store

AI Crawler Checker: GPTBot, robots.txt & llms.txt Policy Audit

Try for free

Check which AI crawlers each website blocks in robots.txt (GPTBot, ClaudeBot, Google-Extended, PerplexityBot), plus llms.txt and TDM opt-out signals.

Pricing

from $2.00 / 1,000 domains

Rating

0.0

(0)

Developer

Ad Astra Connect

Actor stats

Bookmarked

Total users

Monthly active users

3 hours ago

Last modified

AI Crawler Checker: robots.txt, GPTBot & llms.txt Audit API through CLI

The Apify CLI is the official tool that allows you to use AI Crawler Checker: GPTBot, robots.txt & llms.txt Policy Audit locally, providing convenience functions and automatic retries on errors.

Install Apify CLI

Using installation script (macOS/Linux):

$curl -fsSL https://apify.com/install-cli.sh | bash

Using installation script (Windows):

$irm https://apify.com/install-cli.ps1 | iex

Using Homebrew:

$brew install apify-cli

Using npm:

$npm install -g apify-cli

Other API clients include:

AI Crawler Checker: GPTBot, robots.txt & llms.txt Policy Audit API in Python

AI Crawler Checker: GPTBot, robots.txt & llms.txt Policy Audit API in JavaScript

AI Crawler Checker: GPTBot, robots.txt & llms.txt Policy Audit OpenAPI definition

AI Crawler Checker: GPTBot, robots.txt & llms.txt Policy Audit API

AI Crawler Checker – robots.txt Rules for GPTBot & ClaudeBot

locomint/ai-crawler-checker

AI crawler checker: read each website's robots.txt and see whether GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and 13 other AI crawlers are allowed or blocked, with the line that decided each verdict, plus Content-Signal and llms.txt.

Locomint

Dead-Link Watchdog

practicaltools/dead-link-crawler

Automatically monitor your website for broken, redirected, or slow links. Configure which HTTP status codes to flag, schedule recurring scans, and get structured reports — all for just $0.0008 per link. Keep your site clean, fast, and error-free.

Practical Tools

AI Bot & LLM Crawler Exposure Auditor

glowing_glove/ai-bot-llm-crawler-exposure-auditor

Audit robots.txt, llms.txt, AI crawler directives, public content exposure, and policy gaps for websites in the AI search era.

Ushba Khan

AI Crawler Robots.txt Audit & llms.txt Checker

davidbenittah/ai-crawler-robots-audit

Check robots.txt and llms.txt on up to 50 sites and see which AI crawlers are blocked: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, GPTBot and ClaudeBot. Returns a 0-100 AI search access score, the rule matched per bot, and a paste-ready robots.txt fix.

David

AI Readiness Auditor

rationalistic_counsel/ai-readiness-auditor

Check how AI-ready any website is. Get an AI Readiness Score (0-100) checking llms.txt, robots.txt AI crawler directives, Schema.org structured data, and meta tags. No API key needed.

J N

GEO Site Audit - AI Readiness Checker

dltik/geo-site-audit

Audit your website for AI crawler accessibility: robots.txt (GPTBot, ClaudeBot, Perplexity), llms.txt, sitemap, Schema.org, meta tags, content extractability, TTFB. Get an AI-readiness score 0-100 with prioritized recommendations.

Walid

5.0

(1)

ChatGPT Ads & AI Crawler Policy Auditor

x2ppi/ai-crawler-readiness-auditor

ChatGPT Ads landing-page policy audit: bulk-check path-specific OAI-AdsBot access, AI bot rules, llms.txt, sitemap, and metadata.

Yota

AI-Ready Website Crawler

optimus-fulcria/ai-ready-website-crawler

Crawl websites and convert to clean markdown for AI/RAG, LLM fine-tuning, and document pipelines.

Fulcria Labs

Robots.txt & AI Crawler Policy Auditor

merlin-hermes/robots-ai-crawler-policy-auditor

Parses a site's public robots.txt into structured crawler groups, allow/disallow counts, crawl delays, sitemap declarations, and explicit or inherited policies for major AI crawlers.