Pricing

from $3.00 / 1,000 site checkeds

Try for free

Go to Apify Store

AI Crawler Access Checker

Try for free

See which AI bots a website lets in: GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more, per its robots.txt. Also checks llms.txt and X-Robots-Tag. Bulk, fast, polite.

Pricing

from $3.00 / 1,000 site checkeds

Rating

0.0

(0)

Developer

Helvetic Labs

Actor stats

Bookmarked

Total users

Monthly active users

6 days ago

Last modified

AI Crawler Checker | robots.txt for GPTBot & ClaudeBot API through CLI

The Apify CLI is the official tool that allows you to use AI Crawler Access Checker locally, providing convenience functions and automatic retries on errors.

Install Apify CLI

Using installation script (macOS/Linux):

$curl -fsSL https://apify.com/install-cli.sh | bash

Using installation script (Windows):

$irm https://apify.com/install-cli.ps1 | iex

Using Homebrew:

$brew install apify-cli

Using npm:

$npm install -g apify-cli

Other API clients include:

AI Crawler Access Checker API in Python

AI Crawler Access Checker API in JavaScript

AI Crawler Access Checker OpenAPI definition

AI Crawler Access Checker API

AI Crawler Checker – robots.txt Rules for GPTBot & ClaudeBot

locomint/ai-crawler-checker

AI crawler checker: read each website's robots.txt and see whether GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and 13 other AI crawlers are allowed or blocked, with the line that decided each verdict, plus Content-Signal and llms.txt.

Locomint

Hreflang Tag Checker API | SEO Validator

andok/hreflang-checker

Bulk validate website hreflang tags for international SEO. Audit multi-language sites for missing x-default tags, bidirectional errors, and ISO code compliance.

Andok

HTTP Status Code Checker & Bulk URL Checker API

pink_comic/http-status-code-checker

Bulk URL checker and HTTP status scraper for SEO audits. Check status codes, 301/302 redirect chains, response times, content types, and broken URLs. Use the API for website status checks, migrations, monitoring workflows, and link QA.

Ava Torres

ChatGPT Ads & AI Crawler Policy Auditor

x2ppi/ai-crawler-readiness-auditor

ChatGPT Ads landing-page policy audit: bulk-check path-specific OAI-AdsBot access, AI bot rules, llms.txt, sitemap, and metadata.

Yota

Robots AI Crawler Policy Checker

seeb/robots-ai-crawler-policy-checker

Analyze robots.txt and crawler policy text for AI bot access rules, disallow paths, sitemap references, and blocked model-training crawlers.

Techionik

AI Crawler Access Auditor

kingii98/ai-crawler-access-auditor

Audit which AI crawlers can access a site via robots.txt, llms.txt, and sitemap signals for GPTBot, ClaudeBot, PerplexityBot, and more.

kingii98

llms.txt Checker: AI Search Readiness Audit

apifmcpfactory/llms-txt-checker

Audits whether a website is readable by AI assistants (llms.txt, robots.txt AI-crawler access, sitemap) and returns a 0-100 score with concrete fixes - use when asked whether a site is visible to ChatGPT/Claude or how to improve that; $0.005 per site audited.

Howth Technology Factory

SiteVerdict — robots.txt & Sitemap Monitor

siteverdict/siteverdict-robots-sitemap-monitor

Catch SEO disasters before Google does: watches robots.txt, XML sitemaps and key pages for Disallow: /, noindex, canonical and status changes, sitemap URL drops and errors. Diffs vs last run, Slack/webhook alerts, HTML report.

Aman Thakran

AI Crawler Robots.txt Auditor

phoenix2810/ai-crawler-robots-auditor

Fetch a public website's robots.txt and report which AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot, Bytespider, and more) are blocked, allowed, or defaulted. Returns an AI-visibility readiness score with recommendations for GEO and AI-SEO strategy.