Pricing

from $2.00 / 1,000 domain auditeds

Try for free

Go to Apify Store

AI Crawler Access Auditor (robots.txt + llms.txt)

Try for free

For each domain: which AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot, Bytespider, etc.) robots.txt allows or blocks, whether /llms.txt and /llms-full.txt exist, and sitemap presence. One row per domain.

Pricing

from $2.00 / 1,000 domain auditeds

Rating

0.0

(0)

Developer

Honest Agent Kit

Actor stats

Bookmarked

Total users

Monthly active users

13 days ago

Last modified

AI Crawler Checker - robots.txt GPTBot & llms.txt Audit API through CLI

The Apify CLI is the official tool that allows you to use AI Crawler Access Auditor (robots.txt + llms.txt) locally, providing convenience functions and automatic retries on errors.

Install Apify CLI

Using installation script (macOS/Linux):

$curl -fsSL https://apify.com/install-cli.sh | bash

Using installation script (Windows):

$irm https://apify.com/install-cli.ps1 | iex

Using Homebrew:

$brew install apify-cli

Using npm:

$npm install -g apify-cli

Other API clients include:

AI Crawler Access Auditor (robots.txt + llms.txt) API in Python

AI Crawler Access Auditor (robots.txt + llms.txt) API in JavaScript

AI Crawler Access Auditor (robots.txt + llms.txt) OpenAPI definition

AI Crawler Access Auditor (robots.txt + llms.txt) API

Axe Accessibility Tester

hanamira/axe-accessibility-tester

Test any website for WCAG accessibility issues using axe-core. Scan single pages or crawl entire sites. Get detailed reports with violation severity, affected elements, and remediation guidance. Supports WCAG 2.0, 2.1, and 2.2 at A, AA, and AAA levels.

hannah mira

AI Crawler Checker – robots.txt Rules for GPTBot & ClaudeBot

locomint/ai-crawler-checker

AI crawler checker: read each website's robots.txt and see whether GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and 13 other AI crawlers are allowed or blocked, with the line that decided each verdict, plus Content-Signal and llms.txt.

Locomint

Dead-Link Watchdog

practicaltools/dead-link-crawler

Automatically monitor your website for broken, redirected, or slow links. Configure which HTTP status codes to flag, schedule recurring scans, and get structured reports — all for just $0.0008 per link. Keep your site clean, fast, and error-free.

Practical Tools

llms.txt Generator & LLMO AI Search Readiness Auditor

invaluable_rondeau/llmo-ai-search-readiness-auditor

llms.txt generator and AI search readiness auditor for ChatGPT Search, Perplexity, and Claude. Check /llms.txt, robots.txt AI crawlers, and JSON-LD schema.

PROOFNEXA

AI Crawler Robots.txt Audit & llms.txt Checker

davidbenittah/ai-crawler-robots-audit

Check robots.txt and llms.txt on up to 50 sites and see which AI crawlers are blocked: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, GPTBot and ClaudeBot. Returns a 0-100 AI search access score, the rule matched per bot, and a paste-ready robots.txt fix.

David

AI Crawler Access Checker

q_services/ai-crawler-access-checker

Audit robots.txt policies for GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot and other AI crawlers.

Q Services

AI Crawler Access Auditor

kingii98/ai-crawler-access-auditor

Audit which AI crawlers can access a site via robots.txt, llms.txt, and sitemap signals for GPTBot, ClaudeBot, PerplexityBot, and more.

kingii98

Robots AI Crawler Policy Checker

seeb/robots-ai-crawler-policy-checker

Analyze robots.txt and crawler policy text for AI bot access rules, disallow paths, sitemap references, and blocked model-training crawlers.

Techionik

AI Crawler Access and llms.txt Visibility Panel

lavender_whistle/ai-visibility-panel

Track which AI crawlers each website blocks or allows (GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, Bytespider and more) plus whether it adopted the llms.txt standard. One clean row per domain from public robots.txt and llms.txt. Zero-config default run, no login, no PII.