Pricing

from $10.00 / 1,000 domain auditeds

Try for free

Go to Apify Store

AI Crawler & robots.txt Readiness Audit

Try for free

Checks each domain's robots.txt for 21 vendor-documented AI crawler tokens (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and more): allowed, blocked or not mentioned, with the deciding rule and line. Also reports Sitemap lines, llms.txt presence and Content-Signal lines.

Pricing

from $10.00 / 1,000 domain auditeds

Rating

0.0

(0)

Developer

AI Crawl Ready

Actor stats

Bookmarked

Total users

Monthly active users

4 hours ago

Last modified

Categories

SEO tools

Developer tools

You can access the AI Crawler & robots.txt Readiness Audit programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1from apify_client import ApifyClient
2
3# Initialize the ApifyClient with your Apify API token
4# Replace '<YOUR_API_TOKEN>' with your token.
5client = ApifyClient("<YOUR_API_TOKEN>")
6
7# Prepare the Actor input
8run_input = { "domains": [
9        "vercel.com",
10        "anthropic.com",
11    ] }
12
13# Run the Actor and wait for it to finish
14run = client.actor("aicrawlready/ai-crawler-robots-readiness-audit").call(run_input=run_input)
15
16# Fetch and print Actor results from the run's dataset (if there are any)
17print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
18for item in client.dataset(run.default_dataset_id).iterate_items():
19    print(item)
20
21# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

AI Crawler robots.txt Checker: GPTBot, ClaudeBot & more API in Python

The Apify API client for Python is the official library that allows you to use AI Crawler & robots.txt Readiness Audit API in Python, providing convenience functions and automatic retries on errors.

Install the apify-client

$pip install apify-client

Other API clients include:

AI Crawler & robots.txt Readiness Audit API in JavaScript

AI Crawler & robots.txt Readiness Audit API through CLI

AI Crawler & robots.txt Readiness Audit OpenAPI definition

AI Crawler & robots.txt Readiness Audit API

AI Search Readiness Audit (AEO/GEO)

burly_bat/ai-search-readiness-audit

Bulk-audit URLs for AI search readiness (AEO/GEO). Checks 13 AI crawlers against robots.txt with quoted evidence, llms.txt, JSON-LD structured data, no-JS renderability and citation signals. PASS/WARN/FAIL + 0-100 score per URL and a client-ready HTML report. Input: URLs, sitemap or dataset.

Burly Bat

Website AI Readiness Audit: Visitor Experience + AI Search

askmywebsiteai/website-ai-readiness-audit

Scored AI-readiness report per website, in plain English. Checks whether visitors can get answers (chat/FAQ), find contact and pricing, plus speed, SEO, and AI search visibility (GPTBot/ClaudeBot access, llms.txt, Schema.org). For agencies and site owners.

Radu Coman

GEO Site Audit - AI Readiness Checker

dltik/geo-site-audit

Audit your website for AI crawler accessibility: robots.txt (GPTBot, ClaudeBot, Perplexity), llms.txt, sitemap, Schema.org, meta tags, content extractability, TTFB. Get an AI-readiness score 0-100 with prioritized recommendations.

Walid

5.0

(1)

GEO / AEO Website Audit — AI Search Readiness

rtworule/ai-search-readiness-auditor

Audit public websites for GEO and AEO readiness: AI crawler access, robots.txt, llms.txt, sitemaps, Schema.org, social metadata, scores, and prioritized fixes.

Kunteper Koyu

AI Readiness Auditor

rationalistic_counsel/ai-readiness-auditor

Check how AI-ready any website is. Get an AI Readiness Score (0-100) checking llms.txt, robots.txt AI crawler directives, Schema.org structured data, and meta tags. No API key needed.

J N

AI Readiness Checker for Website AEO & GEO

muhammadafzal/ai-readiness-checker

Audit websites for AI crawler access, llms.txt, structured data, answer-ready content, trust signals, scores, and prioritized fixes.

Muhammad Afzal

SEO Audit Tool & AI Search Readiness (GEO) – 0-100 Page Scores

fixandfocus/seo-geo-audit

Audit web pages for technical SEO and AI search readiness (ChatGPT, Perplexity, Google AI Overviews): 0-100 SEO and GEO scores per page, issues by priority, a concrete fix for each, as JSON. Respects robots.txt; pay only for audited pages.

Santiago Alfonso Ramírez

llms.txt Checker: AI Search Readiness Audit

apifmcpfactory/llms-txt-checker

Audits whether a website is readable by AI assistants (llms.txt, robots.txt AI-crawler access, sitemap) and returns a 0-100 score with concrete fixes - use when asked whether a site is visible to ChatGPT/Claude or how to improve that; $0.005 per site audited.

Howth Technology Factory

AI Crawler Readiness Auditor

nectared_screen/ai-crawler-readiness-auditor

Audit domains for llms.txt, robots.txt AI bot rules, sitemap hints, and public AI crawler readiness signals.

jaehan byun

AI Agent Readiness Audit — llms.txt, robots.txt & Schema

paoe/ai-agent-readiness-audit

Check whether AI agents can use a website: robots.txt AI-agent blocks, llms.txt, JSON-LD structured data, sitemap and summarisation signals.

Rashad Flet

Robots Txt Audit

apage/robots-txt-audit

Audit robots.txt files for AI crawler access. Get an AI Readiness Score (0-100), analyze 61+ AI crawlers (ChatGPT, Claude, Perplexity, Gemini), detect syntax errors, security concerns, and get actionable recommendations. Batch audit multiple domains at once with optional subdomain scanning.