Pricing

from $10.00 / 1,000 domain auditeds

Try for free

Go to Apify Store

AI Crawler & robots.txt Readiness Audit

Try for free

Checks each domain's robots.txt for 21 vendor-documented AI crawler tokens (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and more): allowed, blocked or not mentioned, with the deciding rule and line. Also reports Sitemap lines, llms.txt presence and Content-Signal lines.

Pricing

from $10.00 / 1,000 domain auditeds

Rating

0.0

(0)

Developer

AI Crawl Ready

Actor stats

Bookmarked

Total users

Monthly active users

6 hours ago

Last modified

Categories

SEO tools

Developer tools

Domains or URLs

domains

Required

Domains (example.com) or URLs (https://www.example.com/page). Each entry is reduced to its origin (scheme + host); bare domains use https. robots.txt and /llms.txt are fetched from that origin. Duplicates (same origin) are skipped. Maximum 1000 entries per run.

Type:array

Min. items:1

Max. items:1000

Bots to check (optional)

bots

Optional

Limit the audit to these AI crawler tokens. Leave empty to check all 21 vendor-documented tokens.

Type:string[]

Default:

[]

Extra paths to check (optional)

checkPaths

Optional

Additional URL paths to evaluate besides / (for example /blog/ or /docs/page?x=1). Max 20. Paths are matched against the rules; they are not fetched.

Type:array

Default:

[]

Include raw robots.txt

includeRawRobots

Optional

Add the fetched robots.txt text (up to 500 KiB) to each dataset item.

Type:boolean

Default:false

Max concurrency

maxConcurrency

Optional

How many domains are audited in parallel. Each domain gets at most two requests (robots.txt, /llms.txt), made one after the other.

Type:integer

Minimum:1

Maximum:20

Default:5

AI Search Readiness Audit (AEO/GEO)

burly_bat/ai-search-readiness-audit

Bulk-audit URLs for AI search readiness (AEO/GEO). Checks 13 AI crawlers against robots.txt with quoted evidence, llms.txt, JSON-LD structured data, no-JS renderability and citation signals. PASS/WARN/FAIL + 0-100 score per URL and a client-ready HTML report. Input: URLs, sitemap or dataset.

Burly Bat

Website AI Readiness Audit: Visitor Experience + AI Search

askmywebsiteai/website-ai-readiness-audit

Scored AI-readiness report per website, in plain English. Checks whether visitors can get answers (chat/FAQ), find contact and pricing, plus speed, SEO, and AI search visibility (GPTBot/ClaudeBot access, llms.txt, Schema.org). For agencies and site owners.

Radu Coman

GEO Site Audit - AI Readiness Checker

dltik/geo-site-audit

Audit your website for AI crawler accessibility: robots.txt (GPTBot, ClaudeBot, Perplexity), llms.txt, sitemap, Schema.org, meta tags, content extractability, TTFB. Get an AI-readiness score 0-100 with prioritized recommendations.

Walid

5.0

(1)

GEO / AEO Website Audit — AI Search Readiness

rtworule/ai-search-readiness-auditor

Audit public websites for GEO and AEO readiness: AI crawler access, robots.txt, llms.txt, sitemaps, Schema.org, social metadata, scores, and prioritized fixes.

Kunteper Koyu

AI Readiness Auditor

rationalistic_counsel/ai-readiness-auditor

Check how AI-ready any website is. Get an AI Readiness Score (0-100) checking llms.txt, robots.txt AI crawler directives, Schema.org structured data, and meta tags. No API key needed.

J N

AI Readiness Checker for Website AEO & GEO

muhammadafzal/ai-readiness-checker

Audit websites for AI crawler access, llms.txt, structured data, answer-ready content, trust signals, scores, and prioritized fixes.

Muhammad Afzal

SEO Audit Tool & AI Search Readiness (GEO) – 0-100 Page Scores

fixandfocus/seo-geo-audit

Audit web pages for technical SEO and AI search readiness (ChatGPT, Perplexity, Google AI Overviews): 0-100 SEO and GEO scores per page, issues by priority, a concrete fix for each, as JSON. Respects robots.txt; pay only for audited pages.

Santiago Alfonso Ramírez

llms.txt Checker: AI Search Readiness Audit

apifmcpfactory/llms-txt-checker

Audits whether a website is readable by AI assistants (llms.txt, robots.txt AI-crawler access, sitemap) and returns a 0-100 score with concrete fixes - use when asked whether a site is visible to ChatGPT/Claude or how to improve that; $0.005 per site audited.

Howth Technology Factory

AI Crawler Readiness Auditor

nectared_screen/ai-crawler-readiness-auditor

Audit domains for llms.txt, robots.txt AI bot rules, sitemap hints, and public AI crawler readiness signals.

jaehan byun

AI Agent Readiness Audit — llms.txt, robots.txt & Schema

paoe/ai-agent-readiness-audit

Check whether AI agents can use a website: robots.txt AI-agent blocks, llms.txt, JSON-LD structured data, sitemap and summarisation signals.

Rashad Flet

Robots Txt Audit

apage/robots-txt-audit

Audit robots.txt files for AI crawler access. Get an AI Readiness Score (0-100), analyze 61+ AI crawlers (ChatGPT, Claude, Perplexity, Gemini), detect syntax errors, security concerns, and get actionable recommendations. Batch audit multiple domains at once with optional subdomain scanning.