AI Search Readiness Checker: can ChatGPT, Perplexity, Claude and Google's AI read your site?

AI Search Readiness Checker audits a list of websites for the technical basics that decide whether AI search engines can fetch and cite them: robots.txt rules for 17 AI and search bots, server-level blocking of AI crawlers, llms.txt, sitemap, structured data and whether your content is in the raw HTML. Each site gets a 0-100 readiness score and a ranked fix list. No browser, no login, no cookies.

• 🤖 17 bots tested, split into two kinds. Search and answer bots (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, Googlebot, Bingbot and more) decide whether you can be cited. Training bots (GPTBot, ClaudeBot, Google-Extended, CCBot and more) are reported but never scored, because blocking them is your choice.
• 🧱 Catches blocks robots.txt cannot show. Optionally requests the page with the published user agents of OAI-SearchBot, PerplexityBot and ClaudeBot and flags a 403 or 429 while a normal browser gets the page (typical of CDN bot rules).
• 🧾 One row per site, plain-English fixes. Every issue says what is wrong and how to fix it.
• 💸 $5 per 1,000 sites (from $0.0035 on higher Apify plans). Sites that cannot be fetched are free.
• 🆓 Free Apify plan: runs are limited to 20 sites per run. Any paid Apify plan removes the limit and adds volume discounts.

Wrenfield Search Data: keyword and AI-search visibility data, priced per result

• Keyword data: Keyword Search Volume · Keyword Difficulty · Keyword Research · Autocomplete Expander · Google Trends · People Also Ask · Competitor Keywords · Keyword Gap
• AI search visibility: AI Search Visibility Tracker · Google AI Overview Scraper · AI Answer Scraper · AI Search Readiness Checker · Agency AI Visibility Reports
• Site and domain SEO: Domain Authority & Traffic · On-Page SEO Checker · Bulk PageSpeed · SEO Toolkit · SEO Agency Report

Sample output

Site	Score	AI search bots blocked in robots.txt	llms.txt	Sitemap	Words in HTML
apify.com	100	none	yes	yes	1,590
stripe.com	100	none	yes	yes	1,992
hubspot.com	98	none	no	yes	2,190
github.com	94	none	yes	yes	1,126
bbc.com	68	OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User	no	yes	2,700
example.com	66	none	no	no	25

What does it check?

Check	What it looks at	Counts toward the score
AI bot access	robots.txt rules for 17 user agents, for the homepage and the page you gave	Yes (search and answer bots only)
Whole-site block	User-agent: * with Disallow: /	Yes, critical
Server blocking	HTTP status when requesting the page as OAI-SearchBot, PerplexityBot and ClaudeBot	Yes
Indexability	noindex in the robots meta tag or the X-Robots-Tag header	Yes, critical
Content in raw HTML	Words of visible text in the first HTML response, script-heavy shells	Yes
Structured data	JSON-LD types, Organization / LocalBusiness / WebSite, sameAs links	Yes
Sitemap	Sitemap: line in robots.txt or /sitemap.xml	Yes
Page basics	Title, meta description, canonical, H1, language, Open Graph	Yes
llms.txt	/llms.txt and /llms-full.txt exist and look valid	Low weight: a proposed convention, no major engine has confirmed it affects citations

Bots tested

User agent	Vendor	Kind	What it does
OAI-SearchBot	OpenAI	Search and answers	Builds the ChatGPT search index
ChatGPT-User	OpenAI	Search and answers	Fetches a page when a ChatGPT user asks about it
PerplexityBot	Perplexity	Search and answers	Builds the Perplexity answer index
Perplexity-User	Perplexity	Search and answers	Fetches a page when a Perplexity user asks about it
Claude-SearchBot	Anthropic	Search and answers	Builds the Claude search index
Claude-User	Anthropic	Search and answers	Fetches a page when a Claude user asks about it
Googlebot	Google	Search and answers	Google Search crawler; also feeds Google AI Overviews and AI Mode
Bingbot	Microsoft	Search and answers	Bing crawler; feeds Bing and Copilot answers
DuckAssistBot	DuckDuckGo	Search and answers	Fetches pages for DuckDuckGo AI answers
GPTBot	OpenAI	Model training	Collects content to train OpenAI models
ClaudeBot	Anthropic	Model training	Collects content to train Anthropic models
Google-Extended	Google	Model training	Control token for Gemini training and grounding; does not affect Google Search
Applebot-Extended	Apple	Model training	Control token for Apple AI training
CCBot	Common Crawl	Model training	Common Crawl dataset used to train many models
Bytespider	ByteDance	Model training	ByteDance crawler
Meta-ExternalAgent	Meta	Model training	Meta AI training crawler
cohere-ai	Cohere	Model training	Cohere crawler

Vendors add and rename bots. The list was reviewed on Oct 27, 2026 against each vendor's published documentation.

How the score works

Every site starts at 100. Each critical issue costs 30 points, high 12, medium 6, low 2. Grades: A 90+, B 80+, C 70+, D 60+, F below. The score is a checklist of fixable technical blockers, not a prediction of rankings or citations.

How to use it

1. Paste websites or pages, one per line, into Websites or pages to check. A domain checks the homepage; a full URL checks that page. robots.txt, llms.txt and the sitemap are always read from the site root.
2. Leave Test server-level blocking on to catch CDN and firewall bot rules.
3. Click Start. About 2-4 seconds per site; 1,000 sites finish in a few minutes.
4. Open the Readiness results table, or read the issues array on each row for the fixes. The run summary lists the most common issues across your list.

For agencies: run a client book or a list of prospects in one go, sort by score, and lead the pitch with the sites that block AI search bots. Pair it with the AI Search Visibility Tracker to show what the engines actually say today.

Input example

{
    "sites": ["apify.com", "https://www.bbc.com/news", "example.com"],
    "testServerBlocking": true
}

Field	Description
sites	Websites or page URLs, one per line (up to 1,000). https:// is added when missing.
testServerBlocking	Default true. Also requests the page as three AI crawlers to detect server-level blocking.

Output example

One row per site (41 fields; real output for bbc.com on Oct 27, 2026, shortened):

{
    "site": "bbc.com",
    "status": "ok",
    "http_status": 200,
    "readiness_score": 68,
    "grade": "D",
    "search_bots_allowed": [
        "Claude-SearchBot",
        "Claude-User",
        "Googlebot",
        "Bingbot",
        "DuckAssistBot"
    ],
    "search_bots_blocked": [
        "OAI-SearchBot",
        "ChatGPT-User",
        "PerplexityBot",
        "Perplexity-User"
    ],
    "training_bots_blocked": [
        "GPTBot",
        "ClaudeBot",
        "Google-Extended",
        "Applebot-Extended",
        "CCBot",
        "Bytespider",
        "Meta-ExternalAgent",
        "cohere-ai"
    ],
    "firewall": {
        "OAI-SearchBot": 200,
        "PerplexityBot": 200,
        "ClaudeBot": 200
    },
    "llms_txt": false,
    "sitemap": true,
    "schema_types": [
        "WebPage",
        "NewsMediaOrganization"
    ],
    "words_in_html": 2700,
    "issues": [
        {
            "id": "search_bots_blocked",
            "severity": "high",
            "issue": "robots.txt blocks AI search bots: OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User",
            "fix": "Allow these user agents if you want the site to be found and cited in AI answers (a blanket Disallow or a copied \"block AI\" template is the usual cause)."
        },
        {
            "id": "no_entity_schema",
            "severity": "medium",
            "issue": "No Organization, LocalBusiness or WebSite structured data",
            "fix": "Add JSON-LD naming the business, its URL, logo and \"sameAs\" profile links so engines can tie the brand to one entity."
        }
    ]
}

Export results as JSON, CSV, Excel or HTML, or read them through the API.

Pricing

Pay per event, no subscription. $0.005 per site checked on the free Apify plan, lower on paid plans:

Apify plan	Per site	Per 1,000 sites
Free	$0.005	$5.00
Starter	$0.0045	$4.50
Scale	$0.004	$4.00
Business	$0.0035	$3.50

Sites that cannot be fetched (DNS errors, blocked requests, timeouts) are never charged. Set a maximum cost per run in the run options and the Actor stops at that limit.

AI search readiness API: REST, Python, JavaScript, n8n and MCP

Run it from code through the Apify API and read the rows back as JSON.

curl -X POST "https://api.apify.com/v2/acts/wrenfield-data~ai-search-readiness-checker/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"sites": ["apify.com", "example.com"]}'

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("wrenfield-data/ai-search-readiness-checker").call(run_input={"sites": ["apify.com", "example.com"]})
for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(row["site"], row["readiness_score"], row["search_bots_blocked"])

It also works in n8n, Make and Zapier through the Apify integrations, and in any MCP client through Apify's hosted MCP server.

FAQ

Does a high score mean ChatGPT or Perplexity will cite my site? No. The score measures technical access and clarity: whether the bots can fetch your pages and read what is on them. Being cited also depends on content quality, authority and what each engine decides for each question. A low score reliably means something blocks you; a high score means nothing technical does.

Does allowing a bot in robots.txt guarantee it can crawl my site? No. A CDN or firewall can still block it. That is why the Actor also requests the page with three AI crawler user agents. Real crawlers may be verified by IP address, so a pass here is not proof and a block here is a strong warning.

Should I block training bots? That is your decision. Blocking training bots (GPTBot, ClaudeBot, Google-Extended and others) does not affect whether search and answer bots can cite you, so the Actor reports them but does not score them. Note that Googlebot also feeds Google AI Overviews and AI Mode, so blocking it removes you from Google Search too.

Is llms.txt required? No. It is a proposed convention for giving AI systems a Markdown map of your key pages. We report whether it exists and weigh it lightly, because no major AI engine has confirmed that it affects citations.

Why was my site reported as an error and not charged? The site refused our request (HTTP 401, 403, 407 or 429), did not answer in time or returned a server error. Those checks are free.

Does it check my private pages or log in anywhere? No. It requests public URLs only, never logs in, and refuses private network addresses.

Is this affiliated with OpenAI, Anthropic, Perplexity, Google or Microsoft? No. All product names and bot names are trademarks of their owners.

Feedback and support

Missing a bot, a check or a field? Open an issue on the Issues tab and include the run ID. We read every issue and reply there.

Acceptable use

• This Actor is an independent tool. It is not affiliated with, endorsed by or sponsored by OpenAI, Anthropic, Perplexity, Google or Microsoft.
• It makes a small number of ordinary GET requests per site (page, robots.txt, llms.txt, sitemap and three test requests). Only check sites you are allowed to test.