AI Crawler Checker - robots.txt Checker for GPTBot & AI Bots

What does AI Crawler Access Checker do?

AI Crawler Access Checker (an AI crawler checker and bulk robots.txt checker) tells you whether GPTBot and 28 other AI crawlers are blocked on a site: give it one website or up to 10,000 domains, and get each bot's status (allowed, partial, blocked or unknown) from the site's robots.txt, an AI search score, a ready-to-paste robots.txt fix and change tracking week to week, for $2 per 1,000 sites. It also reads Cloudflare Content Signals (Content-Signal: search=yes, ai-train=no), and checks for an llms.txt file and noai directives.

How to check if GPTBot is blocked on your site

1. Paste your website (example.com) into websites; add more lines to check many sites at once.
2. Click Start.
3. Find GPTBot in bots (or in blockedBots): allowed, partial, blocked, or unknown when robots.txt could not be read (not charged).
4. To change it, paste the lines from the Fix snippets view into your robots.txt. This checks robots.txt rules only; for a firewall or CDN block, use GEO Readiness Audit (see below).

Why it matters: if AI search crawlers such as OAI-SearchBot, Claude-SearchBot or PerplexityBot are blocked, the site is unlikely to be cited in ChatGPT, Claude or Perplexity answers. Blocking training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) does not have that effect.

Free daily benchmark: see how 1,005 of the world's most visited websites treat these 29 crawlers, by category and for the top 100, in the AI Crawler Index. It is updated every day with the same checks as this Actor, and you can download the full table as CSV.

What it checks

• 🤖 29 AI crawlers from OpenAI, Anthropic, Perplexity, Google, Apple, Meta, Amazon, DuckDuckGo, Mistral, Common Crawl, ByteDance and Cohere, each labelled as training, search, user (fetches made when a person asks an assistant) or ads (ad review, shown for information and not scored). Crawler list last checked against each vendor's documentation: 2026-10-01 (see the list below)
• 🏷️ Content Signals: Cloudflare's Content-Signal line in robots.txt (search, ai-input, ai-train = yes/no) is returned as contentSignals, mentioned in the verdict, and tracked between runs
• 📜 robots.txt rules evaluated like search engines do: the most specific user-agent group, longest matching rule, * and $ wildcards
• 🚦 Per-bot status: allowed, partial (home page allowed but a tested path is blocked, or the bot's own group has Disallow rules), blocked, or unknown
• 🔒 Honest "unknown": if robots.txt answers 401, 403, 418, 429, a bot challenge page or a CDN edge error (e.g. Cloudflare 520-527), the result is policy: "unknown" with scores null, not "open", and the site is not charged
• 📊 Scores and policy class: aiSearchScore (AI search and assistant crawlers allowed, 0-100), aiAccessScore (all AI crawlers), policy: open, search-only, blocks-search, restrictive, no-robots or unknown
• 🛠️ Fix snippets: recommendedRobotsSnippet re-allows the blocked AI search and assistant crawlers (keeping your other rules); suggestedPolicy is a complete template that blocks training crawlers and allows AI search
• 🔁 Change tracking: give the run a monitor name and each site is compared with its previous result (comparison.changedBots, policy, llms.txt); optionally output only changed sites and get a Slack, Discord or JSON webhook
• 🛣️ Any paths you choose (e.g. /blog/, /products/), not only the home page
• 📄 llms.txt and llms-full.txt: present or not (HTML "not found" pages are not counted)
• 🚫 noai / noimageai in the home page's meta robots or X-Robots-Tag header
• 🗺️ Sitemaps listed in robots.txt, and a plain-language verdict for every site

Which tool should I use?

	AI Crawler Access Checker (this Actor)	GEO Readiness Audit (by TidyTools)
Question	"Do these 5,000 domains allow AI crawlers in robots.txt, and what changed since last week?"	"Why is my site not cited by AI answers, and what should I fix first?"
Depth	robots.txt policy only: custom paths, matched rules, fix snippet, change tracking	Live firewall test (catches CDN "block AI bots" rules and challenge pages), content without JavaScript, structured data, score, HTML report
Best for	Researchers, data teams, agencies watching many domains	Site owners and agencies auditing one site or a few competitors
Price	$0.002 per site	about $0.018 per site

For a firewall diagnosis (robots.txt allows a bot but the CDN blocks it), use GEO Readiness Audit.

How much does it cost?

Event	Price
Checked website	$2.00 / 1,000 websites
Re-checked website, unchanged (monitoring)	$0.50 / 1,000 websites

No start fee. Unreachable websites and sites whose robots.txt cannot be read are not charged. Scores, fix snippets and change tracking are included. With a monitor name, a site whose result did not change since the last run is charged at the re-check price ($0.0005); new and changed sites cost $0.002. Example: watching 1,000 domains weekly where 2% change costs about $0.53 per week. With Only output changed sites or Only output sites with a problem, skipped sites are still checked and charged. Higher Apify plans get volume discounts.

For comparison, other AI crawler checkers in the Apify Store charge $0.005 to $0.02 per site, some with a start fee, and some limit a run to 50 or 100 sites (checked September 2026).

Switching from another AI crawler checker

Paste your existing input as it is. This Actor also reads the field names used by other robots.txt and AI crawler checkers in the Apify Store, and the run log says how each one was read:

Their field	Read here as
websites, urls, domains, url, domain, website, site, sites, startUrls ([{"url": ...}]), checkCompetitors	websites to check (merged and deduplicated). List entries may also be objects such as {"domain": "example.com"}
testPaths	paths
concurrency	maxConcurrency
proxyConfig	proxyConfiguration
previousResults	not needed: set a monitorName and each run is compared with the previous one automatically

For example, this input written for another checker runs unchanged:

{
    "domains": ["example.com", "nytimes.com"],
    "checkCompetitors": ["python.org"],
    "maxConcurrency": 10
}

How the two most-used dedicated AI crawler checkers in the Store compare with this one (public Store pricing and descriptions, checked 2026-10-02, Free plan prices):

	This Actor	zinin/ai-crawler-access-checker	apage/robots-txt-audit
Price per 1,000 sites	$2	$20	$5
Start fee	none	$0.005 per run (per GB of memory)	$0.005 per run (per GB of memory)
Unreachable or unreadable sites	free	free	not stated as free (billed per result row)
Sites per run	up to 10,000	up to 100	batch (no limit stated)
AI crawlers checked	29 (search / training / user / ads)	16	60+
Week-to-week change tracking	built in (monitorName), re-checks of unchanged sites $0.50 / 1,000	by pasting a previous run's results	not yet (planned)
Change alerts	Slack, Discord or JSON webhook	via Apify webhooks / automation tools	no
Ready-to-paste robots.txt fix	yes (recommendedRobotsSnippet, suggestedPolicy)	recommended actions	recommendations with snippets
Also reads	llms.txt, llms-full.txt, Content Signals, noai	llms.txt, llms-full.txt	sitemaps, subdomains, syntax and security checks

If you need 60+ crawlers, subdomain scanning or robots.txt syntax linting, the other tools cover more there. If you check many domains or want weekly alerts, this Actor costs less and does the comparison for you.

Control your cost

• Charged: each website whose robots.txt was read ($0.002), or $0.0005 for an unchanged site in a monitor.
• Free: invalid input lines (including an empty input, localhost and private network addresses such as 192.168.x.x, each with its own invalid_input row; the run still ends normally, it never fails), duplicate lines (merged into one site), unreachable websites (including sites that do not exist or are down behind a CDN: robots.txt and the home page both answer a Cloudflare 520-527/530 or 59x edge error), and robots.txt that could not be read (401, 403, 418, 429, bot challenge, CDN edge error). Every row has charged: true or false, and the error text of free rows ends with "(not charged)".
• At the start, the log and status message show the plan: number of websites × price = the most the run can cost, compared with your maximum charge per run (set in the run options).
• When that maximum is reached, the run stops checking new sites. The status message says how many were not checked, and SUMMARY.notProcessed lists them (count and up to 100 inputs) so you can run them again. A monitor webhook sent from such a run carries incomplete: true.
• If Apify restarts the run (server migration or Resurrect), items already finished are skipped and not charged again (SUMMARY.resumedSkipped).
• Time limit per website (Advanced settings, default 120 seconds): a site whose robots.txt, llms.txt and home page take longer in total gets a row with errorType: "timeout" and is not charged.

How to use it

1. Paste websites, one per line (example.com or full URLs). A line with several domains separated by commas or spaces is split. A line that is not a website gets its own error row (errorType: "invalid_input", not charged) and the other sites are still checked. A URL with a path (e.g. nytimes.com/section/world) also tests that path.
2. Optional: add paths to test.
3. Optional, for monitoring: set a monitor name, schedule the Actor (e.g. weekly), and add a webhook URL (Slack and Discord URLs get a formatted message).
4. Click Start and export the results as JSON, CSV or Excel. The Fix snippets view lists the robots.txt lines to paste for each site.

Input example

{
    "websites": ["nytimes.com", "python.org", "stripe.com, lowes.com"],
    "paths": ["/", "/search"],
    "monitorName": "clients-weekly",
    "outputOnlyChanges": false,
    "webhookUrl": "https://hooks.slack.com/services/..."
}

Output example (real result, September 2026, shortened)

{
    "site": "https://nytimes.com",
    "verdict": "Some AI search/assistant crawlers are blocked (OAI-SearchBot, Claude-SearchBot, PerplexityBot, meta-webindexer, DuckAssistBot, ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher): the site may not appear in those AI answers.",
    "policy": "blocks-search",
    "aiSearchScore": 47,
    "aiAccessScore": 35,
    "blockedBots": ["GPTBot", "OAI-SearchBot", "ChatGPT-User", "ClaudeBot", "..."],
    "partialBots": ["Google-GeminiNotebook", "Google-Agent", "Applebot", "Amazonbot", "..."],
    "contentSignals": null,
    "recommendedRobotsSnippet": "# Allow AI search and assistant crawlers (they decide whether the site can appear in AI answers)\n# Also remove \"Disallow: /\" from the existing group(s) for: OAI-SearchBot, ChatGPT-User, ...\nUser-agent: OAI-SearchBot\nUser-agent: ChatGPT-User\n...\nAllow: /\nDisallow: /ads/\n...",
    "suggestedPolicy": "# AI model training crawlers: blocked (does not affect AI search answers)\nUser-agent: GPTBot\n...\nDisallow: /\n\n# AI search and assistant crawlers: allowed\nUser-agent: OAI-SearchBot\n...",
    "robotsTxt": { "status": 200, "found": true, "state": "found", "reason": null, "sitemaps": ["https://www.nytimes.com/sitemaps/new/news.xml.gz", "..."] },
    "llmsTxt": { "found": false, "url": "https://nytimes.com/llms.txt", "status": 404 },
    "noAiDirective": false,
    "comparison": {
        "status": "changed",
        "changedBots": [{ "bot": "MistralAI-User", "before": "allowed", "after": "partial" }],
        "llmsTxtChanged": false,
        "policyChanged": null,
        "previousCheckedAt": "2026-09-29T02:18:46.163Z"
    },
    "bots": [
        { "bot": "GPTBot", "company": "OpenAI", "purpose": "training", "status": "blocked", "allowedAll": false, "matchedGroup": "GPTBot",
          "paths": [{ "path": "/", "allowed": false, "rule": "Disallow: /" }] }
    ],
    "charged": true
}

A site whose robots.txt refuses the request (lowes.com answered HTTP 403) is reported as unknown and not charged:

{ "site": "https://lowes.com", "policy": "unknown", "aiSearchScore": null, "robotsTxt": { "status": 403, "state": "unavailable", "reason": "robots.txt answered HTTP 403" }, "charged": false }

The SUMMARY record in the key-value store has status (SUCCESS, PARTIAL_RESULTS, FAILED, NO_RESULTS or LIMIT_REACHED), counts (including invalidInputs and duplicateInputs), changedSites, the webhook result, costPlan and notProcessed. Unreachable sites get success: false and an errorType.

Every row carries input, your original line, so results map back to your spreadsheet. When several lines point to the same site (e.g. example.com and EXAMPLE.com), the site is checked and charged once and inputs lists all of them. testedPaths lists the paths checked for every bot. A line that is not a website:

{ "input": "acme", "site": null, "success": false, "errorType": "invalid_input", "error": "\"acme\" is not a website: not a URL or domain. A website needs a domain ending; did you mean acme.com? (not charged)", "charged": false }

A line of words that are not domains (e.g. Acme Inc) stays one line and gives one error row; a line of domains separated by spaces (a.com b.com) is split. If no line is a website, the run still ends normally (no crash): SUMMARY.status is FAILED, nothing is charged and the status message says what to enter. A domain that does not exist or a site with a broken certificate gets a free network row that explains the cause (for example "the domain name does not resolve ... check the spelling"), so a typo is easy to find.

A site that publishes Content Signals (real result, www.cloudflare.com, September 2026):

{
    "site": "https://www.cloudflare.com",
    "verdict": "Open to all listed AI crawlers. Content-Signal: search allowed, AI answers (ai-input) allowed, AI training (ai-train) allowed.",
    "policy": "open",
    "contentSignals": { "search": "yes", "aiInput": "yes", "aiTrain": "yes", "userAgent": "*", "raw": "ai-train=yes, search=yes, ai-input=yes" },
    "comparison": { "status": "changed", "contentSignalChanged": { "before": null, "after": "ai-train=yes, search=yes, ai-input=yes" } }
}

contentSignals is null when robots.txt has no Content-Signal line. A missing signal means "no preference" (neither allowed nor refused). When different user-agent groups declare different signals, all of them are listed in contentSignalGroups.

Policy classes

policy	Meaning
open	No listed AI crawler is blocked at the home page
search-only	Only training crawlers are blocked; AI search and assistants are allowed
blocks-search	Some AI search or assistant crawlers are blocked
restrictive	All AI search and assistant crawlers are blocked (or the site's own server answers robots.txt with a 5xx error)
no-robots	No robots.txt: everything is allowed
unknown	robots.txt could not be read (401, 403, 418, 429, a bot challenge or a CDN edge error)

Crawlers checked (last checked 2026-10-01)

Company	Training	AI search	User-triggered	Ads review
OpenAI	GPTBot	OAI-SearchBot	ChatGPT-User*	OAI-AdsBot
Anthropic	ClaudeBot	Claude-SearchBot	Claude-User
Perplexity		PerplexityBot	Perplexity-User*
Google	Google-Extended	Google-CloudVertexBot (Vertex AI Agents, crawls requested by the site owner)	Google-GeminiNotebook*, Google-Agent*
Apple	Applebot-Extended	Applebot (follows Googlebot's rules when not named)
Meta	Meta-ExternalAgent	meta-webindexer	meta-externalfetcher*	meta-externalads
Amazon	Amazonbot	Amzn-SearchBot	Amzn-User*
DuckDuckGo		DuckAssistBot
Mistral	MistralAI-Training	MistralAI-Index	MistralAI-User
Common Crawl	CCBot
ByteDance, Cohere	Bytespider, cohere-ai (no vendor documentation; widely listed in robots.txt files)

Every token was checked against the company's own crawler documentation. * The vendor says this user-triggered fetcher generally does not follow robots.txt; such bots carry ignoresRobots: true in bots, and robots.txt rules for them express your wish rather than a guarantee.

Ads review crawlers (purpose: "ads") check ad landing pages or improve advertising products. They are listed so you can see your rules for them, but they count in no score and no policy class. Amazonbot is labelled training since 2026-10-01: Amazon's page says it "may be used to train Amazon AI models", while Amzn-SearchBot is the crawler for search experiences such as Alexa.

Use with AI agents (MCP)

Connect Apify's MCP server (https://mcp.apify.com?tools=tidytools/ai-crawler-access-checker) to Claude, Cursor or any MCP client, then ask e.g. "Which of these 50 domains block ChatGPT search in robots.txt, and what should they paste to fix it?"

{ "websites": ["example.com", "example.org"] }

Failed items are not charged and carry an errorType.

Use it from code and integrations

Run it from your own code with the Apify API. This call waits for the run and returns the results as JSON (replace YOUR_TOKEN with your Apify API token):

curl -X POST "https://api.apify.com/v2/acts/tidytools~ai-crawler-access-checker/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"websites":["https://www.nytimes.com","python.org","docs.anthropic.com"]}'

The synchronous endpoint waits up to 5 minutes. For bigger runs, start the run with POST https://api.apify.com/v2/acts/tidytools~ai-crawler-access-checker/runs and read the dataset when it finishes, or use the apify-client package for JavaScript or Python.

Schedules and integrations: run it daily or weekly with Apify Schedules, get a webhook when a run finishes, or send the results to Zapier, Make, n8n, Google Sheets, Slack and other apps with Apify integrations. Results can be exported as JSON, CSV, Excel or XML.

FAQ

Does the Actor collect usage data? The Actor sends one anonymous usage counter per run (input type, item count range, outcome) to improve the tool; no URLs, content or account data.

How do I check if a website blocks GPTBot? Paste the domain (or a list of up to 10,000) and run. Each row has bots with a status for GPTBot and the other 28 crawlers (allowed, partial, blocked or unknown), the robots.txt rule that matched, and a plain-language verdict.

How do I block AI bots in robots.txt without losing ChatGPT search traffic? Block the training crawlers and allow the search and assistant crawlers. Every row has this template in suggestedPolicy (with your own sitemaps and * rules kept). Shortened:

# AI model training crawlers: blocked (does not affect AI search answers)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
...
Disallow: /

# AI search and assistant crawlers: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
...
Allow: /

Can I monitor competitors' robots.txt for AI crawler changes? Yes. Put their domains in the list, set a monitor name and schedule the Actor (e.g. weekly). Each run compares every site with its previous result (comparison.changedBots, policy class, Content Signals, llms.txt) and can send a Slack, Discord or JSON webhook when something changed. Unchanged sites cost $0.0005.

Will blocking GPTBot remove my site from ChatGPT answers? No. GPTBot collects training data. ChatGPT search uses OAI-SearchBot, and ChatGPT-User fetches pages when a person asks. You can block GPTBot and still allow the other two; suggestedPolicy does exactly that.

What is Content-Signal? An extension Cloudflare introduced for robots.txt: a line such as Content-Signal: search=yes, ai-input=no, ai-train=no inside a user-agent group states how content may be used after it is fetched: search (search index with links and snippets), ai-input (feeding AI answers, e.g. RAG or grounding) and ai-train (model training). It does not block crawling by itself; it states the site's preference. This Actor returns the signals for all crawlers (*) in contentSignals.

My site has no llms.txt. Should it? It is an emerging proposal, not yet used by the major AI search engines. If you want one, llms.txt Generator (by TidyTools) builds llms.txt and llms-full.txt from your sitemap.

What does "partial" mean? The home page is allowed, but one of your tested paths is blocked, or the bot has its own group with Disallow rules.

I changed the paths to test (or the crawler list was updated). Will every site show as changed? No. Each monitor snapshot stores the tested paths and the crawler list version. When the paths changed, only bots that became blocked or unblocked are reported; when the crawler list changed, new crawlers and the policy class are not compared that time. comparison.note says so, and a site with no other change is charged the re-check price.

robots.txt allows AI search. Can the site still block AI crawlers? Yes, a CDN or firewall rule can block them regardless of robots.txt. The verdict mentions this; GEO Readiness Audit (by TidyTools) runs a live firewall test.

Use cases

• Your own site: confirm AI search crawlers can read it (and training crawlers are handled the way you want), and copy the fix snippet if not
• SEO/GEO agencies: audit client sites in bulk and get a weekly alert when a client's robots.txt starts blocking AI search
• Research: measure how many sites in a list block AI crawlers, and how that changes over time (the AI Crawler Index does this daily for 1,005 popular sites)

Tips

• Advanced settings: add a Proxy to read robots.txt and llms.txt through Apify Proxy (billed to your Apify account) for sites that refuse data-center requests. Plain HTTP requests from controls how the home page is read for the noai check.
• The webhook can be signed: set a webhook signing secret and verify the X-Signature-256: sha256=<hex> header (HMAC-SHA256 of the raw body).

Limitations

• robots.txt is a request, not an enforcement: this Actor reports what the site asks crawlers to do.
• A domain typed without http:// or https:// is read over https; if the site only speaks plain http (connection refused or not a TLS server), robots.txt is read over http and the row says so in note. Certificate errors are not retried over http and count as unreachable (free).
• At most the first 2 MB of robots.txt and llms.txt are read (Google also ignores robots.txt content past 500 KiB).
• Redirects are followed (e.g. http:// to https://, example.com to www.example.com); a redirect into a private network address is refused and the site is reported unreachable.
• Firewalls or bot protection (e.g. blocking by IP or user agent at the CDN) are not detected here; use GEO Readiness Audit for that.
• The crawler list covers the major AI companies (checked 2026-10-01); others may exist.
• Adding crawlers to the list changes scores: a site that names only a few AI crawlers in robots.txt is judged on all of them (29 since 2026-10-01; ad-review crawlers are not scored).

Support

Open an issue in the Issues tab with the website. Issues are checked regularly.

Found this useful? A short review on the Store helps other users find it.