AI Crawler Access Checker avatar

AI Crawler Access Checker

Pricing

from $0.50 / 1,000 checked domains

Try for free
Go to Apify Store
AI Crawler Access Checker

AI Crawler Access Checker

Try for free

Audit robots.txt policies for GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot and other AI crawlers.

Pricing

from $0.50 / 1,000 checked domains

Rating

0.0

(0)

Developer

Q Services

Q Services

Maintained by Community

Actor stats

0

Bookmarked

2

Total users

1

Monthly active users

3 days ago

Last modified

Categories

AI

Share

Check whether websites allow, block or partially restrict common AI crawlers in robots.txt.

The Actor checks GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot and Bytespider.

Example result:

{
    "domain": "example.com",
    "status": "mixed_policy",
    "blockedBots": ["GPTBot", "CCBot"],
    "partialBots": ["Google-Extended"],
    "allowedBots": ["PerplexityBot"],
    "robotsTxtUrl": "https://example.com/robots.txt",
    "scrapedAt": "2026-07-09T10:30:00.000Z"
}

How to use it

  1. Paste domains or website URLs.
  2. Start the Actor.
  3. Use the dataset to monitor GEO / AI visibility access policies over time.

Pricing

EventPrice
Actor start$0.005
Checked domain$0.0005

1,000 checked domains cost about $0.505.

Output fields

FieldTypeAlways present?
domainstringYes
statusstringYes
hasRobotsTxtbooleanYes
blockedBotsarrayYes
partialBotsarrayYes
allowedBotsarrayYes
unspecifiedBotsarrayYes
policiesarrayYes
sitemapsarrayYes
scrapedAtISO dateYes

Limitations

  • Reads robots.txt; it does not verify how each AI company actually crawls the site.
  • Uses deterministic robots rules, not AI-generated recommendations.
  • partial means the bot has one or more path-specific restrictions.

You might also like

AC

AI Crawler Access and llms.txt Visibility Panel

lavender_whistle/ai-visibility-panel

Track which AI crawlers each website blocks or allows (GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, Bytespider and more) plus whether it adopted the llms.txt standard. One clean row per domain from public robots.txt and llms.txt. Zero-config default run, no login, no PII.

User avatar

Cathal Judge

2

robots.txt Parser & AI Crawler Block Checker avatar

robots.txt Parser & AI Crawler Block Checker

taroyamada/robotstxt-ai-checker

robots.txt parser that audits AI crawler block rules (GPTBot, ClaudeBot, anthropic-ai, PerplexityBot) across thousands of websites in one run. Returns per-bot allow/disallow disposition and crawl-delay.

User avatar

naoki anzai

4

GEO Site Audit - AI Readiness Checker avatar

GEO Site Audit - AI Readiness Checker

dltik/geo-site-audit

Audit your website for AI crawler accessibility: robots.txt (GPTBot, ClaudeBot, Perplexity), llms.txt, sitemap, Schema.org, meta tags, content extractability, TTFB. Get an AI-readiness score 0-100 with prioritized recommendations.

User avatar

Walid

37

5.0

G&

GEO & llms.txt Readiness Auditor

gp005/geo-llms-auditor

Audit which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Googlebot, and 30+ more) your site allows or blocks, validate your llms.txt, and score AI-citability — with an exact prioritized fix list. Non-scraper, reads public well-known files.

User avatar

Geo

2

LG

📄 llms.txt Generator & Validator — AI Search Visibility

upstanding_biobot/llms-txt-generator

Generate llms.txt files for AI search visibility. Fetches sitemap, validates structure, checks AI bot access (OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended). 2 blue ocean competitors only.

User avatar

Alexander Maksimchuk

2

Robots Txt Audit avatar

Robots Txt Audit

apage/robots-txt-audit

Audit robots.txt files for AI crawler access. Get an AI Readiness Score (0-100), analyze 61+ AI crawlers (ChatGPT, Claude, Perplexity, Gemini), detect syntax errors, security concerns, and get actionable recommendations. Batch audit multiple domains at once with optional subdomain scanning.

User avatar

Andy Page

4

Ai Visibility Suite - Dark Visitors Alternative avatar

Ai Visibility Suite - Dark Visitors Alternative

alizarin_refrigerator-owner/ai-visibility-suite---dark-visitors-alternative

Comprehensive AI bot monitoring, robots.txt analysis, LLMs.txt generation & AI shopping optimization. Monitor AI crawlers visits, check AI compliance, generate AI-friendly configurations, and optimize for AI shopping agents. AI Bot Directory Robots.txt LLMs.txt AI Shopping Competitor AI Audit

User avatar

The Howlers

25

AI Visibility Site Audit — robots.txt & llms.txt checks avatar

AI Visibility Site Audit — robots.txt & llms.txt checks

43sunsets/ai-visibility-site-audit

Audit how visible your own website is to AI crawlers and assistants: robots.txt rules for 13 AI bots, llms.txt, structured data, meta hygiene and sitemap — with a scored report and a ready-to-edit llms.txt draft.

User avatar

Ken Yao

2

AC

AI Crawler Readiness Auditor

nectared_screen/ai-crawler-readiness-auditor

Audit domains for llms.txt, robots.txt AI bot rules, sitemap hints, and public AI crawler readiness signals.

User avatar

jaehan byun

2

RI

Robots Indexability Audit

toronto_777/robots-indexability-audit

Audit public robots.txt, sitemap declarations, homepage robots directives, and crawler allow/block signals for SEO and AI visibility checks.

User avatar

Steven Feng

2