AI Crawler & Agent Access Checker (robots.txt, llms.txt)
Pricing
from $5.00 / 1,000 website checkeds
Check which AI crawlers and agents a website lets in: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended and more. Detects bot walls (Cloudflare, Akamai), llms.txt, sitemap and JSON-LD. Bulk audits for SEO and GEO. $5 per 1,000 sites.
You can access the AI Crawler & Agent Access Checker (robots.txt, llms.txt) programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.
$echo '{< "shops": [< "ankerkraut.de",< "zalando.de"< ]<}' |<apify call fast_data_asap/ai-crawler-access-checker --silent --output-datasetThe Apify CLI is the official tool that allows you to use AI Crawler & Agent Access Checker (robots.txt, llms.txt) locally, providing convenience functions and automatic retries on errors.
Using installation script (macOS/Linux):
$curl -fsSL https://apify.com/install-cli.sh | bashUsing installation script (Windows):
$irm https://apify.com/install-cli.ps1 | iexUsing Homebrew:
$brew install apify-cliUsing npm:
$npm install -g apify-cliOther API clients include: