Pricing

from $2.00 / 1,000 successful lookups

Try for free

Go to Apify Store

AI Crawler Policy Checker: robots.txt & llms.txt

Try for free

Check what a website's robots.txt says about AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, and more) and whether it publishes an llms.txt. Reads the site's own public files directly: no API key, no third-party service. Charged only when an explicit AI policy or llms.txt is found.

Pricing

from $2.00 / 1,000 successful lookups

Rating

0.0

(0)

Developer

Adrian Voss

Actor stats

Bookmarked

Total users

Monthly active users

a day ago

Last modified

Robots.txt Checker: AI Crawler & llms.txt Policy API through CLI

The Apify CLI is the official tool that allows you to use AI Crawler Policy Checker: robots.txt & llms.txt locally, providing convenience functions and automatic retries on errors.

Install Apify CLI

Using installation script (macOS/Linux):

$curl -fsSL https://apify.com/install-cli.sh | bash

Using installation script (Windows):

$irm https://apify.com/install-cli.ps1 | iex

Using Homebrew:

$brew install apify-cli

Using npm:

$npm install -g apify-cli

Other API clients include:

AI Crawler Policy Checker: robots.txt & llms.txt API in Python

AI Crawler Policy Checker: robots.txt & llms.txt API in JavaScript

AI Crawler Policy Checker: robots.txt & llms.txt OpenAPI definition

AI Crawler Policy Checker: robots.txt & llms.txt API

AI Crawler Policy Audit — robots.txt, llms.txt, noai

multiplex/ai-crawl-policy

See exactly which AI crawlers a site allows. Per domain it reads robots.txt for 25 named AI agents (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot…), plus ai.txt, llms.txt, noai meta, X-Robots-Tag and tdmrep.json, and quotes the rule behind every answer. Alerts on policy changes.

Daniel James

AI Crawler Access and llms.txt Visibility Panel

lavender_whistle/ai-visibility-panel

Track which AI crawlers each website blocks or allows (GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, Bytespider and more) plus whether it adopted the llms.txt standard. One clean row per domain from public robots.txt and llms.txt. Zero-config default run, no login, no PII.

Cathal Judge

AI Crawler Checker & llms.txt Generator — Website AI Audit

dev-hoss/ai-search-readiness-audit

Check if AI search engines can crawl and cite your website: robots.txt rules for GPTBot, ClaudeBot, PerplexityBot and 17 more AI crawlers, llms.txt validation, sitemap, schema.org, meta tags and content extractability. 0-100 score, prioritized fixes, ready llms.txt.

Hossam Mohamed

AI Access Audit (robots.txt + llms.txt)

conserving_mastodon/ai-access-audit

How does a site present itself to AI? Per-crawler robots.txt verdicts for 13 major AI user agents (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot and more), llms.txt presence and shape, sitemap signals, and a concrete fix list. Charged only when the audit completes.

Chris Arsenault

llms.txt Auditor & AI Crawler Policy Checker

zinin/llms-txt-auditor

Audit public llms.txt, llms-full.txt, and root robots.txt rules for nine named AI crawlers. Get evidence-backed presence, policy, confidence, gaps, and review actions for each complete domain audit.

Tim Zinin

AI Crawler Robots.txt Audit & llms.txt Checker

davidbenittah/ai-crawler-robots-audit

Check robots.txt and llms.txt on up to 50 sites and see which AI crawlers are blocked: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, GPTBot and ClaudeBot. Returns a 0-100 AI search access score, the rule matched per bot, and a paste-ready robots.txt fix.

David

AI Crawler Policy Audit API | 🤖 $1/1K

webdata_labs/ai-crawler-policy-audit-api

[💵 $1.00 / 1K] Audit robots.txt and llms.txt to see whether GPTBot, ClaudeBot, PerplexityBot, Google-Extended and other AI crawlers are allowed.

WebData Labs

AI Crawler Checker – robots.txt Rules for GPTBot & ClaudeBot

locomint/ai-crawler-checker

AI crawler checker: read each website's robots.txt and see whether GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and 13 other AI crawlers are allowed or blocked, with the line that decided each verdict, plus Content-Signal and llms.txt.