Pricing

$25.00 / 1,000 crawler policies

Robots AI Crawler Policy Checker

Analyze robots.txt and crawler policy text for AI bot access rules, disallow paths, sitemap references, and blocked model-training crawlers.

Pricing

$25.00 / 1,000 crawler policies

Rating

0.0

(0)

Developer

Techionik

Actor stats

Bookmarked

Total users

Monthly active users

3 days ago

Last modified

Categories

SEO tools

You can access the Robots AI Crawler Policy Checker programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1from apify_client import ApifyClient
2
3# Initialize the ApifyClient with your Apify API token
4# Replace '<YOUR_API_TOKEN>' with your token.
5client = ApifyClient("<YOUR_API_TOKEN>")
6
7# Prepare the Actor input
8run_input = { "startUrls": [{ "url": "https://example.com/robots.txt" }] }
9
10# Run the Actor and wait for it to finish
11run = client.actor("seeb/robots-ai-crawler-policy-checker").call(run_input=run_input)
12
13# Fetch and print Actor results from the run's dataset (if there are any)
14print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
15for item in client.dataset(run.default_dataset_id).iterate_items():
16    print(item)
17
18# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

Robots AI Crawler Policy Checker API in Python

The Apify API client for Python is the official library that allows you to use Robots AI Crawler Policy Checker API in Python, providing convenience functions and automatic retries on errors.

Install the apify-client

$pip install apify-client

Other API clients include:

Robots AI Crawler Policy Checker API in JavaScript

Robots AI Crawler Policy Checker API through CLI

Robots AI Crawler Policy Checker OpenAPI definition

Robots AI Crawler Policy Checker API

AI Crawler Policy Audit — robots.txt, llms.txt, noai

multiplex/ai-crawl-policy

See exactly which AI crawlers a site allows. Per domain it reads robots.txt for 25 named AI agents (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot…), plus ai.txt, llms.txt, noai meta, X-Robots-Tag and tdmrep.json, and quotes the rule behind every answer. Alerts on policy changes.

Daniel James

Public AI Crawler Policy Signal Agent

jacksu/public-ai-crawler-policy-signal-agent

Analyze public robots.txt and llms.txt files for AI crawler allow/block policy evidence, LLM guidance files, stable hashes, and useful-result pricing.

jack su

llms.txt Auditor & AI Crawler Policy Checker

zinin/llms-txt-auditor

Audit public llms.txt, llms-full.txt, and root robots.txt rules for nine named AI crawlers. Get evidence-backed presence, policy, confidence, gaps, and review actions for each complete domain audit.

Tim Zinin

Agent Accessibility Auditor llms txt Robots AI Schema Check

mambalabs/agent-accessibility-auditor

Can an AI agent read this site. Checks llms.txt, robots AI crawler rules and the Content Signal policy, structured data, render mode, and machine readable endpoints like sitemap, OpenAPI and well known files. Every field is a fact read off a fetch, never a model opinion.

Mamba Labs

robots.txt Parser & AI Crawler Block Checker

taroyamada/robotstxt-ai-checker

Audit robots.txt files across thousands of websites to detect specific crawl policies, disallowed paths, and user-agents for GPTBot and ClaudeBot.

naoki anzai

AI Tooling Detector Company AI Adoption Signals for Clay

mambalabs/ai-tooling-detector

Detects whether a company declares, deploys, or charges for AI. Returns a flat Clay-ready row with an AI maturity tier, detected AI vendors, llms.txt status, and pricing-page evidence.

Mamba Labs

AI Crawler Access Checker

zinin/ai-crawler-access-checker

Audit up to 100 sites for 16 AI crawler policies. Get search vs training verdicts, robots.txt and llms.txt evidence, confidence, unknown states, priorities, and next actions for GEO/AEO reviews.

Tim Zinin

AI Access Audit (robots.txt + llms.txt)

conserving_mastodon/ai-access-audit

How does a site present itself to AI? Per-crawler robots.txt verdicts for 13 major AI user agents (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot and more), llms.txt presence and shape, sitemap signals, and a concrete fix list. Charged only when the audit completes.

Chris Arsenault

llms.txt Checker: AI Search Readiness Audit

apifmcpfactory/llms-txt-checker

Audits whether a website is readable by AI assistants (llms.txt, robots.txt AI-crawler access, sitemap) and returns a 0-100 score with concrete fixes - use when asked whether a site is visible to ChatGPT/Claude or how to improve that; $0.005 per site audited.

Howth Technology Factory

AI Crawler Robots.txt Audit & llms.txt Checker

davidbenittah/ai-crawler-robots-audit

Check robots.txt and llms.txt on up to 50 sites and see which AI crawlers are blocked: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, GPTBot and ClaudeBot. Returns a 0-100 AI search access score, the rule matched per bot, and a paste-ready robots.txt fix.