Pricing

$1.00 / 1,000 ai crawler policy audit rows

AI Crawler Policy Audit API | 🤖 $1/1K

[💵 $1.00 / 1K] Audit robots.txt and llms.txt to see whether GPTBot, ClaudeBot, PerplexityBot, Google-Extended and other AI crawlers are allowed.

Pricing

$1.00 / 1,000 ai crawler policy audit rows

Rating

0.0

(0)

Developer

WebData Labs

Actor stats

Bookmarked

Total users

Monthly active users

4 days ago

Last modified

AI Crawler Policy Audit API | 🤖 $1/1K API through CLI

The Apify CLI is the official tool that allows you to use AI Crawler Policy Audit API | 🤖 $1/1K locally, providing convenience functions and automatic retries on errors.

Install Apify CLI

Using installation script (macOS/Linux):

$curl -fsSL https://apify.com/install-cli.sh | bash

Using installation script (Windows):

$irm https://apify.com/install-cli.ps1 | iex

Using Homebrew:

$brew install apify-cli

Using npm:

$npm install -g apify-cli

Other API clients include:

AI Crawler Policy Audit API | 🤖 $1/1K API in Python

AI Crawler Policy Audit API | 🤖 $1/1K API in JavaScript

AI Crawler Policy Audit API | 🤖 $1/1K OpenAPI definition

AI Crawler Policy Audit API | 🤖 $1/1K API

AI Crawler Access Checker

q_services/ai-crawler-access-checker

Audit robots.txt policies for GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot and other AI crawlers.

Q Services

AI Crawler Access Checker — robots.txt & llms.txt Audit

zinin/ai-crawler-access-checker

Check which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended & more) can access your website. Bulk audit of robots.txt rules, llms.txt presence and sitemap for AI search visibility (GEO/AEO).

Tim Zinin

AI Crawler Access and llms.txt Visibility Panel

lavender_whistle/ai-visibility-panel

Track which AI crawlers each website blocks or allows (GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, Bytespider and more) plus whether it adopted the llms.txt standard. One clean row per domain from public robots.txt and llms.txt. Zero-config default run, no login, no PII.

Cathal Judge

AI Visibility Site Audit — robots.txt & llms.txt checks

43sunsets/ai-visibility-site-audit

Audit how visible your own website is to AI crawlers and assistants: robots.txt rules for 13 AI bots, llms.txt, structured data, meta hygiene and sitemap — with a scored report and a ready-to-edit llms.txt draft.

Ken Yao

GEO & llms.txt Readiness Auditor

gp005/geo-llms-auditor

Audit which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Googlebot, and 30+ more) your site allows or blocks, validate your llms.txt, and score AI-citability — with an exact prioritized fix list. Non-scraper, reads public well-known files.

Geo

GEO Site Audit - AI Readiness Checker

dltik/geo-site-audit

Audit your website for AI crawler accessibility: robots.txt (GPTBot, ClaudeBot, Perplexity), llms.txt, sitemap, Schema.org, meta tags, content extractability, TTFB. Get an AI-readiness score 0-100 with prioritized recommendations.

Walid

5.0

robots.txt Parser & AI Crawler Block Checker

taroyamada/robotstxt-ai-checker

Audit robots.txt files across thousands of websites to detect specific crawl policies, disallowed paths, and user-agents for GPTBot and ClaudeBot.

naoki anzai

AI Crawler Robots.txt Checker & Audit

davidbenittah/ai-crawler-robots-audit

Audit robots.txt for ChatGPT, Claude and Perplexity crawler access. Get a 0–100 AI search access score, separate search visibility from model-training rules, and copy ready-to-use fixes across up to 50 websites.

David

Ai Visibility Suite - Dark Visitors Alternative

alizarin_refrigerator-owner/ai-visibility-suite---dark-visitors-alternative

Comprehensive AI bot monitoring, robots.txt analysis, LLMs.txt generation & AI shopping optimization. Monitor AI crawlers visits, check AI compliance, generate AI-friendly configurations, and optimize for AI shopping agents. AI Bot Directory Robots.txt LLMs.txt AI Shopping Competitor AI Audit