Pricing

from $10.00 / 1,000 results

Robots.txt & AI Crawler Policy Auditor

Parses a site's public robots.txt into structured crawler groups, allow/disallow counts, crawl delays, sitemap declarations, and explicit or inherited policies for major AI crawlers.

Pricing

from $10.00 / 1,000 results

Rating

0.0

(0)

Developer

Sean Burke

Actor stats

Bookmarked

Total users

Monthly active users

4 days ago

Last modified

Robots.txt & AI Crawler Policy Auditor API through CLI

The Apify CLI is the official tool that allows you to use Robots.txt & AI Crawler Policy Auditor locally, providing convenience functions and automatic retries on errors.

Install Apify CLI

Using installation script (macOS/Linux):

$curl -fsSL https://apify.com/install-cli.sh | bash

Using installation script (Windows):

$irm https://apify.com/install-cli.ps1 | iex

Using Homebrew:

$brew install apify-cli

Using npm:

$npm install -g apify-cli

Other API clients include:

Robots.txt & AI Crawler Policy Auditor API in Python

Robots.txt & AI Crawler Policy Auditor API in JavaScript

Robots.txt & AI Crawler Policy Auditor OpenAPI definition

Robots.txt & AI Crawler Policy Auditor API

Robots.txt Analyzer

mahogany_songbird/robots-txt-analyzer

Read robots.txt disallow rules and sitemap declarations.

Britton Furness

Robots Indexability Audit

toronto_777/robots-indexability-audit

Audit public robots.txt, sitemap declarations, homepage robots directives, and crawler allow/block signals for SEO and AI visibility checks.

Steven Feng

Robots.txt Validator - Check Rules, Sitemaps & Crawl Directives

scrappy_garden/robots-txt-validator

Validate robots.txt for one or more websites: fetches /robots.txt per host, parses directive groups (User-agent/Allow/Disallow/Crawl-delay/Sitemap), reports common errors and warnings, and can test URLs against the chosen User-Agent.

Bikram Adhikari

AI Crawler Policy Auditor

junipr/ai-crawler-policy-auditor

Audit robots and AI crawler policy signals for AI crawler access, blocking, and site-readiness.

junipr

AI Crawler Readiness Auditor

nectared_screen/ai-crawler-readiness-auditor

Audit domains for llms.txt, robots.txt AI bot rules, sitemap hints, and public AI crawler readiness signals.

jaehan byun

Robots.txt Auditor

junipr/robots-txt-auditor

Fetch and audit robots.txt syntax, user-agent rules, blocked paths, sitemap declarations, and crawl risks.

junipr

Robots.txt Auditor & Sitemap Finder

andok/robotstxt-auditor

Scan robots.txt files in bulk to extract sitemap URLs and verify crawler directives for technical SEO compliance.

Andok

Public AI Crawler Policy Signal Agent

jacksu/public-ai-crawler-policy-signal-agent

Analyze public robots.txt and llms.txt files for AI crawler allow/block policy evidence, LLM guidance files, stable hashes, and useful-result pricing.

jack su

robots.txt Parser & AI Crawler Block Checker

taroyamada/robotstxt-ai-checker

robots.txt parser that audits AI crawler block rules (GPTBot, ClaudeBot, anthropic-ai, PerplexityBot) across thousands of websites in one run. Returns per-bot allow/disallow disposition and crawl-delay.