Pricing

from $4.90 / 1,000 robots txt policy auditor robots file checkeds

Go to Apify Store

Robots.txt Policy Auditor

Try for free

Fetch and audit robots.txt files, parse crawl directives, test important paths, and report sitemap and policy risks.

Pricing

from $4.90 / 1,000 robots txt policy auditor robots file checkeds

Rating

0.0

(0)

Developer

junipr

Actor stats

Bookmarked

Total users

Monthly active users

a day ago

Last modified

Robots.txt Policy Auditor — technical SEO, web, and domain API through CLI

The Apify CLI is the official tool that allows you to use Robots.txt Policy Auditor locally, providing convenience functions and automatic retries on errors.

Install Apify CLI

Using installation script (macOS/Linux):

$curl -fsSL https://apify.com/install-cli.sh | bash

Using installation script (Windows):

$irm https://apify.com/install-cli.ps1 | iex

Using Homebrew:

$brew install apify-cli

Using npm:

$npm install -g apify-cli

Other API clients include:

Robots.txt Policy Auditor API in Python

Robots.txt Policy Auditor API in JavaScript

Robots.txt Policy Auditor OpenAPI definition

Robots.txt Policy Auditor API

Robots.txt & AI Crawler Policy Auditor

merlin-hermes/robots-ai-crawler-policy-auditor

Parses a site's public robots.txt into structured crawler groups, allow/disallow counts, crawl delays, sitemap declarations, and explicit or inherited policies for major AI crawlers.

Sean Burke

Robots Policy Auditor

frank_conifer_af4/robots-policy-auditor

Audit robots.txt and crawl policy signals for domains before scraping, crawling, SEO checks, or data collection.

Juddroid

AI Crawler Policy Auditor

junipr/ai-crawler-policy-auditor

Audit robots and AI crawler policy signals for AI crawler access, blocking, and site-readiness.

junipr

ChatGPT Ads & AI Crawler Policy Auditor

x2ppi/ai-crawler-readiness-auditor

ChatGPT Ads landing-page policy audit: bulk-check path-specific OAI-AdsBot access, AI bot rules, llms.txt, sitemap, and metadata.

Yota

llms.txt Auditor & AI Crawler Policy Checker

zinin/llms-txt-auditor

Audit public llms.txt, llms-full.txt, and root robots.txt rules for nine named AI crawlers. Get evidence-backed presence, policy, confidence, gaps, and review actions for each complete domain audit.

Tim Zinin

AI Crawler Robots.txt Auditor

phoenix2810/ai-crawler-robots-auditor

Fetch a public website's robots.txt and report which AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot, Bytespider, and more) are blocked, allowed, or defaulted. Returns an AI-visibility readiness score with recommendations for GEO and AI-SEO strategy.

Sanskar Jaiswal

GEO / AEO Website Audit — AI Search Readiness

rtworule/ai-search-readiness-auditor

Audit public websites for GEO and AEO readiness: AI crawler access, robots.txt, llms.txt, sitemaps, Schema.org, social metadata, scores, and prioritized fixes.

Kunteper Koyu

RAG Dataset Builder

zinin/sitemap-to-knowledge

Turn buyer-authorized public website sitemaps into bounded, source-linked text chunks for RAG, search, migration, and review. Each delivered chunk includes provenance, hashes, crawl-gap flags, confidence, and billing metadata. Partial and homepage-fallback chunks are explicitly labeled.

Tim Zinin

Federal Website Compliance Auditor — GSA Site Scanning API

malonestar/gsa-site-scanning-auditor

Audit 29,000+ federal .gov websites via the GSA Site Scanning API. Per-agency compliance scorecards (USWDS, DAP, HTTPS/HSTS, IPv6, mobile, required links, Section 508), flat site exports and gap lists of sites missing a technology — with explicit coverage and per-scan status on every row.