Pricing

from $10.00 / 1,000 robots policy results

Go to Apify Store

Robots Policy Auditor

Try for free

Audit robots.txt and crawl policy signals for domains before scraping, crawling, SEO checks, or data collection.

Pricing

from $10.00 / 1,000 robots policy results

Rating

0.0

(0)

Developer

Juddroid

Actor stats

Bookmarked

Total users

Monthly active users

10 days ago

Last modified

Robots Policy Auditor API through CLI

The Apify CLI is the official tool that allows you to use Robots Policy Auditor locally, providing convenience functions and automatic retries on errors.

Install Apify CLI

Using installation script (macOS/Linux):

$curl -fsSL https://apify.com/install-cli.sh | bash

Using installation script (Windows):

$irm https://apify.com/install-cli.ps1 | iex

Using Homebrew:

$brew install apify-cli

Using npm:

$npm install -g apify-cli

Other API clients include:

Robots Policy Auditor API in Python

Robots Policy Auditor API in JavaScript

Robots Policy Auditor OpenAPI definition

Robots Policy Auditor API

Robots.txt Policy Auditor

junipr/robots-txt-policy-auditor

Fetch and audit robots.txt files, parse crawl directives, test important paths, and report sitemap and policy risks.

junipr

Sitemap & Robots SEO Index Auditor

glowing_glove/sitemap-robots-index-auditor

Audit robots.txt, sitemap discovery, indexability signals, canonical tags, and SEO crawl readiness for business websites.

Ushba Khan

Robots Indexability Audit

toronto_777/robots-indexability-audit

Audit public robots.txt, sitemap declarations, homepage robots directives, and crawler allow/block signals for SEO and AI visibility checks.

Steven Feng

Robots.txt Auditor

junipr/robots-txt-auditor

Fetch and audit robots.txt syntax, user-agent rules, blocked paths, sitemap declarations, and crawl risks.

junipr

AI Crawler Policy Auditor

junipr/ai-crawler-policy-auditor

Audit robots and AI crawler policy signals for AI crawler access, blocking, and site-readiness.

junipr

Robots.txt Evidence Auditor

gifted_wagon/robots-txt-evidence-auditor

Parse robots.txt rules into normalized policy evidence, sitemap references, tested paths, and truthful failures.

Michael Olmos

Robots Meta X-Robots Auditor

junipr/robots-meta-xrobots-auditor

Audit page-level robots directives from meta tags and X-Robots-Tag headers and detect conflicting indexability signals.

junipr

Indexability Audit

zerobreak/indexability-audit

Indexability audit tool that checks robots.txt, meta robots tags, X-Robots-Tag headers, and canonical URLs for any list of pages, so SEO teams know which ones Google can actually crawl and index.

ZeroBreak

Robots.txt & Sitemap Discovery API

213x/robots-sitemap-discovery

Discover robots.txt rules, sitemap URLs, crawl delays, and basic crawlability signals for domains or website URLs.

llms.txt Auditor & AI Crawler Policy Checker

zinin/llms-txt-auditor

Audit public llms.txt, llms-full.txt, and root robots.txt rules for nine named AI crawlers. Get evidence-backed presence, policy, confidence, gaps, and review actions for each complete domain audit.