Pricing

from $4.90 / 1,000 robots txt policy auditor robots file checkeds

Go to Apify Store

Robots.txt Policy Auditor

Try for free

Fetch and audit robots.txt files, parse crawl directives, test important paths, and report sitemap and policy risks.

Pricing

from $4.90 / 1,000 robots txt policy auditor robots file checkeds

Rating

0.0

(0)

Developer

junipr

Actor stats

Bookmarked

Total users

Monthly active users

a day ago

Last modified

Categories

SEO tools

Site Origins

siteOrigins

Optional

siteOrigins used by Robots.txt Policy Auditor.

Type:string[]

Default:

[
  "https://www.wikipedia.org"
]

Robots Urls

robotsUrls

Optional

robotsUrls used by Robots.txt Policy Auditor.

Type:string[]

Default:

[]

Raw Robots Txt Inputs

rawRobotsTxtInputs

Optional

rawRobotsTxtInputs used by Robots.txt Policy Auditor.

Type:string[]

Default:

[]

Important Paths

importantPaths

Optional

importantPaths used by Robots.txt Policy Auditor.

Type:string[]

Default:

[
  "/",
  "/admin",
  "/products"
]

User Agents

userAgents

Optional

userAgents used by Robots.txt Policy Auditor.

Type:string[]

Default:

[
  "*"
]

Sitemap Urls

sitemapUrls

Optional

sitemapUrls used by Robots.txt Policy Auditor.

Type:string[]

Default:

[]

Allowed Domains

allowedDomains

Optional

allowedDomains used by Robots.txt Policy Auditor.

Type:string[]

Default:

[]

Test Important Paths

testImportantPaths

Optional

Enable testImportantPaths behavior.

Type:boolean

Default:true

Include Sitemap Discovery

includeSitemapDiscovery

Optional

Enable includeSitemapDiscovery behavior.

Type:boolean

Default:true

Include Unknown Directives

includeUnknownDirectives

Optional

Enable includeUnknownDirectives behavior.

Type:boolean

Default:true

Request Delay Ms

requestDelayMs

Optional

Numeric limit for Robots.txt Policy Auditor.

Type:number

Minimum:0

Maximum:1000000

Default:0

Timeout Ms

timeoutMs

Optional

Numeric limit for Robots.txt Policy Auditor.

Type:number

Minimum:0

Maximum:1000000

Default:10000

Robots.txt & AI Crawler Policy Auditor

merlin-hermes/robots-ai-crawler-policy-auditor

Parses a site's public robots.txt into structured crawler groups, allow/disallow counts, crawl delays, sitemap declarations, and explicit or inherited policies for major AI crawlers.

Sean Burke

Robots Policy Auditor

frank_conifer_af4/robots-policy-auditor

Audit robots.txt and crawl policy signals for domains before scraping, crawling, SEO checks, or data collection.

Juddroid

AI Crawler Policy Auditor

junipr/ai-crawler-policy-auditor

Audit robots and AI crawler policy signals for AI crawler access, blocking, and site-readiness.

junipr

ChatGPT Ads & AI Crawler Policy Auditor

x2ppi/ai-crawler-readiness-auditor

ChatGPT Ads landing-page policy audit: bulk-check path-specific OAI-AdsBot access, AI bot rules, llms.txt, sitemap, and metadata.

Yota

llms.txt Auditor & AI Crawler Policy Checker

zinin/llms-txt-auditor

Audit public llms.txt, llms-full.txt, and root robots.txt rules for nine named AI crawlers. Get evidence-backed presence, policy, confidence, gaps, and review actions for each complete domain audit.

Tim Zinin

AI Crawler Robots.txt Auditor

phoenix2810/ai-crawler-robots-auditor

Fetch a public website's robots.txt and report which AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot, Bytespider, and more) are blocked, allowed, or defaulted. Returns an AI-visibility readiness score with recommendations for GEO and AI-SEO strategy.

Sanskar Jaiswal

GEO / AEO Website Audit — AI Search Readiness

rtworule/ai-search-readiness-auditor

Audit public websites for GEO and AEO readiness: AI crawler access, robots.txt, llms.txt, sitemaps, Schema.org, social metadata, scores, and prioritized fixes.

Kunteper Koyu

RAG Dataset Builder

zinin/sitemap-to-knowledge

Turn buyer-authorized public website sitemaps into bounded, source-linked text chunks for RAG, search, migration, and review. Each delivered chunk includes provenance, hashes, crawl-gap flags, confidence, and billing metadata. Partial and homepage-fallback chunks are explicitly labeled.

Tim Zinin

Federal Website Compliance Auditor — GSA Site Scanning API

malonestar/gsa-site-scanning-auditor

Audit 29,000+ federal .gov websites via the GSA Site Scanning API. Per-agency compliance scorecards (USWDS, DAP, HTTPS/HSTS, IPv6, mobile, required links, Section 508), flat site exports and gap lists of sites missing a technology — with explicit coverage and per-scan status on every row.