Pricing

from $4.90 / 1,000 robots txt policy auditor robots file checkeds

Go to Apify Store

Robots.txt Policy Auditor

Try for free

Fetch and audit robots.txt files, parse crawl directives, test important paths, and report sitemap and policy risks.

Pricing

from $4.90 / 1,000 robots txt policy auditor robots file checkeds

Rating

0.0

(0)

Developer

junipr

Actor stats

Bookmarked

Total users

Monthly active users

a day ago

Last modified

Robots.txt & AI Crawler Policy Auditor

merlin-hermes/robots-ai-crawler-policy-auditor

Parses a site's public robots.txt into structured crawler groups, allow/disallow counts, crawl delays, sitemap declarations, and explicit or inherited policies for major AI crawlers.

Sean Burke

Robots Policy Auditor

frank_conifer_af4/robots-policy-auditor

Audit robots.txt and crawl policy signals for domains before scraping, crawling, SEO checks, or data collection.

Juddroid

AI Crawler Policy Auditor

junipr/ai-crawler-policy-auditor

Audit robots and AI crawler policy signals for AI crawler access, blocking, and site-readiness.

junipr

ChatGPT Ads & AI Crawler Policy Auditor

x2ppi/ai-crawler-readiness-auditor

ChatGPT Ads landing-page policy audit: bulk-check path-specific OAI-AdsBot access, AI bot rules, llms.txt, sitemap, and metadata.

Yota

llms.txt Auditor & AI Crawler Policy Checker

zinin/llms-txt-auditor

Audit public llms.txt, llms-full.txt, and root robots.txt rules for nine named AI crawlers. Get evidence-backed presence, policy, confidence, gaps, and review actions for each complete domain audit.

Tim Zinin

AI Crawler Robots.txt Auditor

phoenix2810/ai-crawler-robots-auditor

Fetch a public website's robots.txt and report which AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot, Bytespider, and more) are blocked, allowed, or defaulted. Returns an AI-visibility readiness score with recommendations for GEO and AI-SEO strategy.

Sanskar Jaiswal

GEO / AEO Website Audit — AI Search Readiness

rtworule/ai-search-readiness-auditor

Audit public websites for GEO and AEO readiness: AI crawler access, robots.txt, llms.txt, sitemaps, Schema.org, social metadata, scores, and prioritized fixes.

Kunteper Koyu

RAG Dataset Builder

zinin/sitemap-to-knowledge

Turn buyer-authorized public website sitemaps into bounded, source-linked text chunks for RAG, search, migration, and review. Each delivered chunk includes provenance, hashes, crawl-gap flags, confidence, and billing metadata. Partial and homepage-fallback chunks are explicitly labeled.

Tim Zinin

Federal Website Compliance Auditor — GSA Site Scanning API

malonestar/gsa-site-scanning-auditor

Audit 29,000+ federal .gov websites via the GSA Site Scanning API. Per-agency compliance scorecards (USWDS, DAP, HTTPS/HSTS, IPv6, mobile, required links, Section 508), flat site exports and gap lists of sites missing a technology — with explicit coverage and per-scan status on every row.