AI Crawler Policy Auditor
Pricing
from $4.90 / 1,000 page auditeds
Go to Apify Store
AI Crawler Policy Auditor
Audit robots and AI crawler policy signals for AI crawler access, blocking, and site-readiness.
AI Crawler Policy Auditor
Pricing
from $4.90 / 1,000 page auditeds
Audit robots and AI crawler policy signals for AI crawler access, blocking, and site-readiness.
Robots.txt and page-signal snapshots to audit. Defaults to a local fixture that makes no network calls.
[ { "sourceId": "sample-ai-ready-site", "sourceUrl": "https://example.com", "targetPath": "/", "robotsTxt": "User-agent: *\nAllow: /\n\nUser-agent: GPTBot\nAllow: /\n\nUser-agent: ChatGPT-User\nAllow: /\n\nUser-agent: OAI-SearchBot\nAllow: /\n\nUser-agent: ClaudeBot\nAllow: /\n\nUser-agent: Claude-User\nAllow: /\n\nUser-agent: anthropic-ai\nAllow: /\n\nUser-agent: PerplexityBot\nAllow: /\n\nUser-agent: Google-Extended\nAllow: /\n\nUser-agent: GoogleOther\nAllow: /\n\nUser-agent: CCBot\nAllow: /\n\nUser-agent: Bytespider\nAllow: /\n\nUser-agent: Meta-ExternalAgent\nAllow: /\n\nSitemap: https://example.com/sitemap.xml", "llmsTxt": "# Example AI crawler guidance\n\nThis fixture represents a site that allows major AI crawlers and exposes a sitemap.", "headers": { "x-robots-tag": "index, follow" }, "html": "<!doctype html><html><head><title>Example</title><meta name=\"robots\" content=\"index, follow\"></head><body>Example</body></html>" }]Optional site URLs to fetch only when fetchUrls is true. The actor fetches /robots.txt and /llms.txt with strict byte and timeout caps.
[]Fetch robots.txt and llms.txt over HTTP. Default is false to keep zero-config runs local, deterministic, and cheap.
Use allow for AI crawler readiness checks, block for opt-out audits, or neutral for inventory-only reporting.
Crawler user-agent tokens to evaluate against robots.txt.
[ "GPTBot", "ChatGPT-User", "OAI-SearchBot", "ClaudeBot", "Claude-User", "anthropic-ai", "PerplexityBot", "Google-Extended", "GoogleOther", "CCBot", "Bytespider", "Meta-ExternalAgent", "Googlebot", "Bingbot"]Compare previousRobotsTxt against current robotsTxt when a previous snapshot is supplied.
Flag targets that do not include llms.txt content.
Alias for Max Targets/Maximum Records used by the Junipr actor baseline. When both are provided, actor-specific limits still apply.
Write summary JSON, full result JSON, and Markdown report artifacts to the key-value store.
Maximum robots or URL targets to audit.
HTTP timeout for optional robots.txt and llms.txt fetches.
Maximum bytes to read from a fetched robots.txt file.
Hard local spending cap for actor-start, per-item, and report events. The actor stops before work or withholds uncharged output when the next event would exceed this amount.
Validate input and write a dry-run summary without PPE charges or dataset output.