Pricing

from $4.90 / 1,000 page auditeds

Try for free

Go to Apify Store

AI Crawler Policy Auditor

Try for free

Audit robots and AI crawler policy signals for AI crawler access, blocking, and site-readiness.

Pricing

from $4.90 / 1,000 page auditeds

Rating

0.0

(0)

Developer

junipr

Actor stats

Bookmarked

Total users

Monthly active users

3 months ago

Last modified

Configure MCP server with AI Crawler Policy Auditor — technical SEO, web, and domain

Add the configuration above to your project's .mcp.json file for Claude Code, then run /mcp in Claude Code to sign in to Apify. It connects to the hosted Apify MCP server with the AI Crawler Policy Auditor — technical SEO, web, and domain Actor already loaded. Clients without OAuth support can authenticate with an Authorization: Bearer header carrying an API token from API & Integrations in Apify Console.

For other MCP clients, get a client-specific configuration with the AI Crawler Policy Auditor — technical SEO, web, and domain Actor preconfigured at https://mcp.apify.com/?tools=fetch-actor-details,junipr/ai-crawler-policy-auditor.

You can connect to the Apify MCP server using an MCP client that supports Streamable HTTP.

If you want to learn more about our Apify MCP implementation, check out our MCP documentation. To learn more about the Model Context Protocol in general, refer to the official MCP documentation or read our blog post.

Website SEO, Accessibility & AI Readiness Auditor

feeng/website-seo-ai-readiness-auditor

🔎 Crawl websites and turn technical SEO, accessibility, security, performance, broken-link, structured-data, robots.txt, llms.txt, and AI crawler signals into prioritized 0–100 audit scores and client-ready reports.

Daniel S.

AI Bot & LLM Crawler Exposure Auditor

glowing_glove/ai-bot-llm-crawler-exposure-auditor

Audit robots.txt, llms.txt, AI crawler directives, public content exposure, and policy gaps for websites in the AI search era.

Ushba Khan

Robots.txt & AI Crawler Policy Auditor

merlin-hermes/robots-ai-crawler-policy-auditor

Parses a site's public robots.txt into structured crawler groups, allow/disallow counts, crawl delays, sitemap declarations, and explicit or inherited policies for major AI crawlers.

Sean Burke

Robots AI Crawler Policy Checker

seeb/robots-ai-crawler-policy-checker

Analyze robots.txt and crawler policy text for AI bot access rules, disallow paths, sitemap references, and blocked model-training crawlers.

Techionik

AI Crawler Access Audit | Robots.txt & User-Agent Checks

a.klimm67/ai-crawler-reality-check

Compare robots.txt policy with live HTTPS responses for 11 named User-Agent strings, including GPTBot and ClaudeBot. Review possible blocks, llms.txt, and page extractability. Tests run from Apify, not verified crawler IPs; results cannot prove actual crawler access.

Andrii Klymenko

Security Headers Auditor

junipr/security-headers-auditor

Audit website security headers such as CSP, HSTS, X-Frame-Options, Referrer-Policy, and Permissions-Policy.

junipr

AI Search Visibility Auditor

junipr/ai-search-visibility-auditor

Audit supplied AI-answer snapshots for brand mentions, domain citations, competitor visibility, sentiment context, and visibility regressions.

junipr

Public AI Crawler Policy Signal Agent

jacksu/public-ai-crawler-policy-signal-agent

Analyze public robots.txt and llms.txt files for AI crawler allow/block policy evidence, LLM guidance files, stable hashes, and useful-result pricing.