AI Crawler Policy Audit API

Audit any website's robots.txt and llms.txt to understand whether major AI crawlers are allowed, blocked, or not explicitly configured. Turn a manual policy review into a structured dataset for AI governance, SEO, legal operations, content licensing, and competitive research.

The Actor checks GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and CCBot and returns one clean row per domain.

🤖 What you get

• AI crawler access status for five major bots
• robots.txt and llms.txt HTTP status codes
• Direct URLs to both policy files
• A clear allowed, blocked, not_specified, or unknown classification
• A short recommendation for every audited domain
• Explicit error rows for unreachable websites instead of silently missing results

Results are ready to download as JSON, CSV, Excel, XML, or RSS and can be consumed through the Apify API.

🎯 Common use cases

• Audit your own website before changing AI crawler policy
• Compare how competitors treat generative-AI crawlers
• Monitor a portfolio of media, ecommerce, SaaS, or publisher domains
• Detect policy changes with scheduled recurring runs
• Build an evidence trail for AI-governance and compliance reviews
• Enrich an SEO or domain-intelligence dataset with crawler-access metadata

🚀 Ready-to-run examples

• Check AI crawler access for your website — inspect GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and CCBot policy for a single domain.
• Audit competitor AI crawler policies — compare crawler rules across OpenAI, Anthropic, and Google.
• Monitor AI bot robots.txt policies — create a reusable portfolio check for scheduled monitoring.

⚙️ Input

Provide domains, full URLs, or a mixture of both. The Actor normalizes and deduplicates them automatically.

{
  "domains": [
    "openai.com",
    "https://www.anthropic.com",
    "perplexity.ai"
  ],
  "maxDomains": 100,
  "timeoutSecs": 15
}

Input fields

Field	Description	Default
domains	Domains or URLs to audit	Required
maxDomains	Maximum unique domains processed in one run, up to 1,000	100
timeoutSecs	Maximum wait for each policy-file request	15

📤 Output

{
  "domain": "example.com",
  "robotsUrl": "https://example.com/robots.txt",
  "robotsStatus": 200,
  "llmsTxtUrl": "https://example.com/llms.txt",
  "llmsTxtStatus": 404,
  "llmsTxtFound": false,
  "gptbot": "allowed",
  "claudebot": "not_specified",
  "perplexitybot": "blocked",
  "googleExtended": "allowed",
  "commonCrawl": "allowed",
  "recommendation": "Review blocked AI crawler policies."
}

Policy values

Value	Meaning
blocked	The applicable user-agent group contains an all-site Disallow: / rule
allowed	An applicable group exists without an all-site block
not_specified	No matching bot-specific or wildcard group was found
unknown	The policy file could not be retrieved or interpreted

This is a technical policy check, not legal advice. It reports published crawler directives; it does not determine whether a crawler actually obeyed them.

🔌 API usage

Start a synchronous run and receive dataset rows directly:

curl "https://api.apify.com/v2/acts/webdata_labs~ai-crawler-policy-audit-api/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{"domains":["openai.com","anthropic.com"],"maxDomains":100,"timeoutSecs":15}'

You can also schedule recurring audits, trigger runs from Make or Zapier, receive webhooks, or use the official Apify JavaScript and Python clients.

💵 Pricing

The Actor costs $1 per 1,000 audited domains ($0.001 per exported row). You pay for delivered audit rows; Apify platform usage is included in the event price shown on the Actor page.

🌍 Coverage and limitations

• The Actor audits publicly accessible robots.txt and llms.txt files.
• Websites can temporarily fail, redirect unexpectedly, or block automated requests; these cases are returned with an error or non-200 status.
• A missing llms.txt is common and is not automatically a problem.
• Policy classifications focus on all-site blocking rules and are intentionally conservative.
• Results can change whenever a website updates its crawler policy, so schedule recurring runs when change detection matters.

❓ FAQ

Does this prove that an AI company used my content?
No. It only audits the crawler directives currently published by the website.

Is llms.txt the same as robots.txt?
No. robots.txt provides crawler directives. llms.txt is an emerging convention for presenting site information to language models and tools.

Can I audit domains without typing https://?
Yes. Plain domains and full URLs are both accepted and normalized.

Can I monitor policy changes?
Yes. Save your portfolio as an Apify Task, run it on a schedule, and compare datasets or send results through a webhook.

🛠️ Support

If a policy format is classified incorrectly or a domain cannot be fetched, open an issue on the Actor page and include the run URL, affected domain, and expected result.