robots.txt & llms.txt Checker: AI Crawler Access Audit

Check which AI crawlers each website allows or blocks: GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and 20+ more, parsed from robots.txt with real matching rules. Also detects llms.txt / llms-full.txt, sitemaps and crawl-delay. Bulk, fast, no browser.

Data source: each website's public /robots.txt, /llms.txt and /llms-full.txt files. It needs no login, proxy or browser. Results are returned as clean, flat JSON that is ready for spreadsheets, databases, BI tools and AI agents.

What can you use it for?

• Audit AI-crawler policies across publishers, competitors or clients
• Find sites that block or allow GPTBot, ClaudeBot, Google-Extended and others
• Check llms.txt adoption for GEO/AI-search optimisation
• Collect sitemap URLs and crawl-delay before crawling
• Use it as a tool from AI agents through the Apify MCP server, or schedule it and send new results to Slack, Google Sheets or a webhook

Input

Field	Name	Description
urls	Websites	Domains or URLs (only the origin is used)
testPath	Path to test	Path checked against the rules for every bot
checkLlmsFull	Check llms-full.txt	Also look for /llms-full.txt
includeRobotsTxt	Include raw robots.txt	Add the robots.txt text (up to 50 KB) to each row
concurrency	Concurrency	Sites in parallel

Example input

{
  "urls": [
    "nytimes.com",
    "apify.com",
    "stripe.com",
    "anthropic.com"
  ]
}

Output

Each result is one dataset item with these fields: domain, origin, robotsTxtFound, robotsTxtStatus, robotsTxtUrl, testPath, allAiBotsBlocked, aiBotsBlocked, aiBotsAllowed, aiBlockedCount, wildcardAllowed, googlebotAllowed, bingbotAllowed, aiPolicy, userAgentsNamed, groupCount, sitemaps, crawlDelay, robotsTxtBytes, botRules, llmsTxtFound, llmsTxtStatus, llmsTxtUrl, llmsTxtTitle, llmsTxtSummary, llmsTxtSections, llmsTxtLinkCount, llmsTxtBytes, llmsFullTxtFound, llmsFullTxtBytes, checkedAt.

Example output (real result, long fields shortened)

{
  "domain": "nytimes.com",
  "origin": "https://nytimes.com",
  "robotsTxtFound": true,
  "robotsTxtStatus": 200,
  "robotsTxtUrl": "https://www.nytimes.com/robots.txt",
  "testPath": "/",
  "allAiBotsBlocked": false,
  "aiBotsBlocked": [
    "GPTBot",
    "ChatGPT-User",
    "OAI-SearchBot",
    "ClaudeBot",
    "..."
  ],
  "aiBotsAllowed": [
    "Amazonbot",
    "MistralAI-User"
  ],
  "aiBlockedCount": 22,
  "wildcardAllowed": true,
  "googlebotAllowed": true,
  "bingbotAllowed": true,
  "aiPolicy": "selective",
  "userAgentsNamed": [
    "googlebot",
    "aliyunsecbot",
    "amazonbot",
    "anthropic-ai",
    "..."
  ],
  "groupCount": 63,
  "sitemaps": [
    "https://www.nytimes.com/sitemaps/new/news.xml.gz",
    "https://www.nytimes.com/sitemaps/new/sitemap.xml.gz",
    "https://www.nytimes.com/sitemaps/new/collections.xml.gz",
    "https://www.nytimes.com/sitemaps/new/video.xml.gz",
    "..."
  ],
  "crawlDelay": null,
  "robotsTxtBytes": 8547,
  "botRules": {
    "GPTBot": {
      "company": "OpenAI",
      "allowed": false,
      "matchedBy": "specific",
      "rule": "Disallow: /"
    },
    "ChatGPT-User": {
      "company": "OpenAI",
      "allowed": false,
      "matchedBy": "specific",
      "rule": "Disallow: /"
    },
    "OAI-SearchBot": {
      "company": "OpenAI",
      "allowed": false,
      "matchedBy": "specific",
      "rule": "Disallow: /"
    },
    "ClaudeBot": {
      "company": "Anthropic",
      "allowed": false,
      "matchedBy": "specific",
      "rule": "Disallow: /"
    },
    "...": "..."
  },
  "llmsTxtFound": false,
  "llmsTxtStatus": 0,
  "llmsTxtUrl": "https://nytimes.com/llms.txt",
  "llmsFullTxtFound": false,
  "llmsFullTxtBytes": 0,
  "checkedAt": "2026-11-02T17:38:19.555Z"
}

Download the results as JSON, CSV, Excel or XML, or read them through the Apify API.

Pricing

This Actor uses pay-per-event pricing. You pay $0.001 per website returned, plus standard Apify platform usage, which is small because the Actor uses lightweight HTTP requests (no browser). You are never charged for results that are not returned. To cap spending, set a maximum cost per run in the run options.

Tips

• Start with a small max-results value to check your filters, then scale up.
• Schedule the Actor (for example daily) to monitor changes over time.
• The data comes straight from the source, so its freshness and completeness depend on that publisher.

Disclaimer

This Actor is not affiliated with or endorsed by any website, platform or data publisher named here. Use it in line with the target sites' terms and applicable privacy laws (e.g. GDPR/CAN-SPAM for contact data). Verify critical decisions against the official source.