AI Crawler Policy Scanner - robots.txt, llms.txt and Bot Rules

What does AI Crawler Policy Scanner do?

Collect source-linked policy text before changing a website’s AI-crawler settings. This Actor finds robots.txt and llms.txt evidence, plus bot-name and AI-policy language in the public responses it checks. Open the Actor to inspect its input and pricing.

Start with one website

1. Paste one public URL into the Input tab, preferably the relevant policy or resource page.
2. Keep evidence enabled and set a run spending limit that covers both event fees below.
3. Run the check, then inspect its sources and errors before expanding the list.

Example input with one public source URL checked on 27 October 2026. Current responses and coverage may differ:

{
  "startUrls": [
    {
      "url": "https://www.cloudflare.com/robots.txt"
    }
  ],
  "maxPagesPerSite": 3,
  "concurrency": 1,
  "requestTimeoutSecs": 10,
  "includeEvidence": true
}

Read the evidence

Review the matched snippets for GPTBot, ClaudeBot or other configured bot names, then read the original policy file. A bot mention or a block-like string does not establish the effective Allow/Disallow rules for a particular path.

Field	How to use it
status, confidence	Signal-detection status and a heuristic score, not a probability of correctness.
extracted.policyFiles	Candidate policy-file locations, response status and a short first line.
findings	Short source snippets when includeEvidence is enabled.
coverage, errors	Request/download limits and problems affecting what was checked.

Export JSON or CSV with retrievedAt and your source notes so a later comparison retains its original context.

Scope and limits

This is not a complete robots-rule interpreter, a crawler-access test or an indexing guarantee. It does not establish whether a model trains on your content.

Checks use static public responses, without browser execution or login. Each site is bounded by 20 seconds, 1 MiB downloaded and 4 MiB decoded text; redirects and failed requests consume maxPagesPerSite attempts. Submit at most 100 combined URL/domain entries, including duplicates. An undetected signal is not proof of absence. For observed Google results rather than policy text, see the AI Overview visibility guide.

Pricing and spending control

Event fees are $0.002 per attempt (website-processed) plus $0.003 per useful report (ai-policy-report): $0.005 when both apply. Check the Pricing tab for the current rates. The report fee requires FOUND or PARTIAL with useful detected signals or extracted evidence. Failed or empty resource probes do not qualify; ordinary NO_SIGNAL/FETCH_FAILED output incurs only the attempt fee.

Budget for both fees before starting a website, and check the Pricing tab for platform terms and plan charges. Review RUN_SUMMARY before repeating billing/output uncertainty, because a retry is a new chargeable check.

Responsible use and support

Use permitted public targets and review their terms. Send a sanitized input through the Issues tab for support or a scope discussion.