AI Crawler Robots.txt Auditor

AI Crawler Robots.txt Auditor fetches a public website's robots.txt and reports which AI crawlers are blocked, allowed, or have no explicit rule. It checks ~20 known AI crawler user-agent tokens (GPTBot, OAI-SearchBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot, Bytespider, Meta-ExternalAgent, AppleBot, and more), computes an AI-visibility readiness score, and returns recommendations for GEO and AI-SEO strategy.

Use cases

• SEO and GEO teams deciding which AI products may use their content for training and retrieval.
• AI-SEO consultants auditing client robots.txt files for ChatGPT, Claude, Perplexity, and Google AI Overviews visibility.
• Site migration QA teams confirming robots.txt AI crawler policy did not regress.
• Content licensing teams documenting which AI crawlers a publisher allows or blocks.
• Developer-tool builders adding AI crawler policy checks to monitoring workflows.

Input

Field	Type	Required	Description
startUrl	string	Yes	Public website URL to audit. The actor fetches /robots.txt. HTTP and HTTPS are supported. Private IP ranges and URL credentials are blocked.
timeoutSeconds	integer	No	Timeout for the robots.txt HTTP request. Default: 10. Minimum: 3. Maximum: 30.

Output

Each run pushes one dataset item.

Field	Type	Description
inputUrl	string	The URL provided as input.
siteOrigin	string	Normalized website origin checked by the actor.
robotsUrl	string	URL of robots.txt that was fetched.
robotsFound	boolean	Whether a parseable robots.txt was returned.
status	integer or null	HTTP status of the robots.txt response.
contentType	string	Response content type for robots.txt.
checkedAt	string	ISO timestamp for the audit.
score	integer	AI-visibility readiness score from 0 to 100.
grade	string	Letter grade: A, B, C, D, or F.
findings	array	Structured checks with name, status, and points.
recommendations	array	Plain-language fixes for missing or unclear AI crawler policy.
sitemaps	array	Sitemap URLs declared in robots.txt.
crawlers	array	Per-crawler results: name, operator, userAgent, purpose, matchedToken, status, disallowPaths, allowPaths.
blocked	array	Names of crawlers explicitly blocked.
allowed	array	Names of crawlers explicitly allowed.
noRule	array	Names of crawlers with no explicit rule in robots.txt.
partial	array	Names of crawlers with partial Disallow rules.
error	string or null	Fetch error message, if any.

Per-crawler status values

Status	Meaning
blocked	An explicit User-agent group for this crawler contains Disallow: / (or equivalent root block).
allowed	An explicit User-agent group for this crawler exists and does not block the root.
partial	An explicit User-agent group for this crawler contains non-root Disallow rules.
no-rule	No explicit User-agent group for this crawler was found.

Example input

{
  "startUrl": "https://example.com",
  "timeoutSeconds": 10
}

Example output

{
  "inputUrl": "https://example.com",
  "siteOrigin": "https://example.com",
  "robotsUrl": "https://example.com/robots.txt",
  "robotsFound": true,
  "status": 200,
  "contentType": "text/plain",
  "checkedAt": "2026-08-05T00:00:00.000Z",
  "score": 62,
  "grade": "D",
  "findings": [
    { "name": "robots.txt found", "status": "pass", "points": 20 },
    { "name": "AI crawler coverage", "status": "warn", "points": 42 }
  ],
  "recommendations": [
    "GPTBot is explicitly blocked. Unblock it to allow inclusion in OpenAI AI products if your GEO/AI-SEO strategy favors visibility."
  ],
  "sitemaps": ["https://example.com/sitemap.xml"],
  "crawlers": [
    {
      "name": "GPTBot",
      "operator": "OpenAI",
      "userAgent": "GPTBot",
      "purpose": "Powers ChatGPT training and retrieval.",
      "matchedToken": "GPTBot",
      "status": "blocked",
      "disallowPaths": ["/"],
      "allowPaths": []
    },
    {
      "name": "ClaudeBot",
      "operator": "Anthropic",
      "userAgent": "ClaudeBot",
      "purpose": "Powers Claude model training.",
      "matchedToken": "ClaudeBot",
      "status": "allowed",
      "disallowPaths": [],
      "allowPaths": ["/"]
    }
  ],
  "blocked": ["GPTBot", "Bytespider"],
  "allowed": ["ClaudeBot"],
  "noRule": ["PerplexityBot"],
  "partial": [],
  "error": null
}

Security

The actor only fetches public HTTP and HTTPS URLs. It rejects URL credentials, private IPv4 literals, private IPv6 literals, and hostnames that resolve to private IP ranges. Redirects are followed manually and revalidated before each follow-up request. Response bodies are capped before parsing.

Pricing

Suggested pay-per-event pricing:

Event	Price
Actor start	$0.005
Site audited	$0.01

This keeps single-domain AI crawler policy checks inexpensive while charging proportionally for successful dataset results.

FAQ

Does this actor block or unblock crawlers on my site?

No. It only reads and reports the existing robots.txt policy. It does not modify any files or submit any changes.

Which AI crawlers does it check?

It checks ~20 known AI crawler user-agent tokens including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, anthropic-ai, Google-Extended, Googlebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, Meta-ExternalAgent, AppleBot, Amazonbot, YouBot, Diffbot, Omgilibot, ImagesiftBot, and Timpibot.

Does it require login or credentials?

No. It only fetches the public robots.txt file.

Does it enforce robots.txt semantics for crawling?

No. It is an audit and reporting tool. It surfaces directives so GEO and AI-SEO teams can make explicit policy choices. It is not a robots.txt enforcement engine.

What keep metric should be used?

Keep or improve this actor if it receives organic Store impressions, runs, revenue, or inbound questions within 60 to 90 days. Archive it if there is no discovery and no usage.