AI Crawler Access Checker

Enter a domain and see which major AI crawlers its robots.txt allows or blocks.

Use this when

• You need a clear crawler-by-crawler robots.txt policy check.
• You want to verify declared AI crawler access before changing site policy.

What you get

• Allowed and blocked status for known AI crawlers.
• The relevant robots.txt evidence and policy interpretation.

Quick start

1. Enter a public domain.
2. Click Start and inspect the crawler results.

Related Actors

• AI Website Intelligence Scanner — full website assessment.
• llms.txt Website Auditor — AI discovery files.
• Bulk AI Crawler Policy Checker — up to 100 domains.

Check which known AI crawlers a domain's robots.txt policy allows or blocks.

What it checks

• robots.txt rules for named AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, and others)
• Whether robots.txt itself is reachable and parseable

This reports declared policy only, not traffic. It does not measure or infer whether any crawler has crawled the domain — only what the domain's robots.txt currently permits or disallows.

Pricing

This is a paid, pay-per-event Actor: $0.03 per completed check.

Exactly one crawler-access-check event is charged per run, only after a real result exists (success or a useful partial). There is no charge for invalid input, a target blocked by the public-network policy, or a run that fails before a result is produced.

Input

• domain (required): a public domain or HTTPS origin to check, e.g. example.com.

Example input:

{
  "domain": "example.com"
}

Output

The default dataset receives one result envelope per domain; RUN_SUMMARY in the default key-value store records processing, delivery, billing, and dependency totals.

Representative abbreviated output:

{
  "status": "success",
  "target": {"identifier": "example.com"},
  "result": {
    "domain": "example.com",
    "crawlers": {"GPTBot": "blocked", "ClaudeBot": "allowed", "PerplexityBot": "allowed"},
    "robotsUrl": "https://example.com/robots.txt"
  }
}

restricted means API-IFY rejected the target locally before calling Verify. unreachable means the domain could not be reached. dependency_unavailable means Verify was temporarily unavailable.

How it works

This Actor is a thin wrapper around Sentinel Verify's POST /v1/utilities/inspect-ai-crawler-policy endpoint. Each domain is checked locally against API-IFY's public-network policy before being sent to Verify.

Limitations

• Public HTTPS domains only; private, loopback, link-local, metadata, reserved, and CGNAT targets are rejected.
• Reports declared robots.txt policy, not observed crawler activity.
• Audit availability depends on Sentinel Verify.

Privacy and security

The target domain is sent to Sentinel's live Utility API (/v1/utilities/inspect-ai-crawler-policy) for analysis after local public-network validation. Results are stored in the customer's Apify run dataset and summary store. VERIFY_API_KEY is an operator-managed Actor secret — a scoped utilities:read Intelligence API key sent as a bearer token on every request, the same credential mechanism used by any other Utility API consumer; VERIFY_BASE_URL is an operator setting, not an Actor input.

Support

For a reproducible support request, provide the Apify run ID, result itemId, and status. Contact Sentinel Signal Systems through the support link on the Actor page.