Robots AI Crawler Policy Checker

Analyze robots.txt and crawler policy text for AI bot access rules, disallow paths, sitemap references, and blocked model-training crawlers.

Why This Actor Is Useful

This actor audits robots.txt and crawler-policy text for AI search and model-training crawler rules. It is useful for publishers, SEO teams, and AI visibility consultants who need to know which bots can access which paths.

It reads public robots/crawler-policy text and does not bypass private access controls.

What It Extracts

• siteName - Site.
• crawlerName - Crawler.
• policyAction - Action.
• disallowedPath - Disallowed path.
• allowedPath - Allowed path.
• sitemapUrl - Sitemap.
• crawlDelay - Crawl delay.
• aiTrainingSignal - AI training signal.
• riskLevel - Risk level.
• sourceUrl - Source URL.
• evidenceText - Evidence.

Practical Workflows

• Check whether GPTBot, Google-Extended, ClaudeBot, CCBot, or other AI crawlers are blocked.
• Collect sitemap URLs and crawl-delay rules.
• Prepare AI visibility and crawler-access audits.
• Compare policy differences across publisher domains.

Input

Add robots.txt URLs or AI crawler policy pages in Pages to scan. If you already have the robots.txt content, paste it into Paste robots.txt text instead. Keep the default result limit for most crawler-policy checks.

• startUrls - Add robots.txt URLs or public AI crawler policy pages. Example: https://example.com/robots.txt
• sourceText - Optional: paste robots.txt lines or AI crawler policy text that mentions GPTBot, Google-Extended, ClaudeBot, CCBot, or sitemap rules.
• maxResults - Stop after this many useful rows. Most users can leave this at 100.
• requestTimeoutSecs - How long to wait for each URL. Increase only for slow public pages.

Example Input

{
    "sourceText": [
        "ExampleNews robots.txt User-agent: GPTBot Disallow: / User-agent: Google-Extended Disallow: /premium Sitemap: https://examplenews.com/sitemap.xml Crawl-delay: 5",
        "Publisher policy states AI training crawlers are not permitted. User-agent: CCBot Disallow: /archive/"
    ],
    "maxResults": 10,
    "requestTimeoutSecs": 20
}

Example Output

{
    "siteName": "ExampleNews",
    "crawlerName": "GPTBot",
    "policyAction": "Disallow",
    "disallowedPath": "/",
    "allowedPath": null,
    "sitemapUrl": "https://examplenews.com/sitemap.xml",
    "crawlDelay": "5",
    "aiTrainingSignal": null,
    "riskLevel": "disallow: /",
    "sourceUrl": "pasted://source-1",
    "evidenceText": "ExampleNews robots.txt User-agent: GPTBot Disallow: / User-agent: Google-Extended Disallow: /premium Sitemap: https://examplenews.com/sitemap.xml Crawl-delay: 5"
}

Reliability Notes

The actor is intentionally lightweight and fast. It does not try to bypass captcha systems, private pages, paywalls, or login walls. When a site blocks direct fetching, paste the public visible text into sourceText; the same actor-specific parser will still produce structured rows.

Limitations

• The actor extracts only information present in the supplied source.
• Missing source details remain blank instead of being guessed.
• Highly unusual wording may require pasted text or cleaner source snippets for best results.

Output

Rows are written to the default Apify dataset with actor-specific fields and source evidence. The output is designed for spreadsheets, dashboards, client audits, procurement notes, SEO reports, developer research, and other marketplace buyer workflows.