Robots.txt and AI Crawler Auditor

Robots.txt and AI Crawler Auditor

This Actor audits a list of domains and tells you, in one table, who is allowed to crawl each site and what the site publishes for crawlers. For every domain it reads only public well-known files: robots.txt, the sitemaps listed there (or /sitemap.xml), llms.txt, /.well-known/security.txt and ads.txt. It does not load any web page content.

The robots.txt rules are evaluated the way RFC 9309 describes: group selection by user-agent token, longest matching rule wins, Allow wins a tie, * and $ wildcards, percent-encoding normalization, and the rules for missing or failing files. You get an allowed or blocked verdict for 15 named bots plus the wildcard agent, tested against a path you choose (default /).

Who is it for

• Publishers and marketers who want to check whether GPTBot, ClaudeBot, PerplexityBot and other AI crawlers can reach their site, or confirm that a block is working.
• SEO and GEO agencies that audit many client domains and need a repeatable, scheduled check.
• Developers and researchers who want machine-readable robots.txt and sitemap facts through the Apify API.

What it checks

Check	Result fields
robots.txt (RFC 9309 parsing)	robots_found, robots_status, robots_mode, rules_per_agent, ai_bots_blocked, ai_bots_allowed, search_bots_blocked
Sitemaps	sitemap_urls_listed, sitemap_count, sitemap_checks, plus url_count, lastmod_min, lastmod_max in deep mode
llms.txt	llms_txt_present, llms_txt_title
security.txt (RFC 9116)	security_txt_present, security_txt_fields
ads.txt	ads_txt_present
Findings and score	issues, score, score_breakdown

Bots tested: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, anthropic-ai, PerplexityBot, Google-Extended, Googlebot, Bingbot, CCBot, Bytespider, Applebot-Extended, Amazonbot, meta-externalagent, and the wildcard agent *. Each entry carries a short purpose label (for example ai-training, ai-search, search-engine). The labels are descriptive and the vendors' own documentation is the authority. Google-Extended and Applebot-Extended are robots.txt control tokens, not separate crawlers. You can add up to 20 more tokens in the input.

How to use it

1. Open the Actor and enter your domains, one per line (a full URL is reduced to its host).
2. Optionally change the test path, for example /blog/post to test one section.
3. Switch on Crawl sitemaps in depth if you want URL counts and lastmod dates.
4. Click Start. The results appear in the Overview table, and the Issues view lists the findings in plain language.

Input

Field	Meaning
domains	Up to 1,000 domains or URLs.
testPath	Path used for every verdict. A query string is allowed. Default /.
deepSitemaps	Follow nested sitemap indexes and count URLs. Default off.
maxSitemapUrls	Stop counting at this many URLs per domain, at most 50,000.
maxSitemapFiles	Read at most this many sitemap files per domain. Default 25.
customUserAgents	Extra user-agent tokens to test.

Example input:

{
    "domains": ["apify.com", "nytimes.com", "bbc.co.uk"],
    "testPath": "/",
    "deepSitemaps": false
}

Output

One record per domain. This abridged record was produced by a real run for apify.com with deep sitemaps on, capped at 2,000 URLs:

{
    "domain": "apify.com",
    "status": "ok",
    "test_path": "/",
    "robots_found": true,
    "robots_status": 200,
    "robots_mode": "parsed",
    "rules_per_agent": [
        { "agent": "GPTBot", "purpose": "ai-training", "verdict": "allowed", "allowed": true, "matched_group": "wildcard", "matched_rule": "Allow: /", "crawl_delay": null }
    ],
    "ai_bots_blocked": [],
    "search_bots_blocked": [],
    "sitemap_urls_listed": ["https://apify.com/sitemap.xml"],
    "sitemap_count": 3,
    "url_count": 2000,
    "url_count_capped": true,
    "llms_txt_present": true,
    "security_txt_present": true,
    "security_txt_fields": { "Contact": ["mailto:***@apify.com"], "Expires": ["2035-01-01T00:00:00.000Z"] },
    "ads_txt_present": false,
    "issues": ["INFO: None of the 13 AI crawlers tested are named in robots.txt, so each one follows the wildcard group (which allows them)."],
    "score": 83
}

A blocked verdict from the same kind of run (nytimes.com, GPTBot) looks like this:

{ "agent": "GPTBot", "purpose": "ai-training", "verdict": "blocked", "allowed": false, "matched_group": "named", "matched_rule": "Disallow: /", "crawl_delay": null }

status is ok, robots_unavailable (server error or refusal), unreachable, invalid_domain, skipped_budget or error. Only ok records are charged.

How robots.txt problems are handled

• 2xx: the file is parsed (the first 512 KiB, RFC 9309 requires at least 500 KiB).
• 404 and 410: treated as "no rules", so everything is allowed. This is reported as a finding.
• 5xx or network failure after retries: RFC 9309 says to assume the whole site is off limits, and the verdicts show that. The record is robots_unavailable and is not charged.
• Any other 4xx (for example 401, 403, 418, 429): the server refused this Actor, often because of bot protection. No verdicts are produced and the domain is not charged.
• An HTML page served at /robots.txt is reported as a soft 404.

Score

The score from 0 to 100 measures hygiene and readiness. It does not judge whether blocking AI crawlers is right for you.

Part	Points
robots.txt present and clean	20
Sitemap declared in robots.txt (12, or 6 if only found at /sitemap.xml) and reachable (8)	20
AI policy clarity: a wildcard group (8) plus the share of the 13 AI bots named explicitly (up to 17)	25
Googlebot and Bingbot not blocked on the test path	15
llms.txt present	10
security.txt present with Contact and a valid Expires	10

Pricing

This Actor uses pay-per-event pricing. Platform usage is included in the event price.

Event	Price	When it is charged
audit-domain	$0.01	Once per domain with a definitive robots.txt result (found, or a 404 or 410). Covers robots.txt verdicts, sitemap availability, llms.txt, security.txt and ads.txt.
sitemap-deep	$0.03	Once per domain, only when Crawl sitemaps in depth is on and at least one sitemap file was read.

You are not charged for invalid domains, unreachable domains, refused or failing robots.txt requests, or domains skipped because your maximum cost per run was reached.

Politeness and data source

The Actor sends at most one request per second to any host and identifies itself in the User-Agent: JetlaunchRobotsAuditor/1.0 (+https://apify.com/jetlaunch/robots-ai-crawl-auditor; contact chris@jetlaunch.net). It retries politely on 429 and 5xx, never solves CAPTCHAs, and does not try to get around blocks. The files it reads are the ones sites publish for automated clients, so it fetches them (and sitemaps the site lists) even if the site's own robots.txt would disallow its user agent. The one exception is the guessed /sitemap.xml, which is skipped when robots.txt disallows it. Only sitemaps on the same site are followed. The data in your results is whatever the audited sites publish, and any rights in it belong to those sites.

Email addresses and phone numbers in Contact lines of security.txt are redacted, so the output contains no individual contact details.

FAQ

Does it check whether a bot really obeys robots.txt? No. It reads what a site publishes, not what any crawler does.

Why is a verdict "allowed" when the site has no robots.txt? Under RFC 9309 a missing file means no restrictions.

Why do the sitemap numbers differ between modes? Without deep mode, sitemap_count is the number of listed sitemaps that responded with a valid sitemap, and url_count is null. With deep mode it counts every file read, including nested ones, and the URL entries inside them. Entries are counted, not de-duplicated.

Does Content-Signal or meta robots matter? Content-Signal lines are reported but not used for verdicts. Meta robots tags and X-Robots-Tag headers are not checked, because that would need page requests.

Can I run it on a schedule or call it from code? Yes, use Apify schedules or the API.

Limitations

• A site's firewall may treat this Actor differently from a real crawler, so a refusal does not prove how Googlebot or GPTBot is treated.
• Verdicts follow RFC 9309. Some crawlers interpret edge cases differently, and some ignore directives such as Crawl-delay.
• Sitemap crawling stops at your limits (default 25 files, at most 50,000 URLs) and sitemaps on other domains are not followed.
• Bot names and labels may change over time as vendors update their crawlers.

This Actor is not affiliated with or endorsed by any of the companies whose crawler names appear in its results.

Support

Found a problem or want another file or bot added? Open a ticket in the Issues tab of this Actor.