Robots.txt, Sitemap & AI Crawler Access Analyzer

Give it a list of domains. For each site you get:

• which AI crawlers its robots.txt allows or blocks at / (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and 10 more), and at any extra paths you list
• the same for Googlebot and Bingbot
• the parsed robots.txt: groups, Allow and Disallow rules, Crawl-delay, Sitemap lines, and lines crawlers cannot understand
• whether /llms.txt exists
• the sitemaps: files read, number of URLs, oldest and newest <lastmod>, invalid entries, URLs on other hosts
• a short list of findings and suggestions in plain English

Useful for SEO audits, for publishers who want to check that their AI opt-out does what they think, and for anyone who collects data and wants to know what a site permits.

Input

{
    "domains": ["nytimes.com", "github.com", "wordpress.org"],
    "paths": ["/blog/", "/search?q=test"],
    "maxSitemapFiles": 10,
    "maxSitemapUrls": 50000,
    "timeoutSecs": 15
}

• domains (required): domains or URLs. Only the hostname is used. robots.txt belongs to one host, so www.example.com and example.com are different sites, but redirects are followed (nytimes.com is answered by www.nytimes.com). Duplicates are checked once, invalid lines are skipped with a warning.
• paths (optional): extra paths to check for every crawler, besides /. A query string is allowed.
• maxSitemapFiles (default 10): sitemap files downloaded per site, index files included.
• maxSitemapUrls (default 50,000): stop counting sitemap URLs at this number.
• timeoutSecs (default 15): limit for each request.

What the Actor requests

Per site, in this order and one at a time:

1. https://<host>/robots.txt (plain http only if https cannot connect)
2. /llms.txt
3. the sitemaps listed in robots.txt, or /sitemap.xml when robots.txt lists none, then the child sitemaps of index files, up to maxSitemapFiles

With the defaults that is at most 12 requests per site. No HTML pages are fetched. When robots.txt gets no answer at all (DNS error, connection refused), the other requests are skipped. Requests identify themselves with the User-Agent Mozilla/5.0 (compatible; RobotsSitemapAIAnalyzer/1.0; +https://apify.com/isnard/robots-sitemap-ai-analyzer).

How robots.txt is read

The rules follow RFC 9309, which is also what Google and Bing document:

• A crawler uses every group that names its user-agent token (case-insensitive, /version ignored), merged. Only when no group names it does it use the User-agent: * group. So a GPTBot group replaces the * rules for GPTBot completely.
• The most specific rule wins: the longest matching path pattern. When an Allow and a Disallow are equally long, Allow wins.
• * matches any characters and a $ at the end anchors the end of the URL.
• An empty Disallow: blocks nothing.
• A robots.txt answering 4xx (usually 404) means no restrictions: everything is allowed.
• A robots.txt answering 5xx, or not answering, gives unknown. Google treats that case as "do not crawl" for a while.

Each crawler entry looks like this:

{
    "bot": "GPTBot",
    "operator": "OpenAI",
    "purpose": "training",
    "matchedGroup": "GPTBot",
    "root": "blocked",
    "rootRule": "disallow: /",
    "disallowRuleCount": 1,
    "paths": { "/blog/": "blocked", "/search?q=test": "blocked" }
}

• matchedGroup: the token of the group that applies, * when the * group is used, null when there is no group for it at all (everything allowed).
• root: allowed, blocked or unknown for the path /. A site can be allowed at / and still block most sections; disallowRuleCount and paths show that.
• rootRule: the rule that decided /, null when no rule matched.

Crawlers checked

Token	Operator	Purpose
GPTBot	OpenAI	training
OAI-SearchBot	OpenAI	search (ChatGPT search results)
ChatGPT-User	OpenAI	user (fetches a page when a user asks)
ClaudeBot	Anthropic	training
Claude-SearchBot	Anthropic	search
Claude-User	Anthropic	user
Claude-Web	Anthropic	legacy, no longer used
anthropic-ai	Anthropic	legacy, no longer used
Google-Extended	Google	training (Gemini). A control token only, it does not crawl and does not affect Google Search
PerplexityBot	Perplexity	search
CCBot	Common Crawl	training (the dataset is widely used to train models)
Bytespider	ByteDance	training
Applebot-Extended	Apple	training. A control token only, Applebot does the crawling
Meta-ExternalAgent	Meta	training
Amazonbot	Amazon	training and product use
Googlebot	Google	search engine
Bingbot	Microsoft	search engine

Sources for the tokens, checked in October 2026:

• OpenAI: https://developers.openai.com/api/docs/bots
• Anthropic: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler (lists ClaudeBot, Claude-User and Claude-SearchBot; anthropic-ai and Claude-Web are not listed there any more)
• Google: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers
• Perplexity: https://docs.perplexity.ai/guides/bots
• Common Crawl: https://commoncrawl.org/ccbot
• Apple: https://support.apple.com/en-us/119829
• Meta: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/
• Amazon: https://developer.amazon.com/amazonbot
• Bing: https://www.bing.com/webmasters/help/?topicid=8c184ec0
• Bytespider: ByteDance publishes no documentation. The token Bytespider is the one seen in server logs and used in robots.txt files.

robots.txt is a request, not a lock. This Actor tells you what the file says, not what each crawler really does. OpenAI and Perplexity say their user-triggered fetchers (ChatGPT-User, Perplexity-User) may not follow robots.txt, and Bytespider has been reported to ignore it.

Sitemaps

• XML <urlset>, XML <sitemapindex>, plain text (one URL per line) and gzip files are read. Gzip is detected by content, so .xml.gz files and servers that send gzip without saying so both work.
• Index files are followed breadth first: first all sitemaps listed in robots.txt, then their children. The walk stops at maxSitemapFiles files or maxSitemapUrls URLs, and truncated is set to true. A large site such as nytimes.com has thousands of sitemap files, so its count is always a lower bound.
• Invalid entries: <loc> missing, relative, not http(s), longer than 2,048 characters, or a <lastmod> that is not a W3C date.
• URLs on other hosts are compared without a leading www.. Search engines ignore those unless both hosts are verified by the same owner.
• lastmodMin and lastmodMax cover the page entries, not the dates in index files.
• Each file is read up to 50 MB uncompressed, the limit of the sitemap protocol.

Output

One dataset item per site. Shortened example (real output for github.com, full examples in ejemplos/salida.json in the source):

{
    "input": "github.com",
    "host": "github.com",
    "checkedAt": "2026-10-29T09:41:02+00:00",
    "summary": {
        "robotsTxt": "found",
        "aiBlocked": ["Bytespider"],
        "aiAllowed": ["GPTBot", "OAI-SearchBot", "ChatGPT-User", "ClaudeBot", "..."],
        "aiTrainingBlockedCount": 1,
        "aiTrainingTotal": 8,
        "aiNamedInRobots": ["GPTBot", "OAI-SearchBot", "ClaudeBot", "anthropic-ai", "PerplexityBot", "Bytespider"],
        "googlebot": "allowed",
        "bingbot": "allowed",
        "llmsTxtFound": true,
        "sitemapFound": true,
        "sitemapListedInRobots": true,
        "sitemapUrls": 376,
        "sitemapUrlsTruncated": false,
        "reachable": true
    },
    "recommendations": [
        "robots.txt blocks some AI training crawlers by name (Bytespider) but these are not named and still allowed: Google-Extended, CCBot, Applebot-Extended, Meta-ExternalAgent, Amazonbot. If the goal is to opt out of AI training, add groups for them too.",
        "The sitemap URLs have no <lastmod>. Adding it helps search engines recrawl changed pages first."
    ],
    "aiCrawlers": ["one entry per crawler, see above"],
    "searchCrawlers": ["Googlebot and Bingbot"],
    "robots": { "url": "https://github.com/robots.txt", "status": 200, "state": "found", "groups": ["..."], "sitemaps": ["..."], "invalidLineCount": 0 },
    "llmsTxt": { "url": "https://github.com/llms.txt", "found": true, "status": 200, "sizeBytes": 28658 },
    "sitemaps": { "source": "robots.txt", "filesFetched": 1, "totalUrls": 376, "truncated": false, "invalidEntries": 0, "otherHostUrls": 0, "lastmodMin": null, "lastmodMax": null, "files": ["..."] },
    "errors": []
}

summary.robotsTxt is found, missing (4xx) or unreachable. llmsTxt.found is true only for a 200 answer that is not an HTML page, because many sites answer every unknown path with their homepage.

A site that does not answer still gets an item, with reachable: false and the reason in errors. The Actor does not fail because of one bad site.

Pricing

Pay per event: site-analyzed, once per site, only when robots.txt or a sitemap answered. A domain that does not exist or does not answer is not charged.

Limitations

• Paths are compared as given. Percent-encoded and plain characters (/%7Ea and /~a) are not treated as equal.
• User-agent matching is by exact token. Google's fallback from a specific crawler to a general one (Googlebot-News to Googlebot) is not modeled, because only the main tokens are checked.
• Meta tags (noai, noimageai), X-Robots-Tag headers and TDM reservation files are not read, only robots.txt.
• llms.txt is only checked for presence, not validated.
• Sites behind bot protection may answer robots.txt with 403. That is reported as missing, which is what crawlers would assume too. A 429 (rate limited) is unreachable, as Google treats it.
• Requests come from Apify's servers. A site that serves a different robots.txt by country or by user-agent is seen as Apify sees it.

Run locally

python -m venv .venv
.venv/Scripts/pip install -r requirements.txt   # on Linux/macOS: .venv/bin/pip
# put input in storage/key_value_stores/default/INPUT.json
.venv/Scripts/python -m my_actor
python -m unittest discover -s tests -t .

Author: Matias Isnard.