AI Crawler Access Checker

Find out which AI bots a website lets in, for any number of sites in one run. The checker reads each site's robots.txt the way the bots do and reports, per bot, whether it may fetch the home page and whether the site has a rule just for that bot. It also checks llms.txt, llms-full.txt and the X-Robots-Tag header.

Bots checked

Bot	Operator	Purpose
GPTBot, OAI-SearchBot, ChatGPT-User	OpenAI	training, search, user requests
ClaudeBot, Claude-SearchBot, Claude-User	Anthropic	training, search, user requests
Google-Extended	Google	Gemini training control
Applebot-Extended	Apple	Apple Intelligence training control
PerplexityBot, Perplexity-User	Perplexity	search, user requests
Meta-ExternalAgent	Meta	training
CCBot	Common Crawl	open datasets used for training
Bytespider	ByteDance	training
Amazonbot	Amazon	search
DuckAssistBot	DuckDuckGo	AI answers
cohere-ai	Cohere	training

What you get per site

• Per bot: allowed or blocked for the home page, and whether robots.txt names the bot itself
• Summary: how many AI bots are allowed and blocked, which training bots are blocked, which AI search bots are allowed
• llms.txt and llms-full.txt: present or not, size in bytes and lines
• X-Robots-Tag: noai, noimageai, noindex, nofollow on the home page
• robots.txt status: read, missing (everything allowed) or unreachable (unknown)

Typical uses: checking whether your content is open to AI training, search and answer engines, auditing clients for "AI visibility" (GEO), and tracking how sites in a market treat AI bots.

How it works

Per site it reads robots.txt, /llms.txt, /llms-full.txt and the home page headers. It never stores or returns the files' text, only the facts above. It respects robots.txt for its own requests and keeps a polite pace per host.

Input

{
  "sites": ["https://www.example.com/", "example.org"],
  "timeoutSecs": 15
}

Output

One item per site (shortened):

{
  "site": "https://www.example.com/",
  "robots_txt": {"status": "parsed", "sitemaps": 1, "error": ""},
  "agents": [
    {"agent": "GPTBot", "operator": "OpenAI", "purpose": "training", "allowed": false, "own_rule": true},
    {"agent": "OAI-SearchBot", "operator": "OpenAI", "purpose": "search", "allowed": true, "own_rule": false}
  ],
  "allowed_count": 13,
  "blocked_count": 3,
  "training_blocked": ["CCBot", "GPTBot", "Google-Extended"],
  "search_allowed": ["Amazonbot", "Claude-SearchBot", "DuckAssistBot", "OAI-SearchBot", "PerplexityBot"],
  "llms_txt": {"present": true, "bytes": 2140, "lines": 48, "truncated": false},
  "llms_full_txt": {"present": false, "bytes": null, "lines": null},
  "x_robots_tag": []
}

Pricing

You pay per site checked. Inputs that are not a public site are listed free of charge. Set a maximum cost for the run, and the checker stops at your limit.

Note

robots.txt is a request, not a lock: it tells well-behaved bots what to do. This report shows what a site asks for, not what every bot actually does.

Use it from code or AI agents

Run it through the Apify API, schedule it, connect it to Make, Zapier or n8n, or call it from AI agents through Apify's MCP server.

Provider, privacy and fair use

Provider: HelveticLabs, sole proprietorship of Marvin Krieg, c/o SwissMailBox 170, 12 Rue Le Corbusier, 1208 Genève, Switzerland. E-mail: contact@helveticlabs.org. Imprint: https://www.helveticlabs.org/imprint

Privacy: This Actor runs on the Apify platform, in your Apify account. It processes only the URLs or domains you enter and the public technical data the checked servers return (status codes, headers, certificates, DNS records, sitemap URLs, links). It does not collect personal data: it never stores or returns page text (where a page's HTML is read, it is read in memory only, to find links and measure tags), and it leaves out e-mail addresses, cookie names and values and similar values. Your input and results stay in your Apify account and are stored by Apify under its own terms (https://apify.com/privacy-policy). The provider sees only the usage statistics Apify shows to Actor authors (such as the number of users and runs), not your input or results, unless you share a run for support. Questions about privacy: contact@helveticlabs.org.

Fair use: Check only sites you own or may check. Do not use the results to attack systems or people. The Actor respects robots.txt, identifies itself, and keeps a polite pace per site.

No warranty: The results are automated checks and can be incomplete. They are not a penetration test, legal advice or a compliance certificate.