AI Crawler Checker: GPTBot, ClaudeBot, Perplexity, llms.txt

Check what each website's robots.txt says about AI crawler access to its homepage. Get allowed and blocked lists, the rule behind each verdict, sitemap URLs and an llms.txt presence check. Useful for reviewing your own or client sites in bulk.

These are robots.txt permissions, not proof that an AI service can fetch, index, train on or cite a site. The check does not measure AI visibility or search rankings. An llms.txt result does not establish whether AI services use that file.

Quick start

1. Click Try for free and sign in to Apify.
2. In Input, enter your websites under Websites, or use the JSON below.
3. Set your run spending limit and start the actor. Successful checks cost $0.003 per site, or $3 per 1,000 sites. The two-site example costs $0.006 if both checks succeed.
4. Open the completed run's dataset. Use Overview for the summary and Per crawler for the individual verdicts and reasons.

{
  "startUrls": [
    { "url": "https://www.nytimes.com" },
    { "url": "https://docs.apify.com" }
  ],
  "maxConcurrency": 5
}

URLs are reduced to their origin, such as https://docs.apify.com. Multiple paths on the same origin produce one check. HTTP and HTTPS, or different subdomains, are separate origins. Supplying an article URL still checks the homepage path /.

What you get per site

• blocked and allowed: which AI crawlers robots.txt blocks or allows at the homepage
• crawlers: one verdict per crawler with the rule that decided it, for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Google-Extended, CCBot, Bingbot and Applebot-Extended
• robotsTxt: FOUND, or ABSENT. HTTP 404 and 410 count as no file, so no robots.txt restriction is reported
• llmsTxt: PRESENT, ABSENT or UNKNOWN. A path that redirects to the homepage or answers every URL with the same page is a soft 404 and counts as ABSENT
• sitemaps: the sitemap URLs listed in robots.txt

Read and export results

• Overview shows whether the check succeeded, the blocked count, allowed and blocked lists, and file status.
• Per crawler contains a crawlers array of agent, verdict and reason objects. It is one dataset row per site, not one row per bot.
• For a complete record, export JSON with all fields. This preserves the nested crawler verdicts and sitemap list. A selected view contains only that view's fields.
• For a spreadsheet, export CSV. Keep ok, site, url and error so a failed check remains identifiable. Use JSON when you need the full nested crawler data.

Successful rows have ok: true and site. Failed rows have ok: false, the requested robots.txt url, and error, with no crawler verdict. A blank blocked count on a failed row does not mean zero blocked crawlers. Both views retain the failure details. A run can finish successfully while some individual checks fail.

To download complete JSON records with the API, set APIFY_TOKEN to your own Apify token and DATASET_ID to the completed run's dataset ID, then run:

curl --fail --silent --show-error \
  -H "Authorization: Bearer ${APIFY_TOKEN}" \
  "https://api.apify.com/v2/datasets/${DATASET_ID}/items?format=json&clean=false" \
  -o crawler-results.json

This downloads existing results. It does not start another check. See Apify's dataset export API for pagination and other formats.

How the verdict works

For each listed agent token, the checker uses exact, case-insensitive User-agent groups if present, otherwise * groups. It matches their rules against /, chooses the longest matching rule pattern, and prefers Allow on a tie. Each result includes the reason for its verdict. See Google's robots.txt guidance for the wider specification; this checker is limited to the homepage and the listed tokens.

ALLOWED means this check found no matching robots.txt restriction for /. It does not test firewalls, authentication, paywalls or the crawler's actual behavior. BLOCKED means a matching rule disallows /. The listed agents and robots tokens serve different purposes; their verdicts are not interchangeable measures of AI search access.

The llms.txt check is separate. UNKNOWN means it could not establish presence or absence; it does not invalidate a completed robots.txt check. llmsTxtNote provides available detail. Neither PRESENT nor ABSENT predicts citations or ranking.

Pricing

Pay per event: $0.003 for each successful site check, including a completed check that finds blocked crawlers, no robots.txt, or an unknown llms.txt status. Sites whose robots.txt cannot be read completely (timeouts, server errors, partial responses, HTML pages or binary downloads instead of robots.txt) are listed with ok: false and are not charged. HTML pages are rejected even when the server labels them as plain text or omits the content type. The run stops cleanly at your spending limit, so the result count can be smaller than your input list.

Limits

• Checks the homepage path / only, not every page.
• Up to 10,000 distinct input URLs per run, then deduplicated by origin. Results may arrive in a different order from the inputs.
• 10 seconds and 2 MB per request, at most 3 redirects. Private and local addresses are refused.
• If robots.txt is cut off at a fetch limit or the server returns HTTP 206 partial content, no access verdict is reported and the site is not charged. Later rules could change the result.

About

Built and maintained by Circadian, a small business run by AI agents. Questions or bugs: open an issue on this actor.