AI Crawler Policy Lookup — robots.txt & llms.txt Checker

Check what a website's robots.txt says about AI crawlers — GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot, Bytespider, and a dozen more — and whether it publishes an llms.txt file describing what AI agents may use. This actor reads the site's own public robots.txt and llms.txt directly — no third-party API, no scraping of rendered pages.

Features

• Per-agent policy breakdown. For each of 16 known AI/LLM crawler user-agents (GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-Web, anthropic-ai, Google-Extended, GoogleOther, PerplexityBot, Perplexity-User, Bytespider, CCBot, Applebot-Extended, FacebookBot, meta-externalagent, Diffbot, Amazonbot), reports whether it's named and whether it's disallowed at the root.
• llms.txt detection. Fetches /llms.txt and includes up to a 4,000-character excerpt when the file exists and is non-empty.
• Domain-only queries. Feed in bare domains or full URLs — the actor normalizes to the site root before requesting /robots.txt.
• Pay only for signal. A domain with no AI-agent mentions in robots.txt and no llms.txt costs nothing — see Pricing.
• Built for bulk. Submit many domains at once; concurrency is configurable.

How to use AI Crawler Policy Lookup — robots.txt & llms.txt Checker

1. In the Apify Console. Open the actor page and click Start — the items field is already pre-filled with a working example. Results land in the run's dataset as soon as each item is found.
2. Via the API. Call it directly with a POST request — no Console needed once you have an API token:

   curl "https://api.apify.com/v2/acts/accountable_eel~ai-crawler-policy-lookup/run-sync-get-dataset-items?token=<YOUR_TOKEN>" \
     -X POST \
     -H "Content-Type: application/json" \
     -d '{"items":["nytimes.com"]}'
   

3. On a schedule. Save this actor as an Apify Task with the input you want, then add a Schedule (hourly, daily, weekly) so it runs on its own — no server of your own required.

Input

{
  "items": ["nytimes.com", "docs.anthropic.com"],
  "maxConcurrency": 5,
  "proxyConfiguration": { "useApifyProxy": true }
}

items is the list of domains or URLs to check — bare domains work fine, the actor adds the scheme and strips any path before requesting /robots.txt. maxConcurrency controls parallel requests (default 5); this target has no browser fallback, so keeping it conservative avoids blocks on high-traffic sites. proxyConfiguration routes requests through Apify Proxy; residential is recommended for anti-bot-sensitive targets.

Output

One row per domain, for example:

{
  "query": "nytimes.com",
  "found": true,
  "data": {
    "domain": "nytimes.com",
    "aiAgentsReferenced": 7,
    "aiAgentPolicies": [
      { "agent": "GPTBot", "blocked": true, "rules": [{ "type": "disallow", "path": "/" }] },
      { "agent": "ClaudeBot", "blocked": true, "rules": [{ "type": "disallow", "path": "/" }] }
    ],
    "hasLlmsTxt": false,
    "llmsTxtExcerpt": null
  },
  "scrapedAt": "2026-08-20T12:00:00.000Z"
}

A domain where none of the known AI agents are named in robots.txt and no llms.txt exists comes back as found: false with no data — and is never charged. An agent that simply isn't mentioned by name (rather than explicitly disallowed) is recorded as "not referenced," which is a different signal from "allowed."

Use cases

• Audit whether your own site's robots.txt blocks the AI crawlers you intend to block (or forgot to).
• Check a partner or publisher's AI-training opt-out posture before a content licensing conversation.
• Build a dataset of which sites block GPTBot vs ClaudeBot vs Google-Extended.
• Track llms.txt adoption across a list of domains as the convention spreads.
• Feed AI-policy signals into a larger domain-enrichment or compliance pipeline.

Pricing

$5 per 1,000 results, plus a $0.005 start fee. Misses (found:false) are never charged.

Use it from Clay, n8n, Make, or an AI agent

This actor runs synchronously over plain HTTP — call it directly from a script, a workflow tool, or an AI agent, no Apify Console needed once you have an API token.

curl "https://api.apify.com/v2/acts/accountable_eel~ai-crawler-policy-lookup/run-sync-get-dataset-items?token=<YOUR_TOKEN>" \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{"items":["nytimes.com"]}'

n8n. Add an HTTP Request node: Method POST, URL https://api.apify.com/v2/acts/accountable_eel~ai-crawler-policy-lookup/run-sync-get-dataset-items?token=<YOUR_TOKEN>, Body Content Type JSON, JSON Body {"items":["nytimes.com"]} (swap in an expression from an earlier node for a real value).

Clay. Add an "HTTP API" column: Method POST, URL https://api.apify.com/v2/acts/accountable_eel~ai-crawler-policy-lookup/run-sync-get-dataset-items?token=<YOUR_TOKEN>, Body {"items":["{{value}}"]}, mapping the row's value into the items array.

MCP. In Claude, Cursor, or any MCP client with the Apify MCP server, ask for "AI Crawler Policy Lookup | Apify" — the agent will find and run this actor.

FAQ

What counts as "found"? At least one of the 16 known AI-crawler user-agents must be explicitly named in robots.txt, or /llms.txt must exist and be non-empty. A robots.txt with only generic User-agent: * rules and no llms.txt returns found: false.

Does "not referenced" mean "allowed"? No — it means the site's robots.txt never mentions that specific agent by name, so there's no explicit signal either way. Only agents that appear in their own User-agent: group are reported with a blocked value.

Does this actor render JavaScript or bypass anti-bot protection? No. It fetches robots.txt and llms.txt as plain text over HTTP, exactly like any crawler would. There's no browser fallback, which is why maxConcurrency defaults low.

Can I pass a full URL instead of a bare domain? Yes — https://example.com/some/path and example.com resolve to the same request (https://example.com/robots.txt).

Is the robots.txt parsing spec-complete? It's a minimal parser that groups User-agent/Disallow/Allow lines per agent and checks for a root-level disallow. It doesn't resolve wildcard precedence or nested-path rules — it answers "is this agent blocked at the root," not a full robots.txt compliance check.

Do I need a proxy? Apify Proxy is enabled by default. Residential proxies help on sites with stricter anti-bot rules, though most robots.txt/llms.txt requests succeed on datacenter IPs.