Robots.txt Checker – Test URLs, Googlebot & AI Crawlers

Check the robots.txt of one website or 10,000 at once. For each site you get, in plain words:

• Is there a robots.txt, and what Google does when it is missing, broken or the server errors.
• Which search bots are blocked: Googlebot, Bingbot, Applebot, DuckDuckBot, YandexBot, Baiduspider.
• Which AI crawlers are blocked or allowed: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, meta-externalagent and Amazonbot.
• Test your own URLs: is /blog/ or /search?q=shoes blocked, for which bot, and which rule decides it.
• The sitemaps it lists, lines Google cannot read, crawl-delay, files over Google's 500 KiB limit, and whether the site has an llms.txt.

The rules are read the way Google reads them (RFC 9309): the most specific user-agent group, the longest matching rule, Allow wins a tie, * and $ wildcards, and the common typos Google forgives.

Built for SEO audits before and after a launch, checking that a deploy did not block Google by mistake, monitoring competitors' AI crawler policy, publishers deciding what to allow, and AI agents that must respect robots.txt.

Example output row

{"website": "https://www.nytimes.com", "status": 200, "found": true, "verdict": "robots.txt found", "sitemap_count": 25, "blocks_whole_site": false,
 "search_bots_blocked": [], "ai_bots_blocked": ["GPTBot", "OAI-SearchBot", "ChatGPT-User", "ClaudeBot", "PerplexityBot", "Google-Extended", "CCBot", "..."],
 "ai_bots_allowed": ["Amazonbot"], "invalid_lines": 0, "has_llms_txt": false,
 "url_tests": [{"url": "https://www.nytimes.com/section/technology", "blocked_for": ["GPTBot"], "rules": {"GPTBot": "Disallow: /"}}]}

How to use

1. Click Try for free (or Start) at the top of this page to open the input form.
2. Fill in Websites. Domains or URLs, up to 10,000. The form comes with a working example, so you can also run it as it is.
3. Optional: adjust URLs or paths to test (optional), Bots for the URL tests (optional), Also check for llms.txt. Every field has a sensible default.
4. Click Start and wait for the run to finish.
5. Download the results as Excel, CSV or JSON from the Output tab, or get them through the Apify API, a schedule or an integration (Make, Zapier, Google Sheets ...).

Pricing

Pay per event, no subscription:

• $0.0005 per website checked ($0.50 per 1,000).
• $0.001 per run start.

A site that does not answer is still a result (it tells you Google sees it as blocked). Text that is not a website is free. Works on the free Apify plan.

Input

Field	What it does
websites	Domains or URLs, up to 10,000. www and non-www are different sites, as for Google.
testUrls	Optional paths (/blog/, tested on every site) or full URLs.
bots	Optional user agents for the URL tests. Empty = Googlebot, Bingbot and the 12 AI crawlers.
checkLlmsTxt	Also look for /llms.txt (default on).

Good to know

• What Google does without a file: a 404 or other 4xx means "crawl everything"; a 5xx, 429 or no answer means Google treats the whole site as blocked until it can read the file. The verdict says which applies.
• Refused: some sites block automated checkers or whole countries (HTTP 401/403). Then the row says refused and draws no conclusion, because Googlebot may get a different answer.
• robots.txt controls crawling, not indexing: a blocked page can still show up in Google if other sites link to it. To check indexing, use the Google Index Checker.
• AI crawler names change; the list is updated when the big AI companies add new ones. Use bots to test any other user agent.

More tools from Rowfeed

• SEO Site Audit – every page's on-page SEO, noindex and broken links.
• Sitemap URL Extractor – every URL in the sitemaps found here.
• AI Search Visibility Tracker – whether ChatGPT, Gemini and Perplexity mention your brand.
• Redirect Checker – status codes and redirect chains in bulk.