Robots.txt Tester — Is a URL Allowed to Crawl?

Check whether URLs are allowed or blocked by a site's robots.txt with the Robots.txt Tester. Pass a list of URLs and a crawler user-agent, and get back — for each URL — whether it's crawlable, the exact rule that matched, the crawl-delay and the site's sitemaps. Export as JSON, CSV or Excel.

Built for SEO audits, crawler and scraper compliance, indexing checks and site migrations, it applies real robots.txt matching rules so you know exactly what a bot is allowed to fetch.

What it does

• ✅ Allowed / blocked — a clear verdict for each URL and user-agent.
• 🎯 Matched rule — the exact Allow/Disallow line that decided it (longest-match, Allow-wins precedence).
• 🤖 Any user-agent — test as *, Googlebot, Bingbot, GPTBot or any bot.
• ⏱️ Crawl-delay — the crawl-delay declared for that agent, if any.
• 🗺️ Sitemaps — the sitemap URLs listed in the site's robots.txt.
• ⚡ Bulk & cached — tests many URLs and caches each site's robots.txt for speed.
• 📤 Export anywhere — JSON, CSV or Excel, or straight into an SEO tool or report.

Example output

{
  "url": "https://en.wikipedia.org/wiki/Special:Search",
  "userAgent": "*",
  "allowed": false,
  "matchedRule": "disallow: /wiki/Special:",
  "robotsFound": true,
  "sitemaps": ["https://en.wikipedia.org/sitemap.xml"]
}

How to use it

1. Add the URLs you want to test.
2. Set the crawler user-agent (e.g. Googlebot).
3. Run, and download the results as JSON, CSV or Excel.

Input fields

Field	Description
URLs to test	Full URLs (or bare domains) to test.
Crawler user-agent	The bot to test as (*, Googlebot, GPTBot…).
Max URLs	Maximum URLs to test (0 = no limit).

Output fields

url, userAgent, allowed, matchedRule, crawlDelay, robotsUrl, robotsFound, sitemaps, error.

Popular use cases

• SEO audits — confirm important pages aren't accidentally blocked.
• Crawler compliance — respect robots.txt before scraping a site.
• Indexing checks — verify what search bots can and can't fetch.
• Migrations — catch robots.txt mistakes after a site move.
• AI crawler policy — see whether bots like GPTBot are allowed.

FAQ

Do I need an account or key? No. Add URLs and run.

How is the verdict decided? By standard robots.txt matching — the most specific (longest) matching rule wins, and Allow beats Disallow on a tie, for the selected user-agent.

Can I test different bots? Yes — set any user-agent (Googlebot, Bingbot, GPTBot, etc.) to see rules that apply to it.

Can I export to Excel or Google Sheets? Yes — results download as JSON, CSV or Excel and integrate with SEO tools and reports.

---

Test robots.txt allow/block rules for any URL and crawler — with the matched rule, crawl-delay and sitemaps — clean and export-ready for SEO and compliance.