Robots.txt Validator — Blocked Bots & Bad Lines
Pricing
$2.00 / 1,000 site robots.txt checkeds
Robots.txt Validator — Blocked Bots & Bad Lines
Check the robots.txt of the sites you give: found or not, groups and rules, sitemaps, crawl-delay, whether all bots are blocked and lines crawlers cannot read.
Pricing
$2.00 / 1,000 site robots.txt checkeds
Rating
0.0
(0)
Developer
Kadro Tools
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
9 days ago
Last modified
Categories
Security
Limited permissionsShare
Robots.txt Validator
Make sure a leftover Disallow: / or a typo in robots.txt is not hiding your site from Google or AI bots.
What you get
| Field | What it tells you |
|---|---|
site, robotsUrl | The site you gave and its robots.txt address |
statusCode, error | HTTP status and any request error |
found | Whether a robots.txt exists |
groups | Number of user-agent groups |
rules | Number of Allow/Disallow lines |
disallowAll | True when User-agent: * has Disallow: / |
crawlDelay | Crawl-delay for *, if set |
sitemaps | Sitemap URLs declared in the file |
issues | List of problems, or ok when nothing is wrong |
How to use it in 3 steps
- Paste domains or URLs, one per line (up to 500).
- Leave
followRedirectson to read the final robots.txt. - Press Start and check
disallowAllandissues; export as JSON, CSV or Excel.
Pricing
$2 per 1,000 sites checked. Example: 100 sites = $0.20. Sites whose robots.txt could not be requested are free; a missing robots.txt (404) is charged because it is a valid answer. Set a maximum cost per run and the robot stops when it is reached.
Integrations
- Apify API and webhooks: start a run and read the dataset from any language.
- Make, n8n and Zapier: use the Apify module or node, run this robot and pass the dataset on.
- Google Sheets: export the dataset or use the Apify integration to fill a sheet on a schedule.
- Claude, Cursor and other agents (MCP): Use the Apify MCP server (
https://mcp.apify.com?actors=kadro/robots-txt-validator) from Claude or Cursor and ask: "Check the robots.txt of example.com and tell me if any important bot is blocked"
FAQ
What counts as an issue? Lines a crawler cannot understand, rules before any User-agent, a robots.txt over 500 KB (Google reads only the first 500 KiB), unusable sitemap URLs and similar.
Does it test a specific URL against the rules? No. It validates the file and its common rules; it does not simulate how each search engine treats every edge case.
Does it crawl my site? No. It reads only /robots.txt.
What is not charged? Requests that failed or were rejected (private addresses).
Is it legal? robots.txt is a public file meant to be read by crawlers.
Limitations
- Checks syntax and the most common rules, not every search-engine edge case.
- Up to 500 sites per run.
- Reads only the first 500 KB of the file.
- Only public URLs: private and local addresses are blocked.
Other Kadro tools
Part of the Kadro SEO toolkit: check crawl rules here, then validate the sitemap they point to.
- XML Sitemap Validator: checks that a sitemap follows the protocol and Google limits.
- Sitemap URL Extractor: exports every URL from a sitemap or sitemap index.
- SEO On-Page Audit Crawler: crawls a whole site and gives every page a 0-100 SEO score.
- Hreflang & Canonical Checker: finds canonical and hreflang errors on multilingual sites.
- Website Links Extractor: lists every internal and external link of a page with anchor and nofollow.
Found it useful? Please leave a review ⭐ — it helps a lot. Bugs or feature ideas: open an issue, I answer within 24 h.
