Pricing

$2.00 / 1,000 site robots.txt checkeds

Try for free

Go to Apify Store

Robots.txt Validator — Blocked Bots & Bad Lines

Try for free

Check the robots.txt of the sites you give: found or not, groups and rules, sitemaps, crawl-delay, whether all bots are blocked and lines crawlers cannot read.

Pricing

$2.00 / 1,000 site robots.txt checkeds

Rating

0.0

(0)

Developer

Kadro Tools

Actor stats

Bookmarked

Total users

Monthly active users

9 days ago

Last modified

Categories

SEO tools

Developer tools

Security

You can access the Robots.txt Validator — Blocked Bots & Bad Lines programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1from apify_client import ApifyClient
2
3# Initialize the ApifyClient with your Apify API token
4# Replace '<YOUR_API_TOKEN>' with your token.
5client = ApifyClient("<YOUR_API_TOKEN>")
6
7# Prepare the Actor input
8run_input = { "sites": ["https://www.python.org/"] }
9
10# Run the Actor and wait for it to finish
11run = client.actor("kadro/robots-txt-validator").call(run_input=run_input)
12
13# Fetch and print Actor results from the run's dataset (if there are any)
14print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
15for item in client.dataset(run.default_dataset_id).iterate_items():
16    print(item)
17
18# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

Robots.txt Tester: Check Disallow Rules & Sitemaps API in Python

The Apify API client for Python is the official library that allows you to use Robots.txt Validator — Blocked Bots & Bad Lines API in Python, providing convenience functions and automatic retries on errors.

Install the apify-client

$pip install apify-client

Other API clients include:

Robots.txt Validator — Blocked Bots & Bad Lines API in JavaScript

Robots.txt Validator — Blocked Bots & Bad Lines API through CLI

Robots.txt Validator — Blocked Bots & Bad Lines OpenAPI definition

Robots.txt Validator — Blocked Bots & Bad Lines API

Sitemap Extractor

apify/sitemap-extractor

This Apify Actor extracts all URLs from a website's sitemaps and checks their status codes via lightweight HTTP requests. It provides a clean list of valid links, acting as an ideal pre-processor to ensure your larger crawling projects target only active URLs.

Apify

439

3.2

(8)

Robots.txt & Sitemap Analyzer

automation-lab/robots-sitemap-analyzer

This actor fetches and parses robots.txt and sitemap.xml files for any list of websites. It extracts crawl directives (user-agent rules, allowed/disallowed paths, crawl-delay), discovers sitemap URLs, and counts the number of pages listed in each sitemap. Use it for SEO audits, competitive...

Automation Lab

Ads.txt & App-ads.txt Validator

gifted_wagon/ads-txt-evidence-auditor

Audit public ads.txt or app-ads.txt files for syntax, seller relationships, duplicate tuples, variables, and exact line evidence.

Michael Olmos

Robots.txt Checker & Parser - Crawl Rules API

pink_comic/robots-txt-validator

Check, parse, and validate robots.txt files in bulk. Extract crawl rules, sitemaps, crawl-delay, blocked paths, and per-user-agent allow/disallow results for SEO audits and crawler compliance.

Ava Torres

robots.txt & Sitemap Analyzer

bgfc97/robots-txt-sitemap-analyzer

Fetch and parse robots.txt for many domains: user-agent groups, allow/disallow rules, crawl-delay, declared sitemaps and whether crawlers are blocked. Technical SEO auditing. No key.

Bruno

Broken Link Checker — 404s, Redirects & SEO Audit

khadinakbar/broken-link-checker

Check links from a website crawl, URL list, or sitemap. Receive HTTP status, redirect chains, response timing, source pages, anchor text, and diagnostic classes for SEO quality assurance and site migrations.

Khadin Akbar

122

Robots.txt Tester — Is a URL Allowed to Crawl?

hipersoft/robots-txt-checker

Test whether URLs are allowed or blocked by a site's robots.txt for any crawler user-agent. Returns allowed/blocked, the matched rule, crawl-delay and the site's sitemaps. Great for SEO audits and crawler compliance. One row per URL as JSON, CSV or Excel.

Hipersoft

Robots.txt Monitor

datawinder/robots-txt-monitor

Stateful robots.txt monitoring with baseline awareness and severity-classified alerts. Detects meaningful policy changes over time — not noisy diffs.

DatawinderLabs

Sitemap & Robots SEO Index Auditor

glowing_glove/sitemap-robots-index-auditor

Audit robots.txt, sitemap discovery, indexability signals, canonical tags, and SEO crawl readiness for business websites.

Ushba Khan

Schema Markup Validator - JSON-LD Structured Data Checker

tidytools/structured-data-validator

Validate JSON-LD and Microdata schema markup on a URL list, a sitemap or a whole site against Google rich result rules (Product, FAQ, Recipe, Event, Job...), plus Open Graph. $2 per 1,000 pages.