Pricing

from $16.00 / 1,000 robots.txt scans

Try for free

Go to Apify Store

robots.txt Analyzer - Parse Rules, Sitemaps & Crawl Blocks

Try for free

Fetches and parses a site's robots.txt from one site URL or a bulk list. Returns every user-agent group with its Allow and Disallow rules, crawl delays, declared sitemap URLs, whether all crawlers are blocked and common mistakes. For SEO specialists and crawler developers.

Pricing

from $16.00 / 1,000 robots.txt scans

Rating

0.0

(0)

Developer

Broke to Built

Actor stats

Bookmarked

Total users

Monthly active users

6 days ago

Last modified

Categories

SEO tools

Developer tools

Security

You can access the robots.txt Analyzer - Parse Rules, Sitemaps & Crawl Blocks programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1from apify_client import ApifyClient
2
3# Initialize the ApifyClient with your Apify API token
4# Replace '<YOUR_API_TOKEN>' with your token.
5client = ApifyClient("<YOUR_API_TOKEN>")
6
7# Prepare the Actor input
8run_input = {
9    "url": "https://www.apify.com",
10    "urls": [],
11}
12
13# Run the Actor and wait for it to finish
14run = client.actor("eliai/robots-txt-analyzer").call(run_input=run_input)
15
16# Fetch and print Actor results from the run's dataset (if there are any)
17print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
18for item in client.dataset(run.default_dataset_id).iterate_items():
19    print(item)
20
21# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

robots.txt Analyzer: Rules, Sitemaps, Crawl Blocks API API in Python

The Apify API client for Python is the official library that allows you to use robots.txt Analyzer - Parse Rules, Sitemaps & Crawl Blocks API in Python, providing convenience functions and automatic retries on errors.

Install the apify-client

$pip install apify-client

Other API clients include:

robots.txt Analyzer - Parse Rules, Sitemaps & Crawl Blocks API in JavaScript

robots.txt Analyzer - Parse Rules, Sitemaps & Crawl Blocks API through CLI

robots.txt Analyzer - Parse Rules, Sitemaps & Crawl Blocks OpenAPI definition

robots.txt Analyzer - Parse Rules, Sitemaps & Crawl Blocks API

Robots.txt & Sitemap Analyzer

automation-lab/robots-sitemap-analyzer

This actor fetches and parses robots.txt and sitemap.xml files for any list of websites. It extracts crawl directives (user-agent rules, allowed/disallowed paths, crawl-delay), discovers sitemap URLs, and counts the number of pages listed in each sitemap. Use it for SEO audits, competitive...

Automation Lab

Robots Txt Audit

apage/robots-txt-audit

Audit robots.txt files for AI crawler access. Get an AI Readiness Score (0-100), analyze 61+ AI crawlers (ChatGPT, Claude, Perplexity, Gemini), detect syntax errors, security concerns, and get actionable recommendations. Batch audit multiple domains at once with optional subdomain scanning.

Andy Page

SEO Analyzer — Website SEO Audit API

vivid_astronaut/seo-analyzer

Audit any website for SEO issues: scores, meta tags, headings, links, and images with actionable recommendations. Input a URL, get a structured report — useful for agencies, developers, and automated site monitoring.

BRAINIALL Team

Robots.txt & Sitemap Analyzer 🕷️

perryay/robots-txt-sitemap-analyzer

Fetch, parse, and analyze robots.txt and sitemap.xml for any domain. Extract crawl directives, test URL compliance against robots.txt rules, and discover all URLs from sitemaps including nested sitemap indexes. Supports batch analysis with structured JSON output.

Perry AY

URL Redirect Chain & Loop Analyzer

andok/redirect-chain-analyzer

Trace HTTP redirects across any domain. Identify broken links, infinite loops, and SEO-damaging redirect chains.

Andok

robots.txt & Sitemap Analyzer

bgfc97/robots-txt-sitemap-analyzer

Fetch and parse robots.txt for many domains: user-agent groups, allow/disallow rules, crawl-delay, declared sitemaps and whether crawlers are blocked. Technical SEO auditing. No key.

Bruno

Robots.txt Checker & Parser - Crawl Rules API

pink_comic/robots-txt-validator

Check, parse, and validate robots.txt files in bulk. Extract crawl rules, sitemaps, crawl-delay, blocked paths, and per-user-agent allow/disallow results for SEO audits and crawler compliance.

Ava Torres

AI Crawler Checker – robots.txt Rules for GPTBot & ClaudeBot

locomint/ai-crawler-checker

AI crawler checker: read each website's robots.txt and see whether GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and 13 other AI crawlers are allowed or blocked, with the line that decided each verdict, plus Content-Signal and llms.txt.

Locomint

SEO Meta Tag Analyzer

constant_quadruped/seo-meta-analyzer

Audit any URL for SEO: title length, meta description, Open Graph tags, heading hierarchy, image alt texts, keyword density, mobile viewport, canonical URL, schema markup, and more.

Font Detector - Extract Fonts from Website URLs in Bulk

eliai/webpage-font-detector

Extract fonts from website URLs in bulk: detects each page's font families, @font-face files and whether each comes from Google, Adobe, self-hosting or the system. Returns one row per URL with font file URLs; failed fetches are free. For designers, brand managers, performance engineers and agencies.