Pricing

from $16.00 / 1,000 robots.txt scans

Try for free

Go to Apify Store

robots.txt Analyzer - Parse Rules, Sitemaps & Crawl Blocks

Try for free

Fetches and parses a site's robots.txt from one site URL or a bulk list. Returns every user-agent group with its Allow and Disallow rules, crawl delays, declared sitemap URLs, whether all crawlers are blocked and common mistakes. For SEO specialists and crawler developers.

Pricing

from $16.00 / 1,000 robots.txt scans

Rating

0.0

(0)

Developer

Broke to Built

Actor stats

Bookmarked

Total users

Monthly active users

6 days ago

Last modified

Configure MCP server with robots.txt Analyzer: Rules, Sitemaps, Crawl Blocks API

The configuration above connects to the hosted Apify MCP server with the robots.txt Analyzer: Rules, Sitemaps, Crawl Blocks API Actor already loaded. It uses OAuth, so your client opens a browser window to sign you in to Apify on first use. No API token is needed. Clients without OAuth support can authenticate with an Authorization: Bearer header carrying an API token from API & Integrations in Apify Console.

Get a ready-to-use configuration for your MCP client with the robots.txt Analyzer: Rules, Sitemaps, Crawl Blocks API Actor preconfigured at https://mcp.apify.com/?tools=fetch-actor-details,eliai/robots-txt-analyzer.

You can connect to the Apify MCP Server using clients like Tester MCP Client, or any other MCP client of your choice.

If you want to learn more about our Apify MCP implementation, check out our MCP documentation. To learn more about the Model Context Protocol in general, refer to the official MCP documentation or read our blog post.

Robots.txt & Sitemap Analyzer

automation-lab/robots-sitemap-analyzer

This actor fetches and parses robots.txt and sitemap.xml files for any list of websites. It extracts crawl directives (user-agent rules, allowed/disallowed paths, crawl-delay), discovers sitemap URLs, and counts the number of pages listed in each sitemap. Use it for SEO audits, competitive...

Automation Lab

Robots Txt Audit

apage/robots-txt-audit

Audit robots.txt files for AI crawler access. Get an AI Readiness Score (0-100), analyze 61+ AI crawlers (ChatGPT, Claude, Perplexity, Gemini), detect syntax errors, security concerns, and get actionable recommendations. Batch audit multiple domains at once with optional subdomain scanning.

Andy Page

SEO Analyzer — Website SEO Audit API

vivid_astronaut/seo-analyzer

Audit any website for SEO issues: scores, meta tags, headings, links, and images with actionable recommendations. Input a URL, get a structured report — useful for agencies, developers, and automated site monitoring.

BRAINIALL Team

Robots.txt & Sitemap Analyzer 🕷️

perryay/robots-txt-sitemap-analyzer

Fetch, parse, and analyze robots.txt and sitemap.xml for any domain. Extract crawl directives, test URL compliance against robots.txt rules, and discover all URLs from sitemaps including nested sitemap indexes. Supports batch analysis with structured JSON output.

Perry AY

URL Redirect Chain & Loop Analyzer

andok/redirect-chain-analyzer

Trace HTTP redirects across any domain. Identify broken links, infinite loops, and SEO-damaging redirect chains.

Andok

robots.txt & Sitemap Analyzer

bgfc97/robots-txt-sitemap-analyzer

Fetch and parse robots.txt for many domains: user-agent groups, allow/disallow rules, crawl-delay, declared sitemaps and whether crawlers are blocked. Technical SEO auditing. No key.

Bruno

Robots.txt Checker & Parser - Crawl Rules API

pink_comic/robots-txt-validator

Check, parse, and validate robots.txt files in bulk. Extract crawl rules, sitemaps, crawl-delay, blocked paths, and per-user-agent allow/disallow results for SEO audits and crawler compliance.

Ava Torres

AI Crawler Checker – robots.txt Rules for GPTBot & ClaudeBot

locomint/ai-crawler-checker

AI crawler checker: read each website's robots.txt and see whether GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and 13 other AI crawlers are allowed or blocked, with the line that decided each verdict, plus Content-Signal and llms.txt.

Locomint

SEO Meta Tag Analyzer

constant_quadruped/seo-meta-analyzer

Audit any URL for SEO: title length, meta description, Open Graph tags, heading hierarchy, image alt texts, keyword density, mobile viewport, canonical URL, schema markup, and more.

Font Detector - Extract Fonts from Website URLs in Bulk

eliai/webpage-font-detector

Extract fonts from website URLs in bulk: detects each page's font families, @font-face files and whether each comes from Google, Adobe, self-hosting or the system. Returns one row per URL with font file URLs; failed fetches are free. For designers, brand managers, performance engineers and agencies.