Pricing

from $16.00 / 1,000 robots.txt scans

Try for free

Go to Apify Store

robots.txt Analyzer — Rules, Sitemaps & Crawl Blocks

Try for free

Parse and audit any site's robots.txt in one call: user-agent groups, allow/disallow rules, declared sitemaps, and crawl-block issues like an accidental site-wide Disallow. $0.02 per site analyzed. No API key.

Pricing

from $16.00 / 1,000 robots.txt scans

Rating

0.0

(0)

Developer

Broke to Built

Actor stats

Bookmarked

Total users

Monthly active users

10 days ago

Last modified

Categories

SEO tools

Developer tools

You can access the robots.txt Analyzer — Rules, Sitemaps & Crawl Blocks programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1import { ApifyClient } from 'apify-client';
2
3// Initialize the ApifyClient with your Apify API token
4// Replace the '<YOUR_API_TOKEN>' with your token
5const client = new ApifyClient({
6    token: '<YOUR_API_TOKEN>',
7});
8
9// Prepare Actor input
10const input = {
11    "url": "https://www.apify.com",
12    "urls": []
13};
14
15// Run the Actor and wait for it to finish
16const run = await client.actor("eliai/robots-txt-analyzer").call(input);
17
18// Fetch and print Actor results from the run's dataset (if any)
19console.log('Results from dataset');
20console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
21const { items } = await client.dataset(run.defaultDatasetId).listItems();
22items.forEach((item) => {
23    console.dir(item);
24});
25
26// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

robots.txt Analyzer - Rules, Sitemaps, Crawl Blocks API in JavaScript

The Apify API client for JavaScript is the official library that allows you to use robots.txt Analyzer — Rules, Sitemaps & Crawl Blocks API in JavaScript or TypeScript, providing convenience functions and automatic retries on errors.

Install the apify-client

$npm install apify-client

Other API clients include:

robots.txt Analyzer — Rules, Sitemaps & Crawl Blocks API in Python

robots.txt Analyzer — Rules, Sitemaps & Crawl Blocks API through CLI

robots.txt Analyzer — Rules, Sitemaps & Crawl Blocks OpenAPI definition

robots.txt Analyzer — Rules, Sitemaps & Crawl Blocks API

robots.txt & Sitemap Analyzer

bgfc97/robots-txt-sitemap-analyzer

Fetch and parse robots.txt for many domains: user-agent groups, allow/disallow rules, crawl-delay, declared sitemaps and whether crawlers are blocked. Technical SEO auditing. No key.

Bruno

Robots.txt Checker & Parser - Crawl Rules API

pink_comic/robots-txt-validator

Check, parse, and validate robots.txt files in bulk. Extract crawl rules, sitemaps, crawl-delay, blocked paths, and per-user-agent allow/disallow results for SEO audits and crawler compliance.

Ava Torres

Robots Sitemap Analyzer - SEO Crawl Rules

benthepythondev/robots-sitemap-analyzer

Analyze robots.txt files and discover sitemap URLs, user-agent groups, allow rules, disallow rules and crawl-delay directives.

Ben

Robots.txt Analyzer

sootesting/robots-txt-analyzer

Analyze robots.txt for any list of domains — crawl rules per user-agent, declared sitemaps, crawl-delay, and more. Outputs structured JSON. Pay-per-event.

soot

Robots.txt Validator - Check Rules, Sitemaps & Crawl Directives

scrappy_garden/robots-txt-validator

Validate robots.txt for one or more websites: fetches /robots.txt per host, parses directive groups (User-agent/Allow/Disallow/Crawl-delay/Sitemap), reports common errors and warnings, and can test URLs against the chosen User-Agent.

Bikram Adhikari

Robots.txt Analyzer

mahogany_songbird/robots-txt-analyzer

Read robots.txt disallow rules and sitemap declarations.

Britton Furness

robots.txt Parser & URL Tester

scrapeworks/robots-txt

Fetch and parse robots.txt for any site: user-agent rules, crawl-delay, and declared sitemaps. Optionally test whether specific URLs are allowed for a given user-agent, using correct longest-match rules.

Nicolas van Arkens

Robots.txt & AI Crawler Policy Auditor

merlin-hermes/robots-ai-crawler-policy-auditor

Parses a site's public robots.txt into structured crawler groups, allow/disallow counts, crawl delays, sitemap declarations, and explicit or inherited policies for major AI crawlers.

Sean Burke

Website Tech Audit Scraper: robots.txt, Sitemaps & Meta

arman-bd/website-tech-audit-scraper

Audit any site's public technical files: robots.txt rules, sitemap discovery and URL counts, ads.txt sellers, security.txt contacts, meta tags, structured data and the detected tech stack with evidence.

Arman Hossain

Robots.txt & Sitemap Analyzer 🕷️

perryay/robots-txt-sitemap-analyzer

Fetch, parse, and analyze robots.txt and sitemap.xml for any domain. Extract crawl directives, test URL compliance against robots.txt rules, and discover all URLs from sitemaps including nested sitemap indexes. Supports batch analysis with structured JSON output.