Pricing

from $1.30 / 1,000 robots.txt & sitemap audits

Try for free

Go to Apify Store

Robots.txt & XML Sitemap Deep Analyzer

Try for free

Parses robots.txt crawl rules and XML sitemaps. Detects disallowed paths, sitemap locations, AI web scraper permissions (GPTBot, ClaudeBot, CCBot), and index URL counts.

Pricing

from $1.30 / 1,000 robots.txt & sitemap audits

Rating

0.0

(0)

Developer

David Sandor

Actor stats

Bookmarked

Total users

Monthly active users

21 hours ago

Last modified

Categories

Lead generation

MCP servers

You can access the Robots.txt & XML Sitemap Deep Analyzer programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1import { ApifyClient } from 'apify-client';
2
3// Initialize the ApifyClient with your Apify API token
4// Replace the '<YOUR_API_TOKEN>' with your token
5const client = new ApifyClient({
6    token: '<YOUR_API_TOKEN>',
7});
8
9// Prepare Actor input
10const input = {
11    "domainOrUrl": "https://github.com",
12    "parseSitemap": true,
13    "includeKeywords": "",
14    "excludeKeywords": "",
15    "enableRagEnrichment": true,
16    "outputFormat": "JSON"
17};
18
19// Run the Actor and wait for it to finish
20const run = await client.actor("trusted_tech_spot/robots-txt-sitemap-crawler").call(input);
21
22// Fetch and print Actor results from the run's dataset (if any)
23console.log('Results from dataset');
24console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
25const { items } = await client.dataset(run.defaultDatasetId).listItems();
26items.forEach((item) => {
27    console.dir(item);
28});
29
30// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

Robots.txt & XML Sitemap Deep Audit Analyzer API in JavaScript

The Apify API client for JavaScript is the official library that allows you to use Robots.txt & XML Sitemap Deep Analyzer API in JavaScript or TypeScript, providing convenience functions and automatic retries on errors.

Install the apify-client

$npm install apify-client

Other API clients include:

Robots.txt & XML Sitemap Deep Analyzer API in Python

Robots.txt & XML Sitemap Deep Analyzer API through CLI

Robots.txt & XML Sitemap Deep Analyzer OpenAPI definition

Robots.txt & XML Sitemap Deep Analyzer API

Sitemap Detector

coder_zoro/sitemap-detector

Find sitemap URLs fast with our free Sitemap Finder tool. Instantly detect sitemaps from any website for SEO audits, indexing checks, and crawl planning. Improve visibility, site structure insights, and search engine performance in just seconds

Zoro

182

5.0

(3)

Sitemap Generator - Crawl Website & Create XML Sitemap

scrappy_garden/sitemap-generator

Generate an XML sitemap for any website. Crawls internal pages from start URLs (with depth + page limits), deduplicates URLs, and stores a ready-to-submit sitemap.xml plus a structured dataset and summary for SEO audits.

Bikram Adhikari

Robots.txt & Sitemap Analyzer

automation-lab/robots-sitemap-analyzer

This actor fetches and parses robots.txt and sitemap.xml files for any list of websites. It extracts crawl directives (user-agent rules, allowed/disallowed paths, crawl-delay), discovers sitemap URLs, and counts the number of pages listed in each sitemap. Use it for SEO audits, competitive...

Automation Lab

XML Sitemap Finder & Extractor API

andok/find-sitemap-from-url

Find and extract all XML sitemaps for any domain. Automatically parses robots.txt, scans HTML tags, and recursively follows indexes. Perfect for SEO & web scraping.

Andok

Expired Domains Finder — GoDaddy Auctions with SEO Metrics

hipersoft/godaddy-auctions-scraper

Find expired and auction domains from GoDaddy Auctions with price, bids, age, valuation, Majestic Trust/Citation Flow, backlinks, referring domains, SEMrush Authority Score, keyword search volume and CPC. Filter by TLD, keyword, price, age and SEO metrics. Export CSV/JSON.

hiper soft

robots.txt & Sitemap Analyzer

bgfc97/robots-txt-sitemap-analyzer

Fetch and parse robots.txt for many domains: user-agent groups, allow/disallow rules, crawl-delay, declared sitemaps and whether crawlers are blocked. Technical SEO auditing. No key.

Bruno

Robots.txt Checker & Parser - Crawl Rules API

pink_comic/robots-txt-validator

Check, parse, and validate robots.txt files in bulk. Extract crawl rules, sitemaps, crawl-delay, blocked paths, and per-user-agent allow/disallow results for SEO audits and crawler compliance.

Ava Torres

Robots.txt & Sitemap Analyzer 🕷️

perryay/robots-txt-sitemap-analyzer

Fetch, parse, and analyze robots.txt and sitemap.xml for any domain. Extract crawl directives, test URL compliance against robots.txt rules, and discover all URLs from sitemaps including nested sitemap indexes. Supports batch analysis with structured JSON output.

Perry AY

Sitemap Generator

alizarin_refrigerator-owner/sitemap-generator

Generate XML sitemaps by crawling any website. Discover all pages, images, & videos with configurable crawl depth, URL filters, & multiple output formats. Full Site Crawling ,Image Sitemap, Video Sitemap, Multiple Output Formats, URL Filtering, Configurable Depth, Last Modified, Webhook Integration