Pricing

$10.00 / 1,000 audits

Try for free

Go to Apify Store

Sitemap & Robots.txt Checker — URLs, Errors, Rules · $0.01/site

Try for free

Check a website's robots.txt and XML sitemaps in one run. Lists blocked paths and sitemap files, counts URLs, finds duplicates, off-host and non-HTTPS entries, missing lastmod and broken listed pages, and returns a ranked list of issues. Clean JSON for SEO audits. Failed runs are not charged.

Pricing

$10.00 / 1,000 audits

Rating

0.0

(0)

Developer

Ambolt

Actor stats

Bookmarked

Total users

Monthly active users

18 days ago

Last modified

Categories

Agents

SEO tools

You can access the Sitemap & Robots.txt Checker — URLs, Errors, Rules · $0.01/site programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1from apify_client import ApifyClient
2
3# Initialize the ApifyClient with your Apify API token
4# Replace '<YOUR_API_TOKEN>' with your token.
5client = ApifyClient("<YOUR_API_TOKEN>")
6
7# Prepare the Actor input
8run_input = {
9    "site": "https://www.sitemaps.org",
10    "sampleUrls": 3,
11}
12
13# Run the Actor and wait for it to finish
14run = client.actor("ambolt/sitemap-robots-doctor").call(run_input=run_input)
15
16# Fetch and print Actor results from the run's dataset (if there are any)
17print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
18for item in client.dataset(run.default_dataset_id).iterate_items():
19    print(item)
20
21# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

Sitemap & Robots.txt Checker — URLs, Errors, Rules ·… API in Python

The Apify API client for Python is the official library that allows you to use Sitemap & Robots.txt Checker — URLs, Errors, Rules · $0.01/site API in Python, providing convenience functions and automatic retries on errors.

Install the apify-client

$pip install apify-client

Other API clients include:

Sitemap & Robots.txt Checker — URLs, Errors, Rules · $0.01/site API in JavaScript

Sitemap & Robots.txt Checker — URLs, Errors, Rules · $0.01/site API through CLI

Sitemap & Robots.txt Checker — URLs, Errors, Rules · $0.01/site OpenAPI definition

Sitemap & Robots.txt Checker — URLs, Errors, Rules · $0.01/site API

Sitemap Extractor

apify/sitemap-extractor

This Apify Actor extracts all URLs from a website's sitemaps and checks their status codes via lightweight HTTP requests. It provides a clean list of valid links, acting as an ideal pre-processor to ensure your larger crawling projects target only active URLs.

Apify

403

3.2

(8)

Robots.txt & Sitemap Analyzer

automation-lab/robots-sitemap-analyzer

This actor fetches and parses robots.txt and sitemap.xml files for any list of websites. It extracts crawl directives (user-agent rules, allowed/disallowed paths, crawl-delay), discovers sitemap URLs, and counts the number of pages listed in each sitemap. Use it for SEO audits, competitive...

Automation Lab

Website Sitemap Extractor

glassventures/website-sitemap-extractor

Extract all URLs from any website's sitemap. Auto-discovers sitemaps from robots.txt, supports sitemap index files and .gz compression. Filter by URL pattern, date range.

Glass Ventures

Robots.txt Checker & Parser - Crawl Rules API

pink_comic/robots-txt-validator

Check, parse, and validate robots.txt files in bulk. Extract crawl rules, sitemaps, crawl-delay, blocked paths, and per-user-agent allow/disallow results for SEO audits and crawler compliance.

Ava Torres

Sitemap Generator

alizarin_refrigerator-owner/sitemap-generator

Generate XML sitemaps by crawling any website. Discover all pages, images, & videos with configurable crawl depth, URL filters, & multiple output formats. Full Site Crawling ,Image Sitemap, Video Sitemap, Multiple Output Formats, URL Filtering, Configurable Depth, Last Modified, Webhook Integration

The Howlers

Broken Link Checker

jancurn/find-broken-links

Crawls a website and finds broken links. Unlike other similar SEO analysis tools, the actor also reports broken URL #fragments. The results are stored in a JSON and HTML report.

Jan Čurn

808

Robots.txt & Sitemap Analyzer 🕷️

perryay/robots-txt-sitemap-analyzer

Fetch, parse, and analyze robots.txt and sitemap.xml for any domain. Extract crawl directives, test URL compliance against robots.txt rules, and discover all URLs from sitemaps including nested sitemap indexes. Supports batch analysis with structured JSON output.

Perry AY

Small Sitemap Inventory

sapph1re/small-sitemap-inventory

Build a bounded same-host URL inventory from small public HTTPS sitemaps with explicit completeness and source receipts.

Roman V

robots.txt & Sitemap Analyzer

bgfc97/robots-txt-sitemap-analyzer

Fetch and parse robots.txt for many domains: user-agent groups, allow/disallow rules, crawl-delay, declared sitemaps and whether crawlers are blocked. Technical SEO auditing. No key.

Bruno

Sitemap URL Extractor

gifted_wagon/sitemap-url-inventory

Traverse public sitemap indexes into a deduplicated URL inventory with source sitemap, lastmod, changefreq, priority, media counts, and partial-failure evidence.