Pricing

$5.00 / 1,000 url auditeds

Robots & Sitemap Inspector

Fetch and validate robots.txt + sitemap.xml for any site. Parses directives, counts sitemap URLs, flags disallow-all mistakes, stale lastmod, missing sitemaps. A-F grade. Use for SEO audits, crawl-budget checks, migration QA. Input: url or urls[]. Output: JSON report.

Pricing

$5.00 / 1,000 url auditeds

Rating

0.0

(0)

Developer

Coleton Patton

Actor stats

Bookmarked

Total users

Monthly active users

7 days ago

Last modified

Categories

SEO tools

Developer tools

You can access the Robots & Sitemap Inspector programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in Integrations settings in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1import { ApifyClient } from 'apify-client';
2
3// Initialize the ApifyClient with your Apify API token
4// Replace the '<YOUR_API_TOKEN>' with your token
5const client = new ApifyClient({
6    token: '<YOUR_API_TOKEN>',
7});
8
9// Prepare Actor input
10const input = {
11    "url": "https://example.com"
12};
13
14// Run the Actor and wait for it to finish
15const run = await client.actor("pattonholdings/robots-sitemap-inspector").call(input);
16
17// Fetch and print Actor results from the run's dataset (if any)
18console.log('Results from dataset');
19console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
20const { items } = await client.dataset(run.defaultDatasetId).listItems();
21items.forEach((item) => {
22    console.dir(item);
23});
24
25// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

Robots.txt and Sitemap Checker API API in JavaScript

The Apify API client for JavaScript is the official library that allows you to use Robots & Sitemap Inspector API in JavaScript or TypeScript, providing convenience functions and automatic retries on errors.

Install the apify-client

$npm install apify-client

Other API clients include:

Robots & Sitemap Inspector API in Python

Robots & Sitemap Inspector API through CLI

Robots & Sitemap Inspector OpenAPI definition

Robots & Sitemap Inspector API

Sitemap URL Extractor

automationagents/web-sitemap

Extract every URL from a website via sitemap.xml, robots.txt, or crawl discovery. Feed clean URL lists straight into your scrapers.

Alex Jordan

Robots.txt & Sitemap Analyzer

automation-lab/robots-sitemap-analyzer

This actor fetches and parses robots.txt and sitemap.xml files for any list of websites. It extracts crawl directives (user-agent rules, allowed/disallowed paths, crawl-delay), discovers sitemap URLs, and counts the number of pages listed in each sitemap. Use it for SEO audits, competitive...

Stas Persiianenko

Robots.txt & Sitemap Auditor

phoenix2810/robots-sitemap-auditor

Audit a website's robots.txt directives and sitemap discovery in one fast API call. Detect blocked paths, missing sitemaps, and crawlability issues. Built for SEO agencies, technical SEO consultants, and site migration QA teams.

Sanskar Jaiswal

Sitemap URL Extractor — robots.txt + sitemap.xml Crawl

v0iddo/sitemap-url-extractor

Discover every URL a site exposes via its public sitemap chain. Reads robots.txt, follows Sitemap declarations, recursively descends sitemap-index files, extracts URLs with lastmod, changefreq, priority.

vøiddo

Sitemap URL Extractor

wiry_kingdom/sitemap-url-extractor

Extract every URL from any website's sitemap.xml with lastmod, changefreq, priority. Recursively expands sitemap index files, reads robots.txt, handles gzipped sitemaps. SEO audits, content migration, site inventory, competitor research.

Mohieldin Mohamed

XML Sitemap Scraper & URL Extractor API - SEO Crawler

pink_comic/sitemap-url-extractor

Extract URLs from XML sitemaps and robots.txt for SEO crawls, audits, content migrations, and RAG indexing. Auto-discovers sitemap files, parses nested sitemap indexes, and exports URL, lastmod, priority, changefreq, and image metadata in bulk.

Ava Torres

Robots.txt Auditor & Sitemap Finder

andok/robotstxt-auditor

Scan robots.txt files in bulk to extract sitemap URLs and verify crawler directives for technical SEO compliance.

Andok

Sitemap URL Discovery (sitemap.xml + robots.txt → all URLs)

gochujang/sitemap-url-discovery

Given a domain, finds sitemap.xml / sitemap_index.xml (also via robots.txt), recursively expands sitemap indexes, returns one row per discovered URL with lastmod / changefreq / priority. SEO audits, crawl-target prep, content cataloging. $0.0001/URL + $0.01 site fee.

Hojun Lee

Sitemap & URL Extractor — Get Every URL of a Website

dataquarry/sitemap-url-extractor

Get every URL of a website: parses sitemap.xml and sitemap-indexes (discovered via robots.txt or the default location), with a same-site crawl fallback when there's no sitemap. Returns each URL + lastmod. No API key.

Daniel Brenner

Sitemap URL Extractor: Every URL, Recursive

thoob/sitemap-extractor

Reads sitemap.xml, sitemap index files, .gz compressed sitemaps, and robots.txt Sitemap directives, and returns one clean row per URL with lastmod, changefreq, and priority. Billed only per delivered URL.

Pono Data