Pricing

from $0.70 / 1,000 item scrapeds

Try for free

Go to Apify Store

Robots.txt & XML Sitemap Discrepancy Checker

Try for free

Audit robots.txt directives (Disallow, Allow, Crawl-delay) against XML Sitemap URLs. Identifies blocked sitemap URLs, missing sitemaps, and indexing discrepancies.

Pricing

from $0.70 / 1,000 item scrapeds

Rating

0.0

(0)

Developer

LiteZero

Actor stats

Bookmarked

Total users

Monthly active users

6 days ago

Last modified

🤖 Robots.txt & XML Sitemap Discrepancy Checker

Audit technical SEO compliance by cross-checking robots.txt directives (Disallow, Allow, User-agent, Crawl-delay) against XML Sitemap URLs. Instantly spot critical indexing errors such as URLs listed in your sitemap that are blocked from search engine crawlers by robots.txt.

🥊 Why This Actor Beats the Competition

Feature	This Actor (Pay-Per-Result)	Enterprise SEO Suites (Ahrefs, Semrush)	Manual Audit / Spreadsheets
Cost	💰 $1.00 / 1,000 URLs checked	$200–$500/month recurring fees	High labor cost & error-prone
Speed	⚡ Under 15 seconds per audit	Hours of site-wide crawler processing	Slow manual pattern matching
API Integration	✅ Direct REST API, Python & Node.js SDKs	Complex web dashboards	None
Discrepancy Detection	✅ Identifies blocked sitemap URLs automatically	Buried in technical crawl error logs	Manual regex required

🛠️ Step-by-Step: How to Use

Method 1: Apify Web Console

Enter your domain (e.g. https://apify.com or apify.com) in Domains or Website URLs.
Set Max Sitemap URLs to Audit (e.g. 30).
Click Start to run the check.

Method 2: Python SDK

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("litezero/robots-txt-sitemap-checker").call(run_input={
    "domainsOrUrls": ["https://apify.com"],
    "maxUrlsToAudit": 50
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    if item["isBlockedByRobots"]:
        print(f"CRITICAL DISCREPANCY: {item['url']} is in sitemap but blocked by: {item['matchedRule']}")

📥 Input Parameters

Parameter	Type	Default	Description
`domainsOrUrls`	Array	`["https://apify.com"]`	Domains or website homepages to audit.
`maxUrlsToAudit`	Integer	`30`	Maximum sitemap URLs to check against robots.txt rules.

📤 Output Sample JSON

{
  "domain": "https://apify.com",
  "url": "https://apify.com/store",
  "isBlockedByRobots": false,
  "matchedRule": "None (Allowed by default)",
  "hasSitemapInRobots": true,
  "sitemapsFound": ["https://apify.com/sitemap.xml"],
  "disallowRulesCount": 8,
  "allowRulesCount": 2,
  "robotsTxtUrl": "https://apify.com/robots.txt",
  "auditedAt": "2026-10-03T09:20:00.000Z"
}

💡 Top Commercial Use Cases

Pre-Launch Technical SEO QA: Ensure dev/staging Disallow rules were removed before production deployment.
Crawl Budget Optimization: Confirm high-priority sitemap landing pages are 100% crawlable by Googlebot.
Agency Client Audits: Deliver automated robots.txt vs sitemap audit reports in client discovery phases.\n

Sitemap Extractor

apify/sitemap-extractor

This Apify Actor extracts all URLs from a website's sitemaps and checks their status codes via lightweight HTTP requests. It provides a clean list of valid links, acting as an ideal pre-processor to ensure your larger crawling projects target only active URLs.

Apify

403

3.2

(8)

Robots.txt & Sitemap Analyzer

automation-lab/robots-sitemap-analyzer

This actor fetches and parses robots.txt and sitemap.xml files for any list of websites. It extracts crawl directives (user-agent rules, allowed/disallowed paths, crawl-delay), discovers sitemap URLs, and counts the number of pages listed in each sitemap. Use it for SEO audits, competitive...

Automation Lab

Website Sitemap Extractor

glassventures/website-sitemap-extractor

Extract all URLs from any website's sitemap. Auto-discovers sitemaps from robots.txt, supports sitemap index files and .gz compression. Filter by URL pattern, date range.

Glass Ventures

Sitemap Generator

datawinder/sitemap-generator

Automatically crawl a website and generate an SEO-ready sitemap in XML, HTML, or TXT format. Supports crawl depth limits, URL include/exclude patterns, and optional merging with an existing sitemap.xml. Ideal for SEO audits, site migrations, and automation.

DatawinderLabs

Sitemap Change Detector

tri_angle/sitemap-change-detector

Identify and monitor sitemaps for specified websites. Retrieve only the new, updated, or removed URLs since the last crawl.

Tri⟁angle

robots.txt & Sitemap Analyzer

bgfc97/robots-txt-sitemap-analyzer

Fetch and parse robots.txt for many domains: user-agent groups, allow/disallow rules, crawl-delay, declared sitemaps and whether crawlers are blocked. Technical SEO auditing. No key.

Bruno

Robots.txt Checker & Parser - Crawl Rules API

pink_comic/robots-txt-validator

Check, parse, and validate robots.txt files in bulk. Extract crawl rules, sitemaps, crawl-delay, blocked paths, and per-user-agent allow/disallow results for SEO audits and crawler compliance.

Ava Torres

Robots.txt & Sitemap Analyzer 🕷️

perryay/robots-txt-sitemap-analyzer

Fetch, parse, and analyze robots.txt and sitemap.xml for any domain. Extract crawl directives, test URL compliance against robots.txt rules, and discover all URLs from sitemaps including nested sitemap indexes. Supports batch analysis with structured JSON output.

Perry AY

Sitemap Generator

igview-owner/sitemap-generator

Automatically crawl any website and generate XML, HTML, and text sitemaps for SEO optimization. Perfect for submitting to Google Search Console, Bing Webmaster Tools, and improving search engine indexing. no manual work required. Free sitemap generator tool for WordPress, Blogger, and all website.

Sachin Kumar Yadav

Sitemap URL Extractor — recursive, filters, .gz

x402opklaar/sitemap-url-extractor

Extract every URL from any sitemap: recursive sitemap-index expansion, gzipped sitemaps, robots.txt auto-discovery, include/exclude filters, lastmod output. Feed the URL list to any crawler or audit. Pay per sitemap file processed.