This Actor‘s pricing details are outlined in the table below. The unit prices you pay depend on your Apify subscription plan, with higher tiers offering better rates. Visit our pricing page to compare plans and see how much you can save.

Pricing	Free No discount	Starter Bronze discount	Scale Silver discount	Business Gold discount

Inspected domain	$2.00 / 1,000	$2.00 / 1,000	$2.00 / 1,000	$2.00 / 1,000
Actor start Infrequent	$0.005	$0.005	$0.005	$0.005
Additional costs
Platform usage	Free	Free	Free	Free

This Actor's pricing details are outlined in the table below. Visit our pricing page to compare plans and see how much you can save.

Pricing

Inspected domain One domain or website URL inspected for robots.txt and sitemaps.	$2.00 / 1,000
Actor startInfrequent Charged when the Actor starts running. Number of events charged depends on Actor memory (one event per GB, minimum one event).	$0.005
Additional costs
Platform usageIncluded Platform usage costs are tied to the Actor's performance in terms of the underlying platform resources it consumes. Learn more	Free

Robots.txt & Sitemap Analyzer 🕷️

perryay/robots-txt-sitemap-analyzer

Fetch, parse, and analyze robots.txt and sitemap.xml for any domain. Extract crawl directives, test URL compliance against robots.txt rules, and discover all URLs from sitemaps including nested sitemap indexes. Supports batch analysis with structured JSON output.

Perry AY

Robots.txt Analyzer

mahogany_songbird/robots-txt-analyzer

Read robots.txt disallow rules and sitemap declarations.

Britton Furness

Robots.txt and Sitemap.xml Change Monitor

eager_vitamin/robots-sitemap-change-monitor

Checks public robots.txt and sitemap.xml metadata for crawlability regressions, URL drops, and changed crawler directives.

Lau Kuan Ee

Sitemap URL Intelligence

toronto_777/sitemap-url-intelligence

Discover robots.txt sitemap entries and classify public sitemap URLs by page type.

Steven Feng

Robots Sitemap Analyzer - SEO Crawl Rules

benthepythondev/robots-sitemap-analyzer

Analyze robots.txt files and discover sitemap URLs, user-agent groups, allow rules, disallow rules and crawl-delay directives.

Ben

Robots.txt Auditor & Sitemap Finder

andok/robotstxt-auditor

Scan robots.txt files in bulk to extract sitemap URLs and verify crawler directives for technical SEO compliance.

Andok

Sitemap & Robots SEO Index Auditor

glowing_glove/sitemap-robots-index-auditor

Audit robots.txt, sitemap discovery, indexability signals, canonical tags, and SEO crawl readiness for business websites.

Ushba Khan

Robots.txt Auditor

junipr/robots-txt-auditor

Fetch and audit robots.txt syntax, user-agent rules, blocked paths, sitemap declarations, and crawl risks.

junipr

Sitemap URL Extractor

lnlenost/sitemap-url-extractor

Extract page URLs from robots.txt and sitemap.xml files for SEO audits, URL discovery crawl planning, and data pipelines.

Niccolò Salerno

Sitemap URL Extractor — robots.txt + sitemap.xml Crawl

v0iddo/sitemap-url-extractor

Discover every URL a site exposes via its public sitemap chain. Reads robots.txt, follows Sitemap declarations, recursively descends sitemap-index files, extracts URLs with lastmod, changefreq, priority.