Pricing

from $0.80 / 1,000 results

Robots Sitemap Analyzer - SEO Crawl Rules

Analyze robots.txt files and discover sitemap URLs, user-agent groups, allow rules, disallow rules and crawl-delay directives.

Pricing

from $0.80 / 1,000 results

Rating

0.0

(0)

Developer

Ben

Actor stats

Bookmarked

Total users

Monthly active users

13 days ago

Last modified

Categories

SEO tools

Automation

Business

Apify Store pricing discounts use a tiered approach. Higher subscription plans unlock lower Actor costs. See our subscription plans.

Pricing	Free No discount	Starter Bronze discount	Scale Silver discount	Business Gold discount

Result	$1.00 / 1,000	$0.90 / 1,000	$0.85 / 1,000	$0.80 / 1,000
Actor start Infrequent	$0.00005	$0.000045	$0.000043	$0.00004
Additional costs
Platform usage	Free	Free	Free	Free

This Actor's pricing details are outlined in the table below. The unit prices you pay depend on your subscription plan, with higher tiers offering better rates. Visit our pricing page to compare plans and see how much you can save.

Pricing	Free No discount	Business Gold discount

Result Charged per robots analysis result pushed to the default dataset.	$1.00 / 1,000	$0.80 / 1,000
Actor startInfrequent Charged when the Actor starts running. Number of events charged depends on Actor memory (one event per GB, minimum one event).	$0.00005	$0.00004
Additional costs
Platform usageIncluded Platform usage costs are tied to the Actor's performance in terms of the underlying platform resources it consumes. Learn more	Free

Robots.txt Checker & Parser - Crawl Rules API

pink_comic/robots-txt-validator

Check, parse, and validate robots.txt files in bulk. Extract crawl rules, sitemaps, crawl-delay, blocked paths, and per-user-agent allow/disallow results for SEO audits and crawler compliance.

Ava Torres

Robots.txt Validator - Check Rules, Sitemaps & Crawl Directives

scrappy_garden/robots-txt-validator

Validate robots.txt for one or more websites: fetches /robots.txt per host, parses directive groups (User-agent/Allow/Disallow/Crawl-delay/Sitemap), reports common errors and warnings, and can test URLs against the chosen User-Agent.

Bikram Adhikari

robots.txt & Sitemap Analyzer

bgfc97/robots-txt-sitemap-analyzer

Fetch and parse robots.txt for many domains: user-agent groups, allow/disallow rules, crawl-delay, declared sitemaps and whether crawlers are blocked. Technical SEO auditing. No key.

Bruno

Robots.txt Analyzer

mahogany_songbird/robots-txt-analyzer

Read robots.txt disallow rules and sitemap declarations.

Britton Furness

Robots.txt & Sitemap Analyzer

automation-lab/robots-sitemap-analyzer

This actor fetches and parses robots.txt and sitemap.xml files for any list of websites. It extracts crawl directives (user-agent rules, allowed/disallowed paths, crawl-delay), discovers sitemap URLs, and counts the number of pages listed in each sitemap. Use it for SEO audits, competitive...

Stas Persiianenko

Robots.txt & Sitemap Analyzer 🕷️

perryay/robots-txt-sitemap-analyzer

Fetch, parse, and analyze robots.txt and sitemap.xml for any domain. Extract crawl directives, test URL compliance against robots.txt rules, and discover all URLs from sitemaps including nested sitemap indexes. Supports batch analysis with structured JSON output.

Perry AY

Robots.txt & Sitemap Discovery API

213x/robots-sitemap-discovery

Discover robots.txt rules, sitemap URLs, crawl delays, and basic crawlability signals for domains or website URLs.

robots.txt Analyzer — Rules, Sitemaps & Crawl Blocks

eliai/robots-txt-analyzer

Parse and audit any site's robots.txt in one call: user-agent groups, allow/disallow rules, declared sitemaps, and crawl-block issues like an accidental site-wide Disallow. $0.02 per site analyzed. No API key.

Anthony Snider

Robots.txt Auditor

junipr/robots-txt-auditor

Fetch and audit robots.txt syntax, user-agent rules, blocked paths, sitemap declarations, and crawl risks.

junipr

robots.txt Parser & URL Tester

scrapeworks/robots-txt

Fetch and parse robots.txt for any site: user-agent rules, crawl-delay, and declared sitemaps. Optionally test whether specific URLs are allowed for a given user-agent, using correct longest-match rules.