Pricing

from $0.005 / actor start

Robots.txt & Sitemap Analyzer 🕷️

Fetch, parse, and analyze robots.txt and sitemap.xml for any domain. Extract crawl directives, test URL compliance against robots.txt rules, and discover all URLs from sitemaps including nested sitemap indexes. Supports batch analysis with structured JSON output.

Pricing

from $0.005 / actor start

Rating

0.0

(0)

Developer

Perry AY

Actor stats

Bookmarked

Total users

Monthly active users

3 days ago

Last modified

Categories

Developer tools

Domain

domain

Optional

Single domain to analyze (e.g., example.com). If batchMode is enabled, use the 'domains' array instead.

Type:string

Domains (Batch)

domains

Optional

List of domains for batch analysis. When provided with batchMode: true, all domains are processed in a single run.

Type:array

Max. items:20

Analyze robots.txt

analyzeRobotsTxt

Optional

Fetch and parse robots.txt — extract crawl rules, sitemap directives, and test URL compliance.

Type:boolean

Default:true

Analyze sitemap

analyzeSitemap

Optional

Fetch and parse sitemap.xml — discover URLs, detect sitemap indexes, and extract child sitemap locations.

Type:boolean

Default:true

Test URLs

testUrls

Optional

Optional list of URL paths to test against robots.txt rules. Each URL is checked for Allow/Disallow compliance. Relative paths are resolved against the domain.

Type:array

Extract sitemap URLs

extractSitemapUrls

Optional

If true, extract all URLs from sitemap(s) including resolving sitemap indexes. If false, only report sitemap metadata and URL count.

Type:boolean

Default:true

Batch Mode

batchMode

Optional

Enable batch processing for multiple domains. When enabled, provide domains via the 'domains' array field.

Type:boolean

Default:false

Sitemap URL Extractor - XML Sitemap Scraper

benthepythondev/sitemap-url-extractor

Extract URLs from XML sitemaps and sitemap indexes. Get URL, lastmod, changefreq, priority and source sitemap.

Ben

SEO Intelligence Suite - Complete Analysis with AI

viralanalyzer/seo-intelligence-suite

Complete SEO audit: meta tags, headings, links, structured data, AI recommendations.

viralanalyzer

5.0

(4)

Robots.txt & Sitemap Analyzer

automation-lab/robots-sitemap-analyzer

This actor fetches and parses robots.txt and sitemap.xml files for any list of websites. It extracts crawl directives (user-agent rules, allowed/disallowed paths, crawl-delay), discovers sitemap URLs, and counts the number of pages listed in each sitemap. Use it for SEO audits, competitive...

Stas Persiianenko

robots.txt & Sitemap Analyzer

bgfc97/robots-txt-sitemap-analyzer

Fetch and parse robots.txt for many domains: user-agent groups, allow/disallow rules, crawl-delay, declared sitemaps and whether crawlers are blocked. Technical SEO auditing. No key.

Bruno

Domain Age Checker

automation-lab/domain-age-checker

Check domain age and registration dates in bulk using RDAP — the modern WHOIS replacement. Returns registration date, expiration date, registrar, nameservers, and age in days/years. Ideal for SEO analysis, fraud detection, and domain research.

Stas Persiianenko

Sitemap URL Extractor

automation-lab/sitemap-url-extractor

This actor parses XML sitemaps and extracts all URLs with their metadata. It handles both regular sitemaps and sitemap indexes (recursively follows child sitemaps up to 3 levels deep). For each URL, it captures the last modified date, change frequency, priority, and whether the entry...

Stas Persiianenko

Robots Sitemap Analyzer - SEO Crawl Rules

benthepythondev/robots-sitemap-analyzer

Analyze robots.txt files and discover sitemap URLs, user-agent groups, allow rules, disallow rules and crawl-delay directives.

Ben

Sitemap Analyzer

eliai/sitemap-analyzer

Anthony Snider

robots.txt Analyzer

eliai/robots-txt-analyzer

Anthony Snider

OSINT Website Intelligence Analyzer

onescales/website-intelligence-analyzer-osint

All-in-one website analysis tool. Run 30 OSINT checks on any URL — DNS, SSL, WHOIS, tech stack, security headers, email security, open ports, and more. Get a complete site profile in seconds.