Pricing

from $5.00 / 1,000 domain analyzeds

robots.txt & Sitemap Analyzer

Fetch and parse robots.txt for many domains: user-agent groups, allow/disallow rules, crawl-delay, declared sitemaps and whether crawlers are blocked. Technical SEO auditing. No key.

Pricing

from $5.00 / 1,000 domain analyzeds

Rating

0.0

(0)

Developer

Bruno

Actor stats

Bookmarked

Total users

Monthly active users

3 days ago

Last modified

Output per domain

Whether robots.txt exists + HTTP status + size
User-agent groups with their Allow / Disallow rules
Crawl-delay
Declared Sitemaps
blocks_all_crawlers flag (Disallow: / for *)

Input

{ "domains": ["github.com", "apify.com"] }

Output (example)

{
  "domain": "apify.com",
  "exists": true,
  "user_agent_groups": 2,
  "total_rules": 14,
  "blocks_all_crawlers": false,
  "sitemaps": ["https://apify.com/sitemap.xml"]
}

Use cases

Technical SEO audits at scale
Verify your own crawl directives before launch
Discover sitemaps for further crawling
Competitor crawl-policy research

Billed per domain analyzed.

Robots.txt & Sitemap Analyzer

automation-lab/robots-sitemap-analyzer

This actor fetches and parses robots.txt and sitemap.xml files for any list of websites. It extracts crawl directives (user-agent rules, allowed/disallowed paths, crawl-delay), discovers sitemap URLs, and counts the number of pages listed in each sitemap. Use it for SEO audits, competitive...

Stas Persiianenko

Domain Age Checker

automation-lab/domain-age-checker

Check domain age and registration dates in bulk using RDAP — the modern WHOIS replacement. Returns registration date, expiration date, registrar, nameservers, and age in days/years. Ideal for SEO analysis, fraud detection, and domain research.

Stas Persiianenko

Sitemap URL Extractor

automation-lab/sitemap-url-extractor

This actor parses XML sitemaps and extracts all URLs with their metadata. It handles both regular sitemaps and sitemap indexes (recursively follows child sitemaps up to 3 levels deep). For each URL, it captures the last modified date, change frequency, priority, and whether the entry...

Stas Persiianenko

Robots Sitemap Analyzer - SEO Crawl Rules

benthepythondev/robots-sitemap-analyzer

Analyze robots.txt files and discover sitemap URLs, user-agent groups, allow rules, disallow rules and crawl-delay directives.

Ben

Sitemap URL Extractor - XML Sitemap Scraper

benthepythondev/sitemap-url-extractor

Extract URLs from XML sitemaps and sitemap indexes. Get URL, lastmod, changefreq, priority and source sitemap.

Ben

Sitemap Analyzer

eliai/sitemap-analyzer

Anthony Snider

robots.txt Analyzer

eliai/robots-txt-analyzer

Anthony Snider

OSINT Website Intelligence Analyzer

onescales/website-intelligence-analyzer-osint

All-in-one website analysis tool. Run 30 OSINT checks on any URL — DNS, SSL, WHOIS, tech stack, security headers, email security, open ports, and more. Get a complete site profile in seconds.

One Scales

118

5.0

(3)

Sitemap Structure Analyzer

onescales/sitemap-structure-analyzer

Analyze any website's sitemap in seconds using sitemap.xml data. Get URL counts by type (product, blog, docs), content freshness, URL patterns, and SEO anomalies — no page fetching required.

One Scales

5.0

(2)

🗺️ Sitemap Scraper & Analyzer

taroyamada/sitemap-analyzer

Extract thousands of URLs from complex nested XML sitemaps to audit website structure, HTTP status codes, and indexability for technical SEO.

naoki anzai

Sitemap Analyzer — Recursive Parse, Health Check, AI Tags

and_krm/sitemap-analyzer

Parse any sitemap.xml recursively, extract all URLs with metadata, check HTTP health status, and optionally cluster URLs by topic using Claude AI. Perfect for SEO audits and site migration.