Under maintenance

Pricing

Pay per event

Try for free

Go to Apify Store

Robots.txt Analyzer

Under maintenance

Try for free

Analyze robots.txt for any list of domains — crawl rules per user-agent, declared sitemaps, crawl-delay, and more. Outputs structured JSON. Pay-per-event.

Pricing

Pay per event

Rating

0.0

(0)

Developer

soot

Actor stats

Bookmarked

Total users

Monthly active users

3 days ago

Last modified

🚀 What it does

This Actor automates a task that would otherwise take hours of manual work. It is built for batch processing — run it once on a list of inputs and get a structured, downloadable report.

✨ Features

Batch processing — handle one or many inputs in a single run
Structured output — clean tables, ready for CSV/JSON export
Pay-per-event pricing — you only pay for what you use
API & CLI ready — integrate it into any workflow

📥 Input configuration

Configure the Actor with these fields:

Field	Type	Required	Description
`startUrls`	array	✅ Required	List of root URLs or domains to analyze (e.g. "https://example.com"). One output row per domain.

💡 Usage example

Run via Apify API (cURL)

curl -X POST "https://api.apify.com/v2/acts/sootesting~robots-txt-analyzer/runs?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
  "startUrls": [
    "https://example.com"
  ]
}'

Run via Apify CLI

apify run --actor-name sootesting~robots-txt-analyzer --input '{
  "startUrls": [
    "https://example.com"
  ]
}'

📤 Output schema

The Actor produces the following structured data:

Field	Type	Description
`url`	string	url
`root`	string	root
`robotsUrl`	string	robotsUrl
`found`	boolean	found
`status`	integer	status
`allowsCrawl`	boolean	allowsCrawl
`blocksAll`	boolean	blocksAll
`groupCount`	integer	groupCount
`sitemaps`	array	sitemaps
`groups`	object	groups
`warnings`	array	warnings

💰 Pricing

Pay-per-event. See the Actor's pricing tab for current rates. You are only charged for successful events — platform usage is passed through transparently.

Part of the Sootesting SEO & Data Toolkit — a suite of focused actors for technical SEO, security auditing, and data extraction.

Robots.txt Analyzer

mahogany_songbird/robots-txt-analyzer

Read robots.txt disallow rules and sitemap declarations.

Britton Furness

robots.txt Analyzer — Rules, Sitemaps & Crawl Blocks

eliai/robots-txt-analyzer

Parse and audit any site's robots.txt in one call: user-agent groups, allow/disallow rules, declared sitemaps, and crawl-block issues like an accidental site-wide Disallow. $0.02 per site analyzed. No API key.

Broke to Built

Robots Txt Analyzer

zerobreak/robots-txt-analyzer

Robots txt analyzer that fetches and parses crawl rules from any website in bulk, so SEO teams and developers can audit blocked paths, user agents, and sitemap locations across hundreds of domains without manual work.

ZeroBreak

Robots Sitemap Analyzer - SEO Crawl Rules

benthepythondev/robots-sitemap-analyzer

Analyze robots.txt files and discover sitemap URLs, user-agent groups, allow rules, disallow rules and crawl-delay directives.

Ben

Robots.txt & Sitemap Analyzer

automation-lab/robots-sitemap-analyzer

This actor fetches and parses robots.txt and sitemap.xml files for any list of websites. It extracts crawl directives (user-agent rules, allowed/disallowed paths, crawl-delay), discovers sitemap URLs, and counts the number of pages listed in each sitemap. Use it for SEO audits, competitive...

Stas Persiianenko

Robots.txt & Sitemap Analyzer 🕷️

perryay/robots-txt-sitemap-analyzer

Fetch, parse, and analyze robots.txt and sitemap.xml for any domain. Extract crawl directives, test URL compliance against robots.txt rules, and discover all URLs from sitemaps including nested sitemap indexes. Supports batch analysis with structured JSON output.

Perry AY

robots.txt & Sitemap Analyzer

bgfc97/robots-txt-sitemap-analyzer

Fetch and parse robots.txt for many domains: user-agent groups, allow/disallow rules, crawl-delay, declared sitemaps and whether crawlers are blocked. Technical SEO auditing. No key.

Bruno

Robots & Sitemap Analyzer

ib4ngz/robots-sitemap-analyzer

Analyze robots.txt and XML sitemaps into normalized crawlability intelligence.

Iqbal R

Domain Age Checker

automation-lab/domain-age-checker

Check domain age and registration dates in bulk using RDAP — the modern WHOIS replacement. Returns registration date, expiration date, registrar, nameservers, and age in days/years. Ideal for SEO analysis, fraud detection, and domain research.

Stas Persiianenko

4.0

(1)

Sitemap URL Extractor

automation-lab/sitemap-url-extractor

This actor parses XML sitemaps and extracts all URLs with their metadata. It handles both regular sitemaps and sitemap indexes (recursively follows child sitemaps up to 3 levels deep). For each URL, it captures the last modified date, change frequency, priority, and whether the entry...

Stas Persiianenko