Pricing

$20.00 / 1,000 robots.txt analyzeds

Go to Apify Store

robots.txt Analyzer

Try for free

Pricing

$20.00 / 1,000 robots.txt analyzeds

Rating

0.0

(0)

Developer

Anthony Snider

Actor stats

Bookmarked

Total users

Monthly active users

a day ago

Last modified

What it does

Fetches and parses /robots.txt
Lists user-agent groups and their allow/disallow rules
Extracts all declared Sitemap URLs
Flags issues: no sitemap declared, Disallow: / for * (blocks the whole site), missing/HTML robots.txt

Input

{ "url": "https://example.com" }

or bulk:

{ "urls": ["https://a.com", "https://b.com"], "maxUrls": 25 }

Output (per site)

{
  "url": "https://example.com",
  "robotsUrl": "https://example.com/robots.txt",
  "hasRobots": true,
  "userAgentGroups": 3,
  "totalRules": 12,
  "sitemaps": ["https://example.com/sitemap.xml"],
  "blocksAllCrawlers": false,
  "issues": [],
  "summary": "3 groups, 1 sitemap(s), 0 issue(s)"
}

Use cases

SEO audits, crawl-control reviews, and catching accidental site-wide blocks — by hand or wired into an agent. Pairs with our Sitemap Analyzer and SEO & Tech Auditor.

Robots Txt Analyzer

zerobreak/robots-txt-analyzer

Robots txt analyzer that fetches and parses crawl rules from any website in bulk, so SEO teams and developers can audit blocked paths, user agents, and sitemap locations across hundreds of domains without manual work.

ZeroBreak

Robots.txt & Sitemap Analyzer

automation-lab/robots-sitemap-analyzer

This actor fetches and parses robots.txt and sitemap.xml files for any list of websites. It extracts crawl directives (user-agent rules, allowed/disallowed paths, crawl-delay), discovers sitemap URLs, and counts the number of pages listed in each sitemap. Use it for SEO audits, competitive...

Stas Persiianenko

Domain Age Checker

automation-lab/domain-age-checker

Check domain age and registration dates in bulk using RDAP — the modern WHOIS replacement. Returns registration date, expiration date, registrar, nameservers, and age in days/years. Ideal for SEO analysis, fraud detection, and domain research.

Stas Persiianenko

Sitemap URL Extractor

automation-lab/sitemap-url-extractor

This actor parses XML sitemaps and extracts all URLs with their metadata. It handles both regular sitemaps and sitemap indexes (recursively follows child sitemaps up to 3 levels deep). For each URL, it captures the last modified date, change frequency, priority, and whether the entry...

Stas Persiianenko

Redirect Chain Analyzer

automation-lab/redirect-chain-analyzer

This actor follows HTTP redirects step-by-step and reports the full chain for any list of URLs. It also tests HTTP-to-HTTPS and www/non-www variants to verify canonical redirects are properly configured. Use it for SEO migration audits, redirect debugging, or monitoring redirect health.

Stas Persiianenko

Sitemap Diff Tool

automation-lab/sitemap-diff-tool

Compare two XML sitemaps and find added, removed, or changed URLs. Detects lastmod, priority, and changefreq changes. Supports sitemap index files. Export results as JSON, CSV, or Excel.

Stas Persiianenko

Robots.txt Checker & Parser - Crawl Rules API

pink_comic/robots-txt-validator

Check, parse, and validate robots.txt files in bulk. Extract crawl rules, sitemaps, crawl-delay, blocked paths, and per-user-agent allow/disallow results for SEO audits and crawler compliance.

Ava Torres

Sitemap Analyzer

eliai/sitemap-analyzer

Anthony Snider

Robots.txt Auditor & Sitemap Finder

andok/robotstxt-auditor

Scan robots.txt files in bulk to extract sitemap URLs and verify crawler directives for technical SEO compliance.

Andok

Website Metadata Extractor(sitemap, socialLinks, robotsTxt)

codescraper/website-metadata-extractor

A very fast metadata extractor to get all meta tags, robots.txt, sitemaps, social links, H1s, word count, and JSON-LD data. Also provides technology detection for a full analysis. Get your data fast for just $3/month.