Pricing

$2.99/month + usage

Robots Txt Analyzer

Robots txt analyzer that fetches and parses crawl rules from any website in bulk, so SEO teams and developers can audit blocked paths, user agents, and sitemap locations across hundreds of domains without manual work.

Pricing

$2.99/month + usage

Rating

0.0

(0)

Developer

ZeroBreak

Actor stats

Bookmarked

Total users

Monthly active users

4 days ago

Last modified

Categories

SEO tools

This Actor‘s pricing details are outlined in the table below. The unit prices you pay depend on your Apify subscription plan, with higher tiers offering better rates. Visit our pricing page to compare plans and see how much you can save.

Pricing	Free No discount	Starter Bronze discount	Scale Silver discount	Business Gold discount
3 days trial	Free	Free	Free	Free
Price	$2.99 / month	$2.99 / month	$2.99 / month	$2.99 / month
Additional costs
Compute units (CU)	$0.30 / CU	$0.30 / CU	$0.25 / CU	$0.20 / CU
Actor RAM	8 GB	32 GB	128 GB	256 GB
Max concurrent runs	25	32	128	256
Rented Actors		Deducted from prepaid usage	Deducted from prepaid usage	Deducted from prepaid usage
Residential proxies	$8.00 / GB	$8.00 / GB	$7.50 / GB	$7.00 / GB
Datacenter proxies	5 IPs included	30 IPs included then $1.00 / IP	200 IPs included then $0.80 / IP	500 IPs included then $0.60 / IP
SERPs proxy	$2.50 / 1,000 SERPs	$2.50 / 1,000 SERPs	$2.00 / 1,000 SERPs	$1.70 / 1,000 SERPs

This Actor's pricing details are outlined in the table below. The unit prices you pay depend on your Apify subscription plan, with higher tiers offering better rates. Visit our pricing page to compare plans and see how much you can save.

3 days trial	Free
Price To use this Actor, you have to pay a monthly rental fee to the developer. The rent is subtracted from your prepaid usage every month after the free trial period.	$2.99 / month
Additional costs
Platform usage Platform usage costs are tied to the Actor's performance in terms of the underlying platform resources it consumes. Learn more	Variable costs cheaper for higher plans.

Robots.txt & Sitemap Analyzer

automation-lab/robots-sitemap-analyzer

This actor fetches and parses robots.txt and sitemap.xml files for any list of websites. It extracts crawl directives (user-agent rules, allowed/disallowed paths, crawl-delay), discovers sitemap URLs, and counts the number of pages listed in each sitemap. Use it for SEO audits, competitive...

Stas Persiianenko

Robots.txt Validator - Check Rules, Sitemaps & Crawl Directives

scrappy_garden/robots-txt-validator

Validate robots.txt for one or more websites: fetches /robots.txt per host, parses directive groups (User-agent/Allow/Disallow/Crawl-delay/Sitemap), reports common errors and warnings, and can test URLs against the chosen User-Agent.

Bikram Adhikari

Robots.txt Auditor & Sitemap Finder

andok/robotstxt-auditor

Scan robots.txt files in bulk to extract sitemap URLs and verify crawler directives for technical SEO compliance.

Andok

Indexability Audit

zerobreak/indexability-audit

Indexability audit tool that checks robots.txt, meta robots tags, X-Robots-Tag headers, and canonical URLs for any list of pages, so SEO teams know which ones Google can actually crawl and index.

ZeroBreak

Robots.txt Validator

predictable_function/my-actor-3

List of website base URLs whose robots.txt files will be validated

riya rawat

5.0

Robots.txt Checker - CMS-Aware Analysis with AI Recommendations

alizarin_refrigerator-owner/robots-txt-checker

The Robots.txt Checker provides comprehensive analysis of your robots.txt file: Syntax Validation CMS Detection - Identify WordPress, Shopify, Drupal,& 6+ other CMS platforms Best Practice Check Companion File Checks - sitemap.xml, llms.txt, security.txt AI Recommendations - CMS-specific suggestions

The Howlers

Fast Sitemap Generator

eunit/sitemap-generator

Boost SEO with this automatic Sitemap Generator. Crawl any site to create XML, HTML, & TXT sitemaps. Supports custom depth, regex filters, & robots.txt. Compatible with Google Search Console.

Emmanuel Uchenna

5.0

Website Metadata Extractor (meta tags, sitemap, robots) 🔎

powerful_bachelor/website-metadata-extractor

🔍 Website Metadata Extractor 🌐 Extract essential website data: meta tags, robots.txt, and sitemap.xml in one scan. 📊 Analyze SEO elements, crawler directives, and site structure. ✅ Perfect for SEO audits, 🔎 competitor research, and 🚀 understanding how search engines view your website.

Powerful Bachelor

Find Sitemap from url

eesti/find-sitemap-from-url

A powerful [Apify Actor] that finds sitemap URLs for any website. This Actor helps you discover XML sitemaps by checking common locations, robots.txt files, and analyzing HTML content for sitemap links.

ando

206

1.0

Robots Txt Audit

apage/robots-txt-audit

Audit robots.txt files for AI crawler access. Get an AI Readiness Score (0-100), analyze 61+ AI crawlers (ChatGPT, Claude, Perplexity, Gemini), detect syntax errors, security concerns, and get actionable recommendations. Batch audit multiple domains at once with optional subdomain scanning.