This Actor‘s pricing details are outlined in the table below. The unit prices you pay depend on your Apify subscription plan, with higher tiers offering better rates. Visit our pricing page to compare plans and see how much you can save.

Pricing	Free No discount	Starter Bronze discount	Scale Silver discount	Business Gold discount
Comparison done	$2.00 / 1,000	$2.00 / 1,000	$2.00 / 1,000	$2.00 / 1,000
Actor Start Infrequent	$0.00005	$0.00005	$0.00005	$0.00005
Additional costs
Platform usage	Free	Free	Free	Free

This Actor's pricing details are outlined in the table below. Visit our pricing page to compare plans and see how much you can save.

Pricing
Comparison done Charged when the Actor done comparing changes between previous result with current	$2.00 / 1,000
Actor StartInfrequent Charged when the Actor starts running. Number of events charged depends on Actor memory (one event per GB, minimum one event).	$0.00005
Additional costs
Platform usageIncluded Platform usage costs are tied to the Actor's performance in terms of the underlying platform resources it consumes. Learn more	Free

Robots.txt Analyzer

mahogany_songbird/robots-txt-analyzer

Read robots.txt disallow rules and sitemap declarations.

Britton Furness

Robots.txt Policy Auditor

junipr/robots-txt-policy-auditor

Fetch and audit robots.txt files, parse crawl directives, test important paths, and report sitemap and policy risks.

junipr

Sitemap, robots.txt & RSS Change Monitor

enfex/source-change-monitor

Monitor public robots.txt, sitemap XML, and RSS/Atom source changes for SEO and content intelligence without browser automation.

Marcel K

AI Crawler Policy Audit — robots.txt, llms.txt, noai

multiplex/ai-crawl-policy

See exactly which AI crawlers a site allows. Per domain it reads robots.txt for 25 named AI agents (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot…), plus ai.txt, llms.txt, noai meta, X-Robots-Tag and tdmrep.json, and quotes the rule behind every answer. Alerts on policy changes.

Daniel James

Robots.txt Validator

predictable_function/my-actor-3

List of website base URLs whose robots.txt files will be validated

riya rawat

5.0

Robots Policy Auditor

frank_conifer_af4/robots-policy-auditor

Audit robots.txt and crawl policy signals for domains before scraping, crawling, SEO checks, or data collection.

Juddroid

Website Tech Audit Scraper: robots.txt, Sitemaps & Meta

arman-bd/website-tech-audit-scraper

Audit any site's public technical files: robots.txt rules, sitemap discovery and URL counts, ads.txt sellers, security.txt contacts, meta tags, structured data and the detected tech stack with evidence.

Arman Hossain

AI Crawler Policy Drift Monitor

checksmithcats/ai-crawler-policy-drift-monitor

Tracks changes in public robots.txt, noindex, sitemap, and llms.txt signals against a stored baseline. Public crawler-facing signals only.

Checksmith Cats

Robots.txt Tester & Validator

gifted_wagon/robots-txt-evidence-auditor

Parse robots.txt rules into normalized policy evidence, sitemap references, tested paths, and truthful failures.

Michael Olmos

Robots.txt & Sitemap Analyzer 🕷️

perryay/robots-txt-sitemap-analyzer

Fetch, parse, and analyze robots.txt and sitemap.xml for any domain. Extract crawl directives, test URL compliance against robots.txt rules, and discover all URLs from sitemaps including nested sitemap indexes. Supports batch analysis with structured JSON output.