Find declared sitemaps and audit website crawl rules
Compare a robots.txt file with a missing-file response. Export user-agent groups, allow/disallow directives, sitemap declarations and parsing warnings, then replace the two example sites with your own. $0.01 per complete two-site run; platform usage included. HTTP 404/410 is a charged absence result. Inventories directives; it does not decide whether a crawler may fetch a specific URL.
Input
Output fields
Sign up on Apify01
Create your Apify account to access the Robots.txt Auditor | Crawl Rules & Sitemap Discovery.
Start the run02
The Actor will start running based on the input automatically.
Receive the output03
Monitor the progress in real-time. You will be notified as soon as your dataset is complete and ready for review.
Integrate into your workflow04
The final output is delivered in JSON, CSV, or Excel format, ready to be plugged into your workflow.
