Find declared sitemaps and audit website crawl rules

Created by

Austin Aryain

Actor

Robots.txt Auditor | Crawl Rules & Sitemap Discovery

Compare a robots.txt file with a missing-file response. Export user-agent groups, allow/disallow directives, sitemap declarations and parsing warnings, then replace the two example sites with your own. $0.01 per complete two-site run; platform usage included. HTTP 404/410 is a charged absence result. Inventories directives; it does not decide whether a crawler may fetch a specific URL.

Try for free

Robots.txt Auditor | Crawl Rules & Sitemap Discoverysbio-apps/robots-txt-auditor

inputUrl

httpStatus

exists

groups

+2 fields

Input

Public URLs(required):https://www.wikipedia.org/+1

Output fields

inputUrl

httpStatus

exists

groups

sitemaps

warnings

How it works

Sign up on Apify or log in

The Robots.txt Auditor | Crawl Rules & Sitemap Discovery Actor is ready to run with the inputs already set for "Find declared sitemaps and audit website crawl rules". Sign up or log in and you'll land in Apify Console with a copy of this exact task.

Start the "Find declared sitemaps and audit website crawl rules" task

The Robots.txt Auditor | Crawl Rules & Sitemap Discovery Actor will start running based on the input automatically. You can also change all the input fields, like public urls, to match your specific use case.

Receive the "Find declared sitemaps and audit website crawl rules" output

Monitor the progress in real time. You will be notified as soon as your dataset is complete and ready for review. Each result includes data such as inputurl, httpstatus, exists, and more.

Integrate into your workflow

The final output is delivered in JSON, CSV, or Excel format, ready to be plugged into your workflow. You can also run the "Find declared sitemaps and audit website crawl rules" task via API.