Quick Start: Audit a Sitemap & Robots.txt

Created by

Zaden

Actor

Sitemap & Robots.txt Auditor

Ready-to-run task: audit any site's sitemap.xml and robots.txt for errors, broken URLs, and crawl-blocking rules -- with a letter grade and one-line summary.

Try for free

Sitemap & Robots.txt Auditorzaden/sitemap-robots-auditor

Site

Grade

Score

Severity

+6 fields

Input

Sites to audit(required):https://example.com

URLs to spot-check per site:15

Max child sitemaps to follow:5

Request timeout (seconds):15

Concurrent site audits:5

Output fields

Site

Grade

Score

Severity

robots.txt found

Blocks all crawlers

URLs in sitemaps

URLs sampled

Broken sampled URLs

Summary

How it works

Sign up on Apify or log in

The Sitemap & Robots.txt Auditor Actor is ready to run with the inputs already set for "Quick Start: Audit a Sitemap & Robots.txt". Sign up or log in and you'll land in Apify Console with a copy of this exact task.

Start the "Quick Start: Audit a Sitemap & Robots.txt" task

The Sitemap & Robots.txt Auditor Actor will start running based on the input automatically. You can also change all the input fields, like sites to audit, URLs to spot-check per site, and more, to match your specific use case.

Receive the "Quick Start: Audit a Sitemap & Robots.txt" output

Monitor the progress in real time. You will be notified as soon as your dataset is complete and ready for review. Each result includes data such as site, grade, score, and more.

Integrate into your workflow

The final output is delivered in JSON, CSV, or Excel format, ready to be plugged into your workflow. You can also run the "Quick Start: Audit a Sitemap & Robots.txt" task via API.