Pricing

Pay per usage

Website Sitemap & Robots Audit

Find robots.txt, declared sitemaps, homepage metadata, canonical URL, and basic crawlability signals for a list of websites.

Pricing

Pay per usage

Rating

0.0

(0)

Developer

Marcus Harlid Davin

Actor stats

Bookmarked

Total users

Monthly active users

5 days ago

Last modified

Use cases

Build quick SEO/crawlability inventories.
Check whether prospect/customer websites expose usable sitemaps.
Prepare URL discovery inputs for other crawlers.
Audit competitors or portfolios without browser automation.

Input

Provide one or more website URLs. Domains without a scheme are treated as https://.

{
  "urls": ["https://example.com", "apify.com"],
  "timeoutSecs": 15,
  "includeHomepageHtmlSample": false
}

Output

Each dataset item contains:

input URL and final resolved URL
HTTP status
page title and meta description
canonical URL
robots.txt status
sitemap URLs declared in robots.txt
common sitemap URL guesses that exist
errors, if any

Notes

This Actor only fetches public URLs supplied by the user. It does not log in, bypass paywalls, or attempt aggressive crawling.

Robots.txt & Sitemap Discovery API

213x/robots-sitemap-discovery

Discover robots.txt rules, sitemap URLs, crawl delays, and basic crawlability signals for domains or website URLs.

Sitemap & Robots SEO Index Auditor

glowing_glove/sitemap-robots-index-auditor

Audit robots.txt, sitemap discovery, indexability signals, canonical tags, and SEO crawl readiness for business websites.

Ushba Khan

Robots.txt & Sitemap Auditor

phoenix2810/robots-sitemap-auditor

Audit a website's robots.txt directives and sitemap discovery in one fast API call. Detect blocked paths, missing sitemaps, and crawlability issues. Built for SEO agencies, technical SEO consultants, and site migration QA teams.

Sanskar Jaiswal

robots.txt Analyzer

eliai/robots-txt-analyzer

Anthony Snider

Robots.txt Analyzer

mahogany_songbird/robots-txt-analyzer

Read robots.txt disallow rules and sitemap declarations.

Britton Furness

Sitemap Extractor

johnlenflure/sitemap-extractor

Extract and parse XML sitemaps from any website. Discovers sitemaps via robots.txt and common paths. Outputs structured URL data with metadata.

Sinan Donmez

Robots.txt Validator

predictable_function/my-actor-3

List of website base URLs whose robots.txt files will be validated

riya rawat

5.0

Indexability Audit

zerobreak/indexability-audit

Indexability audit tool that checks robots.txt, meta robots tags, X-Robots-Tag headers, and canonical URLs for any list of pages, so SEO teams know which ones Google can actually crawl and index.