Robots.txt & Sitemap SEO Auditor

Audit the crawl-discovery layer of public websites in one structured run. The Actor checks robots.txt, follows declared or fallback XML sitemaps, counts discovered URLs, samples page availability, and returns issue flags plus a simple readiness score.

Why use this Actor

Robots and sitemap problems are easy to overlook and tedious to inspect across many domains. This Actor turns those checks into consistent JSON records that can feed SEO audits, monitoring jobs, agency reports, or technical QA pipelines.

Key features

• Checks public robots.txt availability and extracts user-agent, allow, disallow, and sitemap directives.
• Follows sitemap indexes and URL sets up to configurable limits.
• Falls back to /sitemap.xml when no sitemap is declared in robots.txt.
• Counts discovered page URLs and optionally samples their HTTP availability.
• Reports broken/invalid sitemap conditions and missing crawl-discovery signals.
• Produces a 0-100 heuristic score for quick triage.
• Blocks localhost, private-network targets, and unsafe redirects.

Input

{
  "domains": ["https://example.com"],
  "maxSitemapsPerDomain": 10,
  "maxUrlsPerDomain": 200,
  "checkSampleUrls": 5
}

Output

One dataset item is produced per domain with the base URL, robots status and directives, sitemap reports, discovered URL count, sampled URL results, issue list, and score.

Example

{
  "baseUrl": "https://example.com",
  "ok": true,
  "score": 65,
  "robots": {"status": 200, "declaredSitemaps": []},
  "discoveredUrlCount": 1,
  "issues": ["sitemap_unavailable_or_invalid"]
}

Use cases

• Technical SEO audits across many domains
• Website-launch QA
• Agency site-health checks
• Sitemap discovery for downstream crawlers
• Monitoring crawl-discovery regressions
• Preflight checks before larger scraping or indexing workflows

Pricing

Pay per result. The current price is $0.0005 per audited domain, or $0.50 per 1,000 results.

Limitations

• The score is a practical heuristic, not a search-engine ranking score.
• Very large sitemap estates are intentionally capped by the input limits.
• The Actor does not emulate Googlebot or guarantee how a search engine will interpret directives.
• Authentication, CAPTCHAs, WAF rules, or temporary upstream failures can affect results.
• Only public HTTP/HTTPS targets are supported.

Responsible use

Use the Actor for public websites you are permitted to inspect. It does not bypass authentication or private-network controls.