Robots.txt & Sitemap SEO Auditor

Crawl Policy and Indexability Intelligence preserving robots/sitemap auditing while adding health scores, fingerprints, drift and SEO release gating. Returns structured, machine-readable results for repeatable research, monitoring, automation, Apify Tasks, API workflows, and AI.

Why use this Actor

SEO and web teams need a quick preflight of crawl discovery, sitemap health and sample URL availability before deeper audits.

Key features

• Buyer-native inputs aligned to robots.txt.
• Predictable structured output in the default Apify Dataset.
• Pay Per Event pricing tied to measurable result rows or reports.
• Optional MCP connector delivery for downstream automation and AI agents.

Quick start

Start with this working example and replace the target values with your own:

{
  "domains": [
    "https://example.com"
  ],
  "maxSitemapsPerDomain": 10,
  "maxUrlsPerDomain": 200,
  "checkSampleUrls": 5
}

The Actor writes structured results to the default Apify Dataset and can be used from the Store, API, schedules, Tasks, automations and MCP-compatible AI workflows.

Input

• domains — Domains or website URLs: Public HTTP/HTTPS websites to audit. Duplicate targets are processed once. Entries must be non-empty strings.
• maxSitemapsPerDomain — Maximum sitemaps per domain: Maximum number of XML sitemap files or sitemap indexes to fetch for each website.
• maxUrlsPerDomain — Maximum discovered URLs per domain: Maximum number of page URLs to collect from XML sitemaps for each website.
• checkSampleUrls — Sample URLs to verify: Number of discovered sitemap URLs to fetch as a lightweight availability sample.
• previous_results — Previous SEO/crawl snapshots: Optional map keyed by domain or input URL. Enables crawl-policy and sitemap/indexability drift monitoring.

All integration and advanced analysis fields are optional. The default example is intentionally runnable without configuring MCP or a previous-run baseline.

Output

The Dataset exposes predictable machine-readable output. Representative fields include results.

Pricing

This Actor uses Pay Per Event. The current factory base price is $0.000500 per result event. The Apify Store remains the source of truth for the price and plan/tier details shown to the buyer.

Use cases

• Run the buyer-ready workflows exposed as Apify Tasks without preparing a custom integration first.
• Use the Actor from API or schedules for recurring collection, comparison or monitoring.
• Feed the structured Dataset output into spreadsheets, databases, automations or AI agents.
• Compare repeat runs when the product supports snapshots or previous-run inputs.

Automation and AI

Use the same Actor through Apify API, schedules, public Tasks and the Apify MCP server. Outputs are structured for downstream workflows and AI agents rather than requiring manual copy/paste.

Limitations

• Public websites and upstream APIs can change markup, access rules, rate limits or field availability without notice.
• Fields that are not publicly available are returned as unavailable or omitted rather than fabricated.
• Analytical outputs depend on the quality and coverage of the supplied or collected source data.
• Treat marketplace, reputation, workforce, safety or commercial signals as decision support and validate material decisions against the underlying source evidence.

Detailed documentation

Why use this Actor

Robots and sitemap problems are easy to overlook and tedious to inspect across many domains. This Actor turns those checks into consistent JSON records that can feed SEO audits, monitoring jobs, agency reports, or technical QA pipelines.

Key features

• Checks public robots.txt availability and extracts user-agent, allow, disallow, and sitemap directives.
• Follows sitemap indexes and URL sets up to configurable limits.
• Falls back to /sitemap.xml when no sitemap is declared in robots.txt.
• Counts discovered page URLs and optionally samples their HTTP availability.
• Reports broken/invalid sitemap conditions and missing crawl-discovery signals.
• Produces a 0-100 heuristic score for quick triage.
• Blocks localhost, private-network targets, and unsafe redirects.

Input

{
  "domains": ["https://example.com"],
  "maxSitemapsPerDomain": 10,
  "maxUrlsPerDomain": 200,
  "checkSampleUrls": 5
}

Output

One dataset item is produced per domain with the base URL, robots status and directives, sitemap reports, discovered URL count, sampled URL results, issue list, and score.

Example

{
  "baseUrl": "https://example.com",
  "ok": true,
  "score": 65,
  "robots": {"status": 200, "declaredSitemaps": []},
  "discoveredUrlCount": 1,
  "issues": ["sitemap_unavailable_or_invalid"]
}

Use cases

• Technical SEO audits across many domains
• Website-launch QA
• Agency site-health checks
• Sitemap discovery for downstream crawlers
• Monitoring crawl-discovery regressions
• Preflight checks before larger scraping or indexing workflows

Pricing

Pay per result. The current price is $0.0005 per audited domain, or $0.50 per 1,000 results.

Limitations

• The score is a practical heuristic, not a search-engine ranking score.
• Very large sitemap estates are intentionally capped by the input limits.
• The Actor does not emulate Googlebot or guarantee how a search engine will interpret directives.
• Authentication, CAPTCHAs, WAF rules, or temporary upstream failures can affect results.
• Only public HTTP/HTTPS targets are supported.

Responsible use

Use the Actor for public websites you are permitted to inspect. It does not bypass authentication or private-network controls.

Input reliability update — 24 October 2026

Target arrays accept non-empty strings. Identical targets are processed once per run to avoid duplicate requests and result charges. Whitespace around targets is removed. Baseline maps must be JSON objects. Existing result fields, pricing and target limits are preserved.