Robots.txt & Sitemap Auditor

Audit a website's robots.txt directives and sitemap discovery in a single API call. Detects blocked paths, missing sitemaps, sitewide disallow rules, and crawlability issues. Returns structured findings with severity levels.

Use cases

• SEO agencies - run crawlability audits across client sites
• Technical SEO - identify robots.txt misconfigurations and missing sitemaps
• Site migration QA - verify robots.txt and sitemap health before and after migrations
• Dev shops - automated crawlability checks in deployment pipelines
• Website monitoring - continuous robots.txt and sitemap monitoring

Input

Field	Type	Required	Default	Description
startUrl	string	yes	-	Public website URL to audit
timeoutSeconds	integer	no	10	Per-request timeout (3-30 seconds)

Example input

{
  "startUrl": "https://example.com",
  "timeoutSeconds": 10
}

Output

A single dataset item with the full audit:

Field	Type	Description
inputUrl	string	The URL provided
normalizedUrl	string	Normalized site origin
checkedAt	string	ISO timestamp
robots	object	Robots.txt analysis (see below)
sitemaps	array	Sitemap probe results (see below)
findings	array	Severity-tagged findings with codes and messages

robots object

Field	Type	Description
url	string	URL where robots.txt was fetched
ok	boolean	Whether the fetch succeeded
status	integer	HTTP status code
finalUrl	string	Final URL after redirects
contentType	string	HTTP content-type
sitemapDirectives	array	Sitemap URLs declared in robots.txt
disallowCount	integer	Number of disallow rules
allowCount	integer	Number of allow rules
crawlDelay	array	Crawl-delay values found
error	string	Error message if fetch failed

sitemaps array

Each entry contains:

Field	Type	Description
url	string	Sitemap URL probed
ok	boolean	Whether the fetch succeeded
status	integer	HTTP status code
contentType	string	HTTP content-type
finalUrl	string	Final URL after redirects
bytesSampled	integer	Bytes read from the response
error	string	Error message if fetch failed

findings array

Each finding has a severity (critical, warning, info, or ok), a code, and a message.

Code	Severity	Meaning
SITEWIDE_DISALLOW	critical	robots.txt contains Disallow: /
ROBOTS_NOT_OK	warning	robots.txt returned an error status
NO_SITEMAP_DISCOVERED	warning	No sitemap directive and /sitemap.xml not reachable
CRAWL_DELAY_SET	info	Crawl-delay values found
COMMON_PRIVATE_PATHS_BLOCKED	info	CMS/admin paths are blocked
BASIC_DISCOVERY_OK	ok	Basic robots/sitemap signals are healthy

Example output

{
  "inputUrl": "https://example.com",
  "normalizedUrl": "https://example.com/",
  "checkedAt": "2026-07-06T12:00:00.000Z",
  "robots": {
    "url": "https://example.com/robots.txt",
    "ok": true,
    "status": 200,
    "sitemapDirectives": ["https://example.com/sitemap.xml"],
    "disallowCount": 3,
    "allowCount": 0,
    "crawlDelay": []
  },
  "sitemaps": [
    {
      "url": "https://example.com/sitemap.xml",
      "ok": true,
      "status": 200,
      "contentType": "application/xml",
      "bytesSampled": 45000
    }
  ],
  "findings": [
    { "severity": "info", "code": "COMMON_PRIVATE_PATHS_BLOCKED", "message": "Common private/CMS paths are blocked from crawlers" },
    { "severity": "ok", "code": "BASIC_DISCOVERY_OK", "message": "example.com has basic robots/sitemap discovery signals" }
  ]
}

Security

• Only public HTTP/HTTPS URLs are accepted
• SSRF protection: localhost, private IPv4/IPv6, and DNS-resolving-to-private IPs are blocked
• URLs with embedded credentials are rejected
• Redirects are revalidated before following
• No browser automation, no proxies

Pricing

Pay per event:

Event	Price
Actor start	$0.005
Site audited	$0.01

A single-site audit costs approximately $0.015.

FAQ

What sitemap paths does it check? Sitemaps declared in robots.txt, plus /sitemap.xml and /sitemap_index.xml as fallback probes.

Can I audit multiple sites in one run? This actor audits one site per run. For bulk audits, schedule multiple runs.

What if robots.txt is missing? The actor returns a ROBOTS_NOT_OK warning and continues with sitemap probing.