Robots.txt & Sitemap Discovery API

Discover robots.txt crawl rules and sitemap URLs for any domain.

This Actor checks each supplied domain or URL, fetches /robots.txt, extracts sitemap declarations, evaluates basic crawl policy for a selected user agent, and returns clean JSON/CSV-ready records.

Use cases

• Technical SEO preflight checks
• Find sitemap URLs before running crawlers
• Validate robots.txt availability and crawl-delay directives
• Build automated domain audit workflows
• Collect crawl policy data for many websites

Input

{
  "startUrls": ["https://example.com", "apify.com"],
  "userAgent": "*",
  "includeRobotsText": false
}

Output

{
  "input": "https://example.com",
  "origin": "https://example.com",
  "robotsUrl": "https://example.com/robots.txt",
  "statusCode": 200,
  "hasRobots": true,
  "sitemaps": ["https://example.com/sitemap.xml"],
  "crawlDelay": null,
  "disallowRules": ["/admin"],
  "allowRules": ["/"],
  "issues": []
}

Output fields

• input - original user input.
• origin - normalized website origin.
• robotsUrl - robots.txt URL checked.
• statusCode - robots.txt HTTP response code.
• hasRobots - whether a robots.txt file was found.
• sitemaps - sitemap URLs declared in robots.txt.
• crawlDelay - matched crawl-delay directive for the selected user agent.
• allowRules / disallowRules - matched rules for the selected user agent.
• issues - labels such as robots_not_found or no_sitemaps_declared.

SEO keywords

robots.txt API, sitemap discovery, robots parser, technical SEO API, crawlability checker, sitemap finder, SEO preflight, crawl rules checker.