Robots Sitemap Analyzer - SEO Crawl Rules

Analyze robots.txt files at scale and extract the crawl rules and sitemap URLs that search engines, SEO tools and crawlers use before visiting a website. Paste domains or URLs and the actor returns a structured row for each target with status code, final robots URL, sitemap links, user-agent groups, allow rules, disallow rules and crawl-delay directives.

What is the Robots Sitemap Analyzer?

The robots.txt file is a small text file with a big impact. It tells crawlers which parts of a website are allowed, which paths are blocked, and where XML sitemaps can be found. For SEO audits, migration checks, technical monitoring and crawler setup, this file is often the first place to inspect.

This actor turns robots files into structured data. Instead of opening files manually or writing one-off scripts, you can run a clean Apify job across many domains and export the results to JSON, CSV, Excel or your own API workflow. It is fast, browserless and inexpensive to run.

Common use cases

Use this actor to audit client websites, monitor whether a production deploy accidentally blocked important paths, discover sitemap URLs for further crawling, compare crawl rules across competitors, check crawl-delay directives before running a crawler, or build a list of sitemap indexes for SEO discovery.

Agencies can run it before technical SEO audits. Data teams can use it as the first step in a web discovery pipeline. Developers can add it to release checks to catch broken robots files before search visibility is affected.

Input

Provide domains, hostnames or full URLs. The actor normalizes each target and fetches /robots.txt.

{
  "domains": [
    "apify.com",
    "https://example.com/blog"
  ]
}

Output

Each dataset item represents one analyzed website.

{
  "input": "apify.com",
  "domain": "apify.com",
  "robots_url": "https://apify.com/robots.txt",
  "final_url": "https://apify.com/robots.txt",
  "status_code": 200,
  "exists": true,
  "sitemap_urls": ["https://apify.com/sitemap.xml"],
  "sitemap_count": 1,
  "user_agent_groups": [
    {
      "user_agent": "*",
      "allow": [],
      "disallow": ["/private"],
      "crawl_delay": null,
      "other": []
    }
  ],
  "user_agent_count": 1,
  "disallow_count": 1,
  "allow_count": 0
}

Output fields

• input - the original domain or URL.
• domain - normalized domain from the fetched robots URL.
• robots_url - the robots URL the actor attempted.
• final_url - final URL after redirects.
• status_code - HTTP status code.
• exists - true when a non-empty robots file is found.
• sitemap_urls - sitemap declarations found in the file.
• sitemap_count - number of unique sitemap URLs.
• user_agent_groups - parsed crawl rules grouped by user agent.
• disallow_count and allow_count - quick summary counts.

Why use this actor?

Manual robots checks do not scale. This actor gives you repeatable structured results, easy scheduling, exports, API access and webhooks. You can run a weekly audit across your own websites, trigger alerts when sitemap declarations disappear, or feed discovered sitemap URLs into a sitemap extraction actor.

Example workflows

For an SEO audit, run the actor across every client domain, export the results, and quickly spot missing robots files, blocked sections, crawl-delay rules and sitemap declarations. For migration monitoring, run it before and after a launch to make sure the staging block did not accidentally ship to production. For crawler setup, use the discovered sitemap URLs as inputs for a sitemap extraction run.

For agencies, this actor is a lightweight first-pass audit tool. It helps answer practical questions quickly: does the site expose a sitemap, are important folders blocked, are there different rules for Googlebot, and did the robots file redirect correctly from HTTP to HTTPS? Those answers are useful before you spend time on a deeper crawl.

Integration ideas

Schedule the actor weekly and compare sitemap_urls, disallow_count and status_code against the previous run. Send changes to Slack, email, Make, Zapier or n8n. Store the output in a database if you manage many websites. Use Apify webhooks to trigger a sitemap scraping workflow whenever a new sitemap appears.

The output is intentionally compact: one row per target domain. That makes it easy to join with CRM records, client lists, domain inventories or uptime monitoring tables.

Notes and limits

The actor parses common robots directives and preserves less common directives in the other array. It does not attempt to decide whether every URL on a site is crawlable. Robots semantics can be crawler-specific, especially for conflicting allow and disallow paths. Use the parsed data as a reliable audit input, then apply your crawler's exact matching rules if needed.

Best practices

Run the actor against canonical production domains, not random deep URLs. Review both the status_code and exists fields; a 404 robots file is not always an error, but it means no explicit sitemap declaration was available there. If you find sitemap URLs, pass them into a sitemap extraction actor to continue discovery. For high-value domains, monitor robots changes on a schedule because a single mistaken Disallow: / can remove important pages from search results.

FAQ

Does it crawl the website?

No. It only fetches and parses /robots.txt. This keeps it fast and safe.

Does it extract sitemap URLs?

Yes. Every Sitemap: directive is returned as a structured array.

Can I combine it with a sitemap scraper?

Yes. Use this actor to discover sitemap URLs, then pass them to a sitemap URL extractor.

Can it check many domains?

Yes. Add multiple domains in the input and export the combined dataset.

You might also like

• Sitemap URL Extractor - collect URLs from XML sitemaps.
• Website Contact Extractor - extract contacts from websites.
• RSS Feed Scraper - monitor feeds and content updates.

Keywords

robots.txt analyzer, robots scraper, sitemap finder, sitemap extractor, SEO crawl rules, technical SEO audit, crawl audit, robots API, sitemap API, Apify SEO tools.