Robots & Sitemap Analyzer

What does Robots & Sitemap Analyzer do?

Robots & Sitemap Analyzer checks robots.txt and XML sitemaps for one or many websites and returns structured crawlability data. Use it to discover sitemap files, evaluate robots rules for a selected user-agent, inspect sitemap coverage, and detect common technical problems without crawling website pages.

Run the Actor on the Apify platform and connect its results to your workflows through the API, schedules, webhooks, and integrations.

Why use Robots & Sitemap Analyzer?

• Audit multiple websites in one run.
• Check whether important website areas are restricted by robots.txt.
• Discover declared and common sitemap locations.
• Follow sitemap indexes and summarize their child files.
• Identify missing files, invalid XML, duplicate URLs, invalid dates, and cross-domain entries.
• Feed normalized crawlability data into SEO monitoring, migration checks, lead enrichment, or Website Intelligence products.
• Route requests through Apify Proxy or your own proxies when needed.

How to use Robots & Sitemap Analyzer

1. Add one or more websites in the Websites field.
2. Keep the default user-agent or enter the crawler you want to evaluate.
3. Adjust the per-website sitemap and URL limits if needed.
4. Optionally enable Apify Proxy or provide custom proxies.
5. Start the Actor and open the Dataset to view one result for each website.

Input

Field	Type	Default	Description
startUrls	array	https://apify.com	Websites to analyze. There is no Actor-defined limit on the number of entries.
userAgent	string	*	User-agent used to evaluate robots.txt rules.
maxSitemaps	integer	50	Maximum sitemap files processed per website.
maxUrls	integer	10000	Maximum sitemap URL entries analyzed in detail per website.
followSitemapIndex	boolean	true	Follow child files referenced by sitemap indexes.
requestTimeoutSecs	integer	30	Timeout for each request.
proxyConfiguration	object	proxy disabled	Apify Proxy or custom proxy settings.

Example input:

{
    "startUrls": [{ "url": "https://apify.com" }, { "url": "https://crawlee.dev" }],
    "userAgent": "*",
    "maxSitemaps": 50,
    "maxUrls": 10000,
    "followSitemapIndex": true,
    "requestTimeoutSecs": 30,
    "proxyConfiguration": {
        "useApifyProxy": false
    }
}

URLs without a protocol are interpreted as HTTPS. Limits apply separately to every website.

Output

Each website produces one Dataset item:

{
    "inputUrl": "https://apify.com",
    "normalizedUrl": "https://apify.com/",
    "origin": "https://apify.com",
    "domain": "apify.com",
    "robots": {
        "url": "https://apify.com/robots.txt",
        "exists": true,
        "statusCode": 200,
        "evaluatedUserAgent": "*",
        "rules": {
            "allow": [],
            "disallow": []
        },
        "declaredSitemaps": []
    },
    "sitemaps": {
        "found": true,
        "sitemapCount": 1,
        "urlCount": 250,
        "uniqueUrlCount": 248,
        "duplicateUrlCount": 2,
        "sampleUrls": []
    },
    "summary": {
        "robotsTxtExists": true,
        "sitemapExists": true,
        "crawlability": "open",
        "issueCount": 0
    },
    "flags": {
        "robotsMissing": false,
        "robotsBlocksRoot": false,
        "sitemapMissing": false
    },
    "issues": []
}

The default key-value store record OUTPUT contains the run results under results. Dataset results can be downloaded as JSON, CSV, Excel, XML, RSS, or JSONL.

Data table

Field	Description
inputUrl	URL supplied in startUrls.
normalizedUrl	Normalized website URL.
robots	Robots file status, groups, applicable rules, crawl delay, and sitemap declarations.
sitemaps	Sitemap files, URL statistics, metadata coverage, and representative URL samples.
summary	Main crawlability status and totals.
flags	Boolean indicators for common conditions.
issues	Findings with a stable code, severity, message, and optional context.

How much does it cost to analyze website robots and sitemaps?

This Actor uses lightweight HTTP requests instead of a browser. Cost depends mainly on the number of websites, sitemap files, sitemap URLs, response times, and proxy selection. Start with the defaults and lower the per-website limits for large monitoring lists. See the current Apify pricing page for platform allowances and compute pricing.

Tips and advanced options

• Increase maxSitemaps for websites with large sitemap indexes.
• Increase maxUrls when detailed coverage statistics matter.
• Lower the limits for fast portfolio-wide checks.
• Enable Apify Proxy when direct requests are blocked or when geographic routing is required.
• Use a specific userAgent to inspect the rules relevant to your crawler.
• Schedule recurring runs to detect crawlability changes over time.

Crawlability values are:

• open - no applicable crawl restriction was detected.
• partially_restricted - applicable robots rules restrict some paths.
• blocked - applicable robots rules block the website root.
• unknown - robots.txt could not be evaluated reliably.

Issue codes include ROBOTS_NOT_FOUND, ROBOTS_HTTP_ERROR, ROBOTS_INVALID, ROOT_BLOCKED, SITEMAP_NOT_DECLARED, SITEMAP_NOT_FOUND, SITEMAP_HTTP_ERROR, SITEMAP_INVALID_XML, SITEMAP_EMPTY, SITEMAP_LIMIT_REACHED, URL_LIMIT_REACHED, SITEMAP_RECURSION_LOOP, SITEMAP_CROSS_DOMAIN, SITEMAP_INVALID_URL, and SITEMAP_INVALID_LASTMOD.

Run locally:

npm install
npm run start

Run a deployed Actor with the Apify CLI:

$apify call your-username/robots-sitemap-analyzer --input='{"startUrls":[{"url":"https://apify.com"}]}'

FAQ, disclaimers, and support

Does the Actor crawl website pages?

No. It analyzes robots.txt and sitemap resources only.

Can one run process multiple websites?

Yes. Add all websites to startUrls. The Dataset contains one result per website.

Does it support sitemap indexes?

Yes. Child sitemaps are processed when followSitemapIndex is enabled.

Is the crawlability result equivalent to a search engine decision?

No. Search engines can apply provider-specific behavior. The result is structured technical guidance for monitoring and automation.

Only analyze websites when your use complies with applicable laws, terms, and policies. If you find an issue or need a custom integration, use the Actor support channel on Apify Console.