Sitemap URL QA & Crawl Inventory

Turn a website root or XML sitemap into a clean, auditable URL inventory. The Actor recursively discovers sitemap indexes, reads .xml.gz files, deduplicates entries, and optionally checks every public URL for HTTP status, redirects, response time, content type, and Last-Modified metadata.

It uses plain HTTP requests—no browser, proxy, external API, or LLM—so runs stay predictable and the output works well in JSON/CSV exports, Apify API integrations, schedules, and MCP/agent workflows.

What you get

• Recursive sitemapindex and urlset traversal
• robots.txt sitemap discovery with /sitemap.xml fallback
• Gzip-compressed sitemap support
• URL deduplication across sitemap files
• HEAD checks with safe GET fallback for servers that reject HEAD
• Redirect chain, final URL, HTTP status, response time, content type, and HTTP Last-Modified
• Sitemap lastmod, changefreq, and priority
• Wildcard include/exclude filters and hard URL/sitemap limits
• Explicit sitemap_error records instead of silent partial results
• Public-network-only fetching to reduce SSRF risk
• A run summary in the OUTPUT key-value-store record

Quick start

{
  "startUrls": [{ "url": "https://example.com" }],
  "discoverFromRobots": true,
  "checkStatus": true,
  "sameOriginOnly": true,
  "maxUrls": 10000,
  "maxSitemaps": 100,
  "maxConcurrency": 15,
  "requestTimeoutSecs": 20,
  "maxRedirects": 5,
  "includePatterns": [],
  "excludePatterns": []
}

You can also enter a sitemap directly:

{
  "startUrls": [{ "url": "https://example.com/sitemap-index.xml" }],
  "checkStatus": false
}

Input

Field	Default	Description
startUrls	required	Website roots, sitemap files, sitemap indexes, or .xml.gz URLs.
discoverFromRobots	true	Reads Sitemap: directives from robots.txt for website-root inputs.
checkStatus	true	Audits every discovered page URL. Disable for inventory-only runs.
sameOriginOnly	true	Keeps sitemap files and page URLs on the hostname of the matching start URL.
maxUrls	10000	Hard output-work limit, from 1 to 100,000.
maxSitemaps	100	Maximum recursively processed sitemap files, from 1 to 1,000.
maxConcurrency	15	Parallel page checks, from 1 to 50.
requestTimeoutSecs	20	Per-request timeout, from 3 to 120 seconds.
maxRedirects	5	Maximum redirects per request, from 0 to 10.
includePatterns	[]	Optional wildcard patterns. At least one must match when supplied.
excludePatterns	[]	Wildcard patterns that remove matching URLs.

Wildcard examples: */blog/*, *.pdf, *?preview=*.

Dataset output

Each URL produces one visible dataset item. A typical item looks like this:

{
  "recordType": "url",
  "url": "https://example.com/products/blue-widget",
  "sourceSitemap": "https://example.com/product-sitemap.xml.gz",
  "lastmod": "2026-08-20",
  "changefreq": "weekly",
  "priority": 0.8,
  "statusCode": 301,
  "finalUrl": "https://example.com/product/blue-widget",
  "isBroken": false,
  "isRedirect": true,
  "redirectCount": 1,
  "redirectChain": ["https://example.com/products/blue-widget"],
  "responseTimeMs": 182,
  "contentType": "text/html; charset=utf-8",
  "httpLastModified": "Thu, 20 Aug 2026 08:00:00 GMT",
  "issueCodes": ["REDIRECT"],
  "checkedAt": "2026-08-21T12:00:00.000Z",
  "errorCode": null,
  "errorMessage": null
}

Sitemap-level failures are returned as free, explicit sitemap_error records with an errorCode and errorMessage. Common issue/error codes include HTTP_404, REDIRECT, MISSING_LASTMOD, MALFORMED_XML, UNSUPPORTED_SITEMAP, REQUEST_TIMEOUT, DNS_ERROR, RESPONSE_TOO_LARGE, and PRIVATE_NETWORK_BLOCKED.

The OUTPUT record contains aggregate counts such as discovered/processed URLs, broken URLs, redirects, missing lastmod, sitemap errors, and whether a run stopped at a configured or spending limit.

Pricing

Recommended Store setup: $0.0004 per url-audited event ($0.40 per 1,000 audited URL records) plus Apify's default apify-actor-start event. Remove the apify-default-dataset-item synthetic event to prevent double charging. Sitemap-level error records are not charged by the custom event.

The Actor checks the run spending limit after each charged result and stops cleanly when the limit is reached. Actual Store pricing is displayed by Apify before a run starts and controls over this README if they differ.

Responsible use and privacy

• Use the Actor only for websites you own, administer, or are authorized to audit, or for public resources where automated access is allowed.
• Respect website terms, applicable law, rate limits, and server capacity. Lower maxConcurrency when necessary.
• The Actor does not bypass authentication, CAPTCHAs, access controls, or anti-bot systems.
• It does not request residential proxies and does not collect account credentials.
• URL credentials and targets resolving to loopback, link-local, private, multicast, or otherwise unsafe network addresses are blocked.
• Fetched response bodies are not stored. Sitemap XML is held in memory only while parsing; page checks normally use HEAD.

Limitations

• A successful HEAD response does not prove that a JavaScript application renders correctly.
• Some servers intentionally block HEAD or automated clients. The Actor retries 403, 405, and 501 responses with a small ranged GET, but a block may remain.
• sameOriginOnly compares hostnames exactly; www.example.com and example.com are different hostnames.
• The Actor validates sitemap structure and URL availability, not full search-engine indexing eligibility.
• Very slow sites can make large status-check runs expensive. Start with a small maxUrls and use an Apify run spending limit.

Local development

npm install
node --test test/*.test.mjs
node --check src/core.mjs
node --check src/main.mjs

For local pay-per-event behavior, Apify supports ACTOR_TEST_PAY_PER_EVENT=true. Final pricing and cloud usage must be verified in an unpublished Apify test Actor before Store publication.

Support scope

Bug reports should include the run ID, a public reproducible sitemap URL, expected result, and actual result. Private-site access, custom scraping, account login, and site-specific bypasses are outside the support scope.