Sitemap Validator & URL Extractor

Sitemap Doctor discovers XML sitemaps, validates every sitemap file, extracts URL metadata, optionally checks HTTP status, and compares a run with a previous Apify dataset. Parse and HTTP errors stay visible in the SUMMARY key-value-store record instead of silently dropping a sitemap.

Built by lintlab: small, reliable data tools. Tested before release.

What it does

• Accepts site roots, .xml sitemap URLs, and .xml.gz sitemap URLs.
• For a site root, reads Sitemap: lines from /robots.txt; if none exist, tries /sitemap.xml and /sitemap_index.xml.
• Recursively follows sitemap indexes and handles gzip files and namespace-prefixed XML.
• Preserves lastmod, changefreq, and priority, including invalid values for diagnosis.
• Reports malformed XML, non-2xx sitemap responses, invalid child entries, and sitemap protocol limits in SUMMARY.
• Emits duplicate occurrences with duplicateOf set to the first sitemap where the URL appeared. Duplicate occurrences are not charged as extracted URLs.
• Can check URL status with HEAD and a GET fallback for 405/501. Checks honor robots.txt and run at no more than five concurrent requests per host.
• Can mark URLs added, unchanged, or removed relative to a previous Actor dataset. Removed records are not charged.
• Rejects private, loopback, link-local, and other non-public IP ranges before every request and redirect hop.

Input

Only public sitemap and robots.txt files are read. Status checks honor robots.txt, and private or internal network addresses are always rejected.

{
  "startUrls": [
    { "url": "https://www.sitemaps.org/sitemap.xml" }
  ],
  "maxUrls": 10000,
  "maxSitemapFiles": 200,
  "checkStatus": false,
  "statusSampleSize": 0,
  "sameHostOnly": true
}

statusSampleSize: 0 means all unique URLs when checkStatus is enabled. With sameHostOnly: true, cross-host sitemap files referenced by an index are not followed. Cross-host page URLs are still emitted and flagged so they can be fixed.

previousDatasetId is optional. On the Apify platform it is opened from cloud storage. Its non-removed url values form the comparison baseline.

Dataset output

Each sitemap occurrence produces one record. This is an actual item from the local end-to-end run against https://www.sitemaps.org/sitemap.xml on 2026-09-25:

{
  "url": "https://www.sitemaps.org/",
  "sitemap": "https://www.sitemaps.org/sitemap.xml",
  "lastmod": "2016-11-21",
  "changefreq": null,
  "priority": null,
  "lastmodValid": true,
  "hostMatches": true,
  "duplicateOf": null,
  "status": null,
  "finalUrl": null,
  "redirects": null,
  "skippedByRobots": false,
  "diff": null,
  "issues": []
}

Stable issue IDs are:

DUPLICATE_URL, INVALID_LASTMOD, FUTURE_LASTMOD, CROSS_HOST, NON_200, REDIRECTED, ROBOTS_DISALLOWED, INVALID_PRIORITY, INVALID_CHANGEFREQ, and URL_TOO_LONG.

The SUMMARY key-value-store record contains every attempted sitemap file with discovery method, type, gzip flag, uncompressed byte count, URL count, HTTP status, parse error, and protocol-limit violations. It also includes aggregate counts, billing counts, truncation state, and the most frequent issues.

The same end-to-end run parsed one sitemap file and emitted 84 unique URL records in 1.81 seconds:

{
  "totals": {
    "sitemapFilesAttempted": 1,
    "sitemapFilesParsed": 1,
    "urlRecords": 84,
    "uniqueUrls": 84,
    "removedUrls": 0,
    "duplicates": 0,
    "invalidLastmod": 0,
    "non200": 0,
    "crossHost": 0,
    "statusChecksRequested": 0,
    "statusResponses": 0
  },
  "topIssues": [],
  "truncated": false,
  "eventChargeLimitReached": false,
  "billing": {
    "sitemapFileParsed": 1,
    "urlExtracted": 84,
    "urlStatusChecked": 0
  }
}

Pricing

Pay-per-event pricing (you pay only for work that succeeded):

• $0.001 per sitemap file for sitemap-file-parsed, charged once after a file is fetched and parsed successfully.
• $0.0003 per URL for url-extracted, charged after the first current record for a unique URL is pushed. Duplicate occurrences and removed diff records are not charged.
• $0.0005 per status check for url-status-checked, charged after the URL record is pushed only when the check completed with an HTTP response. Network failures and robots skips are not charged.

The Actor stops adding work when an Apify event charge limit is reported.

Limits and behavior

• maxUrls: default 10,000; maximum 200,000 total dataset records, including removed diff records.
• maxSitemapFiles: default 200.
• Sitemap protocol violations are reported above 50,000 entries or 50 MB uncompressed.
• A 55 MB hard response/decompression cap, 20-second request timeout, 10-redirect cap, and SSRF checks bound network work.
• Status checks are limited to five simultaneous requests per origin.
• No proxies, authentication, login flows, or CAPTCHA handling are used.

Local development

Requires Node.js 20 or newer.

npm install
npm test
npm start

For a local Actor run, place INPUT.json in the default local key-value store and set APIFY_LOCAL_STORAGE_DIR. Newer Crawlee releases also recognize CRAWLEE_STORAGE_DIR for the same location.