XML Sitemap Checker, Validator & URL Extractor

Check a public XML sitemap and extract all of its URLs in one run. It works as a sitemap crawler and sitemap extractor: enter a domain or a sitemap URL, and the Actor finds sitemap files through robots.txt, follows sitemap indexes and gzip files, reports validation errors, and returns the sitemap URLs as one record per URL with its sitemap metadata. Optional HTTP status checks flag URLs that don't resolve, and a diff mode compares a run with a previous one. Parse and HTTP errors stay visible in the SUMMARY record instead of a sitemap being silently dropped.

Try a ready-made example: Extract URLs from a sitemap. Open it, click Start, then copy it and swap in your own domain.

Real output from a run on sitemaps.org (2026-10-01): 8 of the 84 URL records, with lastmod, date validity, HTTP status and issues.

What it does

• Accepts site roots, .xml sitemap URLs, and .xml.gz sitemap URLs.
• For a site root, reads Sitemap: lines from /robots.txt; if none exist, tries /sitemap.xml and /sitemap_index.xml.
• Recursively follows sitemap indexes and handles gzip files and namespace-prefixed XML.
• Preserves lastmod, changefreq, and priority, including invalid values for diagnosis.
• Reports malformed XML, non-2xx sitemap responses, invalid child entries, and sitemap protocol limits in SUMMARY.
• Emits duplicate occurrences with duplicateOf set to the first sitemap where the URL appeared. Duplicate occurrences are not charged as extracted URLs.
• Can check URL status with HEAD and a GET fallback for 405/501. Checks honor robots.txt and run at no more than five concurrent requests per host.
• Can mark URLs added, unchanged, or removed relative to a previous Actor dataset. Removed records are not charged.
• Rejects private, loopback, link-local, and other non-public IP ranges before every request and redirect hop.

Input

Only public sitemap and robots.txt files are read. Status checks honor robots.txt, and private or internal network addresses are always rejected.

{
  "startUrls": [
    { "url": "https://www.sitemaps.org/sitemap.xml" }
  ],
  "maxUrls": 10000,
  "maxSitemapFiles": 200,
  "checkStatus": false,
  "statusSampleSize": 0,
  "sameHostOnly": true
}

statusSampleSize: 0 means all unique URLs when checkStatus is enabled. With sameHostOnly: true, cross-host sitemap files referenced by an index are not followed. Cross-host page URLs are still emitted and flagged so they can be fixed.

previousDatasetId is optional. On the Apify platform it is opened from cloud storage. Its non-removed url values form the comparison baseline.

Dataset output

Each sitemap occurrence produces one record. This is an actual item from the local end-to-end run against https://www.sitemaps.org/sitemap.xml on 2026-09-25:

{
  "url": "https://www.sitemaps.org/",
  "sitemap": "https://www.sitemaps.org/sitemap.xml",
  "lastmod": "2016-11-21",
  "changefreq": null,
  "priority": null,
  "lastmodValid": true,
  "hostMatches": true,
  "duplicateOf": null,
  "status": null,
  "finalUrl": null,
  "redirects": null,
  "skippedByRobots": false,
  "diff": null,
  "issues": []
}

Stable issue IDs are:

DUPLICATE_URL, MULTIPLE_LOC, NON_XML_CONTENT_TYPE, INVALID_LASTMOD, FUTURE_LASTMOD, CROSS_HOST, NON_200, REDIRECTED, ROBOTS_DISALLOWED, INVALID_PRIORITY, INVALID_CHANGEFREQ, and URL_TOO_LONG. MULTIPLE_LOC uses the first location and is counted under structuralIssues on an index sitemap's SUMMARY entry. NON_XML_CONTENT_TYPE is also a structuralIssues entry, on any sitemap file that parses as XML but is served with an HTML or plain-text Content-Type (it records the type it saw); the file's URLs are still extracted.

The SUMMARY key-value-store record contains every attempted sitemap file with discovery method, type, gzip flag, uncompressed byte count, URL count, HTTP status, parse error, and protocol-limit violations. It also includes aggregate counts, billing counts, truncation state, and the most frequent issues.

The same end-to-end run parsed one sitemap file and emitted 84 unique URL records in 1.81 seconds:

{
  "totals": {
    "sitemapFilesAttempted": 1,
    "sitemapFilesParsed": 1,
    "urlRecords": 84,
    "uniqueUrls": 84,
    "removedUrls": 0,
    "duplicates": 0,
    "invalidLastmod": 0,
    "non200": 0,
    "crossHost": 0,
    "statusChecksRequested": 0,
    "statusResponses": 0
  },
  "topIssues": [],
  "truncated": false,
  "eventChargeLimitReached": false,
  "billing": {
    "sitemapFileParsed": 1,
    "urlExtracted": 84,
    "urlStatusChecked": 0
  }
}

Pricing

Pay-per-event pricing (you pay only for work that succeeded):

• $0.001 per sitemap file for sitemap-file-parsed, charged once after a file is fetched and parsed successfully.
• $0.0003 per URL for url-extracted, charged after the first current record for a unique URL is pushed. Duplicate occurrences and removed diff records are not charged.
• $0.0005 per status check for url-status-checked, charged after the URL record is pushed only when the check completed with an HTTP response. Network failures and robots skips are not charged.

The Actor stops adding work when an Apify event charge limit is reported.

Limits and behavior

• maxUrls: default 10,000; maximum 200,000 total dataset records, including removed diff records.
• maxSitemapFiles: default 200.
• Sitemap protocol violations are reported above 50,000 entries or 50 MB uncompressed.
• A 55 MB hard response/decompression cap, 20-second request timeout, 10-redirect cap, and SSRF checks bound network work.
• Status checks are limited to five simultaneous requests per origin.
• No proxies, authentication, login flows, or CAPTCHA handling are used.

Local development

Requires Node.js 20 or newer.

npm install
npm test
npm start

For a local Actor run, place INPUT.json in the default local key-value store and set APIFY_LOCAL_STORAGE_DIR. Newer Crawlee releases also recognize CRAWLEE_STORAGE_DIR for the same location.

More lintlab tools

• Broken Link Checker & Technical SEO Audit: page-level SEO findings and broken internal links
• Website Screenshot & Visual Regression Diff: full-page screenshots with pixel diffs
• PDF to Markdown & Text Extractor: page-aware Markdown for RAG and AI agents

Built by lintlab: tools that check web pages, sites and documents. Charged only for completed results. Support: open an issue on this Actor's Issues tab here on Apify.