Sitemap Extractor & Validator

What it does

Sitemap Extractor & Validator by Riprap Tools turns XML sitemaps, gzip sitemaps and nested sitemap indexes into a distinct URL list with declared metadata and validation issues. Built for SEO specialists, migration and QA engineers, and developers building crawlers or monitors, it reads sitemaps—not destination pages.

What you can use it for

• Build a pre-migration inventory of URLs declared in your sitemaps.
• Audit sitemap hygiene: duplicates, fragments, invalid dates and metadata values.
• Feed normalized URLs into your own crawler or monitoring pipeline.
• Inspect declared lastmod dates for freshness analysis and future-date errors.
• Identify cross-host entries by turning off sameHostOnly.

How to use it

1. In Apify Console, open the Actor's Input tab and enter sitemap URLs or site roots. For a root, it reads robots.txt Sitemap: lines, falling back to /sitemap.xml.
2. Set URL and sitemap limits, then start the run. For API use, send the same input JSON through Apify's Actor run API; use the Actor's API tab for integration details.
3. Read the default dataset for URL records and the key-value store's SUMMARY record for counts, skips and truncation. Increase the defaults when you need more than a small sample.

Input

Field	Type	Default	What it does
startUrls	array of strings	["https://www.gov.uk/sitemaps/sitemap_1.xml"]	Required. Sitemap, index or site-root URLs; 1–100 HTTP(S) URLs.
maxUrls	integer	10	Maximum distinct URLs; range 1–100,000.
maxSitemaps	integer	2	Sitemap-file attempt limit, including index files, nested children and failed files; range 1–1,000.
sameHostOnly	boolean	true	Skip sitemap and result URLs whose hostname differs from the starting URL, except www-only differences (allowed for both files and rows).
includeUrlParameters	boolean	false	Include query-string result URLs when enabled; otherwise skip them.

Output

One dataset item per normalized URL. Normalization lowercases scheme and hostname, removes default ports and fragments, and preserves query order. Duplicates encountered within the processed entries are not emitted again; the retained item gets duplicate. Missing metadata is null; invalid metadata is retained for diagnosis.

Field	Meaning
url	Original sitemap URL entry (trimmed).
normalizedUrl	URL used for deduplication.
sourceSitemap	Sitemap containing the entry.
sitemapDepth	Nesting depth; starting sitemap is 0.
lastmod	Declared modification date, or null.
lastmodValid	True/false for date validity; null when absent or empty.
changefreq	Declared change frequency, or null.
priority	Declared priority as text, or null.
issues	Array of validation codes; empty when none found.

{
  "url": "https://www.gov.uk/",
  "normalizedUrl": "https://www.gov.uk/",
  "sourceSitemap": "https://www.gov.uk/sitemaps/sitemap_1.xml",
  "sitemapDepth": 0,
  "lastmod": null,
  "lastmodValid": null,
  "changefreq": null,
  "priority": "0.5",
  "issues": []
}

Issue codes

Code	Meaning
invalid_lastmod	Nonempty date could not be parsed in the accepted date format.
future_lastmod	Parsed date is later than the current time.
duplicate	Another processed entry has the same normalized URL.
priority_out_of_range	Nonempty priority is not numeric within 0–1.
url_too_long	Original URL exceeds 2,048 characters.
has_fragment	Original URL contains a fragment.
invalid_changefreq	Nonempty value is not always, hourly, daily, weekly, monthly, yearly or never.
cross_host	Hostname differs from the starting URL; only delivered when sameHostOnly is false.
suspicious_lastmod	Valid date belongs to a file with suspicious date patterns; heuristic, not proof.
http_in_https_sitemap	HTTP entry in an HTTPS sitemap file.
www_mismatch	Entry host and sitemap-file host differ only by the www prefix; retained even with sameHostOnly: true.

Run summary

SUMMARY contains issueCounts, sitemapsFetched (successfully parsed sitemap/index files with a supported root), sitemapsAttempted (fetch attempts, including failures and indexes), urlsEmitted, skipped (counts by reason), truncated, warnings, failedSitemaps, sitemapFindings and outcome. A spending stop additionally sets billingLimitReached.

urlsEmitted counts only deliveries whose dataset push and charge both completed. A pushed row whose charge fails may be visible in the dataset but is not counted here; charges are never retried. outcome is no_results when no deliveries completed, otherwise partial when truncated or any sitemap failed, otherwise complete. Complete means the selected inventory was processed, not that every entry is valid: inspect skips and findings too.

truncated: true signals a URL, sitemap, depth, per-file or billing limit—not a complete inventory. Sitemap-budget warnings name maxSitemaps as the input to raise. Skips include email_url, invalid_url, url_parameters, cross_host, cycle, depth_limit, sitemap_limit and file_url_limit. Stable failure codes include robots_denied, robots_unavailable, crawl_delay_limit, http_<status>, redirect_limit, invalid_root, html_instead_of_xml, doctype_forbidden, response_size_limit, decompressed_size_limit and unsafe_destination; unknown fetch/parse errors use fetch_or_parse_failed.

failedSitemaps and sitemapFindings are lists of { "url": "https://example.org/sitemap.xml", "code": "invalid_root" } objects, capped at 100 each. Report URLs omit queries/fragments, redact credentials and apparent email-like URLs, and are capped at 2,048 characters. Failure totals remain in skipped even when the report cap is reached.

Per-file findings: invalid_namespace (missing/wrong sitemaps.org namespace), html_instead_of_xml (body begins with HTML/HTML doctype; response headers are unavailable), file_url_limit (overflow truncation: the first 50,000 entries are still processed and delivered rows are charged; its skip count counts affected files, not omitted URLs), and file_size_limit (over the 50 MiB uncompressed safety cap, rejected). Existing DTD and size rejection remains in effect. This is body sniffing, not a Content-Type audit.

Lastmod findings use valid declared dates per file: at least 5 dates all identical (lastmod_all_identical) or all equal to the UTC run date (lastmod_all_today); at least 20 dates with at most 10% distinct calendar days (lastmod_low_day_diversity). Only valid-date rows in those files receive suspicious_lastmod; missing/invalid dates do not. These patterns do not establish actual page modification history. File findings inspect the parsed file, while row findings cover only processed entries.

Pricing

Pay per event: one result per distinct URL delivered. Per 1,000 URLs: $0.30 no discount / $0.25 Bronze / $0.20 Silver / $0.15 Gold. No start fee. No charge for failed, skipped or duplicate URLs. A per-run spending cap stops delivery cleanly. All plans get the same features.

10,000 URLs ≈ $3.00 at no-discount pricing.

Limits and what it does not do

• Reads sitemap XML only, plus robots.txt for discovery and access rules. No page crawling, JavaScript rendering, login or proxy/CAPTCHA bypass.
• Follows robots.txt for sitemap requests; unavailable robots.txt denies access. Crawl delays above 10 seconds are refused.
• Sequential requests, minimum one-second delay, 10-second socket timeout and at most five redirects. Redirect destinations are checked too.
• Private/internal and other non-public network destinations are refused; URL credentials are not accepted.
• Compressed responses and decompressed XML are each capped at 50 MiB. XML DTDs/entities are rejected.
• Processes at most 50,000 URL entries per file and sitemap depths 0–10. Run maxima: 100 starting URLs, 100,000 distinct results and 1,000 sitemap attempts.
• Query-string result URLs are skipped unless enabled. Extraction and deduplication use memory, not streaming output.

Data and responsibility

You must have the right to process the sites you submit and comply with their terms and applicable data-protection rules. Sitemaps can contain URLs with people's names; names are not detected or removed. Apparent email-like URLs, including percent-encoded forms, are dropped. Output is limited to what the sitemap declares.

FAQ

Does it check that each URL is live? No. It does not request destination pages or check their HTTP status.

Does a valid lastmod prove the page changed? No. It validates the declared date, not the page's actual modification history.

Why did I get only 10 URLs or no child sitemap results? Defaults are 10 URLs and two sitemap attempts. An index itself consumes an attempt, leaving one for a child; deeper indexes or additional children need a larger budget. Raise the limits and inspect SUMMARY.

Support

Report reproducible problems in the Actor's Issues tab. Response within 14 days.

Contains public sector information licensed under the Open Government Licence v3.0. Source: GOV.UK; no affiliation or endorsement implied.