Sitemap URL Extractor & Validator

Get every URL a website lists in its XML sitemaps, with last-modified dates, change frequency and priority, in one run. Give it a site and it finds the sitemaps itself through robots.txt, or give it a sitemap URL directly. Sitemap indexes, gzip sitemaps and plain-text sitemaps are read, and a free validation report shows where the sitemaps were found and what is wrong with them. For SEO specialists, developers and data teams who need a clean list of a site's pages.

What is Sitemap URL Extractor & Validator?

It is a sitemap extractor and sitemap URL scraper that reads only what a site publishes for crawlers: its sitemaps. It does not crawl pages, so it is fast to set up, cheap per URL and gentle on the site. For each input it:

1. reads the site's robots.txt and follows its Sitemap: lines,
2. falls back to /sitemap.xml and /sitemap_index.xml when robots.txt lists none,
3. follows sitemap indexes down to the URL lists (nested indexes included),
4. writes every URL with its lastmod, changefreq, priority and source sitemap to the dataset, up to your limit,
5. writes a validation report per sitemap to the key-value store record SUMMARY.

What data do you get?

Per URL (dataset, charged):

• The page URL (loc) exactly as the sitemap lists it.
• lastmod: the last-modified date, when it is in W3C date format.
• changefreq and priority, when the site provides them.
• The sitemap the URL came from.
• The site (your input) the URL belongs to.

Per site and sitemap (key-value store record SUMMARY, free):

• Where the sitemaps were found: in robots.txt, at the default locations, or given directly as input.
• How many URLs were extracted for the site.
• Per sitemap file: whether it is a sitemap index (and how many child sitemaps it lists) or a URL list (and how many URLs it holds), or why it could not be read.
• Validation warnings: more than 50,000 URLs in one file (the protocol's limit), lastmod values not in W3C date format, and URLs on another host than the sitemap.
• Errors per sitemap: HTTP errors such as http_404, files over 50 MB, broken gzip, invalid XML, files that are not sitemaps, and sitemaps blocked by robots.txt.

Use cases

• SEO audits: get the full list of pages a site wants indexed, then check them in bulk with the URL Status & Redirect Checker to find sitemap URLs that are broken or redirect.
• Site migrations: export the complete URL inventory of the old site before a relaunch, as the basis for the redirect map, and check it again after the switch.
• Sitemap validation: before submitting sitemaps to search engines, check that they can be read, are valid XML, stay under 50,000 URLs per file, use W3C dates and list only URLs of their own host.
• Content inventories: count how many pages, posts or products a site lists and see when they were last changed, for content audits, pruning and planning.
• Change monitoring: schedule a run and compare lastmod dates and URL lists between runs to see which pages were added or updated.
• Feeding crawlers and AI pipelines: give a crawler, a search index or a RAG pipeline the exact list of pages a site publishes, instead of crawling it blindly.
• Indexing coverage: compare the sitemap URLs with the pages a crawl finds; the Website Health Audit does that comparison for you.

How to use Sitemap URL Extractor & Validator

1. Open the Actor in Apify Console (click Try for free or Start on this page).
2. Enter one or more sites (https://example.com) or sitemap URLs (https://example.com/sitemap.xml) in Sites or sitemap URLs.
3. Set Max URLs per site to the number of URLs you need (the default is 10,000).
4. Set a maximum cost per run in the run options if you want a hard spending limit.
5. Click Start. The status message shows which site is being read.
6. Download the URLs from the Output tab as JSON, CSV, Excel, XML or HTML. The validation report is the SUMMARY record in the run's key-value store (also linked in the output).
7. Create a Schedule to extract the URLs again daily or weekly.

Input

Field	Type	Default	What it does
startUrls	array of strings	(required)	Sites or sitemap URLs.
maxUrlsPerSite	integer, 1 to 1,000,000	10,000	Stop after this many URLs for one site.
timeoutSecs	integer, 5 to 120	30	How long to wait for each sitemap download, in seconds.

Tips:

• Site or sitemap? An input whose path ends with .xml, .xml.gz, .gz or .txt, or contains the word sitemap, is read directly as a sitemap. Anything else is treated as a site, and its sitemaps are discovered through robots.txt and the default locations.
• Give the sitemap URL when you know it. If a site keeps its sitemap at an unusual address and does not list it in robots.txt, enter that URL directly.
• maxUrlsPerSite protects your budget on very large sites. When it is reached, the extractor stops for that site and goes on with the next one.
• timeoutSecs: large sitemaps on slow servers may need more than the default 30 seconds.
• Limits: up to 1,000 inputs per run; further entries are ignored. Exact duplicates are read once. A missing scheme is completed with https://.
• API input: besides strings, you can pass objects like {"url": "https://example.com/"}.

Example input:

{
  "startUrls": ["https://www.sitemaps.org/", "https://example.com/sitemap.xml"],
  "maxUrlsPerSite": 10000,
  "timeoutSecs": 30
}

Output

One dataset item per URL, in the order the sitemaps list them:

{
  "url": "https://example.com/blog/first-post",
  "lastmod": "2026-09-30",
  "changefreq": "weekly",
  "priority": 0.8,
  "sitemap": "https://example.com/sitemap-posts.xml",
  "site": "https://example.com/"
}

Output fields

Field	Type	Meaning
url	string	The page URL (<loc>) as the sitemap lists it.
lastmod	string or null	The <lastmod> value when it is in W3C date format (for example 2026-09-30 or 2026-09-30T08:15:00+00:00), else null.
changefreq	string or null	The <changefreq> value, when present.
priority	number or null	The <priority> value as a number, when present and numeric.
sitemap	string	The sitemap file the URL was listed in.
site	string	Your input this URL belongs to, cleaned up (for example https://example.com/).

The SUMMARY record

The key-value store record SUMMARY holds one entry per input:

{
  "https://example.com/": {
    "found_from": "robots.txt",
    "urls": 1234,
    "sitemaps": {
      "https://example.com/sitemap_index.xml": {"kind": "sitemapindex", "children": 3},
      "https://example.com/sitemap-posts.xml": {"kind": "urlset", "urls": 1200, "warnings": ["lastmod not in W3C date format"], "site": "example.com"},
      "https://example.com/sitemap-pages.xml.gz": {"kind": "urlset", "urls": 34, "warnings": [], "site": "example.com"},
      "https://example.com/sitemap-old.xml": {"error": "http_404"}
    }
  },
  "not a url": {"error": "invalid_url"}
}

Field	Meaning
found_from	robots.txt, default locations (/sitemap.xml and /sitemap_index.xml) or input (you gave a sitemap URL).
urls	URLs extracted for this input in this run.
sitemaps	One entry per sitemap file that was read or tried.
kind	sitemapindex, urlset (an XML URL list) or text (a plain-text sitemap).
children	For a sitemap index: how many child sitemaps it lists.
urls (per sitemap)	How many URLs the file lists.
warnings	Validation warnings for the file.
site	The host of your input.
error	Why a sitemap or an input could not be read.

Error codes include http_<status> (for example http_404), too_large (over 50 MB), bad_gzip, invalid_xml, not_a_sitemap, doctype_not_allowed, blocked_by_robots, robots_unavailable, timeout, tls_error, connection_error, dns_error, not_public_address, too_many_redirects and invalid_url.

Dataset view

The Overview table shows url, lastmod, changefreq, priority and sitemap.

Supported sitemap formats

• XML sitemaps (<urlset>) and sitemap indexes (<sitemapindex>), nested to any depth, per the sitemaps.org protocol.
• Gzip sitemaps (.xml.gz), recognised by their content, not by the file name.
• Plain-text sitemaps: one URL per line.
• Up to 500 sitemap files per site; each file up to 50 MB, also after decompression.
• Image, video and news sitemap extensions (titles, captions, descriptions) are left out on purpose: they are other people's content. Alternate-language links (hreflang) are not extracted either.
• For safety, XML with a DOCTYPE is refused (doctype_not_allowed).

How much does it cost?

This Actor uses pay-per-event pricing: you pay USD 0.0003 per URL extracted (USD 0.30 per 1,000 URLs), plus Apify's standard actor start event of USD 0.00005 per run. With pay-per-event pricing the platform usage of the run is included in these prices.

URLs extracted	URL events	Actor start	Total per run
100	USD 0.03	USD 0.00005	USD 0.03005
1,000	USD 0.30	USD 0.00005	USD 0.30005
10,000	USD 3.00	USD 0.00005	USD 3.00005
100,000	USD 30.00	USD 0.00005	USD 30.00005
1,000,000	USD 300.00	USD 0.00005	USD 300.00005

What is charged and what is free:

• Charged: each URL written to the dataset.
• Free: the SUMMARY validation report, sitemap indexes themselves, sitemaps that could not be read, and inputs that are invalid or not public. A site without sitemaps costs only the actor start.

Set a maximum cost per run and the extractor stops at your limit; maxUrlsPerSite caps each site on top of that. Apify's free plan includes monthly platform credit, which you can use to try the Actor.

Tips and best practices

• Start small. Run a site with maxUrlsPerSite at 100 first to see the structure and the report, then raise the limit.
• Read the SUMMARY first. It shows at once whether the sitemaps were found, which files failed and which have warnings.
• Check the URLs next. Feed the url column into the URL Status & Redirect Checker: sitemaps should list only URLs that answer 200 without redirects.
• Use the right host. Sitemaps of example.com and www.example.com can differ; enter the host the site really uses.
• Watch the warnings. URLs on another host than their sitemap, and lastmod values in the wrong format, are common reasons why search engines ignore parts of a sitemap.
• Allow the extractor on your own site. It obeys robots.txt rules for the product token HelveticLabs-WebCheck; a site that blocks all bots also blocks its sitemaps for it.

Limitations

• Sitemaps only. It never crawls pages to guess URLs. If a site has no sitemap, the report says which locations were tried and no URLs are extracted.
• No deduplication across sitemaps. A URL listed in two sitemap files appears (and is charged) once per listing; the sitemap field shows where each one came from.
• robots.txt is respected. A sitemap that robots.txt disallows is not downloaded. When a site's robots.txt answers with a server error, nothing on that site is read.
• Polite pace. One request at a time per host, with a pause between requests, and one site after another. Sites with hundreds of sitemap files take a while.
• Caps: 1,000 inputs per run, up to 1,000,000 URLs per site, 500 sitemap files per site, 50 MB per file.
• lastmod values that are not W3C dates are returned as null (the report warns about them).
• Only public sitemaps on public addresses; nothing behind a login.

FAQ

Is it legal to use? Sitemaps are files a site publishes for crawlers. The extractor reads only those, respects robots.txt and leaves out content such as image captions and news titles. Check only sites you own or may check, and see the fair use rules at the end of this page.

Does it respect robots.txt? Yes, for every request. It also uses the Sitemap: lines in robots.txt to find the sitemaps.

Does it store personal data? No. It returns URLs and dates from public sitemaps, never page content.

How fast is it? It reads one sitemap file at a time per site, with a polite pause between requests to the same host. The duration depends on the number and size of the files and on the server.

The site has no sitemap. What now? The SUMMARY record shows the locations that were tried. If you know the sitemap's address, enter it directly. To find pages without a sitemap, use the Website Health Audit, which crawls the site's links.

Can I schedule it? Yes, with a schedule in Apify Console, for example weekly to track new and changed pages.

Can I use it from Python, JavaScript or the API? Yes, see Integrations below.

Can AI agents use it? Yes, through Apify's MCP server, for example to get the list of pages of a documentation site before reading them.

Why was something not charged? Only URLs written to the dataset are charged. The report, the sitemap indexes, failed sitemaps and invalid inputs are free.

How is it different from a generic web crawler? A crawler follows links page by page and reads every page. This extractor reads only the sitemaps, so it is cheaper per URL, finds pages that are not linked anywhere, and also tells you what is wrong with the sitemaps.

What happens on errors and timeouts? A sitemap that fails is recorded with its error in the SUMMARY record, and the extractor goes on with the next sitemap and the next site. Results written before a run timeout stay in the dataset.

Does it read sitemaps behind a login? No. Only public sitemaps.

Integrations

The actor ID is HelveticLabs/sitemap-extractor.

Python (pip install apify-client):

from decimal import Decimal

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("HelveticLabs/sitemap-extractor").call(
    run_input={"startUrls": ["https://example.com/"], "maxUrlsPerSite": 5000},
    max_total_charge_usd=Decimal("2"),  # optional spending limit for this run
)
# apify-client 3.x; with older versions use run["defaultDatasetId"] and run["defaultKeyValueStoreId"]
urls = [item["url"] for item in client.dataset(run.default_dataset_id).iterate_items()]
summary = client.key_value_store(run.default_key_value_store_id).get_record("SUMMARY")
print(len(urls), "URLs")

JavaScript (npm install apify-client):

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_APIFY_TOKEN>' });
const run = await client.actor('HelveticLabs/sitemap-extractor').call(
    { startUrls: ['https://example.com/'], maxUrlsPerSite: 5000 },
    { maxTotalChargeUsd: 2 },
);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
const summary = await client.keyValueStore(run.defaultKeyValueStoreId).getRecord('SUMMARY');
console.log(items.length, summary?.value);

HTTP API: POST https://api.apify.com/v2/acts/HelveticLabs~sitemap-extractor/runs with the input as the JSON body; add maxTotalChargeUsd as a query parameter to cap the cost.

No-code and automation: connect the Actor to Make, Zapier or n8n, or add a webhook that starts the next step when a run finishes, for example a bulk status check of all URLs.

AI agents (MCP): Apify's MCP server lets AI agents and MCP clients run this Actor as a tool; pass maxTotalChargeUsd in the call options to cap the spend.

Related HelveticLabs actors

Actor	What it does
URL Status & Redirect Checker	Status code, redirect chain, final URL and response time for up to 100,000 URLs.
Website Health Audit: Broken Links & SEO Checker	Crawls a site for broken links and SEO issues and compares it with the sitemap.
Website Security Headers & TLS Audit	A+ to F security grade: HTTPS, headers, SSL certificate, SPF and DMARC.
AI Crawler Access Checker	Which AI bots a site's robots.txt allows, plus llms.txt.

Support and feedback

A sitemap that is read wrongly, or a format you need? Open an issue in the Issues tab on this Actor's page, or write to contact@helveticlabs.org. Include the run ID and the sitemap URL if you can.

Provider, privacy and fair use

Provider: HelveticLabs, sole proprietorship of Marvin Krieg, c/o SwissMailBox 170, 12 Rue Le Corbusier, 1208 Genève, Switzerland. E-mail: contact@helveticlabs.org. Imprint: https://www.helveticlabs.org/imprint

Privacy: This Actor runs on the Apify platform, in your Apify account. It processes only the URLs or domains you enter and the public technical data the checked servers return (status codes, headers, certificates, DNS records, sitemap URLs, links). It does not collect personal data: it never stores or returns page text (where a page's HTML is read, it is read in memory only, to find links and measure tags), and it leaves out e-mail addresses, cookie names and values and similar values. Your input and results stay in your Apify account and are stored by Apify under its own terms (https://apify.com/privacy-policy). The provider sees only the usage statistics Apify shows to Actor authors (such as the number of users and runs), not your input or results, unless you share a run for support. Questions about privacy: contact@helveticlabs.org.

Fair use: Check only sites you own or may check. Do not use the results to attack systems or people. The Actor respects robots.txt, identifies itself, and keeps a polite pace per site.

No warranty: The results are automated checks and can be incomplete. They are not a penetration test, legal advice or a compliance certificate.