Sitemap URL Extractor

Give it a website or sitemap URL; get every page URL in its sitemaps, one row each, with lastmod, changefreq, priority and hreflang alternates. It finds sitemaps through robots.txt, follows indexes, reads gzip and streams huge files in flat memory.

What it does

• Discovery: a home page URL reads the Sitemap: lines in robots.txt 1. If there are none, it tries /sitemap.xml. Any other URL is read as a sitemap.
• Formats: <urlset>, <sitemapindex> and plain-text sitemaps (one URL per line) 1. Gzip is detected by its bytes, whatever the file is called.
• Indexes: breadth-first, each file once (loop guard), with maxDepth and maxSitemaps.
• Fields:
  • lastmod exactly as written, plus lastmodUtc: W3C Datetime 3 converted to UTC, or null if invalid;
  • changefreq and priority, kept only when valid per sitemaps.org;
  • xhtml:link alternates 4.
• Faults: malformed XML, HTML served as XML, feeds, damaged gzip and files over 50 MB each give an error row naming the sitemap. URLs read before the fault are kept.
• Politeness:
  • at most 2 requests per site at once (up to 4 if you choose);
  • a User-Agent naming the Actor (SitemapExtractor/1.0);
  • robots.txt Disallow rules obeyed for every sitemap fetch, per RFC 9309 2. A robots.txt that returns 5xx or can't be reached means nothing is fetched.
• Safety: only http and https. Private, loopback, link-local and cloud metadata addresses are refused 5, including after DNS and on every redirect.

Quick start

The prefilled input lists the first 10 URLs on The National Archives' website:

{
  "startUrls": ["https://www.nationalarchives.gov.uk/"],
  "maxUrls": 10
}

First row, from the run on 4 October 2026:

{
  "url": "https://www.nationalarchives.gov.uk/",
  "lastmod": "2026-10-01",
  "lastmodUtc": "2026-10-01",
  "changefreq": null,
  "priority": null,
  "alternates": [],
  "sitemap": "https://www.nationalarchives.gov.uk/sitemaps/sitemap_1.xml",
  "sitemapDepth": 1,
  "site": "https://www.nationalarchives.gov.uk",
  "error": null
}

It read robots.txt, two indexes and one sitemap, then stopped at 10 URLs (capped: true).

Input

Field	Default	Meaning
startUrls	(required)	Websites or sitemaps, up to 100
maxUrls	1,000	Stop after this many URLs (up to 200,000)
maxSitemaps	200	Most sitemap files to read
maxDepth	5	Deepest index nesting; 0 reads only the start files
includeAlternates	true	Return hreflang alternates
removeDuplicates	true	Return each URL once
maxConcurrencyPerHost	2	Requests to one site at a time (1–4)
timeoutSecs	30	Per file

Output

Field	Meaning
url	Page URL (null on error rows)
lastmod, lastmodUtc	As written, and normalised
changefreq, priority	Valid values, else null
alternates	[{hreflang, href}]
sitemap, sitemapDepth, site	Where the URL was listed
error	Why a sitemap couldn't be read

OUTPUT has totals and every sitemap read or skipped.

Use from code

curl -X POST "https://api.apify.com/v2/acts/landlordtools~sitemap-extractor/run-sync-get-dataset-items?token=YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": ["https://www.gov.uk/"], "maxUrls": 500}'

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_API_TOKEN' });
const run = await client.actor('landlordtools/sitemap-extractor').call({ startUrls: ['https://www.gov.uk/'], maxUrls: 500 });
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items.length, items[0].url);

Use with AI agents

Agents can call it through Apify's MCP server (https://mcp.apify.com) to see a site's pages before reading them.

Limits

• RSS and Atom feeds aren't read; image, video and news sitemap extensions are ignored.
• DTDs are never read, so a sitemap that relies on DTD entities is reported as malformed.
• The maxUrls cap counts URLs after duplicates are removed.
• Benchmarked against Python's standard-library parser on 101,759 live GOV.UK, National Archives, Met Office and legislation.gov.uk URLs: identical (tests/benchmark/).

Integrations

Power Platform, OpenAPI, Excel and Google Sheets files are in integrations/README.md. Apify's Zapier, Make and n8n apps work too.

Pricing

Pay per event: $0.50 per 1,000 results ($0.0005 each), plus Apify's minimal start fee. A result is one URL returned. Rows for sitemaps that can't be read are free, and the run stops at your spending limit.

Related Actors

• Link and Citation Checker: check that the URLs work.
• GOV.UK Guidance Fetcher: read GOV.UK pages as clean text.

Sources

1. sitemaps.org: Sitemaps XML format. Accessed 4 October 2026.
2. RFC 9309: Robots Exclusion Protocol. Accessed 4 October 2026.
3. W3C Date and Time Formats. Accessed 4 October 2026.
4. Google Search Central: localized versions in sitemaps. Accessed 4 October 2026.
5. IANA IPv4 Special-Purpose Address Registry. Accessed 4 October 2026.
6. IANA IPv6 Special-Purpose Address Registry. Accessed 4 October 2026.