Sitemap URL Extractor

Give it a list of domains. For each one it finds the sitemaps, reads them, follows sitemap index files down to the nested sitemaps, decompresses gzipped files, and returns one row per page URL with the lastmod date the site reports.

Use it to get a full page list for an SEO audit, to find pages changed since a date, to seed your own crawler, or to compare two sites' structure.

How it finds sitemaps

1. Reads the domain's robots.txt and takes every Sitemap: line.
2. If robots.txt lists none, tries /sitemap.xml, then /sitemap_index.xml.
3. Each file is read as a sitemap index (its child sitemaps are queued), a URL set (its URLs become results), or a plain-text sitemap (one URL per line).
4. Gzipped files are detected by their content, not their file name, so .xml.gz files work whether or not the server marks them as gzip.

You can also give a sitemap URL directly, and the actor reads that file and anything it points to.

Input

Field	Type	Default	What it does
startUrls	array of strings	required, no default	Domains (example.com), homepage URLs, or direct sitemap URLs. A URL with sitemap in the path, or ending in .xml, .txt, or .gz, is read as a sitemap. Any other path is ignored and the domain is used.
maxItems	integer	1000	Total URLs to return across all domains. This is your cost cap.
maxItemsPerDomain	integer	0	URLs to return per domain, so one big site cannot use the whole run. 0 means no per-domain limit.
maxSitemaps	integer	500	Sitemap files to read per domain, index files included.
urlPattern	string	empty	Regular expression, case-insensitive. Only matching URLs are returned. Example: /blog/.
modifiedSince	string	empty	Only URLs with a lastmod on or after this date. A date (2026-01-31) or an age (7 days, 2 weeks, 6 months). URLs with no lastmod are left out when this is set.

Example input:

{
    "startUrls": ["docs.python.org", "wiki.openstreetmap.org", "https://developer.mozilla.org/sitemap.xml"],
    "maxItems": 500,
    "maxItemsPerDomain": 200,
    "modifiedSince": "30 days"
}

Output

One dataset item per unique page URL. A real item from a test run (this one came from a gzipped nested sitemap):

{
    "url": "https://wiki.openstreetmap.org/wiki/Main_Page",
    "lastmod": "2026-10-21T14:26:22Z",
    "changefreq": null,
    "priority": null,
    "source_url": "https://wiki.openstreetmap.org/sitemap-wiki-0.xml.gz",
    "input": "wiki.openstreetmap.org",
    "domain": "wiki.openstreetmap.org",
    "fetched_at": "2026-10-28T06:13:52.483Z"
}

Field	Meaning
url	Page URL as listed in the sitemap.
lastmod	The <lastmod> value exactly as the site wrote it, or null.
changefreq	The <changefreq> value in lower case, or null.
priority	The <priority> value as a number, or null.
source_url	The sitemap file the URL was found in.
input	The startUrls entry this result belongs to.
domain	Host of that entry.
fetched_at	When the sitemap file was read (ISO 8601, UTC).

Two extra records go to the run's key-value store. They are not dataset items, so they are not billed:

• SUMMARY: per entry, where the sitemaps were found (robots.txt, fallback, or input), how many sitemap files were read, how many URLs they listed, how many were returned, and why it stopped (done, maxItems, maxItemsPerDomain, maxSitemaps, no sitemap found, no readable sitemap, robots.txt unavailable).
• FAILED_URLS: every entry or sitemap file that was skipped, with the reason. For example Disallowed by robots.txt, HTTP 404, Got an HTML page, not a sitemap, or No sitemap found.

Behavior and limits

• Respects robots.txt for every file it reads, including nested sitemaps on other hosts. If a site's robots.txt returns a server error or cannot be reached, that site is skipped.
• Waits 300 to 800 ms between requests to the same host. Sites are read one at a time.
• A URL that appears in more than one sitemap is returned once per run.
• Sitemap indexes are followed up to 5 levels deep. Loops (an index that lists itself or a parent) are read once.
• Files over 60 MB, after decompression, are skipped. The sitemap protocol caps files at 50 MB.
• Broken files are handled per file: a cut-off file returns the complete entries before the break, entries without a valid absolute <loc> are skipped, and the run log says how many. One bad file never stops the run.
• Some wiki sites publish index entries with the host written twice (https://host/https://host/sitemap-0.xml.gz). These are repaired to the inner URL.
• lastmod is only as accurate as the site makes it. Some sites stamp every URL with the last build time.
• Image, video, news, and hreflang extensions inside a sitemap are not returned. Only the page <loc> is.
• RSS and Atom feeds used as sitemaps are not supported.
• No page HTML is fetched. If a site has no sitemap, it returns nothing for that site rather than crawling links.

Pricing

Pay per result: each returned URL is one result. You set maxItems (and optionally maxItemsPerDomain), so you know the most a run can cost before it starts. Skipped files and the SUMMARY and FAILED_URLS records are free.

A single sitemap file can list up to 50,000 URLs, and the default maxItems of 1000 caps a run at about $0.50. Set maxItems lower for a quick look.

This actor is not affiliated with any of the sites named above. They are used as examples of public sitemaps.

Use it from an AI agent

Add this actor as a tool through Apify's MCP server: https://mcp.apify.com?tools=comradecoding/sitemap-url-extractor. The agent gets one JSON row per page URL found in the site's sitemaps, with lastmod when the sitemap has it.

Integrations

Anything that can call the Apify API can run this actor: the Apify API and client libraries, and Apify's integrations for Make, n8n, Zapier and LangChain. Results come back as a dataset you can download as JSON, CSV or Excel.

What it costs

Run size	Cost
1,000 URLs	about $0.50
10,000 URLs	about $5
100,000 URLs	about $50

Platform usage is included in the price.

FAQ

Does it follow nested and gzipped sitemaps?

Yes, sitemap index files and .xml.gz files are both read.

What if a site has no sitemap?

It checks robots.txt and the usual sitemap paths. If none exist, the domain is listed in the failures record and nothing is charged.

Can I get the page content too?

Pass the URLs to URL to Markdown Converter, which returns each page as clean Markdown.

Related actors by ComradeCoding

• URL to Markdown Converter
• PDF to Text and JSON