Sitemap URL Extractor: all URLs from XML sitemaps

For SEO and web agencies that track many client and prospect sites. Sitemap URL Extractor lists every URL a site publishes in its sitemaps, with the sitemap file it came from and its lastmod date, for one site or a whole client list in one run.

The job in one line: give it domains, get a clean URL inventory per site, and on later runs only the URLs that are new or changed.

What one run costs (pay per event):

Event	Price
URL record (one delivered URL)	$0.0004
Actor start (once per run, Apify's standard fee)	$0.00005

1,000 URL records cost $0.40. Site summaries and error records are free. A real run on 17 Oct 2026 (wordpress.org, maxUrls 50) took 7.2 seconds, delivered 50 URL records plus the site summary and cost a buyer $0.02005.

Quick start

{
    "sites": ["wordpress.org"],
    "maxUrls": 50
}

For a client portfolio, list every domain under sites, raise maxUrls (default 1,000, up to 1,000,000, counted across all sites) and switch on deltaMode for scheduled runs.

Output example

REAL records from a run of the input above, read on 8 Oct 2026 at 01:53 UTC (one of its 50 url records and its site_summary, sitemaps_listed shortened to 3 of 9 here). The site changes, so a later run can give other URLs and dates.

[
  {
    "id": "be5e3f0b503a8a36269f",
    "record_type": "url",
    "status": "ok",
    "site": "wordpress.org",
    "url": "https://wordpress.org/about/privacy/cookies/",
    "lastmod": "2026-09-01T12:37:54Z",
    "changefreq": null,
    "priority": null,
    "sitemap_url": "https://wordpress.org/sitemap-1.xml",
    "depth": 1,
    "images_count": 0,
    "videos_count": 0,
    "news_count": 0,
    "sitemap_truncated": false,
    "change": null,
    "source_url": "https://wordpress.org/sitemap-1.xml",
    "fetched_at": "2026-10-08T01:53:12Z"
  },
  {
    "id": "c30ba320d4fd9550c3c0",
    "record_type": "site_summary",
    "status": "ok",
    "error_type": null,
    "site": "wordpress.org",
    "source_url": "https://wordpress.org/robots.txt",
    "fetched_at": "2026-10-08T01:53:10Z",
    "discovery": "robots.txt",
    "sitemaps_listed": [
      "https://wordpress.org/sitemap.xml",
      "https://wordpress.org/news-sitemap.xml",
      "https://wordpress.org/themes/sitemap.xml"
    ],
    "files_read": 2,
    "files_failed": 0,
    "urls_found": 50,
    "urls_delivered": 50,
    "duplicates": 0,
    "filtered_out": 0,
    "other_host_skipped": 0,
    "unchanged_skipped": 0,
    "loops_skipped": 0,
    "truncated": false,
    "truncated_files": [],
    "errors_by_type": {},
    "errors": [],
    "stopped_reason": "maxUrls reached"
  }
]

In delta mode change is new or lastmod_changed. id comes from the site and the URL, so it is the same on every run.

Input

Field	Default	Meaning
sites	none	Domains (example.com) or site addresses. Sitemaps are found through robots.txt, then /sitemap.xml, /sitemap_index.xml and /wp-sitemap.xml.
sitemapUrls	none	Sitemap or index files to read directly.
include / exclude	none	Regular expressions to keep or drop URLs (for example only /blog/).
lastmodAfter	none	Keep only URLs with a lastmod on or after this date (YYYY-MM-DD, UTC).
maxUrls	1000	Stop after this many URL records in total, across all sites.
sameHostOnly	true	Keep only URLs on the site's own host or its www form.
deltaMode	false	Deliver only URLs that are new or whose lastmod changed since the last run with the same sites and filters.

Give at least one entry in sites or sitemapUrls.

Three agency use cases

1. Site migration and redesign checks. Pull the full URL list of a client site before a migration and again after it. Compare the two lists to build the redirect map and to catch pages that dropped out of the sitemap.
2. Competitor content watch for clients. Schedule a weekly run over each client's competitors with deltaMode on. Each run returns only the pages that are new or whose lastmod changed, so the strategist sees new landing pages and posts without reading whole sitemaps.
3. Feed for your own audit tools. Use the URL list as the input for a link checker, a crawler or an indexing check across the whole portfolio. site_summary tells you per site why a sitemap gave nothing (not_found, blocked, error, skipped).

Limits and what it does not do

• Reads only robots.txt and sitemap files, with GET requests. It never opens the pages a sitemap lists, so it does not check status codes, titles or indexing.
• Obeys robots.txt: a sitemap file that robots.txt disallows is reported, not read. Some sites refuse automated agents; those come back as blocked with the HTTP code.
• At most 2 requests per second per host, 30 seconds per request, 50 MB per file, 500 sitemap files per site, 5 index levels.
• maxUrls counts across sites: sites after the limit is reached are not read and get a site_summary with status skipped.
• A sitemap file that ends early gives the URLs before the cut, flagged with sitemap_truncated and truncated.
• Delta state is kept in a named key-value store in your account, per site and filter set.

FAQ

Does it follow sitemap index files? Yes, up to 5 levels. An index that points back at itself or at one already read is skipped and counted under loops_skipped.

Why did a site give no URLs? Its site_summary says why: not_found (no sitemap at robots.txt or the usual addresses), blocked, skipped (maxUrls reached before its turn) or error with the error types.

Does truncated: false mean I have every URL? No. It means the files read were complete. Check stopped_reason too: maxUrls reached means the run stopped at your limit.

Are duplicates charged? No. A URL listed twice is delivered and charged once per run and counted under duplicates.

How do I cap my spend? Set maxUrls and a maximum cost per run in the run options. The run stops cleanly when the maximum charge is reached.

Keywords: sitemap URL extractor, sitemap crawler for agencies, XML sitemap to CSV, site migration URL list, new page monitor, competitor content tracking, sitemap index parser