Sitemap URL Extractor: All Website URLs from Sitemaps

What does Sitemap URL Extractor do?

Sitemap URL Extractor gets every URL a website lists in its sitemaps, with last-modified date, change frequency, priority, images, videos, news data and language versions (hreflang). Enter a website such as apify.com and it finds the sitemaps automatically, through robots.txt and the usual locations (WordPress, Yoast, .xml.gz, text sitemaps). You can also paste a direct sitemap, sitemap index or RSS/Atom feed URL.

It is built to finish reliably: broken XML is repaired where possible, a website without a sitemap never fails the whole run, and every problem is explained in plain language. Because it runs on the Apify platform, you get API access, scheduling, webhooks, integrations (Make, Zapier, n8n, Google Sheets) and run monitoring.

You are never charged for a failed or empty result

Every result says what it is, and only real results cost anything.

status	Meaning	Charged
ok	A real result	Yes
empty	Worked, but there was nothing to report	No
failed	Could not be done — the error field says why	No
truncated / partial	Cut short by a limit you set; what was delivered is kept	Only the delivered part
skipped	Not attempted, for example after the maximum cost per run was reached	No

If every input fails, the run itself fails with an explanation instead of reporting success with no data.

Why use Sitemap URL Extractor?

• SEO audits and content inventories: list every page, see what changed recently, and spot old content.
• Crawl planning: feed an exact URL list into your scraper or RAG pipeline instead of crawling the whole site.
• Competitor monitoring: schedule a daily run in changes since last run mode to get only new, updated and removed products, posts, job pages or landing pages.
• Migration checks: export all URLs before and after a redesign and compare them.
• Broken link checks: turn on status checks to get the HTTP status and final URL after redirects.

How to extract all URLs from a website's sitemap

1. Click Try for free.
2. In Websites or sitemap URLs, enter one or more websites (for example apify.com) or sitemap links.
3. Optionally set Maximum URLs per website, Include/Exclude URLs matching or Modified after.
4. Click Start. Most sites finish in seconds.
5. Open the Output tab. Overview shows URL, last modified, change frequency and priority. Details adds images, language versions and the sitemap each URL came from.

Input

Field	Description	Default
Websites or sitemap URLs	Websites, domains, or direct links to a sitemap, sitemap index, .xml.gz, .txt sitemap or RSS/Atom feed. Up to 50 per run.	–
Maximum URLs per website	Stop collecting from a website after this many URLs (up to 200,000)	10,000
Report changes since last run	Off, New and modified pages, or New, modified and removed pages	off
First run in change mode	Return all URLs as initial, or save the baseline only (free)	return all
Removal safety threshold (%)	Don't report removals if a website suddenly lists fewer URLs than this share of last run	50
Include URLs matching	Glob (*/blog/*), plain text (/products/) or regular expression between slashes (/\/2026\//)	all
Exclude URLs matching	Same syntax, for example */tag/*	none
Modified after	Keep only URLs with a sitemap lastmod on or after this date	any
Only URLs on the same website	Drop links to other domains (subdomains are kept)	on
Remove duplicate URLs	Save each URL once across all sitemaps	on
Include images, videos and news data	Add sitemap extension data	on
Check HTTP status of each URL	Request each URL and save status code and final URL	off
Parallel status checks	How many URLs are checked at once	5
Maximum sitemap index depth	Levels of nested sitemap indexes to follow	5

Example input:

{
  "startUrls": ["apify.com", "https://wordpress.org/news/"],
  "maxUrls": 1000,
  "excludePatterns": ["*/tag/*"],
  "lastmodAfter": "2026-01-01"
}

Output

Each URL is one item in the dataset. Real examples from a run on 15 September 2026:

{
  "url": "https://wordpress.org/embed-image/",
  "site": "https://wordpress.org",
  "sitemapUrl": "https://wordpress.org/image-sitemap-1.xml",
  "lastmod": "2022-10-04T14:46:18+00:00",
  "changefreq": null,
  "priority": null,
  "alternates": [],
  "images": [
    { "url": "https://wordpress.org/files/2022/08/embed-image.png", "title": null, "caption": null }
  ],
  "videos": [],
  "news": null,
  "error": null,
  "scrapedAt": "2026-09-15T11:20:24.820134+00:00"
}

With status checks on, each item also has "statusCode": 200, "finalUrl" and "statusError".

A website whose sitemap can't be read gets one item with an error explaining why (for example "No sitemap found in robots.txt or at the usual locations" or "HTTP 403 Forbidden: the site refused automated access") and suggestions. These items are never charged. A per-website summary of every sitemap found, how many URLs each had and any errors is saved in the key-value store as SITEMAPS_REPORT.

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel.

Monitor a website for new and removed pages

Turn on Report changes since last run and put the Actor on a schedule (for example daily) to watch a website without paying for the same URLs again and again:

1. Enter the websites to watch, for example a competitor's shop, blog or careers site. Use Include URLs matching (*/products/*, */jobs/*) to watch only one section.
2. Choose New and modified pages or New, modified and removed pages.
3. Choose what the first run returns: every URL marked initial, or nothing (Save baseline only, free).
4. Save the configuration as a task and add a schedule. Connect a webhook, Slack, email or Google Sheets integration to be notified when a run finds changes.

Every item then has a changeType:

changeType	Meaning
new	The URL was not in the sitemap last time
modified	The URL's lastmod date changed (previousLastmod holds the old date)
removed	The URL was in the sitemap last time but is gone now (only in New, modified and removed pages mode)
initial	First run for this website, returned when Return all URLs is selected

Example new item:

{
  "url": "https://example.com/products/new-running-shoe",
  "site": "https://example.com",
  "sitemapUrl": "https://example.com/sitemap-products.xml",
  "lastmod": "2026-09-15T00:00:00+00:00",
  "changeType": "new",
  "previousLastmod": null,
  "scrapedAt": "2026-09-16T06:00:04+00:00"
}

How it works. After each run, the list of URLs for each website (URL and lastmod only) is saved in a key-value store named sitemap-url-extractor-state in your own Apify account, and the next run compares against it. One list is kept per start URL plus include/exclude patterns and the same-website setting, so changing those starts a new baseline. Modified after only filters what is returned and does not reset the list. To start over, delete the store in Storage → Key-value stores.

Built-in safeguards against false alarms:

• If a website lists far fewer URLs than last time (below the Removal safety threshold), the run assumes a sitemap failed to load: nothing is reported as removed and the saved list is kept.
• If Maximum URLs per website is reached, a website's sitemaps hit one of the Actor's own ceilings (500 sitemap files or 512 MB per website), the run runs out of time, or some sitemaps can't be read, new and modified pages are still reported, but removals are not, and the saved list is only extended.
• If the run stops at your maximum cost, the saved list is not updated, so no change is lost; it is reported on the next run.
• SITEMAPS_REPORT shows per website: previous and current URL counts, new, modified and removed counts, whether this was the baseline, and a note explaining any safeguard that was applied.

You pay only for changes: in change mode only the URLs returned (new, modified, removed, or initial on a first run) are charged. A scheduled run on a website that didn't change returns nothing and costs only the small Actor start fee.

Output fields

Field	Meaning
url	Page URL from the sitemap
site	Website it belongs to
sitemapUrl	Sitemap the URL was found in
lastmod	Last modification date (ISO 8601, UTC), when the sitemap has it
changefreq, priority	Crawl hints from the sitemap
alternates	Language versions: [{ "hreflang": "de", "url": "..." }]
images, videos, news	Image, video and Google News sitemap data
statusCode, finalUrl, statusError	HTTP check results (only when status checks are on; removed URLs are not checked)
changeType, previousLastmod	new, modified, removed or initial in change mode, otherwise null
error, suggestions	Set only on the item for a website whose sitemap couldn't be read

How much does it cost to extract sitemap URLs?

This Actor uses pay-per-event pricing:

• URL extracted: charged for each URL saved to your dataset. Duplicates, filtered URLs and websites without a readable sitemap cost nothing. In change mode, only the changed URLs returned are charged.
• URL status checked: charged only when Check HTTP status is on, for each URL actually checked.

See the Pricing tab for current prices per 1,000 URLs. Platform compute is included. You can set a maximum cost per run and the Actor stops cleanly when it is reached.

Tips

• Big sites: set Maximum URLs per website and use include patterns (for example */products/*) to get only what you need.
• Only new content: use Report changes since last run with a daily schedule. Unlike Modified after, it also catches new pages that have no lastmod date.
• Status checks are slower: they request every URL. Keep Parallel status checks low for small websites.
• No sitemap found? Look for a sitemap link in the site's robots.txt or footer and enter it directly, or use the site's RSS feed.

Related tools

Audit the URLs you extracted:

• Broken Link Checker: crawl a website and find broken links, images and redirect chains.
• SEO Audit & Metadata Checker: check titles, meta descriptions, canonicals and other on-page SEO for pages or whole sites.
• Website Tech Stack Detector: detect the CMS, frameworks, analytics and hosting of any website.

FAQ and support

Does it respect robots.txt? Yes. Sitemaps disallowed for crawlers in robots.txt are skipped and reported.

What happens when a site won't let us read its robots.txt? Some sites put their /robots.txt behind bot protection and answer 403 or time out. We then can't tell what the site allows, so we follow one rule everywhere: the address you entered is still read, and addresses we would have worked out ourselves are not. A site with no robots.txt at all (404) is treated as allowing everything. For this Actor that means a sitemap URL you paste is always read, while the usual locations we try for you (/sitemap.xml, /sitemap_index.xml, …) are not tried at such a site.

Why did a site fail with HTTP 403? Some websites block automated requests from cloud servers. The run still finishes for your other websites, and the reason is in the error field and SITEMAPS_REPORT.

Does it crawl pages that are not in the sitemap? No. It reads sitemaps only, which is fast and light on the website. Pages missing from the sitemap won't appear.

Missing a format or a field? Open an issue in the Issues tab. Custom extraction pipelines are available on request.

Only extract and use data from websites in line with their terms of use.