Sitemap URL Extractor & Status Checker

Extract every URL from a website's XML sitemaps in one run. Give it a homepage or a sitemap URL. It finds the sitemaps, follows every nested sitemap index, reads gzipped sitemaps and returns one row per URL with its lastmod date. Turn on the status check to also get the HTTP status code and final URL of every page.

• Finds the sitemap for you. It checks robots.txt, /sitemap.xml and /sitemap_index.xml.
• Handles big sites. Sitemap indexes with dozens of files, and gzipped sitemaps.
• Status codes on request. Find the 404s, redirects and slow pages across the whole sitemap.
• Ready for code and AI agents. Every output field is typed, and agents can run it through Apify's MCP server.
• Only what changed. Run it on a schedule with Changes only on, and you get just the new URLs, removed URLs and changed status codes.
• $0.50 per 1,000 URLs. Export to CSV, JSON or Excel, or send the list straight to another Actor.

Read this first

Every run ends with a short summary in plain English. It's the first line of the run's status, SUMMARY.md on the Output tab (Summary), and the last row of the dataset (summaryText). With Changes only on, a weekly run reads like this:

3 URLs added, 1 removed, 1 newly failing.

## Added (3)
- https://example.com/blog/new-post
- https://example.com/products/new-mug
- https://example.com/pages/faq

## Removed (1)
- https://example.com/old-page

## Newly failing (1)
- https://example.com/pricing: HTTP 404 (was HTTP 200).

## Run
1,204 URLs read from 6 sitemaps.

A quiet week is one line: "All clear. No changes since the last run." With Changes only off, the summary counts URLs by status code and lists the failing ones.

Quick start

1. Put a homepage or sitemap URL in Websites or sitemap URLs.
2. Click Start. The form starts at 100 URLs, so your first run costs $0.05.
3. Open the Dataset tab and export as CSV.

What does Sitemap URL Extractor do?

Give it a website or a sitemap URL. It finds the sitemaps, follows every nested sitemap index, reads gzipped sitemaps and returns one row per URL. Turn on the status check and each row also gets the HTTP status code and the final URL after redirects.

It looks for sitemaps in three places: the Sitemap: lines in robots.txt, then /sitemap.xml and /sitemap_index.xml if robots.txt allows them.

When something goes wrong, the run doesn't fail silently. A site with no sitemap, a 403, a 429 or a timeout gives you a row that says what happened.

Here's what five rows look like:

url	lastmod	statusCode	finalUrl
https://example.com/	2026-09-20	200	https://example.com/
https://example.com/pricing	2026-09-18	200	https://example.com/pricing
https://example.com/blog/old-post	2025-11-02	301	https://example.com/blog/new-post
https://example.com/careers	2026-06-30	404	https://example.com/careers
https://example.com/docs/start	2026-09-01	200	https://example.com/docs/start

(Example data.)

Who uses it

• SEO audits. Get the full list of pages a site says it has, then check which ones return an error.
• Site migrations. Pull every URL from the old site before you move it, so you can map redirects and check nothing was lost.
• Content inventories. See every page and when it was last changed, in one spreadsheet.
• Feeding a crawler or an LLM pipeline. Hand the URL list to the next Actor or your own code through the Apify API.

Input

Field	What it does	Default
Websites or sitemap URLs	One or more homepages (https://example.com) or sitemaps (https://example.com/sitemap.xml). It works out which one you gave it.	none, required
Check HTTP status of each URL	Adds the status code, final URL, number of redirects and response time to each row. It costs more per URL (see Pricing).	off
Maximum URLs	Stops after this many unique URLs. Use it to test on a big site first.	1,000
Changes only	Output only what changed since this job's last run: URLs added, URLs removed and, with the status check on, URLs with a new status code. See Changes-only mode.	off
Start changes-only over	Forget what this job saved. Every URL counts as new again.	off
Maximum concurrent requests	Status checks in flight at once, never more than 4 to one site.	5
Request timeout (seconds)	How long to wait for any one sitemap or page.	20

Input field names in the API: startUrls, checkStatus, maxUrls, onlyChanges, resetChanges, maxConcurrency, requestTimeoutSecs.

The form starts at 100 URLs, so your first run costs $0.05. Turn on the status check and it's $0.10.

Example input:

{
  "startUrls": [{ "url": "https://example.com" }],
  "checkStatus": true,
  "maxUrls": 1000
}

Output

One row per URL in the run's dataset. Export it from the Console as CSV, JSON, Excel, XML or HTML, or read it through the Apify API.

Field	What it holds
type	url for a page, error for something it couldn't read, summary for the run's summary, always the last row.
url	The page URL, as the sitemap lists it.
lastmod	The last-modified date from the sitemap, if the site gives one.
changefreq	How often the site says the page changes, if given.
priority	The priority the site gives the page (0.0 to 1.0), if given.
sourceSitemap	The sitemap file the URL came from.
startUrl	The input the row came from.
statusCode	The HTTP status code. Only with the status check on.
finalUrl	Where the URL ends up after redirects. Only with the status check on.
redirectCount	How many redirects it followed. Only with the status check on.
responseTimeMs	Time to the response, in milliseconds. Only with the status check on.
error	Empty when the row is fine. Otherwise it says what went wrong, like "no sitemap found" or "403 Forbidden".
change	Only in changes-only mode. new, removed or changed (the status code changed).
previousStatusCode	Only in changes-only mode. The status code from the last run, for a changed or removed URL.
summaryText	Only on the summary row. The run in plain English, the same text as SUMMARY.md (see Read this first).

Example row (JSON):

{
  "url": "https://example.com/blog/old-post",
  "lastmod": "2025-11-02",
  "changefreq": "monthly",
  "priority": "0.5",
  "type": "url",
  "sourceSitemap": "https://example.com/post-sitemap.xml.gz",
  "statusCode": 200,
  "finalUrl": "https://example.com/blog/new-post",
  "redirectCount": 1,
  "error": null
}

Changes-only mode: track new and removed pages

Turn on Changes only and each run outputs only what changed since the same job's last run. Each row gets a change value:

• new: the URL is in the sitemaps now and wasn't last time.
• removed: the URL was there last time and isn't now. This is only reported after a complete run: every sitemap read, and no early stop from Maximum URLs, the time limit or your maximum cost. After a partial run the status message says removals weren't reported, and the old URLs are kept for next time.
• changed: the status check is on and the URL's status code isn't the same as last time. previousStatusCode has the old one.

Error rows are always listed.

The first run has nothing to compare with. Every URL is output, marked new, and charged like a normal run.

A job is the same start URLs (in any order), Maximum URLs and status check setting. Change any of those and it's a new job, with its own first run.

Where the state lives. What each job saw is kept in your own Apify account, in a key-value store named sitemap-url-extractor-changes. It holds the URLs and their status codes, nothing else.

Starting over. Run the job once with Start changes-only over on. Or delete its record (job-...), or the whole store, under Storage in the Apify Console.

Limits. With Changes only on, Maximum URLs can be at most 50,000. A job keeps at most 5 MB, about 45,000 URLs of 100 characters each. A run that would go past that stops before the URLs that don't fit.

It doesn't charge for those URLs, and its status says why it stopped. It counts as a partial run, so it reports no removals.

What it costs: the normal price, even when nothing changed and no rows come back. Every URL is still read from the sitemaps, and checked if the status check is on. That work is what's charged, whether or not the URL is output. Removed URLs aren't charged.

How much does it cost to extract a sitemap?

You pay per URL returned. In changes-only mode you pay per URL read instead, as above. There's no monthly fee.

• URLs only: $0.0005 per URL.
• Status check: $0.0005 per URL that answers with a status code, on top of the URL price. A URL that robots.txt disallows, times out, can't be reached or answers 429 (rate-limited) is listed with the reason and isn't charged for a check.

A worked example. A site has 2,000 pages in its sitemap.

• URLs only: 2,000 × $0.0005 = $1.00.
• With status codes: 2,000 × ($0.0005 + $0.0005) = $2.00.

Apify also charges its standard start fee for each run. Set Maximum URLs, or a maximum cost per run, for a hard ceiling: the run stops cleanly when it reaches your maximum cost.

How it handles the hard cases

• No sitemap. You get one row with the error "no sitemap found" and nothing else is charged. The run finishes normally.
• Nested sitemap indexes. It follows each index down to the sitemaps that hold the URLs.
• Gzipped sitemaps. It reads .xml.gz files directly.
• 403 and 429. A blocked request or a rate limit is recorded in the error field instead of failing the run. A 429 isn't charged as a status check, since it doesn't say what the page's status is. It doesn't retry a 429 in this version.
• Timeouts. A sitemap that doesn't answer in time is recorded as an error row. The timeout is 20 seconds unless you change it.
• robots.txt. It reads robots.txt first and reads the sitemaps it lists. It only tries /sitemap.xml and /sitemap_index.xml when robots.txt allows them, and it never checks a page that robots.txt disallows for its user agent. If robots.txt itself returns a server error or can't be reached, it doesn't look for sitemaps or check pages on that site, and says so in the row. A sitemap URL you give it directly is always read.
• Load on the site. It limits how fast it requests pages from any one host (at most 4 requests at a time, and it honours a Crawl-delay) and identifies itself with an honest user agent that names this Actor (SitemapURLExtractor).

Sitemap URL Extractor vs. doing it by hand

You can do this yourself. Open /sitemap.xml in a browser, open each child sitemap, unzip the gzipped ones and copy the URLs into a spreadsheet. For a small site with one sitemap, that's quick, and a free web tool will do it too.

It gets slow when the site has a sitemap index with dozens of files, or you need status codes for thousands of URLs, or you need the list every week. That's where this Actor fits. It does the whole site in one run, you can schedule it, and it hands the list to your code through the API.

Compared with other sitemap Actors in the store, the difference is how it fails. A site it can't read gives you a row that says why, not a failed run with nothing in it.

FAQ

How do I extract all URLs from a sitemap? Put the sitemap URL, or just the homepage, in the input and run it. The Actor follows every nested sitemap and returns one row per URL. Export the dataset as CSV, JSON or Excel.

How do I get a list of all pages on a website? Most sites list their pages in an XML sitemap. Give the Actor the homepage and it returns every URL the sitemaps list, one row each. A page that isn't in any sitemap won't show up; Site Audit can find those by following links.

How do I find a website's sitemap? Give the Actor the homepage. It checks the Sitemap: lines in robots.txt and the usual locations, /sitemap.xml and /sitemap_index.xml. The sourceSitemap field on each row shows which file the URL came from.

Why did my run return no URLs? The site has no sitemap it could find, or it blocked the request. Check the error field. It says "no sitemap found", a status like 403, or a timeout, and lists every place it looked. If the sitemap lives somewhere unusual, put its full URL in the input instead of the homepage.

Does it respect robots.txt? Yes. It reads the sitemaps robots.txt lists, only tries the usual sitemap locations when robots.txt allows them, and never checks a page robots.txt disallows. It also limits its request rate per site.

Can it check the HTTP status of every URL? Yes. Turn on "Check HTTP status". Each row gets the status code (200, 301, 404, 500 and so on) and the final URL after redirects. That's how you find broken pages and redirect chains in a sitemap.

How do I export a sitemap to CSV? Run the Actor, open the run's Dataset tab and pick CSV. You can also download it through the API with format=csv.

Can I run it on a schedule? Yes. Use Apify Schedules to run it daily or weekly with the same input. Each run makes a new dataset, so you can compare lists over time.

Do I need an API key or a login? No. It reads public sitemaps, so the only required input is the website or sitemap URL.

Can an AI agent run it? Yes. Claude, ChatGPT and other MCP clients can find and run it through the Apify MCP server. The only required input is startUrls, every output field is typed and described in the dataset schema, and it's approved for Apify's agentic payments, so an agent can pay for its own runs.

Can I call it from my own code or connect it to other tools? Yes. Start it and read the results through the Apify API or the Python and JavaScript clients. It also works with Apify's integrations, like webhooks, Make, Zapier and Google Sheets.

Limits

• It reads XML sitemaps (including gzipped ones). It doesn't crawl the site's links, so a page that isn't in any sitemap won't show up. Plain-text (.txt) and RSS or Atom sitemaps aren't read in this version.
• It works on public pages. It doesn't log in.
• A sitemap file can hold up to 50,000 URLs under the sitemap protocol. Very large sites take longer. It runs in 256 MB by default. A sitemap file over 55 MB uncompressed is refused.
• The status check makes one request per URL, so it takes longer and costs more than URLs alone.

Integrations

This Actor runs on Apify, so everything the platform does works with it.

• API. Start a run and read the results from your own code, in any language. Apify API
• Schedules. Run it weekly and sort by lastmod to see which pages changed. Schedules
• Tasks. Save your input once as a task and run it again with one click, or on a schedule. Tasks
• Webhooks. Get a call to your own URL when a run succeeds or fails. Webhooks
• Google Sheets and Drive. Download the results as CSV or Excel and open them in Sheets. Or add a Google Sheets step after this Actor in Make or Zapier. Or send the files to a Google Drive folder. Datasets and exports, Google Drive
• Zapier, Make and n8n. Start a run from another app and send the results on to the next step. Zapier, Make, n8n
• AI agents (MCP). Claude, ChatGPT and other MCP clients can find and run this Actor through the Apify MCP server. Apify MCP server
• Other Actors. Pass this Actor's results to another Actor when a run finishes. Actor-to-Actor

The SEO toolkit

Five Actors that work together. All are priced per result, with no monthly fee.

Actor	What you get	Price
Website SEO Audit & Crawler	A 0–100 SEO score and a fix list for every page, with week-over-week changes	$5 / 1,000 pages
Broken Link Checker	Every 404, server error and redirect chain, with the page it's on	$1 / 1,000 links
Bulk PageSpeed Insights	Lighthouse score and Core Web Vitals for every URL, mobile and desktop	$2.50 / 1,000 tests
Sitemap URL Extractor (this one)	Every URL in a site's sitemaps, with lastmod and status codes	$0.50 / 1,000 URLs
Website Screenshot API	A clean full-page screenshot of every URL, with cookie banners and pop-ups hidden	$8 / 1,000 screenshots

Support

Found a bug or need a feature? Open an issue on this Actor's Issues tab. Include the run ID and the URL you gave it.

If it saved you time, a quick review on the Reviews tab helps other people find it.