Sitemap URL Extractor & Status Checker

Sitemap URL Extractor & Status Checker reads a website's XML sitemaps and returns every URL, with its lastmod date and an optional HTTP status code.

What it does

Give it a website or a sitemap URL. It finds the sitemaps, follows every nested sitemap index, reads gzipped sitemaps and returns one row per URL. Turn on the status check and each row also gets the HTTP status code and the final URL after redirects.

It looks for sitemaps in three places: the Sitemap: lines in robots.txt, then /sitemap.xml and /sitemap_index.xml if robots.txt allows them.

When something goes wrong, the run doesn't fail silently. A site with no sitemap, a 403, a 429 or a timeout gives you a row that says what happened.

Here's what five rows look like:

url	lastmod	statusCode	finalUrl
https://example.com/	2026-09-20	200	https://example.com/
https://example.com/pricing	2026-09-18	200	https://example.com/pricing
https://example.com/blog/old-post	2025-11-02	301	https://example.com/blog/new-post
https://example.com/careers	2026-06-30	404	https://example.com/careers
https://example.com/docs/start	2026-09-01	200	https://example.com/docs/start

(Example data.)

Who it's for

• SEO audits. Get the full list of pages a site says it has, then check which ones return an error.
• Site migrations. Pull every URL from the old site before you move it, so you can map redirects and check nothing was lost.
• Content inventories. See every page and when it was last changed, in one spreadsheet.
• Feeding a crawler or an LLM pipeline. Hand the URL list to the next Actor or your own code through the Apify API.

Input

Field	What it does	Default
Websites or sitemap URLs	One or more homepages (https://example.com) or sitemaps (https://example.com/sitemap.xml). It works out which one you gave it.	none, required
Check HTTP status of each URL	Adds the status code, final URL, number of redirects and response time to each row. It costs more per URL (see Pricing).	off
Maximum URLs	Stops after this many unique URLs. Use it to test on a big site first.	1,000
Maximum concurrent requests	Status checks in flight at once, never more than 4 to one site.	5
Request timeout (seconds)	How long to wait for any one sitemap or page.	20

Input field names in the API: startUrls, checkStatus, maxUrls, maxConcurrency, requestTimeoutSecs.

The form starts at 100 URLs, so your first run costs $0.05. Turn on the status check and it's $0.10.

Example input:

{
  "startUrls": [{ "url": "https://example.com" }],
  "checkStatus": true,
  "maxUrls": 1000
}

Output

One row per URL in the run's dataset. Export it from the Console as CSV, JSON, Excel, XML or HTML, or read it through the Apify API.

Field	What it holds
type	url for a page, error for something it couldn't read.
url	The page URL, as the sitemap lists it.
lastmod	The last-modified date from the sitemap, if the site gives one.
changefreq	How often the site says the page changes, if given.
priority	The priority the site gives the page (0.0 to 1.0), if given.
sourceSitemap	The sitemap file the URL came from.
startUrl	The input the row came from.
statusCode	The HTTP status code. Only with the status check on.
finalUrl	Where the URL ends up after redirects. Only with the status check on.
redirectCount	How many redirects it followed. Only with the status check on.
responseTimeMs	Time to the response, in milliseconds. Only with the status check on.
error	Empty when the row is fine. Otherwise it says what went wrong, like "no sitemap found" or "403 Forbidden".

Example row (JSON):

{
  "url": "https://example.com/blog/old-post",
  "lastmod": "2025-11-02",
  "changefreq": "monthly",
  "priority": "0.5",
  "type": "url",
  "sourceSitemap": "https://example.com/post-sitemap.xml.gz",
  "statusCode": 200,
  "finalUrl": "https://example.com/blog/new-post",
  "redirectCount": 1,
  "error": null
}

Pricing

You pay per URL returned. There's no monthly fee.

• URLs only: $0.0005 per URL.
• Status check: $0.0005 per URL that answers with a status code, on top of the URL price. A URL that robots.txt disallows, or that times out or can't be reached, is listed with the reason and isn't charged for a check.

A worked example. A site has 2,000 pages in its sitemap.

• URLs only: 2,000 × $0.0005 = $1.00.
• With status codes: 2,000 × ($0.0005 + $0.0005) = $2.00.

Apify also charges its standard start fee for each run. Set Maximum URLs, or a maximum cost per run, for a hard ceiling: the run stops cleanly when it reaches your maximum cost.

How it handles the hard cases

• No sitemap. You get one row with the error "no sitemap found" and nothing else is charged. The run finishes normally.
• Nested sitemap indexes. It follows each index down to the sitemaps that hold the URLs.
• Gzipped sitemaps. It reads .xml.gz files directly.
• 403 and 429. A blocked request or a rate limit is recorded in the error field instead of failing the run. It doesn't retry a 429 in this version.
• Timeouts. A sitemap that doesn't answer in time is recorded as an error row. The timeout is 20 seconds unless you change it.
• robots.txt. It reads robots.txt first and reads the sitemaps it lists. It only tries /sitemap.xml and /sitemap_index.xml when robots.txt allows them, and it never checks a page that robots.txt disallows for its user agent. If robots.txt itself returns a server error or can't be reached, it doesn't look for sitemaps or check pages on that site, and says so in the row. A sitemap URL you give it directly is always read.
• Load on the site. It limits how fast it requests pages from any one host (at most 4 requests at a time, and it honours a Crawl-delay) and identifies itself with an honest user agent that names this Actor (SitemapURLExtractor).

Compared with doing it by hand

You can do this yourself. Open /sitemap.xml in a browser, open each child sitemap, unzip the gzipped ones and copy the URLs into a spreadsheet. For a small site with one sitemap, that's quick, and a free web tool will do it too.

It gets slow when the site has a sitemap index with dozens of files, or you need status codes for thousands of URLs, or you need the list every week. That's where this Actor fits. It does the whole site in one run, you can schedule it, and it hands the list to your code through the API.

Compared with other sitemap Actors in the store, the difference is how it fails. A site it can't read gives you a row that says why, not a failed run with nothing in it.

FAQ

How do I extract all URLs from a sitemap? Put the sitemap URL, or just the homepage, in the input and run it. The Actor follows every nested sitemap and returns one row per URL. Export the dataset as CSV, JSON or Excel.

How do I find a website's sitemap? Give the Actor the homepage. It checks the Sitemap: lines in robots.txt and the usual locations, /sitemap.xml and /sitemap_index.xml. The sourceSitemap field on each row shows which file the URL came from.

Why did my run return no URLs? The site has no sitemap it could find, or it blocked the request. Check the error field. It says "no sitemap found", a status like 403, or a timeout, and lists every place it looked. If the sitemap lives somewhere unusual, put its full URL in the input instead of the homepage.

Does it respect robots.txt? Yes. It reads the sitemaps robots.txt lists, only tries the usual sitemap locations when robots.txt allows them, and never checks a page robots.txt disallows. It also limits its request rate per site.

Can it check the HTTP status of every URL? Yes. Turn on "Check HTTP status". Each row gets the status code (200, 301, 404, 500 and so on) and the final URL after redirects. That's how you find broken pages and redirect chains in a sitemap.

How do I export a sitemap to CSV? Run the Actor, open the run's Dataset tab and pick CSV. You can also download it through the API with format=csv.

Can I run it on a schedule? Yes. Use Apify Schedules to run it daily or weekly with the same input. Each run makes a new dataset, so you can compare lists over time.

Can I call it from my own code or connect it to other tools? Yes. Start it and read the results through the Apify API or the Python and JavaScript clients. It also works with Apify's integrations, like webhooks, Make, Zapier and Google Sheets.

Limits

• It reads XML sitemaps (including gzipped ones). It doesn't crawl the site's links, so a page that isn't in any sitemap won't show up Plain-text (.txt) and RSS or Atom sitemaps aren't read in this version.
• It works on public pages. It doesn't log in.
• A sitemap file can hold up to 50,000 URLs under the sitemap protocol. Very large sites take longer It runs in 256 MB by default. A sitemap file over 55 MB uncompressed is refused.
• The status check makes one request per URL, so it takes longer and costs more than URLs alone.

Integrations

This Actor runs on Apify, so everything the platform does works with it.

• API. Start a run and read the results from your own code, in any language. Apify API
• Schedules. Run it weekly and sort by lastmod to see which pages changed. Schedules
• Tasks. Save your input once as a task and run it again with one click, or on a schedule. Tasks
• Webhooks. Get a call to your own URL when a run succeeds or fails. Webhooks
• Google Sheets and Drive. Download the results as CSV or Excel and open them in Sheets. Or add a Google Sheets step after this Actor in Make or Zapier. Or send the files to a Google Drive folder. Datasets and exports, Google Drive
• Zapier, Make and n8n. Start a run from another app and send the results on to the next step. Zapier, Make, n8n
• AI agents (MCP). Claude, ChatGPT and other MCP clients can find and run this Actor through the Apify MCP server. Apify MCP server
• Other Actors. Pass this Actor's results to another Actor when a run finishes. Actor-to-Actor

Works with

• Site Audit: an SEO score from 0 to 100 for every page, with a fix for each issue.
• Broken Link Checker from Sitemap: checks every page in your sitemap for 404s, server errors and redirect chains.
• Bulk PageSpeed Insights for a Sitemap: runs Google PageSpeed Insights on every URL in your sitemap.

Support

Found a bug or need a feature? Open an issue on this Actor's Issues tab. Include the run ID and the URL you gave it.