Sitemap URL Extractor & Scraper: All Website URLs

What does Sitemap URL Extractor do?

Sitemap URL Extractor gives you every page URL a website lists in its sitemaps, as a clean table you can download as JSON, CSV or Excel or read through the API.

Enter a website address. The tool finds the sitemaps on its own, follows sitemap indexes down to the last file, and returns each URL with its last-modified date. You do not need to know where the sitemap is.

• Finds the sitemaps for you: reads robots.txt, then tries the usual locations such as /sitemap.xml, /sitemap_index.xml and /wp-sitemap.xml.
• Follows sitemap indexes: an index that points to 500 sitemap files is read to the end, several files at a time.
• Reads every sitemap format: XML, gzip-compressed files (.xml.gz), RSS and Atom feeds, and plain text lists.
• Optional status check: requests each URL and adds its HTTP status, redirect target and response time, so you can find dead pages and redirects in a sitemap.
• Filters: keep only URLs that match a pattern, skip others, or keep only pages changed since a date.
• No duplicates: a URL listed in several sitemaps is returned once.

What data do you get?

Field	Example	Meaning
url	https://www.example.com/blog/post-1	The page URL from the sitemap
lastModified	2026-09-30T08:00:00.000Z	The last-modified date the sitemap gives, when it has one
changeFrequency	weekly	The changefreq value, when present
priority	0.8	The priority value, when present
imageCount	2	How many images the sitemap lists for the page
alternateCount	3	How many language versions (hreflang) the sitemap lists
sitemapUrl	https://www.example.com/sitemap-posts.xml	The sitemap file the URL came from
site	https://www.example.com	The website
httpStatus	200	With the status check on: the HTTP status after redirects
redirectUrl	https://www.example.com/new-page	With the status check on: where the URL redirects to
responseTimeMs	184	With the status check on: how long the request took
error	null	Empty when all went well, otherwise a code and a message

Turn on Include image and language entries to also get the image URLs and the hreflang links themselves.

How to get all URLs from a sitemap

1. Open the tool and enter one or more websites under Websites or sitemaps, for example https://www.example.com. A direct sitemap address such as https://www.example.com/sitemap.xml works too.
2. Set Maximum URLs if you want to cap the size of the run.
3. Press Start.
4. Open the Output tab and download the result as JSON, CSV, Excel or HTML, or fetch it through the API.

To run it on a schedule, for example every morning to pick up new pages, use the Schedule option in Apify Console.

How much does it cost to extract sitemap URLs?

You pay for results, not for time: $0.30 per 1,000 URLs returned. Apify subscribers on the Silver plan pay $0.25 and on the Gold plan $0.20 per 1,000 URLs. Each run also has a start fee of $0.00005.

URLs returned	Cost
100	$0.03
1,000	$0.30
10,000	$3.00
100,000	$30.00

Apify's free plan includes $5 of credit each month, which covers about 16,000 URLs.

URLs that are filtered out and duplicates are not charged. Set Maximum URLs, or a maximum cost for the run in Apify Console, and the tool stops there and keeps what it has found.

Input

The only required field is Websites or sitemaps.

{
    "startUrls": [{ "url": "https://www.example.com" }],
    "maxUrls": 10000,
    "checkStatus": false,
    "includePatterns": ["/blog/"],
    "excludePatterns": ["*.pdf"],
    "modifiedAfter": "30 days"
}

Patterns are plain text that may appear anywhere in the URL, with * for any characters. For a regular expression, wrap it in slashes, for example /\/product\/\d+$/.

Output example

{
    "url": "https://www.example.com/blog/post-1",
    "lastModified": "2026-09-30T08:00:00.000Z",
    "changeFrequency": "weekly",
    "priority": 0.8,
    "imageCount": 2,
    "alternateCount": 0,
    "sitemapUrl": "https://www.example.com/sitemap-posts.xml",
    "site": "https://www.example.com",
    "error": null
}

Each run also stores a SUMMARY record with the number of sitemaps read, URLs found, duplicates skipped, and any sitemap file that could not be read and why.

What can you use it for?

• SEO audits: get the full list of pages a site wants indexed, then check them for dead pages and redirects with the status check.
• Site migrations: export every URL of the old site before a redesign or domain move, and verify afterwards that nothing was lost.
• Feeding a crawler or an AI pipeline: use the URL list as the start list for a content crawler instead of crawling blindly.
• Content monitoring: run it daily with Changed since set to 1 day to get only new and updated pages.
• Competitor research: see how many pages a site has, how they are organised and how often they change.

Use it from the API

curl -X POST "https://api.apify.com/v2/acts/cyber1nj3ct0r~sitemap-url-extractor/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "startUrls": [{ "url": "https://www.example.com" }], "maxUrls": 1000 }'

The tool also works as a tool for AI agents through the Apify MCP server.

FAQ

The tool says no sitemap was found. Why?

Some sites have no sitemap, or keep it at an unusual address that is not listed in robots.txt. If you know the address, enter it directly. If the site has none, a sitemap cannot be read; you would need a crawler that follows links instead.

Does it find pages that are not in the sitemap?

No. It returns what the sitemaps list. Pages a site left out of its sitemap are not found.

Can it read very large sitemaps?

Yes. Sitemap files are read piece by piece, sitemap indexes are followed up to eight levels deep, and a site with millions of URLs works as long as Maximum URLs allows it. Duplicate removal covers the first 1.5 million URLs of a run.

A site blocks the tool. What can I do?

The tool identifies itself honestly in its User-Agent and does not try to get round blocks. If it is your own site, allow the User-Agent in your firewall, or set your own under More options.

Is it allowed to read sitemaps?

Sitemaps and robots.txt are published so that programs can read them. The tool reads only those files and, when you turn on the status check, requests each listed page once. You are responsible for how you use the results.

Something does not work

Open an issue on the Issues tab with the website you tried. Every report is read and answered.