Sitemap URL Extractor

Every URL listed in a website's sitemap files, with last-modified date, change frequency and priority, following nested sitemap indexes, for SEO audits, crawl planning and content inventories.

What does Sitemap URL Extractor do?

A sitemap is a file that a website publishes to tell search engines which pages it has. Big sites split theirs into many files and list those files in a "sitemap index". This Actor takes a website address (or the address of a sitemap file), finds the sitemap files, reads all of them, and gives you one flat list of page addresses.

It reads plain XML sitemaps, sitemap indexes inside sitemap indexes, gzip-compressed files (.xml.gz) and plain-text sitemaps with one address per line. It reads only the sitemap files, never the pages themselves. It is built for SEO consultants, developers planning a crawl, and anyone who needs a content inventory of a site.

What data does it return?

One row per page address. For an address that can't be read, one row that says why.

Field	Example	Meaning
inputUrl	https://www.gov.uk/	The website or sitemap address from your input that this row belongs to.
status	ok	ok for a page address. Otherwise no-sitemap, not-found, blocked-by-robots or error, explained below.
url	https://www.gov.uk/government/news/workshops-for-creative-industries-professionals	The page address listed in the sitemap. Empty when the row explains a problem.
host	www.gov.uk	The host name of the page address (of your input, when the row explains a problem).
lastmod	2026-06-01T12:58:18+01:00	When the page was last modified, as the sitemap states it (ISO 8601). Empty when the sitemap gives no valid date.
changefreq	weekly	How often the page changes, as the sitemap states it: always, hourly, daily, weekly, monthly, yearly or never. Often empty.
priority	0.29	The page's priority from 0.0 to 1.0, as the sitemap states it. Often empty.
sitemapUrl	https://www.gov.uk/sitemaps/sitemap_1.xml	The sitemap file the address was listed in.
message	HTTP 503: https://example.org/sitemap.xml	Empty for a page address. Otherwise what went wrong, in plain English.

The status of a row that explains a problem is one of these:

• no-sitemap: none of the usual places has a sitemap for the website.
• not-found: the sitemap file you gave doesn't exist.
• blocked-by-robots: the site's robots.txt doesn't allow reading it, or couldn't be read.
• error: anything else, for example a server error, a file that isn't a sitemap, or an address the Actor doesn't read.

A row that explains a problem counts as one result, like any other row. If part of a site's sitemap couldn't be read, you still get the URLs that were read, plus one such row at the end of that site, so a short list is never mistaken for a full one.

How to use it

1. Open the Input tab and add the websites or sitemap addresses. The prefilled values work as they are.
2. Click Start.
3. Download the results as JSON, CSV or Excel, or read them through the Apify API.

Input	Meaning
urls	Websites such as https://example.com, or sitemap files such as https://example.com/sitemap.xml or .xml.gz. For a website the Actor reads the Sitemap: lines in its robots.txt and, if there are none, tries /sitemap.xml, /sitemap_index.xml and /sitemap.xml.gz. A page address such as https://example.com/blog/ is treated as its website; use includePatterns to keep one section.
includePatterns	Keep a URL only if it matches at least one of these patterns. Write * for "anything", for example */blog/*. Not case sensitive, and the pattern can match any part of the URL.
excludePatterns	Drop a URL if it matches any of these patterns, for example *.pdf or */tag/*. Applied after includePatterns.
lastmodAfter	Keep only URLs that changed on or after this date, written YYYY-MM-DD. URLs without a valid last-modified date are left out when you set it. A date written as only a year or a month counts as its first day.
maxDepth	How many levels of sitemap files to follow below the first one, from 1 to 3 (default 3).
maxUrlsPerSite	Stop reading a website after this many URLs (default 10,000).
maxItems	Stop after this many results in total (default 1,000), up to 100,000 per run.

How much does it cost?

You pay $0.50 per 1,000 results, with no monthly fee. New Apify accounts get free monthly credit, so you can try it at no cost.

Example output

A page address from the sitemap of a large website:

{
    "inputUrl": "https://www.gov.uk/",
    "status": "ok",
    "url": "https://www.gov.uk/government/news/workshops-for-creative-industries-professionals",
    "host": "www.gov.uk",
    "lastmod": "2026-06-01T12:58:18+01:00",
    "changefreq": null,
    "priority": 0.29,
    "sitemapUrl": "https://www.gov.uk/sitemaps/sitemap_1.xml",
    "message": null
}

A website with no sitemap:

{
    "inputUrl": "https://example.org/",
    "status": "no-sitemap",
    "url": null,
    "host": "example.org",
    "lastmod": null,
    "changefreq": null,
    "priority": null,
    "sitemapUrl": null,
    "message": "No sitemap was found in robots.txt or at /sitemap.xml, /sitemap_index.xml or /sitemap.xml.gz."
}

Data source and limits

• Source: the sitemap files that each website you list publishes for search engines, following the Sitemaps protocol. Nothing else is read from the site except its robots.txt.
• Robots.txt: the Actor reads each host's robots.txt first and skips every address that it doesn't allow for this Actor or for crawlers in general. If robots.txt can't be read (it answers 401, 403 or a server error), nothing is read from that host. A Crawl-delay is kept, up to 30 seconds; a site that asks for more is skipped.
• Rate: about one request per second in total, and a retry with a pause after a 429 or server error.
• Fresh: the Actor reads the live files every time it runs.
• Size limits: a sitemap file may be up to 50 MB once unzipped (the protocol's limit), a website up to 500 sitemap files, and the Actor stops reading a website's remaining sitemap files after 5 of them fail.
• Not read: addresses with a username or password, ports other than 80, 443, 8080 and 8443, addresses on private or local networks, and social networks such as Facebook, LinkedIn, Instagram and TikTok.
• No contact details: a URL that contains an email address or a phone number is left out, so the output holds no personal contact data.

The data is public. This Actor doesn't collect personal data. It is an independent tool and isn't affiliated with, endorsed by or sponsored by any website you read or by any search engine.

FAQ

Can I run it on a schedule? Yes. Use Apify Schedules to run it daily or hourly and send the results to Google Sheets, Slack, a webhook or your own API. Set lastmodAfter to find pages that changed since the last run.

Why do I get fewer URLs than the site has pages? A sitemap lists only what the site's owner chooses to list. Some sites leave out pages, and some publish no sitemap at all.

Why are lastmod, changefreq and priority empty? They are optional in the Sitemaps protocol and many sites don't publish them. The Actor shows exactly what the sitemap says and doesn't guess.

What if a site blocks it? You get one row with the status blocked-by-robots and the reason. The Actor respects robots.txt and doesn't try to get around it.

Can I get more than 100,000 results in one run? No, maxItems stops at 100,000. For a bigger site, run it several times with includePatterns or lastmodAfter to split the list.

Is it legal? It reads only public sitemap files, which sites publish so that crawlers can read them, at a polite rate and only where robots.txt allows it. It collects no personal data. Check that your own use of the URLs fits the site's terms.