Sitemap URL Extractor – All URLs from XML Sitemaps

What does Sitemap URL Extractor do?

Sitemap URL Extractor pulls every URL out of a website's XML sitemaps and gives them back as a clean list with lastmod, changefreq and priority. Type a domain such as example.com and the Actor finds the sitemaps for you. It reads robots.txt, tries the standard locations, follows nested sitemap indexes, unpacks .xml.gz files and also reads plain-text, RSS and Atom sitemaps.

It is a light sitemap parser API. It uses plain HTTP requests, with no browser and no proxy, so it is fast and cheap. You can also check the HTTP status of each URL in the same run.

Why use Sitemap URL Extractor?

• 🔎 SEO audits and site migrations. Get the full list of indexable URLs, compare the old and new sitemaps, and find stale lastmod dates.
• 🤖 RAG, LLM and crawling pipelines. Build a clean, deduplicated URL list before you run a heavier crawler such as Website Content Crawler. You only pay for the pages you need.
• 📰 Content monitoring. Schedule a daily run with "Modified after: 1 day" to get only new or updated pages, including Google News titles from news sitemaps.
• 🧭 Competitor research. See how a site is organised, how many products or articles it has, and how often it publishes.

Apify platform features work out of the box: scheduling, REST API, webhooks, and integrations with Make, Zapier, n8n, Google Sheets, Slack and more. You can export results to JSON, CSV, Excel or XML.

What it handles that other sitemap tools miss

Problem	What this Actor does
Sitemap is not at /sitemap.xml	Reads every Sitemap: line in robots.txt, then tries /sitemap_index.xml, /sitemap-index.xml, /wp-sitemap.xml, /sitemap.xml.gz, /sitemap/sitemap.xml, /sitemaps.xml and /sitemap.txt
Sitemap index nested 2 or more levels deep	Follows indexes recursively, up to a cap you choose
.gz file that lies about itself	Detects gzip from the file's magic bytes, so it works whether or not the server sends Content-Encoding: gzip, and also when the file is not compressed at all
Index lists http:// children that fail	Retries automatically over https:// (for example, gnu.org returns HTTP 500 on http)
URL returns 200 with an HTML page (soft 404, SPA fallback)	Reported as a failure with a clear reason, not as a silent empty success
Text, RSS 2.0 and Atom sitemaps	All parsed, as allowed by the sitemaps.org protocol
Broken XML (stray &)	The parser recovers and keeps the valid entries
One bad site in a batch	The run continues. Every site gets a status and a reason in the SUMMARY record
Run "succeeds" with 0 results	The run is marked FAILED with the reason, so your schedules and alerts notice it (you can turn this off)
Huge sites	Per-site and total URL caps, a lastmod date filter that skips old child sitemaps before downloading them, include and exclude regexes, and deduplication

What data can Sitemap URL Extractor extract?

Field	Description
url	Page URL listed in the sitemap
lastmod	Last modification date (normalised to ISO 8601)
changefreq	daily, weekly… if present
priority	0.0 to 1.0 if present
sitemapUrl, sitemapDepth, sitemapFormat	Where the URL was found (child sitemap, depth in the index tree, and format: urlset, text, rss or atom)
imageCount, images	Image sitemap entries
newsTitle, newsPublicationDate	Google News sitemap data
alternates	hreflang alternates (multilingual sites)
statusCode, finalUrl	HTTP status and redirect target (optional status check)

How to extract all URLs from a sitemap

1. Click Try for free.
2. In Websites, domains or sitemap URLs, enter one or more domains (example.com), homepages or direct sitemap URLs (https://example.com/sitemap_index.xml).
3. Optionally set Max URLs per site, a Modified after date or include/exclude patterns.
4. Click Start and wait a few seconds.
5. Download the URL list as CSV, Excel or JSON, or get it via the API.

How much does it cost to extract sitemap URLs?

The Actor itself is free to use: there is no Actor fee for now. You only pay the Apify platform usage of your runs (compute units and data transfer), which stays small because the Actor uses plain HTTP requests, without a browser or proxy. The Apify Free plan includes $5 of monthly credit, which covers a lot of test runs.

To keep runs short, set Max URLs per site or Max URLs in total.

Input

See the Input tab for every option. Example:

{
    "urls": ["apify.com", "https://www.lemonde.fr", "https://www.gnu.org/sitemap.xml"],
    "maxUrlsPerSite": 5000,
    "lastmodFrom": "7 days",
    "excludePatterns": ["/tag/", "\\.pdf$"],
    "includeAlternates": false,
    "checkStatus": false
}

Output

You can download the dataset in various formats such as JSON, HTML, CSV or Excel. Example items:

[
    {
        "url": "https://www.lemonde.fr/international/article/2026/10/28/example_6801641_3210.html",
        "lastmod": "2026-10-28T20:01:34.000Z",
        "changefreq": null,
        "priority": null,
        "site": "https://www.lemonde.fr",
        "sitemapUrl": "https://www.lemonde.fr/sitemap_news.xml",
        "sitemapDepth": 0,
        "sitemapFormat": "urlset",
        "imageCount": 1,
        "newsTitle": "Example headline",
        "newsPublicationDate": "2026-10-28T20:01:34.000Z"
    },
    {
        "url": "https://wordpress.org/themes/example-theme/",
        "lastmod": "2026-10-20T09:12:00.000Z",
        "changefreq": "weekly",
        "priority": 0.6,
        "site": "https://wordpress.org",
        "sitemapUrl": "https://wordpress.org/themes/sitemap-1.xml",
        "sitemapDepth": 2,
        "sitemapFormat": "urlset",
        "imageCount": 0,
        "newsTitle": null,
        "newsPublicationDate": null
    }
]

The SUMMARY record in the key-value store contains one entry per input site:

{ "origin": "https://www.gnu.org", "robotsTxtStatus": 200, "sitemapsDeclaredInRobots": 1, "sitemapsFetched": 2,
  "sitemapsFailed": [], "urlsFound": 5381, "urlsPushed": 5381, "status": "ok" }

status is one of ok, partial, limit-reached, no-sitemap, blocked or error. blocked means the site answered with a captcha or bot challenge, or its robots.txt returned HTTP 401/403/429/5xx: the actor treats that as "no bots allowed" and never tries to get past it. Every failed sitemap is listed with its reason (HTTP 404, HTML page instead of a sitemap, invalid gzip, disallowed by robots.txt…).

Tips

• Monitoring new pages: schedule the Actor daily with lastmodFrom: "1 day". Old child sitemaps are skipped before download, so large news sites stay cheap.
• Speed: requests to the same host are spaced by 250 ms by default, and more if robots.txt sets a Crawl-delay. Several sites run in parallel (maxConcurrency).
• Status check sends one HEAD request per URL, so it is slower. Use it on filtered subsets.
• No sitemap found? Look at SUMMARY. If the site really has no sitemap, use a link crawler instead.

FAQ

Does it crawl the pages listed in the sitemap?

No. It downloads only robots.txt and the sitemap files. With Check HTTP status turned on, it also sends a HEAD request to each URL. It never downloads page content.

Does it respect robots.txt?

Yes. It identifies itself with a clear user agent (SitemapUrlExtractor/1.0), honours Crawl-delay (capped at 10 seconds by default), and never fetches guessed locations or checks URLs that robots.txt disallows. Sitemaps that the site itself declares in robots.txt are always allowed, because they are published for bots. Robots rules are checked again at every redirect. If robots.txt or a sitemap answers with a captcha or bot challenge (or robots.txt returns 401/403/429/5xx), the site is skipped and reported as blocked.

Can it read sitemap.xml.gz files?

Yes. It handles gzip whether or not the server sends Content-Encoding, double-compressed files, and .gz files that are really plain XML.

What is the maximum sitemap size?

The sitemaps.org protocol allows 50,000 URLs and 50 MB per file. The Actor accepts files up to 60 MB after decompression, and up to 500 files per site by default (you can raise this to 50,000).

Can I use it through the API or from Python/JavaScript?

Yes. See the API tab for ready-made curl, Python and Node.js snippets, or use the Apify MCP server from your AI agent.

Is it legal to extract sitemaps?

Sitemaps are public files that site owners publish so that robots can read them. This Actor collects no personal data, only page URLs and their metadata.

Our Actors are ethical and do not extract any private user data, such as email addresses, gender, or location. They only extract what the user has chosen to share publicly. We therefore believe that our Actors, when used for ethical purposes by Apify users, are safe. However, you should be aware that your results could contain personal data. Personal data is protected by the GDPR in the European Union and by other regulations around the world. You should not scrape personal data unless you have a legitimate reason to do so. If you're unsure whether your reason is legitimate, consult your lawyers.

Something does not work?

Open an issue in the Issues tab and include the site and the SUMMARY record. Issues are usually answered quickly.

Related actors

More Actors by the same author (innovative_cookie on Apify):

• PageSpeed Insights Bulk Checker: Core Web Vitals and Lighthouse scores for many URLs.
• Bulk Domain Availability Checker: check whether domain names are free via RDAP.
• Logo & Favicon Extractor: get the logo and favicon of any website.
• Bulk Image Converter & Compressor: convert and compress images to WebP, AVIF, JPEG or PNG.
• Broken Link Checker: find 404s and dead links on any website.
• Website Image Extractor: list every image of a page with size, format and alt text.
• Dataset Merge, Dedup & Transform: merge datasets, remove duplicates and clean columns.