Sitemap URL Extractor

Give it a website and it returns every URL the site lists in its XML sitemaps, one row per URL. You don't need to know where the sitemap lives: the Actor reads the Sitemap: lines in robots.txt, falls back to the usual places such as /sitemap.xml and /sitemap_index.xml, and then follows sitemap indexes and .gz files down to the last URL.

Typical uses: an inventory of a site before an SEO audit or a migration, a list of pages to feed into another scraper, or a daily check for pages a competitor has just published.

What you put in

The only required field is the website (or a sitemap address, if you already have one):

{
  "startUrls": [{ "url": "https://crawlee.dev/" }],
  "maxItems": 5000
}

You can list several websites in one run.

Setting	What it does
maxItems	Stops after this many URLs in total. The default is 50 so a first run stays quick; raise it to get the whole sitemap.
includePatterns / excludePatterns	Keep only URLs that contain one of your texts (for example /blog/), or drop URLs that contain one (for example /tag/). Case does not matter.
patternsAreRegex	Read the two lists above as regular expressions instead of plain text.
lastmodSince	Keep only URLs whose lastmod is on or after a date (2026-01-31) or within a period (7 days, 2 weeks, 3 months).
keepUrlsWithoutLastmod	With a date filter, URLs that have no lastmod are dropped unless you turn this on.
onlyNewUrls	Returns only URLs that were not in the sitemap on your previous run. See below.
rememberUrls	Keeps every row but marks each one with isNew: true or false.
maxSitemaps	Safety limit on the number of sitemap files read in one run (default 500).

What you get

One row per unique URL:

{
  "url": "https://crawlee.dev/blog",
  "lastmod": null,
  "changefreq": "weekly",
  "priority": 0.5,
  "images": 0,
  "videos": 0,
  "news": 0,
  "hreflangCount": 0,
  "hreflangs": null,
  "sitemapUrl": "https://crawlee.dev/sitemap.xml",
  "startUrl": "https://crawlee.dev/",
  "isNew": null
}

• lastmod, changefreq and priority are copied as the site publishes them. Many sites leave some of them out, and those come back as null.
• images, videos and news count the image, video and news sitemap entries attached to the URL.
• hreflangs lists the language codes of the alternate versions declared for the URL (x-default is left out), and hreflangCount is their number.
• sitemapUrl tells you which sitemap file the URL came from, which helps when a site splits its sitemap by section.
• isNew is filled only when change tracking is on.

A URL that appears in more than one sitemap is returned once.

The run summary

Next to the table, the run saves one SUMMARY record in its key-value store: how many sitemap files were read, failed or skipped, how many URLs were found and saved, and for each sitemap file where it was found (robots.txt, a sitemap index, a usual location or your input) and what happened to it. When a file can't be used, the reason is written there and the run carries on with the rest. Typical reasons are HTTP 404, "the server returned an HTML page instead of a sitemap", "broken XML" (URLs read before the break are kept) and "disallowed by robots.txt".

Getting only new URLs

Turn on onlyNewUrls and run the same input on a schedule. The first run returns everything and remembers it; each later run returns only the URLs that have appeared since. The memory is a key-value store in your own account, named by stateStoreName, with one record per website. It holds short fingerprints of the URLs, not the URLs themselves. Use a different name to keep separate histories.

In this mode every sitemap file is read even after maxItems is reached, so that the memory stays complete. On large sites that makes the run longer.

Price

Free for now. You only pay Apify's usual platform usage. In our test, a run that read one sitemap of about 3,900 URLs and saved 50 took about 3 seconds, which costs a small fraction of a cent. A per-URL price may be added later. If that happens, it will be shown on this page before it applies to you.

How it treats the sites it reads

• It only reads robots.txt and the sitemap files a site publishes for search engines. It never opens the pages listed in them.
• It follows robots.txt. A sitemap in a location that robots.txt disallows is skipped, and so is everything on a site whose robots.txt answers with a server error or "forbidden". A crawl delay in robots.txt is respected.
• By default it makes at most two requests at a time and waits at least half a second between requests to the same site. You can slow it down further, but not beyond four parallel requests.

Limits

• It reads XML sitemaps, sitemap indexes, gzip-compressed sitemaps and plain-text sitemaps (one URL per line). RSS and Atom feeds are not read.
• A site with no sitemap in robots.txt and none at the usual locations returns no rows. The summary says so.
• Files larger than 50 MB after decompression are skipped, as the sitemap standard sets that limit.
• The result is what the site says in its sitemap. Pages missing from the sitemap are not found, and a lastmod date is only as accurate as the site keeps it.
• A sitemap index nested more than five levels deep is not followed further.

This Actor is an independent tool. It is not made by, or affiliated with, any of the websites it reads or the companies behind sitemap standards.