Sitemap URL Extractor
Pricing
Pay per usage
List every URL a website publishes in its XML sitemaps, found automatically through robots.txt. Filter by text or lastmod date, or get only the URLs added since your last run.
A website home page (for example https://crawlee.dev/) or the address of a sitemap or sitemap index. Several can be listed in one run.
[ { "url": "https://crawlee.dev/" }]Stop after saving this many URLs in total. Raise it to get the whole sitemap.
Keep only URLs that contain at least one of these texts, such as /blog/ or /products/. Case does not matter. Leave empty to keep everything.
Drop URLs that contain any of these texts, such as ?page= or /tag/.
Turn on to write the two lists above as regular expressions (Python syntax, case-insensitive), for example /p/\d+$.
Keep only URLs whose lastmod date is on or after this point. Write a date such as 2026-01-31, or a period such as 7 days, 2 weeks or 3 months. Leave empty to ignore dates.
When a date filter is set, URLs that have no lastmod in the sitemap are dropped. Turn this on to keep them.
Remembers the URLs seen for each website and returns only URLs that were not there last time. The first run returns everything and sets the baseline. Every sitemap is read in this mode, even after the maximum is reached, so the memory stays complete.
Keeps all rows but fills the isNew column (true or false) by comparing with earlier runs. Not needed when the option above is on.
Name of the key-value store in your account that holds the remembered URLs. Use a different name to keep separate histories, for example one per project.
Safety limit on how many sitemap files are read per run, including index files.
How many sitemap files to download at the same time. Capped at 4 to stay gentle on the website.