Sitemap URL Extractor: All Pages of Any Website
Pricing
from $0.40 / 1,000 url extracteds
Get every URL of a website from its XML sitemaps. Auto-discovers sitemaps from robots.txt, follows sitemap indexes and .gz files, and returns lastmod, changefreq, priority and image counts. Filter by URL pattern or last-modified date. USD 0.50 per 1,000 URLs.
A domain (example.com) finds the sitemaps automatically from robots.txt and the usual locations. A sitemap URL (https://example.com/sitemap.xml) starts from that sitemap.
Stop after this many URLs per site. 0 means no limit. Large sites can list hundreds of thousands of URLs, so start with a cap and raise it once you have seen the results.
Keep only URLs that contain any of these texts, for example /blog/. Wrap in slashes for a regular expression, for example //products/[^/]+$/.
Drop URLs that contain any of these texts, for example /tag/ or ?page=.
Only URLs whose sitemap lastmod is on or after this date (YYYY-MM-DD). URLs without a lastmod are skipped when this is set. Handy with a daily schedule to catch new and updated pages.
Add up to 20 image URLs per page from image sitemaps.
Stop after reading this many sitemap files on one site.
How many items to process at the same time.
How long to wait for one request before giving up.
