Sitemap URL Extractor & New Page Monitor

Get every URL of a website from its XML sitemaps, in seconds. Enter a website and the Actor finds its sitemaps for you (robots.txt, /sitemap.xml, /sitemap_index.xml, WordPress, Shopify, Yoast...), opens sitemap indexes and .gz files, and returns one clean row per page with lastmod, changefreq, priority, hreflang language versions, image count and Google News data.

Turn on "Only new URLs since the last run" and schedule it daily: you get a list of every new page, product or blog post a website publishes. Perfect for competitor monitoring and SEO.

• ✅ Auto-discovery: robots.txt Sitemap: lines + the usual locations
• ✅ Sitemap indexes, nested indexes, gzip (.xml.gz), plain-text sitemaps, Google News and image sitemaps
• ✅ lastmod, changefreq, priority, hreflang alternates, image count, news title and date
• ✅ Filters: URL contains / excludes, regular expression, modified in the last X days
• ✅ "Only new URLs" mode for daily new-page alerts
• ✅ Fast and cheap: no browser, streams huge sitemaps (50,000 URLs per file) with little memory

What can you use it for?

• Competitor monitoring: get an alert for every new product, landing page or article a competitor publishes.
• SEO audits: list all indexable URLs, find pages that were not updated for years, check hreflang setup.
• Content inventories and migrations: export the full URL list of a site to Excel or Google Sheets.
• Feeding other scrapers and AI tools: use the URL list as input for a content crawler or RAG pipeline.
• E-commerce: track new products on Shopify or WooCommerce stores (their sitemaps list every product).

How to use it

1. Enter one or more websites (https://www.example.com) or sitemap URLs (https://www.example.com/sitemap_index.xml).
2. Optional: filter with URL must contain (e.g. /products/, /blog/) or Modified in the last X days.
3. Click Start and download the URLs as JSON, CSV, Excel, or via the API.
4. For daily new-page alerts: turn on Only new URLs since the last run and create a Schedule.

Input example

{
    "startUrls": ["https://www.competitor.com", "https://shop.example.com/sitemap_products_1.xml"],
    "urlContains": ["/products/"],
    "urlExcludes": ["?variant="],
    "modifiedWithinDays": 30,
    "maxSitemaps": 500,
    "maxResults": 10000,
    "onlyNewSinceLastRun": true
}

Field	Default	Meaning
startUrls	–	Websites (we find the sitemaps) or sitemap URLs. Max 200.
urlContains	[]	Keep URLs that contain any of these texts (not case-sensitive).
urlExcludes	[]	Skip URLs that contain any of these texts.
urlRegex	empty	Keep URLs that match this regular expression.
modifiedWithinDays	empty	Keep pages with a lastmod (or news date) in the last X days.
includeWithoutLastmod	false	With a date filter: also keep pages without a date.
maxSitemaps	500	Maximum sitemap files to open per website.
maxResults	10000	Maximum URLs (and maximum cost) per run.
onlyNewSinceLastRun	false	Only return URLs that an earlier run with the same start URLs did not return.

Output example

{
    "url": "https://www.allbirds.com/products/mens-wool-runner-up-mizzle-plus",
    "domain": "www.allbirds.com",
    "lastmod": "2026-10-08T03:52:52.000Z",
    "changefreq": "daily",
    "priority": null,
    "alternates": [],
    "imageCount": 1,
    "newsTitle": null,
    "newsPublishedAt": null,
    "sitemapUrl": "https://www.allbirds.com/sitemap_products_1.xml?from=1878193471557&to=7369944137808",
    "startUrl": "https://www.allbirds.com/",
    "scrapedAt": "2026-10-08T10:40:20.363Z"
}

Every row has the same fields. Missing values are null (or an empty list). Dates are converted to UTC ISO format. A URL that appears in several sitemaps is returned (and charged) only once per run.

Pricing

Pay per result: $1 per 1,000 URLs ($0.001 per URL). You only pay for URLs in your dataset; duplicates are never charged. Set a maximum cost per run and the Actor stops cleanly when it is reached.

Free Apify plan: a run returns at most 100 URLs. Upgrade to a paid Apify plan to get all results.

FAQ

Is this legal? Yes. Sitemaps are files that websites publish specifically so that robots can find their pages. The Actor only reads robots.txt and the sitemap files, keeps a polite request rate (max 4 requests per second per site) and does not open the pages themselves. It collects no personal data.

"No sitemap found"? Some websites do not list their sitemap in robots.txt and use an unusual location. Find it in your browser (often linked in the footer or in the SEO plugin) and paste the sitemap URL directly.

Why do I get fewer URLs than the website has? A sitemap only lists what the owner put in it. Also check maxSitemaps and maxResults, and look at the SUMMARY record in the key-value store for warnings.

Some sites return HTTP 403? A few websites block automated access to their sitemaps. The Actor respects that and reports it in the SUMMARY; it does not try to bypass protections.

How does "only new" work? The Actor remembers (per list of start URLs) up to 250,000 URLs it already returned and only returns new ones next time.

Limitations

• Only URLs listed in sitemaps (the Actor does not crawl links on pages).
• Sitemap files larger than 60 MB are skipped (the official limit is 50 MB).
• RSS/Atom feeds used as sitemaps are not supported (use an RSS Actor for those).

Support

Found a problem or a sitemap that is not read correctly? Open an issue on the Issues tab.