Sitemap URL Extractor: Bulk, Lastmod Filter, New URL Alerts

Get every URL a website publishes in its sitemap, for one site or hundreds, in a single run. Then schedule it and get only the URLs that are new since last time: new products, new blog posts, new landing pages of your competitors.

What you get per URL

• URL, site and top-level section (/blog, /products, ...)
• lastmod, changefreq and priority as declared by the site
• isNew flag when change detection is on
• Image URLs and hreflang language versions (optional)
• News sitemap titles when present
• The sitemap file the URL came from

How it finds the sitemaps

1. Reads the Sitemap: lines in robots.txt
2. If none, tries /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /wp-sitemap.xml and /sitemap.xml.gz
3. Follows nested sitemap indexes, gzip files and plain-text sitemaps

You can also paste a direct sitemap URL.

Why people use it

• SEO audits: how many pages does a site really have, which sections are growing
• Competitor monitoring: schedule weekly and receive only the new URLs
• Content research: newest articles by lastmod date
• E-commerce: new product pages of a store
• Feeding crawlers, AI agents and RAG pipelines with a clean URL list

Input example

{
  "startUrls": ["apify.com", "https://www.example.com/sitemap_index.xml"],
  "includePattern": "/blog/",
  "lastmodAfter": "2026-01-01",
  "onlyNewSinceLastRun": true
}

Output example

{
  "url": "https://apify.com/store",
  "site": "https://apify.com",
  "section": "/store",
  "lastmod": "2026-10-01",
  "changefreq": "daily",
  "priority": 0.8,
  "isNew": null,
  "sitemapUrl": "https://apify.com/sitemap.xml"
}

The key-value store record SUMMARY holds, per site: sitemaps found, URLs found and delivered, newest and oldest lastmod, URL count per section.

Change detection

With Only URLs new since the last run on, the Actor stores the URL list of every site in a named key-value store (default sitemap-url-history). The next run returns only URLs that were not there before. The first run returns everything and builds the baseline. Use a different store name per project.

Pricing (pay per result)

You pay only for URLs delivered to the dataset. Filtered-out URLs and URLs that are not new are free.

Event	Price
URL delivered	$0.0015

Example: a 2,000-page site costs about $3 the first time. A weekly change check that finds 40 new URLs costs about $0.06.

Use "Max cost per run" to cap spending.

Limitations

• Only URLs the site lists in its sitemaps. Pages missing from the sitemap are not crawled.
• lastmod is what the site declares, and some sites set it to today for every URL.
• Sites that block automated requests to their sitemap may return nothing.

Data and legality

Sitemaps are public files that websites publish precisely so that machines can read them. This Actor reads only those files and robots.txt, and collects no personal data.

Support

Open an issue on the Actor page with the site and run link.