XML Sitemap Extractor to CSV & New-URL Monitor: get every URL from sitemap.xml, then only the new ones

Give it domains or sitemap URLs and it returns every page URL listed in the site's sitemaps: <lastmod>, change frequency, priority, images, videos, Google News titles and hreflang alternates.

Switch to Changes only mode and schedule it, and each run returns only what changed since the last run:

• new URLs (new products, articles, landing pages, job posts)
• removed URLs (delisted products, deleted pages)
• lastmodChanged URLs (pages the site says were updated)

No browser and no page crawling are involved. It reads the XML that sites publish for search engines, so it is fast (about 9,000 URLs from 5 sites in 4 seconds in our tests) and cheap. Free while in launch; planned pay-per-event price: $0.20 per 1,000 URLs plus $0.002 per site.

What can I use it for?

• Competitor monitoring. Get a daily list of new product pages, blog posts or pricing pages from competitor sites.
• News and content tracking. News sitemaps include the headline, publication time and keywords, so you can follow new articles without scraping article pages.
• SEO audits. Export the full URL inventory, find URLs that redirect or return 404 (optional status check), spot noindex in X-Robots-Tag and check hreflang coverage.
• Feeding crawlers and AI pipelines. Pass only the new URLs to your scraper, RAG indexer or translation workflow instead of recrawling the whole site.
• Ecommerce catalog changes. Track when products are added or delisted on Shopify, WooCommerce or Magento stores.

What data does it return?

One row per URL:

Field	Description
site, url	Host and page URL
changeType	null in "All URLs" mode; baseline, new, removed or lastmodChanged in "Changes only" mode
lastmod, previousLastmod	Last-modified date from the sitemap (normalised to ISO 8601), and the previous value for updated URLs
lastmodIsFetchTime	true when the sitemap stamps every URL with the time of the request (common on Shopify), so that date says nothing about the page
changefreq, priority	As declared in the sitemap
sitemapUrl	The sitemap file the URL came from
imageCount, images	Image sitemap data (total count and first 10 image URLs)
videos	Video sitemap data: title, description, thumbnail, content/player URL, duration (first 5)
news	Google News data: publication name, language, publication date, title, keywords
alternates	hreflang alternates: language and URL (first 50)
statusCode, redirectLocation, xRobotsTag, statusError	Only with "Check HTTP status" on. statusError is BLOCKED_BY_SITE when the site refused the check (403, 429 or a bot-protection page), DISALLOWED_BY_ROBOTS_TXT when robots.txt does not allow the URL, ROBOTS_TXT_UNREACHABLE when robots.txt could not be read, or a network error
extractedAt	Time of extraction

A per-site summary (sitemaps found, parsed and failed, URLs listed and matched, new/removed/changed counts and any issues) is saved as SUMMARY in the run's key-value store. A site that refused our requests has the status blocked_by_site there.

Sitemap to CSV: the Output tab exports the URL list as CSV, Excel, JSON or XML in one click, or you can fetch it through the Apify API.

Example output

From a real "All URLs" run on the default input (2026-10-03, UTC), a row from TechCrunch's news sitemap:

{
  "site": "techcrunch.com",
  "url": "https://techcrunch.com/2026/10/03/federal-judge-calls-flock-indiscriminate-mass-surveillance/",
  "changeType": null,
  "lastmod": null,
  "lastmodIsFetchTime": null,
  "previousLastmod": null,
  "changefreq": null,
  "priority": null,
  "sitemapUrl": "https://techcrunch.com/news-sitemap.xml",
  "news": {
    "publicationName": "TechCrunch",
    "language": "en",
    "publicationDate": "2026-10-03T19:33:15Z",
    "title": "Federal judge calls Flock ‘indiscriminate mass surveillance’"
  },
  "extractedAt": "2026-10-03T23:54:11+00:00"
}

In "Changes only" mode, changeType says what happened. A new, a removed and an updated URL look like this:

{"site": "example.com", "url": "https://example.com/blog/c", "changeType": "new", "lastmod": "2026-10-03", "previousLastmod": null}
{"site": "example.com", "url": "https://example.com/blog/a", "changeType": "removed", "lastmod": null, "previousLastmod": "2026-09-01"}
{"site": "example.com", "url": "https://example.com/blog/b", "changeType": "lastmodChanged", "lastmod": "2026-10-02", "previousLastmod": "2026-09-02"}

How much does it cost?

Free while in launch. The actor charges no fee of its own right now; you pay only the Apify platform usage of your runs (the free Apify plan includes $5 of usage a month). The pay-per-event prices below are planned and apply only once they are switched on; Apify notifies existing users before a price change takes effect.

Planned pricing, pay per event:

Event	Planned price
Site processed (sitemaps found and read)	$0.002
URL in the output	$0.0002 (= $0.20 per 1,000 URLs)
URL status checked (optional)	$0.0005

• Monitoring is cheap. After the first run, only changed URLs are output and charged. At the planned prices, our second run on 5 large sites (about 43,000 URLs in total) output 1 new URL and would cost $0.0102.
• Cheap first run. In "Changes only" mode, set Output all URLs on the first monitoring run to off. The first run then saves the URL list silently and would cost only $0.002 per site. With it on, our first run on the same 5 sites output 42,985 URLs, which would cost $8.61.
• Not charged: sites where no sitemap could be found or read, sites that refused our requests (blocked_by_site), status checks the site refused or robots.txt did not allow, and invalid inputs.
• Cost cap: once pricing is on, set Maximum cost per run and the actor stops cleanly when it reaches it.

For comparison (Apify Store, checked 2026-10-03): the most-used sitemap extractor charges $0.50 per 1,000 URLs, another popular one $30 per 1,000 URLs, and a sitemap change monitor charges $0.005 per change row.

How to use it

1. Enter domains (example.com), site URLs or direct sitemap URLs (https://example.com/sitemap_index.xml).
2. Choose All URLs for a one-off export, or Changes only to monitor.
3. Optionally add include/exclude regex patterns (for example /products/ or /blog/), a date filter (7 days or 2026-09-01) or the HTTP status check.
4. For monitoring, create a schedule (for example daily) with the same input. Use a different Snapshot store name for each independent monitor.

Input example

{
  "sites": ["techcrunch.com", "allbirds.com", "https://wordpress.org/sitemap.xml"],
  "mode": "changes",
  "emitBaselineOnFirstRun": false,
  "includePatterns": ["/products/", "/20\\d\\d/"],
  "excludePatterns": ["/live/"],
  "snapshotStoreName": "competitor-monitor"
}

API example

curl -X POST "https://api.apify.com/v2/acts/adamhaleworks~sitemap-extractor-change-monitor/run-sync-get-dataset-items?token=<TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"sites": ["apify.com"], "maxUrlsPerSite": 5000}'

How it works

• Discovery. It reads Sitemap: lines from robots.txt. If there are none, it tries common locations such as /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml and /sitemap.xml.gz, skipping any that robots.txt disallows.
• Parsing. It follows nested sitemap indexes (up to 6 levels), handles gzip, plain-text sitemaps and RSS/Atom feeds, and recovers URLs from malformed XML. It uses a streaming parser, so sitemaps with 50,000 URLs and large indexes fit in normal memory. External entities are never resolved.
• Change detection. The URL list of each site is saved, compressed, in a named key-value store in your account. Each run compares the new list with the saved one. If some sitemap files fail to load, removed URLs are not reported for that run, which prevents false alarms, and the saved list is merged rather than replaced.
• Fetch-time lastmod detection. Some platforms (for example Shopify product sitemaps and some news sitemaps) set <lastmod> to the time of the request. The actor detects this, marks the rows lastmodIsFetchTime: true and does not report them as updated. Date filters then use the Google News publication date where there is one.
• Reliability. Network errors, HTTP 429 and 5xx get retries with exponential backoff, and Retry-After is respected. Status checks are limited to 2 parallel requests per host. Progress is saved, so a migrated run continues where it stopped.
• Polite by design. Every request identifies the actor as sitemap-extractor-change-monitor/1.0 (+<link to this Store page>); there is no browser disguise, no proxy and no retry with a different identity. Status checks follow the site's robots.txt rules for sitemap-extractor-change-monitor. If a site answers with 403, keeps answering 429 or serves a bot-protection page (Cloudflare, DataDome and similar), the actor records BLOCKED_BY_SITE; after a 429 or a protection page it sends nothing more to that host in the run.

Limitations

• It only knows what the site lists in its sitemaps. Pages missing from the sitemaps are not found, because the actor does not crawl links.
• lastmodChanged is only as reliable as the site's <lastmod> values. Many CMSs update them correctly; some never do.
• Private and internal addresses (for example 127.0.0.1 or 10.0.0.0/8) are refused.
• Memory is the practical limit for very large sites. In our test, 100,000 URLs from bbc.com peaked at 165 MB and the saved snapshot was 1.1 MB. Allow roughly 1 GB of memory per million URLs processed at the same time; the default 1 GB is fine for a few hundred thousand URLs per site.

Is it legal? What about personal data?

Sitemaps are files that sites publish specifically so that machines can read their URL lists. The actor reads only those files and robots.txt, and checks guessed sitemap locations against robots.txt. It does not open the pages themselves unless you turn on the status check, which sends a single HEAD request per URL that robots.txt allows. It identifies itself honestly and does not try to get around bot protection. URLs and publication metadata are not personal data. You are responsible for using the results in line with the laws and terms that apply to you.

FAQ

How is this different from a crawler? A crawler opens every page to find links. This actor reads the site's own URL list, which takes seconds instead of hours and costs a fraction as much.

Why did a site return 0 URLs? Check the SUMMARY record in the key-value store. Common reasons: the site has no sitemap (NO_SITEMAP_FOUND), robots.txt points to a sitemap that returns 404, your filters matched nothing (compare urlsListed with urlsMatched), the date filter was used on a sitemap whose lastmod is the fetch time, or the site refused our requests (blocked_by_site). Some sites refuse all automated clients, or any User-Agent that mentions Apify; the actor accepts that, has no proxy option to get around it, and you are not charged for the site.

I run a website. How do I control this actor? It sends sitemap-extractor-change-monitor in its User-Agent. A robots.txt group for that name controls the status checks and the guessed sitemap locations (for example User-agent: sitemap-extractor-change-monitor with Disallow: /). Sitemaps you list in robots.txt are treated as published for machines and are still read. To refuse it entirely, answer its User-Agent with 403 or 429: the actor records blocked_by_site, does not retry a 403, and after a 429 sends nothing more to your host in that run.

Can I monitor many sites in one run? Yes, up to 10,000 sites per run. Each site has its own saved snapshot.

Something is wrong or missing? Open an issue on the actor's Issues tab and include the site.

---

Author: Adam Hale