Sitemap Extractor & Change Monitor (new/removed/updated URLs)

Sitemap Extractor & Change Monitor: all URLs, lastmod, new/removed pages

An XML sitemap parser and URL change monitor: give it a domain or a sitemap.xml URL and get every URL the site publishes in its XML sitemaps (sitemap indexes, gzip sitemaps, and Google News, image and video sitemaps), with lastmod, changefreq and priority. Run it on a schedule for an SEO sitemap audit, and it reports only what changed since the last run: new pages, removed pages and pages whose lastmod moved. You can also check a list of key pages for content change detection, comparing a hash of the main text so ads, scripts, navigation and footers don't trigger false alarms.

• ✅ Automatic discovery: robots.txt Sitemap: lines, then common paths (/sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml, .gz, .txt …), then the site's RSS/Atom feed as a fallback
• ✅ Every sitemap format: sitemap indexes (nested up to 5 levels), gzip (.xml.gz, or gzip served without a .gz name), plain-text sitemaps, Google News / image / video extensions, RSS 2.0 and Atom
• ✅ Change monitoring between runs, with state kept in your own key-value store: added, removed, lastmodChanged
• ✅ Page content checks: main-text hash plus HTTP status, title and meta description, with a CSS selector and ignore patterns
• ✅ Polite and robust: obeys robots.txt (RFC 9309) and Crawl-delay, applies a per-host delay, retries 429/5xx, and never reports false "removed" URLs after a partial read
• ✅ No browser, no proxy needed: plain HTTP, so it is fast and cheap (tens of thousands of URLs per minute)

Use cases

• SEO: audit what a site submits to Google, find URLs missing a lastmod, and catch sitemap regressions after a deploy
• Competitor monitoring: get alerted when a competitor publishes new product, pricing, landing or blog pages, or removes some
• Content and news tracking: new articles from news sitemaps (title, publication date) as a structured feed
• Crawl seeding: a clean URL list to feed other scrapers, filtered with include/exclude patterns
• Change detection on key pages: pricing pages, terms and policies, docs

Input

Field	Description
startUrls	Domains or any page URLs (example.com, https://www.example.com/blog). Sitemaps are discovered automatically. A URL that is itself a sitemap or feed is read directly.
sitemapUrls	Exact sitemap, index, .gz, RSS or Atom URLs, read as-is with no discovery
includePatterns / excludePatterns	blog or /blog/ = substring (case-insensitive), https://example.com/blog/* = wildcard, /\/20(25|26)\//i = regex
maxUrls	Cap per site (default 10,000; 0 = no limit). When capped, the newest child sitemaps are read first.
maxSitemapsPerSite	Safety cap on sitemap files per site (default 500)
discoverSitemaps, rssFallback	Turn off path probing or the RSS/Atom fallback
includeExtensions	Add news, image and video fields
trackChanges	Compare with the previous run and mark each URL added / lastmodChanged / unchanged, plus removed rows
onlyChanges	Output only the changes (this turns monitoring on). The first run saves a baseline and outputs nothing, unless outputBaseline is set.
stateKey	Monitor name. Use a different name for each schedule or task.
pageCheckUrls	Pages whose content should be compared between runs (max 1,000)
pageCheckOnlyChanged	Fetch only those pages whose sitemap entry is new or changed, plus pages never checked before
pageCheckDetails, contentSelector, ignoreTextPatterns	Also compare title and meta; hash only one part of the page; ignore noisy text such as view counters
maxConcurrency, perHostDelaySecs, maxRequestsPerMinute	Politeness controls (defaults 5 / 1 s / 120)

Example: monitor a competitor's blog and pricing page every day:

{
  "startUrls": ["competitor.com"],
  "includePatterns": ["/blog/", "/pricing"],
  "onlyChanges": true,
  "stateKey": "competitor-daily",
  "pageCheckUrls": ["https://competitor.com/pricing"]
}

Schedule it in Apify (for example daily) and connect a webhook, Slack, email or Google Sheets integration. Each run then delivers only the changes.

Output

URL rows (recordType: "url"):

{
  "recordType": "url",
  "url": "https://example.com/blog/new-post",
  "site": "example.com",
  "lastmod": "2026-10-29",
  "changefreq": "weekly",
  "priority": 0.8,
  "sourceType": "sitemap",
  "sitemapUrl": "https://example.com/post-sitemap.xml.gz",
  "scrapedAt": "2026-10-29T00:19:36.820Z",
  "changeType": "added",
  "previousLastmod": null
}

With includeExtensions, rows also carry title (news title), newsPublicationName, newsPublicationDate, newsLanguage, imageUrls, imageCount, videoTitles and videoCount when the sitemap has them. RSS/Atom rows use the item title (title) and date as lastmod (sourceType: "rss" / "atom").

changeType is added, removed, lastmodChanged, unchanged, or baseline (first run). It is null when monitoring is off.

Page-check rows (recordType: "page"): changeType is new / changed / unchanged / error / blockedByRobots. Each row also has changedFields (text, statusCode, title, metaDescription), the current and previous contentHash, statusCode, title, metaDescription, textLength, plus canonicalUrl, metaRobots and finalUrl.

The dataset has three views: URLs, Changes only and Page checks. A RUN_SUMMARY record in the default key-value store lists, for each site, how sitemaps were found, how many were read, URL counts, whether the read was complete, and any problems. It also records run time and peak memory.

Pricing

This Actor uses pay per event, so you pay only for what you get:

Event	Price	When it's charged
URL result	$0.30 per 1,000 ($0.0003 each)	Each URL row saved to the dataset
Page check	$0.002 per page	Each page from pageCheckUrls that is actually fetched and compared

• Full export: 10,000 URLs = $3.00.
• Monitoring (onlyChanges): you're charged only for change rows (added, removed, lastmodChanged). The first run saves a baseline for free unless you set outputBaseline, and a run with no changes produces no URL charges.
• Page checks: 50 pages checked daily ≈ $0.10/day. Pages blocked by robots.txt or that fail to load aren't charged. Use pageCheckOnlyChanged to fetch only pages whose sitemap entry changed.
• Set Maximum cost per run in the run options to cap spending. The Actor stops cleanly and saves its state.

The live price list is on the Pricing tab of this page.

FAQ

Does it work on any website? Yes, if the site publishes an XML sitemap, a plain-text sitemap or an RSS/Atom feed. Most CMSs (WordPress, Shopify, Wix, Webflow, Next.js sites …) do.

How do I get alerts for new pages? Turn on onlyChanges and add a schedule (e.g. daily). The first run stores a baseline. Later runs output only added, removed and updated URLs. Send them to Slack, email, Google Sheets or a webhook with Apify integrations.

Where is the monitoring state stored? In a named key-value store in your own Apify account, so it survives between runs. Nothing is shared with other users.

Does it render JavaScript? No. Sitemaps don't need a browser, and page checks compare the HTML text the server returns, which keeps runs fast and cheap.

Why are no URLs reported as removed? Removals are only reported after two complete reads in a row, to avoid false alarms (see below).

How change detection stays accurate

• A URL is reported as removed only if the previous run and this run read the site's sitemaps completely: no failed sitemap, no maxUrls cap and no spending limit hit. Otherwise the new URLs are merged into the saved state and removals wait for the next complete run.
• If you change the include/exclude patterns, the monitor resets to a fresh baseline for that site instead of reporting the filtered-out URLs as removed.
• Some sites generate their sitemap on every request, so every lastmod equals "now". The Actor detects this (lastmodLooksDynamic in RUN_SUMMARY). From the next run it ignores lastmod changes for that site, so you only get real added/removed URLs. Use page checks for content changes there.
• Page checks hash the visible main text only. Scripts, styles, hidden elements, header, footer, nav and aside are dropped, and you can narrow further with contentSelector and ignoreTextPatterns.

Privacy & politeness

• Only reads public sitemaps, feeds and the pages you list. It does not extract emails, names or other personal data. Page checks store only a hash, the text length, the title and the meta description.
• robots.txt is respected. The Actor's own User-agent token is SitemapChangeMonitor, falling back to *. Sitemaps a site lists in robots.txt are considered invited. Pages disallowed for crawlers are reported as blockedByRobots and not fetched.
• Crawl-delay is honoured up to 30 s. Note: sites with a long delay (e.g. Hacker News: 30 s) make discovery slow, because each probe waits for the delay.

Limits & tips

• The cap is per site, so one huge site can't starve the others. Raise maxUrls (or set 0) for full exports of big sites. 1M URLs need roughly 1–2 GB memory.
• No sitemap? The Actor falls back to the site's RSS/Atom feed. That usually lists only the latest 10–50 items, which suits "new page" alerts but is not a full URL list.
• Sites that block datacenter IPs can be read through the Apify Proxy (proxyConfiguration).

Legal & data use

• The Actor reads only public sitemaps and feeds, plus the pages you list. It obeys robots.txt and Crawl-delay and doesn't log in or bypass access controls.
• It doesn't collect personal data. Output is URLs and page metadata (status, title, meta description, content hash).
• You're responsible for monitoring only sites you're allowed to, and for using the results in line with those sites' terms and applicable law.

Running on Apify

You can run this Actor on the Apify platform from the Console, the API, a schedule, integrations (webhooks, Slack, Google Sheets, Make, Zapier) or MCP.

Other DatLab actors

• Web Page to Markdown — turn a list of URLs into Markdown for LLMs and RAG
• Website to Markdown Crawler — crawl a site into Markdown, with sitemaps and llms.txt
• ATS Job Listings — Personio, Teamtailor, Workable and Recruitee, one schema
• SAM.gov Contract Opportunities — US federal opportunities from the official daily extract, no API key