Competitor Sitemap Monitor – New, Removed & Updated URLs

Find out what changed on any website since your last check: new products, new landing pages, new blog posts, removed pages and updated content, straight from the site's XML sitemaps. Add your competitors' domains, schedule the Actor daily or weekly, and get one clean row per change.

What it does

1. Finds the sitemaps of each domain automatically: Sitemap: lines in robots.txt, then common paths (/sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml, .xml.gz, /sitemap.txt). Sitemap indexes (nested), gzip, text, RSS and Atom sitemaps are supported. You can also pass a direct sitemap URL (e.g. only the products sitemap of a Shopify store).
2. Collects every URL with its <lastmod> date, optionally filtered by URL patterns such as /products/ or /blog/.
3. Compares the URL list with the snapshot of the previous run, stored in your own named key-value store sitemap-change-monitor-state (one record per stateKey + domain).
4. Outputs one row per change (new, removed, updated) plus one summary row per domain with total URL count and change counts.

The first run of a domain is the baseline: it stores the snapshot and outputs only the summary row (total URLs), not thousands of "new" rows. From the second run on you get the real changes. Set emitBaseline: true if you want all URLs on the first run too.

Who it is for

• E-commerce teams tracking competitors' new products, discontinued products and new category pages.
• SEO and content marketers watching competitors' new blog posts, landing pages, programmatic SEO pages and site restructures.
• Market and investor research: new product lines, careers pages, docs, pricing pages, regional sites.
• Your own sites: catch accidentally removed pages or a broken sitemap after a deploy.

Why this Actor

• Change rows, not URL dumps – no spreadsheet diffing; only what is new, removed or updated since the last run.
• Cheap and fast – plain HTTP, no browser. A blog with 1,000 URLs is checked in about 3 seconds. You pay $0.003 per domain checked, regardless of how many URLs the domain has.
• Safe "removed" detection – if a sitemap fails to load or the URL limit is hit, removed URLs are not reported for that run (status partial) and the old URLs are kept in the snapshot, so a temporary error never floods you with fake removals.
• Pattern filters – track only /products/ or only /blog/; changing the patterns automatically starts a fresh baseline.
• Multiple independent monitors – use different stateKey values (e.g. shop-weekly, blog-daily) for the same domains.
• Flat, AI-friendly output – ideal for Slack/email alerts, Google Sheets, n8n, Make, Zapier or an LLM agent.

Input

Field	Description
domains	Domains, website URLs or direct sitemap URLs. Required.
includePatterns	Only track URLs matching one of these (plain text or regex), e.g. /products/.
excludePatterns	Ignore URLs matching any of these, e.g. /tag/, /author/.
stateKey	Name of the monitor (default default). Same key = compare with the previous run of that key.
emitBaseline	Output all URLs as new rows on the first run (default false).
trackLastmod	Report URLs whose <lastmod> changed as updated (default true).
maxUrlsPerDomain	Safety limit for huge sites (default 100,000).
maxChangeRowsPerDomain	Cap on change rows per domain and run (default 10,000); the summary always has full counts.
proxyConfiguration	Optional. By default sitemaps are fetched directly; an Apify proxy is used only as a fallback when a site blocks the request.

Example:

{
  "domains": ["https://blog.apify.com", "competitor-shop.com"],
  "includePatterns": ["/products/", "/blog/"],
  "stateKey": "competitors-daily",
  "emitBaseline": false
}

Output

Every row has rowType (change or summary), domain, url, changeType, lastmod, previousLastmod, detectedAt, stateKey and isBaseline. Summary rows additionally have totalUrls, previousTotalUrls, newUrls, removedUrls, updatedUrls, previousCheckAt, sitemapsRead, discovery, status (ok, partial, failed) and error.

Real output from a test run on blog.apify.com:

[
  {
    "rowType": "change",
    "domain": "blog.apify.com",
    "url": "https://blog.apify.com/outreacher-linkedin-to-lemlist/",
    "changeType": "new",
    "lastmod": "2026-11-03T14:29:55.000Z",
    "previousLastmod": null,
    "detectedAt": "2026-11-03T16:10:10Z",
    "stateKey": "default",
    "isBaseline": false
  },
  {
    "rowType": "change",
    "domain": "blog.apify.com",
    "url": "https://blog.apify.com/author/magda/",
    "changeType": "updated",
    "lastmod": "2026-11-03T13:08:04.000Z",
    "previousLastmod": "2026-09-01T10:00:00.000Z",
    "detectedAt": "2026-11-03T16:10:10Z",
    "stateKey": "default",
    "isBaseline": false
  },
  {
    "rowType": "summary",
    "domain": "blog.apify.com",
    "url": "https://blog.apify.com",
    "detectedAt": "2026-11-03T16:10:10Z",
    "stateKey": "default",
    "totalUrls": 997,
    "previousTotalUrls": 994,
    "newUrls": 3,
    "removedUrls": 0,
    "updatedUrls": 1,
    "isBaseline": false,
    "previousCheckAt": "2026-11-03T16:09:54Z",
    "sitemapsRead": 5,
    "discovery": "robots.txt",
    "status": "ok",
    "error": null
  }
]

The dataset has two views: Changes and Domain summary. Filter change rows with rowType = change.

Pricing

Pay per event: $0.003 per domain checked (all sitemaps of the domain read and compared). Domains whose sitemaps cannot be loaded are free and marked with status: failed. Checking 20 competitors every day costs about $1.80 per month plus minimal platform usage.

Scheduling

Create a schedule (e.g. daily at 7:00) with your saved input. Each run compares with the previous run of the same stateKey. Connect a webhook or integration (Slack, email, Google Sheets) to get notified about the change rows.

API, n8n and MCP

Run via API and get the results in one call:

curl -X POST "https://api.apify.com/v2/acts/fast_data_asap~sitemap-change-monitor/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"domains": ["competitor.com"], "includePatterns": ["/products/"], "stateKey": "shop"}'

• n8n / Make / Zapier: use the Apify node, Actor fast_data_asap/sitemap-change-monitor, operation "Run Actor and get dataset items", then filter rowType = change.
• MCP (Claude, Cursor, other AI agents): https://mcp.apify.com?tools=fast_data_asap/sitemap-change-monitor

FAQ

Where is the state stored? In the named key-value store sitemap-change-monitor-state in your own Apify account. Delete a record there to reset the baseline of a domain, or simply use a new stateKey.

What if a site has no sitemap? The domain gets status: failed with an explanation and is not charged. Pass a direct sitemap URL if the sitemap lives at an unusual path.

Does it detect content changes without lastmod? updated relies on the sitemap's <lastmod>. Many CMSs (WordPress, Shopify, Ghost, Webflow) maintain it. For page-content diffs use a website change monitor.

Is it legal? It only reads public sitemaps and robots.txt files that websites publish for crawlers. No login, no personal data.