Sitemap Change Monitor — Added, Removed & Updated URLs

Track catalog and website URL changes from XML sitemaps and sitemap indexes. Return baseline, added, removed, or lastmod-changed records. Stores a private per-user baseline, supports scheduled runs, and protects it when a crawl is incomplete.

Quick start

Try the small public demonstration fixture first, then replace its URL with your own public source:

{
  "urls": [
    "https://commerce-actor-demos.vercel.app/sitemap.xml"
  ],
  "maxResults": 10
}

The demo source contains synthetic catalog data labelled as a test fixture. Results go straight into the default Dataset. Export JSON, CSV, or Excel in Apify Console. Diagnostics are saved as RUN_STATS.

What it does

Follows same-origin sitemap index links, supports URL-set XML and gzip content, and deduplicates URLs. First complete run returns baseline records. Later runs compare URL presence and published lastmod; this does not compare page contents or detect price changes. Baselines live in a private named key-value store scoped to your Apify user and source URL. Use an Apify schedule to run again. Partial snapshots never overwrite the baseline and never label unseen URLs as removed. Run the same source sequentially; overlapping scheduled runs can race.

Output

Main dataset fields: change, url, lastmod, previousLastmod, snapshotComplete, sourceUrl. A stable recordKey is included for retry deduplication. Inspect the demo run's dataset for a complete working output example.

Verified demo output

This sample was returned by a successful cloud run against the public synthetic fixture.

[
  {
    "sourceUrl": "https://commerce-actor-demos.vercel.app/sitemap.xml",
    "url": "https://example.com/products/demo-one",
    "lastmod": "2026-10-08",
    "change": "unchanged",
    "previousLastmod": "2026-10-08",
    "snapshotComplete": true
  },
  {
    "sourceUrl": "https://commerce-actor-demos.vercel.app/sitemap.xml",
    "url": "https://example.com/products/demo-two",
    "lastmod": "2026-10-07",
    "change": "unchanged",
    "previousLastmod": "2026-10-07",
    "snapshotComplete": true
  }
]

Pricing

Configured launch price: $0.75 per 1,000 url record records, plus a $0.00005 Actor-start event (memory-based, minimum one event). Check the live Pricing tab before running; discounted tiers may differ. Platform usage is included in pay-per-event pricing. Each dataset record is one event, including a validation result that reports issues. RUN_STATS diagnostics are not dataset items. Spending-limit checks stop new work before the next record.

Limits and honest failures

Default maxResults is 100. This is a preview cap, not a promise to export everything. Set a larger cap only after checking a sample. Up to 100 source URLs, 8 MB per document, 50,000 rows per feed, 30-second request timeouts, 512 KB image probes, conservative shared request spacing, and at most five redirects. Public HTTP(S) only. Private/reserved addresses and credential URLs are rejected; DNS destinations are checked and pinned. robots.txt is respected. 429/503 stops the source for a later retry. Login/CAPTCHA pages, browser-only markup, and denied requests are reported rather than bypassed.

All-source failure ends FAILED after saving diagnostics. Useful partial output remains available with a partial status. Truncation and spending-limit stops are marked separately. Migration/retry checkpoints and existing dataset record IDs prevent repeated output in the same run. New independent runs are charged normally.

Input

• urls: 1–100 public HTTP(S) URLs. No credentials, private networks, custom ports, or access-control bypass.
• maxResults: Global output cap. RUN_STATS reports truncation. Start with a small sample. Default: 100.
• requestDelayMs: Shared host spacing. Requests stop on 429/503 instead of ignoring cooldowns. Default: 500.
• maxSitemaps: Includes root and child XML documents. Reaching this limit prevents baseline replacement and removal detection. Default: 20.
• onlyChanges: Only changes after the first snapshot. Default: false.
• updateBaseline: Update private baseline after a complete crawl. Default: true.

API

Use your Apify token in the Authorization header. For small jobs:

curl -X POST \
  'https://api.apify.com/v2/acts/skyfishdev~sitemap-catalog-change-monitor/run-sync-get-dataset-items' \
  -H 'Authorization: Bearer YOUR_APIFY_TOKEN' \
  -H 'Content-Type: application/json' \
  -d '{"urls":["https://commerce-actor-demos.vercel.app/sitemap.xml"],"maxResults":10}'

For larger jobs, use the asynchronous runs endpoint, poll the run status, and fetch its default dataset.

Local development

npm ci
npm test
apify run --input-file examples/demo.json

No LLM bill, image-host subscription, or additional API key is required. Only process public sources you are authorized to use. This tool does not promise customer demand, Google approval, or complete data on every website.