Sitemap Change Detector

Monitor changes in website sitemaps by comparing current entries against a previous snapshot stored in a named key-value store. Detect new, updated, removed, or unchanged URLs and react to changes automatically.

What is Sitemap Change Detector?

This Apify actor fetches sitemap URLs (or discovers them via robots.txt), parses <loc> and <lastmod> entries, and stores a snapshot of URLs in a key-value store named sitemap-change-detector-snapshots. On subsequent runs, it retrieves the previous snapshot and compares the lists to identify changes.

How it Detects Changes

• NEW: URLs present in the current sitemap but missing from the previous snapshot.
• REMOVED: URLs present in the previous snapshot but missing from the current sitemap.
• UPDATED: URLs present in both snapshots where the current <lastmod> timestamp is later than the previous one.
• SAME: URLs present in both snapshots with identical <lastmod> timestamps.

How to Use

1. Go to the Sitemap Change Detector actor on the Apify Store.
2. Enter one or more Start URLs (direct sitemap URLs or root domains).
3. Enable Discover sitemaps to fetch robots.txt and enqueue sitemap URLs.
4. Select which Change Types to detect: NEW, UPDATED, REMOVED, SAME.
5. (Optional) Provide a URL filter regex to restrict monitored URLs.
6. (Optional) Enable Add removed URLs to KVS to log deleted URLs separately.
7. Save the input and click Run.
8. View the results in the default dataset or fetch via API or webhooks.

Example Input

{
    "startUrls": [
        {
            "url": "https://www.apify.com",
            "method": "GET"
        }
    ],
    "changeTypes": ["NEW", "UPDATED"],
    "addRemovedUrlsToKvs": false,
    "discoverSitemaps": true,
    "snapshotKeyPrefix": "APIFY"
}

Output

Each record in the output dataset represents a URL change:

{
    "url": "https://example.com/page",
    "changeType": "UPDATED"
}

If Add removed URLs to KVS is enabled, check the REMOVED-URLS KVS record for an array of removed URLs.

FAQ

Can I export data using API?

Yes, you can access this actor using your own applications through the Apify API. Click on the API tab for code examples or check out the Apify API reference docs at https://docs.apify.com/api/v2 for full details.

Can I use Sitemap Change Detector through an MCP Server?

This actor, like all Apify actors, works on the Apify MCP server. For more information and instructions, read the Apify MCP server integration guide at https://docs.apify.com/platform/integrations/mcp.

Can I integrate data from Sitemap Change Detector with other apps?

Yes. Sitemap Change Detector can be connected with almost any cloud service or web app. Read more about the possibilities on our integrations page at https://apify.com/integrations.

Is it legal to scrape data using Sitemap Change Detector?

This actor only extracts publicly available data. It does not collect private user data. However, you should ensure your reason for scraping is legitimate. Consult legal counsel if unsure. For more on scraping legality and ethics, see:

• https://blog.apify.com/is-web-scraping-legal/
• https://blog.apify.com/what-is-ethical-web-scraping-and-how-do-you-do-it/

Your feedback

We welcome feedback to improve this actor. If you encounter issues or have suggestions, please create an issue on the actor’s Issues tab.