# Sitemap Change Monitor – New, Removed & Updated Pages Tracker (`skuang/sitemap-change-monitor`) Actor Watch websites' XML sitemaps and get the pages that were added, removed or updated (lastmod) since the previous run. Track competitors' new content, product and landing pages. - **URL**: https://apify.com/skuang/sitemap-change-monitor.md - **Developed by:** [skuang](https://apify.com/skuang) (community) - **Stats:** 2 total users, 1 monthly users, 46.2% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $2.00 / 1,000 change detecteds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## Sitemap Change Monitor – New, Removed & Updated Pages Tracker Watch any website's XML sitemap and get told what changed since the last run: pages that were **added**, pages that were **removed**, and pages whose **lastmod** date changed. Track competitors' new blog posts, product pages, landing pages and docs without crawling their sites. Run it once to save a baseline, then schedule it (for example daily). Each later run compares every site with the previous run and outputs only what changed. ### Who it is for - **SEOs and content marketers** following what competitors publish, update and take down. - **E-commerce teams** watching for new product and category pages on competing stores. - **Agencies** reporting site changes for clients, or checking that a migration did not drop pages. ### What it does 1. Finds each site's sitemaps: the `Sitemap:` lines in `robots.txt`, or else `/sitemap.xml`, `/sitemap_index.xml` and `/wp-sitemap.xml`. You can also give a sitemap URL directly. 2. Reads them all: sitemap indexes (nested up to 3 levels, up to 500 sitemap files per site), gzipped `.xml.gz` files, and plain-text sitemaps (one URL per line). It reads each page's `` and ``. 3. Compares the URLs with the snapshot saved by the previous run for that site and writes one `change` row per added, removed or updated page. 4. Optionally writes every URL as a `url` row. 5. Writes one free `site` status row per site, and a run summary to the `OUTPUT` record. URLs are compared after light normalization: surrounding spaces trimmed, the `#fragment` dropped, the query string kept, the host lower-cased. A URL listed twice counts once, with the latest of its lastmod values. ### Input | Field | Default | What it does | |---|---|---| | `sites` | `["https://astro.build"]` | Up to 20 sites or sitemap URLs. `astro.build` or `https://astro.build` finds the sitemaps itself; `https://example.com/post-sitemap.xml` (ends in `.xml`, `.xml.gz` or `.txt`, or contains "sitemap") is read as given. For a site, any path is ignored. | | `outputMode` | `changes_only` | `changes_only` (cheapest for daily monitoring), `urls_and_changes`, or `urls_only` (no change tracking). | | `maxUrlsPerSite` | 5000 | Stop reading a site after this many unique URLs (max 200,000). | | `detectLastmodChanges` | `true` | Report pages whose `` changed. | Example: ```json { "sites": ["https://astro.build", "nodejs.org", "https://www.fsf.org/sitemap.xml.gz"], "outputMode": "changes_only", "maxUrlsPerSite": 20000 } ``` ### Output All rows go to the run's dataset. The `type` field tells them apart. The dataset has three views: **Changes**, **URLs** and **Site status**. #### Change row (`type: "change"`) ```json { "type": "change", "site": "https://www.fsf.org/sitemap.xml.gz", "url": "https://www.fsf.org/blogs/community/2025-free-software-bulletin-issue-47-now-available", "changeType": "lastmod_changed", "lastmodBefore": "2025-12-18T10:02:11-05:00", "lastmodAfter": "2026-01-05T16:31:35-05:00", "sitemap": "https://www.fsf.org/sitemap.xml.gz", "detectedAt": "2026-10-12T11:40:03.112Z" } ``` | `changeType` | Meaning | `lastmodBefore` → `lastmodAfter` | |---|---|---| | `added` | The URL is listed now and was not listed in the previous run. | `null` → current lastmod (or `null`) | | `removed` | The URL was listed in the previous run and is no longer listed in any of the site's sitemaps. | previous lastmod (or `null`) → `null` | | `lastmod_changed` | The URL is listed in both runs and its `` changed. | previous → current | Lastmod values are shown exactly as the site writes them. Two values that are the same moment in different notations (`2026-10-01` and `2026-10-01T00:00:00+00:00`) are not a change. A page that gains or loses its `` is not reported as a change. #### URL row (`type: "url"`, only in `urls_and_changes` and `urls_only`) ```json { "type": "url", "site": "https://astro.build", "url": "https://astro.build/blog/astro-5/", "lastmod": null, "sitemap": "https://astro.build/sitemap-0.xml", "scrapedAt": "2026-10-12T11:49:24.120Z" } ``` #### Site status row (`type: "site"`, free, one per site in every mode) ```json { "type": "site", "site": "https://astro.build", "input": "https://astro.build", "status": "ok", "baseline": "compared", "urlsFound": 338, "sitemapsRead": 2, "sitemapsFailed": 0, "complete": true, "removalsDetected": true, "lastmodUniform": false, "added": 3, "removed": 2, "lastmodChanged": 0, "changesDetected": 5, "changesPushed": 5, "urlsPushed": 0, "stopReason": null, "errorType": null, "error": null, "checkedAt": "2026-10-12T11:49:24.549Z" } ``` - `status`: `ok`; `partial` (some sitemap files could not be read, or the charge limit or a dataset error stopped the rows); `failed` (no sitemap with URLs could be read, for example none exists, the site blocks automated access, or `robots.txt` could not be read); `skipped` (not processed: the charge limit was already reached, or another input already covers the same sitemap). - `baseline`: `created` (first run for this site: snapshot saved, no changes reported), `compared`, `not-tracked` (`urls_only`), `unknown` (rare: a run restarted after a platform migration whose comparison was already saved), or `null` when nothing could be read. - `complete`: every sitemap file was read in full (none failed, came back empty or was cut off) and no limit was hit. Removed pages are reported only when this is `true`. - `stopReason`: `max-urls` (more URLs than `maxUrlsPerSite`) or `max-sitemaps` (more than 500 sitemap files). The `OUTPUT` record in the default key-value store holds the same per-site details plus the sitemap roots used, how they were found (`robots`, `fallback` or `direct`), the first sitemap errors, and run totals. The run fails only when every site failed. ### How change tracking works - **First run for a site = baseline.** It saves a snapshot and reports no changes. Schedule the actor (daily is typical); changes are measured between consecutive runs. - **Snapshots live in your account**, in a named key-value store called `sitemap-change-monitor-state-erok7ac472yj0bycy` (the name ends with this actor's ID in lowercase, so it does not clash with the state store of another actor that has the same name). It holds URLs and lastmod values only, gzipped and split into several records for large sites. Delete a site's records there to start it over with a new baseline. All runs of this actor in your account share these snapshots, so two schedules watching the same site each see only the changes since the other one ran. - **Removed pages are reported only after a complete read.** If a sitemap file fails (server error, blocked, timeout, an empty body, or a file cut off before its closing tag), added and updated pages are still reported, removed pages are not, and the URLs of the failed file are kept in the snapshot so nothing is reported twice when it comes back. - **A sitemap that suddenly loses half of its URLs** (at least 50 URLs) is treated as a possible glitch: those removals are held back for one run and reported only if the next complete read confirms them. - **Site-wide lastmod stamps are ignored.** Some sites write their build time into every URL's ``, either as one value or as values a few seconds apart. When at least 90% of a site's URLs (and at least 20) share one lastmod value or fall within one 60-second window, in the previous run or in this one, lastmod changes are not reported for that run (`lastmodUniform: true`); added and removed pages still are. - **Large sites.** When a site has more URLs than `maxUrlsPerSite`, removed pages cannot be detected for it, and only URLs whose lastmod is later than the previous run (or that could not be delivered last time) count as added. - **Undelivered changes come back, delivered ones do not.** If the charge limit (or a dataset error) stops a run while change rows are being written, the snapshot is updated only for the changes that were delivered. The next run reports exactly the rest. A run that the platform moves to another server (migration) or that you abort waits up to 20 seconds for this bookkeeping before it stops; see Limitations for the case it cannot cover. ### Pricing Pay per event. Apart from a tiny fixed fee per run, you pay only for rows actually written to your dataset: | Event | When | Price | |---|---|---| | `apify-actor-start` | Once per run | $0.00005 | | `url-listed` | One URL row written (only in `urls_and_changes` and `urls_only`) | $0.0003 ($0.30 per 1,000 URLs) | | `change-detected` | One change row written (`added`, `removed` or `lastmod_changed`) | $0.002 ($2.00 per 1,000 changes) | Site status rows are free. With the default `changes_only` mode, a first run (baseline) costs only the $0.00005 run fee, and later runs cost the run fee plus the changes found. If you set a maximum cost per run, the actor stops cleanly when it is reached and says so in the status message and in `OUTPUT`. Sites cut short are marked `partial`, sites not started are marked `skipped`, and changes that could not be delivered are reported on the next run. Delivered changes are not reported or charged again, also across a platform migration or an abort; the exception is a hard stop described under Limitations. When you abort a run, no further charged rows are written; changes not yet written are reported on the next run. ### Limitations - Only what the sitemaps say. A page that exists but is not in the sitemap is invisible; a page removed from the sitemap is reported as `removed` even if it is still online. Pages are not fetched, so title or content changes are not detected; `lastmod_changed` is only as reliable as the site's own `` values. - Reads `` and `` only. Image, video and news sitemap extensions, `hreflang` alternates, `` and `` are ignored. RSS and Atom feeds used as sitemaps are not supported (the site is reported with that reason). - Respects `robots.txt`: sitemap files the site disallows for crawlers are not read, a `Crawl-delay` up to 30 seconds is honored for that host, and a site whose `robots.txt` cannot be read (server error) is not read at all. `robots.txt` is checked for the site's own host only; sitemap files hosted elsewhere (for example on a CDN) are read without a robots check. - Sites behind bot protection (Cloudflare challenges and similar) usually answer with an HTML page instead of the sitemap; they are reported as failed with that reason. No proxies, no CAPTCHA solving. - Only public internet addresses on the standard ports (80/443) are contacted, also when a sitemap index or a redirect points elsewhere. Sitemap files are limited to 50 MB (after unpacking `.gz`). - Duplicate inputs (the same site twice, or a site and its sitemap URL, also when one of them redirects to the other) are processed once, the first in input order. A site and one of its child sitemaps given as separate inputs are not detected as duplicates and are both read. - Changes are recorded as delivered when the run saves the site's snapshot, right after writing its rows. If the run is killed hard in between (out of memory, or a forced stop that skips the abort handling), the next run reports those changes again and they are charged again. Migrations and normal aborts wait up to 20 seconds for the save. - Plain-text sitemaps have no closing tag, so a text sitemap cut off in transfer cannot be told from a shorter one. - Memory: the default (1 GB) covers three sites reading 50 MB sitemap files at the same moment. If you lower the memory for sites with very large sitemap files, the run can run out of memory. - Up to 20 sites per run, 200,000 URLs per site, 500 sitemap files per site. ### FAQ **How do I check that a site works?** Open `https:///robots.txt` and look for `Sitemap:` lines, or open `https:///sitemap.xml`. If you see XML with `` or `` entries, the actor can read it. **Why did the first run report no changes?** There was nothing to compare with yet. The first run saves the baseline; changes appear from the second run on. **How fast is it?** At most one request every 0.5 seconds per host, shared by all sites on that host (slower if the site sets a `Crawl-delay`), 3 sites in parallel. A site with one sitemap index and a few sitemap files takes a few seconds. **Do I need proxies?** No. Sitemaps are published for crawlers; the actor reads them directly with a clear user agent (`SitemapChangeMonitor`). # Actor input Schema ## `sites` (type: `array`): Websites to watch, one per line. A site address (https://astro.build or just astro.build) finds its sitemaps through robots.txt, or at /sitemap.xml, /sitemap_index.xml or /wp-sitemap.xml. A sitemap address (ending in .xml, .xml.gz or .txt, or containing "sitemap") is read as given. Up to 20 per run. ## `outputMode` (type: `string`): "Changes only": only pages added, removed or updated since the previous run (cheapest for daily monitoring). "URLs and changes": also every URL in the sitemaps. "URLs only": every URL, no change tracking (the saved snapshot is not touched). Every mode also writes one free status row per site. ## `maxUrlsPerSite` (type: `integer`): Stop reading a site's sitemaps after this many unique URLs. When a site has more, removed pages cannot be detected for it, and only pages with a lastmod after the previous run count as added. ## `detectLastmodChanges` (type: `boolean`): Report a page as updated when its date changes. Turn off if a site rewrites lastmod without real changes. Sites that stamp the same date on almost every URL (a build time) are detected and their lastmod changes are skipped automatically. ## Actor input object example ```json { "sites": [ "https://astro.build" ], "outputMode": "changes_only", "maxUrlsPerSite": 5000, "detectLastmodChanges": true } ``` # Actor output Schema ## `changes` (type: `string`): No description ## `urls` (type: `string`): No description ## `sites` (type: `string`): No description ## `allItems` (type: `string`): No description ## `summary` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "sites": [ "https://astro.build" ] }; // Run the Actor and wait for it to finish const run = await client.actor("skuang/sitemap-change-monitor").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "sites": ["https://astro.build"] } # Run the Actor and wait for it to finish run = client.actor("skuang/sitemap-change-monitor").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "sites": [ "https://astro.build" ] }' | apify call skuang/sitemap-change-monitor --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,skuang/sitemap-change-monitor" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/Erok7ac472yJ0byCY/builds/B9yoTzkRiIVwnUcNW/openapi.json