# Sitemap Change Monitor: New & Removed Page Alerts (`scrapemint/sitemap-change-monitor`) Actor Watch any website's sitemap and get one row per change: new pages, removed pages, optional lastmod updates. See competitors' new products, content clusters and landing pages the day they ship. Pay only for changes; quiet sites cost nothing. Keyless, no proxy. - **URL**: https://apify.com/scrapemint/sitemap-change-monitor.md - **Developed by:** [Ken M](https://apify.com/scrapemint) (community) - **Categories:** SEO tools, Marketing - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing Pay per usage This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Sitemap Change Monitor: New & Removed Page Alerts A website's sitemap announces its strategy before anyone reads the content: new product lines, new content clusters, new landing pages, new careers sections all appear there first. This actor watches any list of sites, snapshots their full sitemap URL sets, and emits **one row per change** on every following run: pages added, pages removed, and optionally pages whose `lastmod` moved. No login, no API key, and no proxy — sitemaps are public by definition. You pay per change, not per scan. A competitor who shipped nothing today costs you nothing. Enterprise tools that do this start around $139 a month. ### What you get One row per detected change: | Field | Description | |---|---| | `changeType` | `new_page`, `removed_page`, `updated_page`, or a one-time free `baseline` | | `site` | Which monitored site | | `url` | The page that appeared, disappeared, or changed | | `lastmod` / `oldLastmod`, `newLastmod` | Sitemap timestamps where published | | `checkedAt` | Timestamp | ### How it works Give it domains or direct sitemap URLs. For domains it reads `robots.txt` for `Sitemap:` lines (falling back to `/sitemap.xml`), follows sitemap indexes into child sitemaps (gzipped `.xml.gz` files supported), and collects up to your URL cap per site. The first run per site stores the snapshot and emits a single **free** baseline row; every following run diffs against the stored snapshot and pushes only the differences. Snapshots live in a named key-value store and survive between runs. ### Input - **Sites**: competitor domains, client sites, or explicit sitemap URLs. - **Change toggles**: new pages, removed pages, lastmod updates. - **Max URLs per site**: snapshot depth (default 20,000, up to 50,000). - **Max rows per run**: cost cap for bulk publishing days. ### Pricing Pay per result: **$0.005 per change row**. Baselines are free, the first 2 change rows of every run are free, and unchanged sites cost nothing. Watching 10 competitor sites daily typically costs a few cents a day. Even a big content push (200 new pages) is $1. ### Typical uses - **SEO and content teams**: see every new page a competitor publishes, daily, before rankings move. New content clusters reveal their keyword strategy. - **E-commerce**: new product and collection pages on rival stores signal category expansion. - **Agencies**: automated "what did competitors ship this week" client reporting. - **Affiliates**: new merchant landing pages and product pages the day they go live. - **Comms and analysts**: removed pages are signals too — vanishing products, pulled announcements, retired services. ### Scheduling Run it daily on an Apify schedule. Wire rows to Slack, email, or Google Sheets with Apify integrations, or poll the dataset via API. Pair with the [Website Change Monitor](https://apify.com/scrapemint/website-change-monitor) to watch page **content**, and this actor to watch site **structure**; the [SEO Site Audit Scraper](https://apify.com/scrapemint/seo-site-audit-scraper) covers page quality. ### Data notes Only what sites publish in their sitemaps is visible; pages excluded from sitemaps are not tracked (that is what the Website Change Monitor is for). Very large sitemaps are truncated at your URL cap, newest-first as ordered by the site. A small number of sites publish no sitemap at all; those are skipped with a warning and cost nothing. # Actor input Schema ## `sites` (type: `array`): Domains (sitemap auto-discovered via robots.txt) or direct sitemap URLs, e.g. https://competitor.com or https://competitor.com/sitemap.xml. ## `trackNewPages` (type: `boolean`): Emit a row when a URL appears in the sitemap. ## `trackRemovedPages` (type: `boolean`): Emit a row when a URL disappears from the sitemap. ## `trackUpdatedPages` (type: `boolean`): Emit a row when a page's lastmod timestamp changes. Uses more snapshot space, so the per-site URL cap tightens to 20,000. ## `maxUrlsPerSite` (type: `integer`): Snapshot depth per site (large sitemaps are truncated to this). ## `maxRows` (type: `integer`): Cap on change rows returned. Controls total cost. ## Actor input object example ```json { "sites": [ "https://www.allbirds.com" ], "trackNewPages": true, "trackRemovedPages": true, "trackUpdatedPages": false, "maxUrlsPerSite": 20000, "maxRows": 200 } ```` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "sites": [ "https://www.allbirds.com" ] }; // Run the Actor and wait for it to finish const run = await client.actor("scrapemint/sitemap-change-monitor").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "sites": ["https://www.allbirds.com"] } # Run the Actor and wait for it to finish run = client.actor("scrapemint/sitemap-change-monitor").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "sites": [ "https://www.allbirds.com" ] }' | apify call scrapemint/sitemap-change-monitor --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=scrapemint/sitemap-change-monitor", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Sitemap Change Monitor: New & Removed Page Alerts", "description": "Watch any website's sitemap and get one row per change: new pages, removed pages, optional lastmod updates. See competitors' new products, content clusters and landing pages the day they ship. Pay only for changes; quiet sites cost nothing. Keyless, no proxy.", "version": "0.1", "x-build-id": "taal53BrvcR7W5WnG" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/scrapemint~sitemap-change-monitor/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-scrapemint-sitemap-change-monitor", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/scrapemint~sitemap-change-monitor/runs": { "post": { "operationId": "runs-sync-scrapemint-sitemap-change-monitor", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/scrapemint~sitemap-change-monitor/run-sync": { "post": { "operationId": "run-sync-scrapemint-sitemap-change-monitor", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "sites" ], "properties": { "sites": { "title": "Sites or sitemap URLs", "type": "array", "description": "Domains (sitemap auto-discovered via robots.txt) or direct sitemap URLs, e.g. https://competitor.com or https://competitor.com/sitemap.xml.", "default": [], "items": { "type": "string" } }, "trackNewPages": { "title": "Track new pages", "type": "boolean", "description": "Emit a row when a URL appears in the sitemap.", "default": true }, "trackRemovedPages": { "title": "Track removed pages", "type": "boolean", "description": "Emit a row when a URL disappears from the sitemap.", "default": true }, "trackUpdatedPages": { "title": "Track updated pages (lastmod)", "type": "boolean", "description": "Emit a row when a page's lastmod timestamp changes. Uses more snapshot space, so the per-site URL cap tightens to 20,000.", "default": false }, "maxUrlsPerSite": { "title": "Max URLs per site", "minimum": 10, "maximum": 50000, "type": "integer", "description": "Snapshot depth per site (large sitemaps are truncated to this).", "default": 20000 }, "maxRows": { "title": "Max change rows per run", "minimum": 1, "maximum": 2000, "type": "integer", "description": "Cap on change rows returned. Controls total cost.", "default": 200 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```