# Sitemap Change Monitor: New Pages with Titles (`huffworks/sitemap-change-monitor`) Actor Watch competitor sitemaps: get only new, removed and updated pages, with the title and description of each new page and a per-site digest. - **URL**: https://apify.com/huffworks/sitemap-change-monitor.md - **Developed by:** [Greg Huff](https://apify.com/huffworks) (community) - **Categories:** SEO tools, Marketing - **Stats:** 2 total users, 1 monthly users, 60.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $1.28 / 1,000 change or site summary rows This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ### What does Sitemap Change Monitor do? **Sitemap Change Monitor** watches the sitemaps of the sites you choose and tells you **what changed since the last run**: pages that were **added**, **removed** or **updated**. Every new page arrives with its **title and meta description**, so an alert reads "New page on competitor.com (/blog/): *Pricing for teams, explained*" instead of a bare URL. Each site also gets one **summary row with per-section counts** (/blog/ +4, /products/ +2) and a ready-to-post message for Slack, email or a spreadsheet. - 🆕 **New pages with titles**: title, meta description, H1, canonical, noindex flag, publish date and language for every added page, read from the page itself (robots.txt respected). - 🗂️ **Per-section digest**: one row per site with added, removed and updated counts by section, the newest pages, and a one-line message. - 🔕 **Only changes**: the first run saves a baseline; later runs return only what changed. In "only when changed" mode a quiet site produces no row, so Make, Zapier or Slack fire only when there is news. - 🧭 **Finds the sitemaps for you**: robots.txt, sitemap indexes (nested), gzipped `.xml.gz` files, plain-text and news sitemaps, and the site's **RSS/Atom feed when there is no sitemap**. - 👥 **Many sites per run**: up to 100 competitors or clients in one scheduled run. Generic sitemap scrapers give you the full URL list every time, and most sitemap diff tools give you bare URLs. This Actor gives you only the changes, explains each new page, and groups them by section. #### Who is it for? - **SEO teams**: see a competitor's new landing pages and content clusters the day they appear in the sitemap, and catch pages dropped from your own sitemap after a release. - **Content marketers**: a daily or weekly feed of what competitors published, with titles and descriptions, ready for a Slack channel or a content brief. - **Competitor and market watchers**: new product, pricing, careers or docs pages across a whole list of companies, one summary per company. ### What data do you get? Two kinds of rows, told apart by `type`. **Change rows** (`type: "change"`), one per added, removed or updated page: | Field | Example | | --- | --- | | changeType | added | | domain | simonwillison.net | | url | https://simonwillison.net/2026/Oct/15/latex/ | | section | /2026/ | | title | Tool: LaTeX renderer | | description | LaTeX renderer turns LaTeX source into a rendered view, either a single display equation typeset by MathJax or a full document... | | h1, canonical, noindex, publishedAt, language, httpStatus | as the page declares them | | lastmod / previousLastmod | the sitemap's `` now and at the previous check | | message | New page on simonwillison.net (/2026/): "Tool: LaTeX renderer" https://simonwillison.net/2026/Oct/15/latex/ | **Site rows** (`type: "site"`), one per site and run: `status` (baseline, compared, partial, failed), `urlCount`, `added`, `removed`, `updated`, `sections` (up to 15, most active first), `newPages` (the 5 newest with titles), `changesPending`, `source` (sitemap or feed), `discoveredVia`, `previousCheckAt` and a `message` such as: > simonwillison.net: 2 new since 2026-10-17 17:15 UTC. New by section: /2026/ +2. Latest: "A quote from Addy Osmani, Morgan Westlee Lunt, and Qing Wang" https://simonwillison.net/2026/Oct/16/code-review/ Download as JSON, CSV, Excel or XML, or read it through the Apify API. The dataset has two views: **Changes and site summaries**, and **Site summaries**. ### How to use it 1. Click **Try for free** and list your sites under **Sites to watch** (a domain, a home page, or a sitemap or feed URL). 2. Press **Start**. The first run saves each site's URL list as the **baseline** and returns one summary row per site. 3. Save the input as a **task** and add a **schedule** (daily is typical). Every later run returns only the changes since the run before. 4. For alerts, set **Site summary rows** to **Only for sites that changed** and connect the task to Slack, email, Make or Zapier: post each row's `message`. #### Input example This is the prefilled input: ```json { "sites": ["https://crawlee.dev", "https://astro.build"], "changeTypes": ["added", "removed", "updated"], "fetchPageDetails": true, "maxDetailsPerSite": 20, "maxChangesPerSite": 200, "summaryRows": "always" } ``` Optional: `includeUrlPatterns` (watch only `/blog/`), `excludeUrlPatterns` (skip `/tag/`), `sectionDepth` (2 groups `/docs/guides/` separately), `maxUrlsPerSite`, `stateStoreName`, `resetState` and `maxConcurrency`. #### Output example From a run on 2026-10-17, shortened. To show change rows without waiting a day, two existing posts were taken out of the saved snapshot, so the run reports them as new; the titles and descriptions were read live from the pages. ```json [ { "type": "change", "domain": "simonwillison.net", "changeType": "added", "url": "https://simonwillison.net/2026/Oct/15/latex/", "section": "/2026/", "title": "Tool: LaTeX renderer", "description": "LaTeX renderer turns LaTeX source into a rendered view, either a single display equation typeset by MathJax or a full document or prose fragment typeset by faster-latex, with a live …", "canonical": "https://simonwillison.net/2026/Oct/15/latex/", "noindex": false, "language": "en-gb", "httpStatus": 200, "detailsStatus": "ok", "message": "New page on simonwillison.net (/2026/): \"Tool: LaTeX renderer\" https://simonwillison.net/2026/Oct/15/latex/", "previousCheckAt": "2026-10-17T17:15:07.481Z", "checkedAt": "2026-10-17T17:15:23.685Z" }, { "type": "site", "domain": "simonwillison.net", "status": "compared", "message": "simonwillison.net: 2 new since 2026-10-17 17:15 UTC. New by section: /2026/ +2. Latest: \"A quote from Addy Osmani, Morgan Westlee Lunt, and Qing Wang\" https://simonwillison.net/2026/Oct/16/code-review/", "urlCount": 17079, "added": 2, "removed": 0, "updated": 0, "changesPending": 0, "removalsChecked": true, "sections": [{ "section": "/2026/", "urls": 1073, "added": 2, "removed": 0, "updated": 0 }], "checkedAt": "2026-10-17T17:15:23.685Z" } ] ``` ### How much does it cost? Pay per event, two events. Platform usage is included. | Event | When | Price (Free and Bronze) | Silver | Gold | | --- | --- | --- | --- | --- | | Change or site summary row | One row saved: an added, removed or updated page, or one site's summary | $0.0016 ($1.60 per 1,000) | $0.00144 | $0.00128 | | New page details | One added page opened to read its title and description | $0.001 ($1.00 per 1,000) | $0.0009 | $0.0008 | Plus Apify's standard start fee of $0.00005 per run. Failed checks (no sitemap or feed found, site blocked) are free. Why two events: the summary row pays for downloading a site's sitemap, which is the real work on a quiet day; page details are separate work you can switch off or cap. Examples: - **10 competitors, daily, 3 new pages on a typical day**: 13 rows and 3 page reads, about $0.024 a day, **about $0.72 a month**. - **The same with "Only for sites that changed"**: quiet sites cost nothing; if the 3 new pages are on 2 sites, that is 5 rows and 3 page reads, about $0.011 a day. - **A site migration with 5,000 changed URLs**: capped at 200 rows per site by default ($0.32); the rest come in the next runs. - **An online store whose platform reshuffles product dates** (some Shopify stores serve a different `` on every request): those date changes are recognised and ignored, so they cost nothing. New and removed products are still reported. Set a maximum cost per run and the Actor stops cleanly when it is reached. Changes it could not save are kept and reported by the next run, never charged twice. (Exception: if you abort a run while a site is being saved, that site keeps its previous snapshot and its changes are reported again.) ### Snapshots and how to reset them Each site's last URL list is kept in a **named key-value store in your account**, `huffworks-sitemap-monitor-state` by default (one gzipped record per site, URLs and lastmod dates only). All runs and tasks that use the same store name share it. - **Start one site over**: in Console open **Storage > Key-value stores > huffworks-sitemap-monitor-state** (or the name in the run log, if that name was taken) and delete that site's record (named after the domain), or run once with that site and **Start over** turned on. - **Start everything over**: delete the store, or set a new **Snapshot store name**. - **Separate watchlists** (for example one per client): give each task its own store name. ### Limits and good to know - **Only what the sitemap says.** A page that is live but not in the sitemap is invisible; a page taken out of the sitemap is reported as removed even if it still loads. - **Updated depends on ``.** Many sites publish no lastmod at all (several of our test sites did not), and some stamp the build time on every URL. A page counts as updated only when its lastmod moves **forward** past the latest date seen before. Lastmod changes are ignored, and counted in `updatedSuppressed`, when 90% of a site's URLs share one lastmod, when a sitemap file's dates move backwards (that file stays ignored until you start over), or when half or more of one file (at least 20 URLs) jumps forward at once. - **Big sites and the URL limit.** When a site has more URLs than **Maximum URLs per site**, the newest sitemap files are read first (by the dates in the sitemap index). A page counts as added only if it comes from a sitemap file that was also read in full at the previous check, or from a file that newly appeared in the index. URLs from files read for the first time (for example after you raise the limit) are saved to the snapshot without rows or charges, and counted in `urlsAbsorbed`. The trade-off: on a site bigger than the limit, a new page can occasionally be missed, but an old page is never reported (or charged) as new. To watch every page of a big site, raise the limit above its size or watch only part of it with **Only URLs matching**. - **Removals are reported only after a complete read.** If any sitemap file fails or a limit cuts the read short, added pages are still reported (by the rule above) but removed ones are not. If more than half of a site (at least 50 URLs) vanishes at once, those removals are held back until the next run confirms them. - **Feeds instead of sitemaps** only show recent items, so for feed-based sites only new pages are reported, never removals. - **Polite by design**: one request at a time per site, at least 0.5 s apart (or the site's robots.txt Crawl-delay, up to 10 s). Pages are opened only where robots.txt allows. No proxies, no logins, no CAPTCHA solving: a site behind a bot wall is reported as failed, free of charge. - Up to 100 sites per run, 500 sitemap files and 50,000 URLs per site by default (up to 200,000), 50 MB per sitemap file. ### FAQ #### Is it legal to monitor sitemaps? Sitemaps and feeds are published by site owners for machines to read. The Actor reads only public, logged-out pages, follows robots.txt for every page it opens, and keeps no personal data. How you use the results is up to you and the terms of the sites you watch. #### Why did my first run return no changes? The first run for a site saves the baseline. Changes are measured between two runs, so schedule the task and the next run reports what changed. #### Can I watch only part of a site? Yes: add `/blog/` (or any regular expression) under **Only URLs matching**, or give a specific sitemap URL such as `https://example.com/post-sitemap.xml`. #### Can I get the full URL list too? No. This Actor reports changes and per-section counts, not the full list. For a one-off full URL list, use a sitemap extractor. ### Support Found a site it cannot read, or need another field? Open an issue on the **Issues** tab with the site and the run link, and we will look at it. # Actor input Schema ## `sites` (type: `array`): Domains (crawlee.dev), home pages (https://www.example.com) or a specific sitemap or feed URL (https://example.com/post-sitemap.xml). For a domain, the Actor finds the sitemaps in robots.txt or at the usual addresses, and falls back to the site's RSS/Atom feed when there is no sitemap. Up to 100 sites per run. ## `changeTypes` (type: `array`): Which changes become rows. added: a URL that was not listed last time. removed: a URL that is no longer listed (only reported after every sitemap file was read in full). updated: the page's date changed. ## `fetchPageDetails` (type: `boolean`): Open each ADDED page once (where robots.txt allows it) and read its title, meta description, first H1, canonical URL, noindex flag, publish date and language. Charged per page read. Pages from RSS/Atom feeds and news sitemaps already carry a title and are not opened. ## `maxDetailsPerSite` (type: `integer`): Cost guard for the page details above: at most this many added pages are opened per site and run, newest first. The others are still reported, without a title. ## `maxChangesPerSite` (type: `integer`): Cost guard for a site that changes a lot at once (a migration, a catalogue import). Changes over the limit are not lost: they are reported by the next run. ## `summaryRows` (type: `string`): One summary row per site with counts, per-section breakdown and a ready-to-post message. 'always': every run (good for dashboards and a heartbeat). 'whenChanged': only for sites that changed (good for Slack, email and Zapier/Make: no row, no alert). 'never': change rows only. ## `sectionDepth` (type: `integer`): How many path levels name a section in the summary. 1 groups https://site.com/blog/my-post under /blog/; 2 groups /docs/guides/x under /docs/guides/. A leading language code (/de/, /en-us/) is kept as part of the section. ## `includeUrlPatterns` (type: `array`): Watch only URLs that match one of these regular expressions (case-insensitive), for example /blog/ or /products/. Empty: all URLs. Changing this later makes newly included URLs show up as added. ## `excludeUrlPatterns` (type: `array`): Ignore URLs that match one of these regular expressions, for example /tag/ or ?page=. ## `maxUrlsPerSite` (type: `integer`): Stop reading a site's sitemaps after this many URLs. When a site is cut off, removed pages are not reported for it (the list is incomplete). ## `stateStoreName` (type: `string`): Name of the key-value store in your account that keeps each site's last URL list. Use a different name for each independent watchlist (for example one per client). Delete the store, or a site's record in it, to start that site over. ## `resetState` (type: `boolean`): Forget the saved snapshot of every site in this run and save a new baseline. No change rows are reported in that run. ## `maxConcurrency` (type: `integer`): Different sites are checked side by side; requests to one site are always one at a time, at least 0.5 s apart (or the site's robots.txt Crawl-delay, up to 10 s). ## Actor input object example ```json { "sites": [ "https://crawlee.dev", "https://astro.build" ], "changeTypes": [ "added", "removed", "updated" ], "fetchPageDetails": true, "maxDetailsPerSite": 20, "maxChangesPerSite": 200, "summaryRows": "always", "sectionDepth": 1, "includeUrlPatterns": [], "excludeUrlPatterns": [], "maxUrlsPerSite": 50000, "stateStoreName": "huffworks-sitemap-monitor-state", "resetState": false, "maxConcurrency": 5 } ``` # Actor output Schema ## `results` (type: `string`): Change rows and site summaries as JSON. Add ?format=csv or ?format=xlsx to download another format. ## `overview` (type: `string`): The overview table: one line per change or site. ## `sites` (type: `string`): Counts per site: URLs, added, removed, updated, pending. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "sites": [ "https://crawlee.dev", "https://astro.build" ] }; // Run the Actor and wait for it to finish const run = await client.actor("huffworks/sitemap-change-monitor").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "sites": [ "https://crawlee.dev", "https://astro.build", ] } # Run the Actor and wait for it to finish run = client.actor("huffworks/sitemap-change-monitor").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "sites": [ "https://crawlee.dev", "https://astro.build" ] }' | apify call huffworks/sitemap-change-monitor --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,huffworks/sitemap-change-monitor" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/4oeJ7ThIDkJINIzA0/builds/QxtnZbup2JK85vaMg/openapi.json