# Sitemap Extractor: Every Page URL, with Change Alerts (`rookdev/sitemap-change-monitor`) Actor Get every page URL from any website's sitemaps (robots.txt, sitemap indexes, gzip), or schedule it to get only pages added, updated or removed since the last run. Built for SEO audits and competitor monitoring. - **URL**: https://apify.com/rookdev/sitemap-change-monitor.md - **Developed by:** [Rook Dev](https://apify.com/rookdev) (community) - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.50 / 1,000 page urls This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## Sitemap Extractor: Every Page URL, with Change Alerts **See every page a website publishes, straight from its sitemaps, or schedule it and get only the pages that were added or removed (and, if you want, updated) since the last run.** - **Every page URL in one table.** Give it a website and it finds the sitemaps itself: robots.txt first, then `/sitemap.xml` and `/sitemap_index.xml`. Sitemap indexes, gzip files (`.xml.gz`) and plain-text sitemaps all work. - **Monitoring built in.** Turn on *Only new and removed pages*, schedule it, and you pay only for changes: `new` pages and `gone` pages. Turn on *Also report updated pages* to get `changed` pages too (their last-modified date moved, with the previous date); sitemap files that re-date most of their pages at once are ignored, so you do not pay for noise. - **Safe by design.** A page is reported removed only when the sitemap it was listed in was read in full. A failed sitemap, a robots.txt hiccup or your URL cap never produces a flood of false "gone" rows. - **Alerts out of the box.** Every row has a one-line `summary`, and the run's status message lists the first changes, so the built-in Slack integration posts "New page on competitor.com: …" with no extra tools. It reads only sitemaps, the files websites publish so that their pages can be found. No browser, no proxies, no page downloads, so runs are fast, cheap and light on the sites you watch. ### Sample output ```json { "status": "new", "site": "docs.apify.com", "url": "https://docs.apify.com/api", "lastModified": null, "changeFrequency": "weekly", "priority": 0.5, "sitemapUrl": "https://docs.apify.com/sitemap_base.xml", "firstSeenAt": "2026-10-29T12:00:00Z", "scrapedAt": "2026-10-29T12:00:00Z" } ``` `lastModified` is filled when the site's sitemap has `` dates (many do; this one does not), normalised to UTC ISO 8601. ### Who uses it - **Marketers and SEO teams:** see the moment a competitor launches a landing page, a pricing page, a product or a blog post, and which pages they quietly remove. - **Content and SEO audits:** a complete list of a site's indexed pages, with last-modified dates, in one click. - **Ecommerce and market research:** new product URLs from a retailer's product sitemap, every day. - **Developers:** feed fresh URLs into another crawler, a search index or an LLM pipeline, without crawling the whole site. - **AI agents:** a narrow, predictable tool for "what pages does this site have, and what changed?"; works through the Apify MCP server. ### How to use it 1. Paste websites into **Websites or sitemap URLs**, one per line, for example `https://docs.apify.com`. You can also paste a single sitemap, for example `https://example.com/sitemap-products.xml`. 2. Optionally keep only part of a site with **URL must contain**, for example `/blog/`. 3. Click **Start**. Download the results as JSON, CSV or Excel, or read them through the API. **To monitor:** turn on **Only new and removed pages since the last run**, save the input as a task and add a daily schedule. The first run returns every page marked `new` (or, with **Start quietly**, just records them for free); after that you get only changes. Connect the task to Slack, email, Google Sheets, Make, Zapier or n8n to get alerts. #### Input examples All pages of a website: ```json { "websites": ["https://docs.apify.com"] } ``` Daily monitor of three competitors' blogs and landing pages, starting quietly: ```json { "websites": ["competitor-one.com", "competitor-two.com", "https://competitor-three.com/sitemap.xml"], "onlyNew": true, "skipExistingOnFirstRun": true, "urlContains": ["/blog/", "/solutions/", "/pricing"] } ``` #### Which inputs work | You enter | What happens | | --- | --- | | `https://docs.apify.com`, `docs.apify.com`, or any page of the site | The site's sitemaps are found through robots.txt, then `/sitemap.xml` and `/sitemap_index.xml` | | A sitemap URL ending in `.xml`, `.xml.gz` or `.txt`, or with `sitemap` in its path | That sitemap and any sitemaps it lists | Inputs that cannot be used produce a free row with `status: invalid_input`, `not_found` or `error` and a plain-English `error`. The same site entered twice is read and charged once, and a URL listed in several sitemaps is returned once. ### Output One row per page: `status`, `summary` (one readable line, e.g. `New page on docs.apify.com: https://docs.apify.com/api`), `site`, `url`, `lastModified`, `changeFrequency`, `priority`, `sitemapUrl`, `firstSeenAt` (monitoring) and `scrapedAt`. `status` is `current` in a normal run, or `new`, `gone` and (with **Also report updated pages**) `changed` when monitoring. `changed` rows also have `previousLastModified`. Inputs that produced no data get a free row with `status` `invalid_input`, `not_found`, `error` or `skipped` (not checked because your spending limit was reached). A `RUN_SUMMARY` record in the run's key-value store lists every site with how many sitemaps and URLs were read, the site checks charged, any sitemap that failed, whether the read was complete or cut by your URL cap, and any sitemap files whose dates were ignored as a re-stamp. ### Get alerts in Slack, Google Drive, Make, Zapier or n8n 1. Turn on monitoring, click **Save as a new task**, and add a daily schedule to the task. 2. On the task's **Integrations** tab, pick **Slack**, connect your workspace, choose the **Run succeeded** event and a channel, and paste this as the message: ```handlebars *{{resource.statusMessage}}* ``` The status message reads like *Checked 3 site(s), saved 4 URL row(s) (3 new, 0 changed, 1 gone). New page on competitor.com: https://competitor.com/pricing; …* so the alert itself tells you what changed. - **Google Drive:** the **Upload results to GDrive** integration saves each run's rows as CSV or JSON. - **Make, Zapier, n8n:** use the Apify app's "Actor run finished" trigger and map the `summary` field into a message, a sheet row or a CRM note. - **Webhooks:** send the run to your own endpoint and read the dataset through the API. ### Pricing Pay per event, no platform usage charges on top: | Event | Price | | --- | --- | | Page URL returned | $0.50 per 1,000 URLs | | Website or sitemap checked | $1.00 per 1,000 site checks: one per site, plus one more per started 50,000 URLs read after the first 50,000 | Rows for inputs that could not be used, sites with no sitemap and sites that failed are free. A site check is charged after its rows are delivered, so if your spending limit stops the run before any of a site's rows go out, that check is free. - **One-off export:** a documentation site with about 4,200 pages costs about **$2.10** ($2.10 for the URLs plus $0.001 for the check). - **Daily monitoring:** 20 competitor sites checked every day with about 50 new or removed pages a day in total costs about **$0.045 a day, about $1.35 a month** (20 checks for $0.02 plus 50 rows for $0.025), because you pay only for changes. With **Start quietly**, the first run costs $0.02 instead of paying for every existing page. - **A big shop:** a daily monitor of a 100,000-page site (with **Max URLs per site** at 100,000) pays 2 site checks a run, **$0.002 a day** when nothing changed, plus $0.0005 per new or removed page. A URL filter on that site scans at most 10 times the cap, so a filtered 1,000-URL cap pays at most one check for its 10,000 URLs read. Set a maximum cost per run in the run options and the Actor stops cleanly when it reaches it: sites it could not check get a free `skipped` row, and when monitoring, changes it could not deliver come on the next run. Rows it delivered are never charged twice. ### FAQ **Is this allowed?** Sitemaps are published so that search engines and other tools can find a site's pages; the Actor reads only those files. It follows `Disallow` rules in robots.txt for the sitemap locations it guesses, reads one sitemap at a time per site, and never downloads the pages themselves. Page URLs are not personal data. A site's own terms can still forbid automated monitoring, so watch only sites you have a lawful reason to follow, do not use it on sites that ask robots to stay out, and follow the terms of any service you send results to. **Why do I get no `changed` rows?** Updated pages are opt-in: turn on **Also report updated pages**. They are off by default because many sites re-date pages without changing them: in a test on three Shopify stores, a run one minute after another saw 2,338 pages with a new `` (1,072 of 1,318 on one store), and you would have paid for every one. Turning it on starts a separate monitoring memory, so its first run is a first run again (use **Start quietly**). Many sitemaps also have no `` dates at all, so only new and removed pages can be seen on those. **What if a site has a huge sitemap?** *Max URLs per site* (default 10,000) caps cost, and sitemap files over 60 MB are skipped. With **URL must contain** or **URL must not contain**, a site scans at most 10 times the cap. When a site is cut short, that run reports no removed pages. A capped monitor sees a different slice of a big site when its sitemap is reordered, so a page it has not seen before is reported `new` only when its `` is after the previous run; the others are recorded for free (the status message counts them), and pages it has not seen for 30 days are forgotten, so its memory stays small. For complete monitoring, set the cap above the site's size, or narrow it with **URL must contain** to the part you care about. **Why did a run stop early?** To keep runs fair and cheap, a run stops checking new sites when your spending limit is reached (a site is read only when the limit still covers one site check and one URL), when 50 inputs in a row fail or have no sitemap (a dead list or an outage), or when it has worked a long time without producing any paid results (a safety limit for runs that only meet errors and timeouts). The sites it did not check share one free `skipped` row and are checked on the next run; when monitoring, pages it found but did not deliver come back on the next run, and nothing is charged twice. A run reads at most 1,000 websites; split longer lists across tasks. One request may take at most 60 seconds, a host that fails 10 times in a row is left alone for the rest of the run, and the Actor sends at most 2 requests at a time to one host. If one Apify account's runs keep producing nothing to charge (for example a list of dead sites run again and again), new runs from that account stop early for up to 24 hours, with a free row that says so; normal use never gets near this. **Why was my URL refused?** The Actor reads only public websites. Addresses on private, local or reserved networks (`localhost`, `127.0.0.1`, `10.x`, `192.168.x`, `169.254.169.254`, `.internal` names and other spellings of them) are refused with a free `invalid_input` row, and so are sitemaps and redirects that lead there. **What if a site re-dates every page on each deploy?** With updated pages on, a sitemap file in which half or more of its known pages (at least 50) get a new `` at once is treated as a re-stamp: none of its pages are reported, the new dates are remembered, and `RUN_SUMMARY` lists the file under `restampedSitemaps` (the status message says how many dates were ignored). The same test is applied to the whole site, for sites with many small sitemap files. Real edits in other sitemap files are still reported. You are not charged for thousands of fake changes. **Why is the first monitoring run bigger?** Nothing has been seen yet, so every page counts as new. Turn on **Start quietly** to skip that. Each saved task, and each combination of filters, keeps its own memory, and normal (non-monitoring) runs never touch it. **What if a sitemap is briefly empty?** If a site that had pages suddenly lists none, the Actor waits for a second empty check before reporting them all removed. **Can I use it from code or an AI agent?** Yes: through the Apify API, the Python and JavaScript clients, or the Apify MCP server. Inputs and outputs are stable and documented above. ### Support Open an issue on the Issues tab with the website you tried; I aim to answer within a day. # Changelog This Actor's version history is a separate document: https://apify.com/rookdev/sitemap-change-monitor/changelog.md # Actor input Schema ## `websites` (type: `array`): One per line. A website (https://docs.apify.com or just docs.apify.com) finds its sitemaps through robots.txt, then /sitemap.xml and /sitemap_index.xml. A sitemap URL (https://example.com/sitemap-products.xml, .xml.gz or .txt) reads just that sitemap and any sitemaps it lists. At most 1,000 per run; addresses on private or local networks are refused. ## `onlyNew` (type: `boolean`): Turn on, save as a task and schedule it to get only changes: pages added (new) and pages removed (gone), plus updated pages if you turn that on below. The first run returns every page as new. Normal runs never touch this memory. ## `skipExistingOnFirstRun` (type: `boolean`): Monitoring only: the first run records the pages that exist today without returning or charging for them (you pay only the site check), so you get only real changes from the second run on. ## `reportUpdatedPages` (type: `boolean`): When monitoring, also return pages whose date in the sitemap moved (status changed, with the previous date). Off by default: many shops, Shopify stores among them, re-date most pages all the time, which would be paid noise. When it is on, a sitemap file in which half or more of its pages (at least 50) move at once is treated as a re-stamp and reported as nothing. Turning it on starts a separate monitoring memory, so its first run is a first run again. ## `urlContains` (type: `array`): Keep only page URLs that contain any of these texts (case-insensitive), for example /blog/ or /products/. Leave empty for all pages. At most 100 entries of up to 200 characters. ## `urlExcludes` (type: `array`): Drop page URLs that contain any of these texts (case-insensitive), for example /tag/ or ?lang=. At most 100 entries of up to 200 characters. ## `maxUrlsPerSite` (type: `integer`): Stop reading a site's sitemaps after this many matching URLs, to cap costs. With a URL filter, a site scans at most 10 times this many URLs. When a site is cut short, no pages are reported as removed in that run, and pages seen for the first time are reported as new only when their last-modified date is after the previous run. ## `stateStoreName` (type: `string`): Key-value store that remembers which pages you have already seen (lowercase letters, digits and hyphens). Each saved task and each combination of filters keeps its own memory inside it, so the default works for most people. ## `maxConcurrency` (type: `integer`): How many websites to read at the same time. Sitemaps of one site are always read one after another, to be gentle with it. ## Actor input object example ```json { "websites": [ "https://docs.apify.com" ], "onlyNew": false, "skipExistingOnFirstRun": false, "reportUpdatedPages": false, "maxUrlsPerSite": 200, "stateStoreName": "sitemap-change-monitor-state", "maxConcurrency": 5 } ``` # Actor output Schema ## `results` (type: `string`): One row per page URL # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "websites": [ "https://docs.apify.com" ], "maxUrlsPerSite": 200 }; // Run the Actor and wait for it to finish const run = await client.actor("rookdev/sitemap-change-monitor").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "websites": ["https://docs.apify.com"], "maxUrlsPerSite": 200, } # Run the Actor and wait for it to finish run = client.actor("rookdev/sitemap-change-monitor").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "websites": [ "https://docs.apify.com" ], "maxUrlsPerSite": 200 }' | apify call rookdev/sitemap-change-monitor --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,rookdev/sitemap-change-monitor" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/phwj8Hgi8H5eXTkEA/builds/UQXVckOHjnyCVtFpQ/openapi.json