# XML Sitemap Extractor to CSV & New-URL Monitor (`adamhaleworks/sitemap-extractor-change-monitor`) Actor Extract every URL from any site's XML sitemaps to CSV, Excel or JSON (robots.txt discovery, nested indexes, .gz, txt, RSS, broken XML) with lastmod, images, news and hreflang. Changes mode returns only new, removed and updated URLs since the last run: schedule it to monitor competitors. - **URL**: https://apify.com/adamhaleworks/sitemap-extractor-change-monitor.md - **Developed by:** [Adam Hale](https://apify.com/adamhaleworks) (community) - **Categories:** SEO tools, Developer tools, Automation - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing Pay per usage This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## XML Sitemap Extractor to CSV & New-URL Monitor: get every URL from sitemap.xml, then only the new ones Give it domains or sitemap URLs and it returns **every page URL listed in the site's sitemaps**: ``, change frequency, priority, images, videos, Google News titles and hreflang alternates. Switch to **Changes only** mode and schedule it, and each run returns only what changed since the last run: - **new** URLs (new products, articles, landing pages, job posts) - **removed** URLs (delisted products, deleted pages) - **lastmodChanged** URLs (pages the site says were updated) No browser and no page crawling are involved. It reads the XML that sites publish for search engines, so it is fast (about 9,000 URLs from 5 sites in 4 seconds in our tests) and cheap. **Free while in launch; planned pay-per-event price: $0.20 per 1,000 URLs** plus $0.002 per site. ### What can I use it for? - **Competitor monitoring.** Get a daily list of new product pages, blog posts or pricing pages from competitor sites. - **News and content tracking.** News sitemaps include the headline, publication time and keywords, so you can follow new articles without scraping article pages. - **SEO audits.** Export the full URL inventory, find URLs that redirect or return 404 (optional status check), spot `noindex` in X-Robots-Tag and check hreflang coverage. - **Feeding crawlers and AI pipelines.** Pass only the new URLs to your scraper, RAG indexer or translation workflow instead of recrawling the whole site. - **Ecommerce catalog changes.** Track when products are added or delisted on Shopify, WooCommerce or Magento stores. ### What data does it return? One row per URL: | Field | Description | |---|---| | `site`, `url` | Host and page URL | | `changeType` | `null` in "All URLs" mode; `baseline`, `new`, `removed` or `lastmodChanged` in "Changes only" mode | | `lastmod`, `previousLastmod` | Last-modified date from the sitemap (normalised to ISO 8601), and the previous value for updated URLs | | `lastmodIsFetchTime` | `true` when the sitemap stamps every URL with the time of the request (common on Shopify), so that date says nothing about the page | | `changefreq`, `priority` | As declared in the sitemap | | `sitemapUrl` | The sitemap file the URL came from | | `imageCount`, `images` | Image sitemap data (total count and first 10 image URLs) | | `videos` | Video sitemap data: title, description, thumbnail, content/player URL, duration (first 5) | | `news` | Google News data: publication name, language, publication date, title, keywords | | `alternates` | hreflang alternates: language and URL (first 50) | | `statusCode`, `redirectLocation`, `xRobotsTag`, `statusError` | Only with "Check HTTP status" on. `statusError` is `BLOCKED_BY_SITE` when the site refused the check (403, 429 or a bot-protection page), `DISALLOWED_BY_ROBOTS_TXT` when robots.txt does not allow the URL, `ROBOTS_TXT_UNREACHABLE` when robots.txt could not be read, or a network error | | `extractedAt` | Time of extraction | A per-site summary (sitemaps found, parsed and failed, URLs listed and matched, new/removed/changed counts and any issues) is saved as `SUMMARY` in the run's key-value store. A site that refused our requests has the status `blocked_by_site` there. **Sitemap to CSV:** the Output tab exports the URL list as CSV, Excel, JSON or XML in one click, or you can fetch it through the Apify API. #### Example output From a real "All URLs" run on the default input (2026-10-03, UTC), a row from TechCrunch's news sitemap: ```json { "site": "techcrunch.com", "url": "https://techcrunch.com/2026/10/03/federal-judge-calls-flock-indiscriminate-mass-surveillance/", "changeType": null, "lastmod": null, "lastmodIsFetchTime": null, "previousLastmod": null, "changefreq": null, "priority": null, "sitemapUrl": "https://techcrunch.com/news-sitemap.xml", "news": { "publicationName": "TechCrunch", "language": "en", "publicationDate": "2026-10-03T19:33:15Z", "title": "Federal judge calls Flock ‘indiscriminate mass surveillance’" }, "extractedAt": "2026-10-03T23:54:11+00:00" } ``` In "Changes only" mode, `changeType` says what happened. A new, a removed and an updated URL look like this: ```json {"site": "example.com", "url": "https://example.com/blog/c", "changeType": "new", "lastmod": "2026-10-03", "previousLastmod": null} {"site": "example.com", "url": "https://example.com/blog/a", "changeType": "removed", "lastmod": null, "previousLastmod": "2026-09-01"} {"site": "example.com", "url": "https://example.com/blog/b", "changeType": "lastmodChanged", "lastmod": "2026-10-02", "previousLastmod": "2026-09-02"} ``` ### How much does it cost? **Free while in launch.** The actor charges no fee of its own right now; you pay only the Apify platform usage of your runs (the free Apify plan includes $5 of usage a month). The pay-per-event prices below are planned and apply only once they are switched on; Apify notifies existing users before a price change takes effect. Planned pricing, pay per event: | Event | Planned price | |---|---| | Site processed (sitemaps found and read) | $0.002 | | URL in the output | $0.0002 (= **$0.20 per 1,000 URLs**) | | URL status checked (optional) | $0.0005 | - **Monitoring is cheap.** After the first run, only changed URLs are output and charged. At the planned prices, our second run on 5 large sites (about 43,000 URLs in total) output 1 new URL and would cost $0.0102. - **Cheap first run.** In "Changes only" mode, set **Output all URLs on the first monitoring run** to off. The first run then saves the URL list silently and would cost only $0.002 per site. With it on, our first run on the same 5 sites output 42,985 URLs, which would cost $8.61. - **Not charged:** sites where no sitemap could be found or read, sites that refused our requests (`blocked_by_site`), status checks the site refused or robots.txt did not allow, and invalid inputs. - **Cost cap:** once pricing is on, set **Maximum cost per run** and the actor stops cleanly when it reaches it. For comparison (Apify Store, checked 2026-10-03): the most-used sitemap extractor charges $0.50 per 1,000 URLs, another popular one $30 per 1,000 URLs, and a sitemap change monitor charges $0.005 per change row. ### How to use it 1. Enter domains (`example.com`), site URLs or direct sitemap URLs (`https://example.com/sitemap_index.xml`). 2. Choose **All URLs** for a one-off export, or **Changes only** to monitor. 3. Optionally add include/exclude regex patterns (for example `/products/` or `/blog/`), a date filter (`7 days` or `2026-09-01`) or the HTTP status check. 4. For monitoring, create a schedule (for example daily) with the same input. Use a different **Snapshot store name** for each independent monitor. #### Input example ```json { "sites": ["techcrunch.com", "allbirds.com", "https://wordpress.org/sitemap.xml"], "mode": "changes", "emitBaselineOnFirstRun": false, "includePatterns": ["/products/", "/20\\d\\d/"], "excludePatterns": ["/live/"], "snapshotStoreName": "competitor-monitor" } ``` #### API example ```bash curl -X POST "https://api.apify.com/v2/acts/adamhaleworks~sitemap-extractor-change-monitor/run-sync-get-dataset-items?token=" \ -H "Content-Type: application/json" \ -d '{"sites": ["apify.com"], "maxUrlsPerSite": 5000}' ``` ### How it works - **Discovery.** It reads `Sitemap:` lines from robots.txt. If there are none, it tries common locations such as `/sitemap.xml`, `/sitemap_index.xml`, `/wp-sitemap.xml` and `/sitemap.xml.gz`, skipping any that robots.txt disallows. - **Parsing.** It follows nested sitemap indexes (up to 6 levels), handles gzip, plain-text sitemaps and RSS/Atom feeds, and recovers URLs from malformed XML. It uses a streaming parser, so sitemaps with 50,000 URLs and large indexes fit in normal memory. External entities are never resolved. - **Change detection.** The URL list of each site is saved, compressed, in a named key-value store in your account. Each run compares the new list with the saved one. If some sitemap files fail to load, removed URLs are not reported for that run, which prevents false alarms, and the saved list is merged rather than replaced. - **Fetch-time lastmod detection.** Some platforms (for example Shopify product sitemaps and some news sitemaps) set `` to the time of the request. The actor detects this, marks the rows `lastmodIsFetchTime: true` and does not report them as updated. Date filters then use the Google News publication date where there is one. - **Reliability.** Network errors, HTTP 429 and 5xx get retries with exponential backoff, and `Retry-After` is respected. Status checks are limited to 2 parallel requests per host. Progress is saved, so a migrated run continues where it stopped. - **Polite by design.** Every request identifies the actor as `sitemap-extractor-change-monitor/1.0 (+)`; there is no browser disguise, no proxy and no retry with a different identity. Status checks follow the site's robots.txt rules for `sitemap-extractor-change-monitor`. If a site answers with 403, keeps answering 429 or serves a bot-protection page (Cloudflare, DataDome and similar), the actor records `BLOCKED_BY_SITE`; after a 429 or a protection page it sends nothing more to that host in the run. ### Limitations - It only knows what the site lists in its sitemaps. Pages missing from the sitemaps are not found, because the actor does not crawl links. - `lastmodChanged` is only as reliable as the site's `` values. Many CMSs update them correctly; some never do. - Private and internal addresses (for example `127.0.0.1` or `10.0.0.0/8`) are refused. - Memory is the practical limit for very large sites. In our test, 100,000 URLs from bbc.com peaked at 165 MB and the saved snapshot was 1.1 MB. Allow roughly 1 GB of memory per million URLs processed at the same time; the default 1 GB is fine for a few hundred thousand URLs per site. ### Is it legal? What about personal data? Sitemaps are files that sites publish specifically so that machines can read their URL lists. The actor reads only those files and robots.txt, and checks guessed sitemap locations against robots.txt. It does not open the pages themselves unless you turn on the status check, which sends a single HEAD request per URL that robots.txt allows. It identifies itself honestly and does not try to get around bot protection. URLs and publication metadata are not personal data. You are responsible for using the results in line with the laws and terms that apply to you. ### FAQ **How is this different from a crawler?** A crawler opens every page to find links. This actor reads the site's own URL list, which takes seconds instead of hours and costs a fraction as much. **Why did a site return 0 URLs?** Check the `SUMMARY` record in the key-value store. Common reasons: the site has no sitemap (`NO_SITEMAP_FOUND`), robots.txt points to a sitemap that returns 404, your filters matched nothing (compare `urlsListed` with `urlsMatched`), the date filter was used on a sitemap whose lastmod is the fetch time, or the site refused our requests (`blocked_by_site`). Some sites refuse all automated clients, or any User-Agent that mentions Apify; the actor accepts that, has no proxy option to get around it, and you are not charged for the site. **I run a website. How do I control this actor?** It sends `sitemap-extractor-change-monitor` in its User-Agent. A robots.txt group for that name controls the status checks and the guessed sitemap locations (for example `User-agent: sitemap-extractor-change-monitor` with `Disallow: /`). Sitemaps you list in robots.txt are treated as published for machines and are still read. To refuse it entirely, answer its User-Agent with 403 or 429: the actor records `blocked_by_site`, does not retry a 403, and after a 429 sends nothing more to your host in that run. **Can I monitor many sites in one run?** Yes, up to 10,000 sites per run. Each site has its own saved snapshot. **Something is wrong or missing?** Open an issue on the actor's Issues tab and include the site. *** Author: Adam Hale # Actor input Schema ## `sites` (type: `array`): Domains (example.com), site URLs or direct sitemap URLs (https://example.com/sitemap_index.xml). For domains, sitemaps are found via robots.txt and common paths. ## `mode` (type: `string`): "All URLs" returns every URL in the sitemaps. "Changes only" remembers the previous run and returns only new, removed and updated URLs. Schedule it daily to monitor sites. ## `emitBaselineOnFirstRun` (type: `boolean`): In "Changes only" mode, the first run for a site has nothing to compare with. If on, it outputs all URLs with changeType "baseline". If off, it saves the snapshot silently and outputs nothing for that site. ## `trackLastmodChanges` (type: `boolean`): In "Changes only" mode, also report URLs whose date changed. ## `includeRemoved` (type: `boolean`): In "Changes only" mode, also report URLs that disappeared from the sitemaps. They are skipped automatically when some sitemaps failed to load, to avoid false alarms. ## `snapshotStoreName` (type: `string`): Named key-value store (in your account) that keeps the previous URL list per site for "Changes only" mode. Use different names for independent monitors. ## `includePatterns` (type: `array`): Only keep URLs matching at least one of these regular expressions, for example /blog/ or /products/. ## `excludePatterns` (type: `array`): Drop URLs matching any of these regular expressions, for example /tag/ or ?page=. ## `lastmodAfter` (type: `string`): Keep only URLs whose is on or after this date (2026-09-01) or within a relative period ("7 days", "24 hours"). URLs without lastmod are dropped when this is set. ## `maxUrlsPerSite` (type: `integer`): Stop reading a site's sitemaps after this many URLs (0 = up to 5,000,000). ## `includeExtensions` (type: `boolean`): Add image URLs (first 10 and the total count), video details, Google News title and date, and hreflang alternates when the sitemap has them. ## `checkStatus` (type: `boolean`): Send a HEAD request (no redirects followed) to each output URL and add statusCode, redirectLocation and X-Robots-Tag. Free while in launch; under the planned pricing, charged per URL checked. URLs that robots.txt disallows for this actor are skipped, and URLs the site refuses (HTTP 403, 429 or a bot challenge) are marked BLOCKED_BY_SITE; neither is charged. ## `maxStatusChecks` (type: `integer`): Upper limit on status checks in one run. ## `statusConcurrency` (type: `integer`): Parallel status checks. Keep this low to be gentle with the target sites. ## `respectRobotsTxt` (type: `boolean`): Skip guessed sitemap locations that robots.txt disallows. Sitemaps declared in robots.txt and sitemap URLs you enter are always read. Status checks always follow robots.txt. ## `maxSitemapsPerSite` (type: `integer`): Safety limit for huge sitemap indexes. ## `maxConcurrency` (type: `integer`): How many sites are processed at the same time. ## `sitemapConcurrency` (type: `integer`): How many sitemap files of one site are downloaded at the same time. ## `requestTimeoutSecs` (type: `integer`): Timeout per HTTP request. Large sitemaps can take a while to download. ## `maxRetries` (type: `integer`): Retries for network errors, HTTP 429 and 5xx responses, with exponential backoff. ## Actor input object example ```json { "sites": [ "apify.com", "https://techcrunch.com/news-sitemap.xml" ], "mode": "all", "emitBaselineOnFirstRun": true, "trackLastmodChanges": true, "includeRemoved": true, "snapshotStoreName": "sitemap-monitor-snapshots", "maxUrlsPerSite": 1000, "includeExtensions": true, "checkStatus": false, "maxStatusChecks": 10000, "statusConcurrency": 10, "respectRobotsTxt": true, "maxSitemapsPerSite": 1000, "maxConcurrency": 5, "sitemapConcurrency": 4, "requestTimeoutSecs": 30, "maxRetries": 3 } ``` # Actor output Schema ## `results` (type: `string`): Dataset items, one per URL, with lastmod, change type, images, news and hreflang data. ## `summary` (type: `string`): SUMMARY record in the default key-value store: per-site status, sitemaps read, URL counts, status checks and invalid inputs. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "sites": [ "apify.com", "https://techcrunch.com/news-sitemap.xml" ], "maxUrlsPerSite": 1000 }; // Run the Actor and wait for it to finish const run = await client.actor("adamhaleworks/sitemap-extractor-change-monitor").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "sites": [ "apify.com", "https://techcrunch.com/news-sitemap.xml", ], "maxUrlsPerSite": 1000, } # Run the Actor and wait for it to finish run = client.actor("adamhaleworks/sitemap-extractor-change-monitor").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "sites": [ "apify.com", "https://techcrunch.com/news-sitemap.xml" ], "maxUrlsPerSite": 1000 }' | apify call adamhaleworks/sitemap-extractor-change-monitor --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,adamhaleworks/sitemap-extractor-change-monitor" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/LBivjLpoY7phBCJTj/builds/Rm1l5sPVupzhDeNFI/openapi.json