# Sitemap Extractor & URL Change Monitor (`automa-flow/sitemap-extractor-monitor`) Actor Sitemap extractor API for complete XML, gzip, text and nested sitemap inventories. Export normalized URLs or monitor new, publisher-lastmod-changed and conservatively confirmed removed URLs. Failed or partial trees never manufacture mass removals. - **URL**: https://apify.com/automa-flow/sitemap-extractor-monitor.md - **Developed by:** [Vadim Bezrukov](https://apify.com/automa-flow) (community) - **Categories:** SEO tools, Automation, Developer tools - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.10 / 1,000 delivered sitemap urls This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. - **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use). - **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`. - **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`. - **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`. - **Any other language** — the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Sitemap Extractor & URL Change Monitor This Actor turns public XML, gzip, text, and nested sitemap trees into a complete URL inventory—and turns later runs into reliable added, `lastmod`-changed, and confirmed-removed URL events. Use it for technical SEO audits, site migrations, competitor monitoring, incremental crawlers, link inventories, and RAG ingestion. It is an HTTP sitemap processor, not a browser crawler: no login, proxy, CAPTCHA solving, or external database is required. ### Use from AI agents with MCP Connect the public Actor directly at `https://mcp.apify.com?tools=automa-flow/sitemap-extractor-monitor`. Ask your agent to inventory a site's sitemap tree or compare later complete runs for added, lastmod-changed, and safely confirmed removed URLs. ### Why use this Sitemap Extractor? - Extract up to 5,000,000 URL entries per input from `urlset`, `sitemapindex`, `.xml.gz`, plain-text sitemaps, or every `Sitemap:` line in `robots.txt`. - Keep URL identity stable without collapsing trailing slashes or query strings. - Reuse per-sitemap `ETag` and HTTP `Last-Modified` validators on scheduled runs. - Treat a valid empty sitemap as `SUCCESS`, never as a transport/parser failure. - Preserve the last good state whenever any child sitemap is incomplete. - Confirm removals only after two independent, complete missing observations. - Batch 1–100 websites and keep one bad input isolated from every other result. - Join results to your CRM or pipeline with the echoed `externalId`. ### Quick start: extract a complete sitemap The default run is a stateless snapshot and returns every selected URL. Paste a root website, `robots.txt`, sitemap index, direct sitemap, gzip sitemap, or text sitemap: ```json { "sites": [ { "url": "https://www.shopify.com/sitemap.xml", "externalId": "store-demo" } ] } ``` The Dataset contains normalized `url_observation` records and exactly one `site_summary` for this input. Download it as JSON, JSONL, CSV, XML, RSS, or Excel from Apify. ### Monitor URL changes safely Save this input in a Task and schedule it daily or weekly: ```json { "sites": [ {"url": "https://example.com", "externalId": "competitor-a"} ], "mode": "monitor", "outputMode": "changesOnly" } ``` The first complete run creates a quiet baseline: it emits a summary but does not pretend every existing URL is new. Later complete runs emit only: | Event | Meaning | | --- | --- | | `URL_ADDED` | A normalized URL was absent from the prior good state. | | `URL_LASTMOD_CHANGED` | The publisher-supplied sitemap `` changed. | | `URL_REMOVED_PENDING` | First complete run where the URL is missing. | | `URL_REMOVED_CONFIRMED` | Second independent complete run where it is still missing. | `` is publisher metadata, not proof that page content changed. If a URL returns before the second missing observation, the pending removal is cleared. A partial or failed run never creates or advances removals and never overwrites the last complete state. ### Predictable pay-per-event pricing Two events, both charged only for work that finished. Apify platform usage is included; the current Pricing tab in Apify is authoritative. | Event | Price | Charged for | | --- | ---: | --- | | `site-check` | $0.002 | One per input site whose sitemap tree was fetched and compared completely, including a quiet run that proves nothing changed. | | `url-record` | $0.0001 | One per `url_observation` row delivered. | | Delivered URL rows | Price | | ---: | ---: | | 1,000 | $0.10 | | 10,000 | $1.00 | | 100,000 | $10.00 | | 1,000,000 | $100.00 | You pay nothing for site summaries, retries, invalid inputs, failed or partial sitemap trees, suppressed baseline/unchanged rows, or equivalent duplicate input aliases. A failed or partial tree costs $0, including its `site-check`. A snapshot or `outputMode: all` charges for every URL row returned; a changes-only monitor charges only the change rows returned, so a quiet run costs one `site-check` and nothing else. Worked examples: a one-off snapshot of a 50,000-URL site is $5.002. Monitoring 100 sites hourly with `changesOnly` is $0.20 per quiet run, plus $0.0001 for each URL that actually appears, changes `lastmod`, or is removed. If the run reaches its maximum charge, only the affordable URL rows are pushed, the input summary reports `PARTIAL` with `BILLING_LIMIT_REACHED`, and monitor state is not advanced. The next run can therefore resume from the last good comparison instead of silently losing changes. ### Input reference | Field | Default | Description | | --- | --- | --- | | `sites` | required | 1–100 root, robots, XML/index, gzip, or text sitemap URLs. | | `sites[].externalId` | `null` | Optional client, competitor, migration, or pipeline ID. | | `mode` | `snapshot` | `snapshot` returns current inventory; `monitor` compares good runs in KVS. | | `outputMode` | `all` | `all` returns every URL; `changesOnly` suppresses baseline/unchanged rows in monitor mode. | | `discoverFromRobots` | `true` | For roots, read all sitemap declarations, then use `/sitemap.xml` if none exist. | | `checkPageStatus` | `false` | Make a lightweight status request for each emitted current URL. | | `includePatterns` | `[]` | Case-sensitive URL glob allowlist, such as `*://*/docs/*`. | | `excludePatterns` | `[]` | Case-sensitive glob denylist, applied after includes. | Snapshot mode always returns selected URL rows. Pattern changes use a separate monitor-state namespace so incompatible inventories are never compared. ### Dataset output A URL record is history-ready and includes stable IDs, collection time, source sitemap, normalized metadata, change classification, and an optional page status: ```json { "record_type": "url_observation", "source": "sitemap", "source_id": "e1b2...sha256", "source_url": "https://example.com/products-sitemap.xml", "scraped_at": "2026-09-02T08:00:00Z", "schema_version": 1, "fingerprint": "5a1c...sha256", "site_id": "173c0b9f...", "external_id": "migration-main", "status": "SUCCESS", "change_type": "URL_LASTMOD_CHANGED", "url": "https://example.com/products/red-shoes?size=42", "sitemap_url": "https://example.com/products-sitemap.xml", "lastmod": "2026-09-02T10:00:00Z", "previous_lastmod": "2026-08-28T10:00:00Z", "current_lastmod": "2026-09-02T10:00:00Z", "changefreq": "weekly", "priority": 0.8, "http_status": null, "error": null } ``` Every input also gets one `site_summary`, including completeness, sitemap and URL counts, changes, bytes, HTTP 304 reuse, duration, and a structured error when needed. Dataset views separate **URL changes**, **URL inventory**, and **Site summaries**. ### Run from the API Synchronous HTTP call: ```bash curl -X POST \ "https://api.apify.com/v2/acts/automa-flow~sitemap-extractor-monitor/run-sync-get-dataset-items?token=$APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{"sites":[{"url":"https://example.com/sitemap.xml"}]}' ``` JavaScript: ```javascript import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('automa-flow/sitemap-extractor-monitor').call({ sites: [{ url: 'https://example.com', externalId: 'seo-main' }], mode: 'monitor', outputMode: 'changesOnly', }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); ``` Python: ```python import os from apify_client import ApifyClient client = ApifyClient(os.environ["APIFY_TOKEN"]) run = client.actor("automa-flow/sitemap-extractor-monitor").call( run_input={ "sites": [{"url": "https://example.com/sitemap.xml"}], "mode": "snapshot", } ) items = client.dataset(run["defaultDatasetId"]).list_items().items ``` Keep API tokens in environment variables or Apify secrets, never in Actor input. ### Scheduling and webhooks For monitoring, create a Task from a tested input, add an exclusive daily or weekly Schedule, then add an `ACTOR.RUN.SUCCEEDED` webhook. The webhook run resource exposes `defaultDatasetId`; filter Dataset rows where `record_type == "url_observation"` and route them to Slack, email, Make, n8n, Zapier, your queue, or an incremental crawler. For RAG ingestion, seed the corpus with one snapshot. On scheduled runs, enqueue `URL_ADDED` and `URL_LASTMOD_CHANGED`, and delete downstream documents only after `URL_REMOVED_CONFIRMED`. ### Completeness and failure semantics The Actor returns `SUCCESS` with zero URLs only for a valid, completely parsed empty tree. These conditions return `PARTIAL` or `FAILED` and preserve good monitor state: - malformed XML, DTD/entity declarations, invalid UTF-8, or truncated gzip; - failed child sitemap, timeout, 403, exhausted 429/5xx retries, or challenge HTML; - response, decompression, nesting, file-count, or total-entry limit; - sitemap ancestor cycle or a 304 response whose matching cache is missing; - Dataset delivery, KVS commit, or PPE charge-limit truncation failure. Each structurally invalid item in a batch receives its own `FAILED` summary. `NO_RESULTS` is never used to disguise a source failure. ### Conditional HTTP, memory, and state Monitor mode stores only the last complete state in a named Apify KVS. It uses a small atomic manifest, 256 logical URL buckets coalesced into byte-bounded KVS records, and per-sitemap parsed caches targeting at most 3 MiB per record. New generation records are written before the manifest commit point; interrupted or partial work leaves the old generation active. Raw responses and decompressed XML spill from memory to temporary disk above 1 MiB. XML parsing is incremental. Optional page status checks are globally deduplicated and bounded to 20 overall / 2 per host, but can dramatically increase requests—leave them off for large inventories unless you need them. ### Supported formats and limits - Sitemap XML with or without the standard namespace - Nested `sitemapindex` trees and globally deduplicated children - Gzip by payload magic bytes, including `.xml.gz` - UTF-8 plain-text sitemaps - 50,000 entries and 50 MiB uncompressed per sitemap protocol document - Nesting depth 8, up to 10,000 sitemap files, and 5,000,000 entries per input - Public HTTP(S) ports 80/443 only; maximum input/URL length 2,048 characters The Actor preserves path case, trailing slash, query parameters, and query order. It lowercases and IDNA-normalizes hosts, removes default ports and fragments, and does not invent equivalence between distinct public URLs. ### Troubleshooting - **Only a summary on the first monitor run:** expected with `outputMode: changesOnly`; use a snapshot to export the initial inventory. - **PARTIAL with a child error:** fix or wait for the named sitemap source. The Actor will not compare an incomplete tree. - **No 304 responses:** the origin may ignore conditional headers. Correctness is unchanged; the sitemap is downloaded and parsed again. - **Unexpectedly high request count:** disable `checkPageStatus`; it adds a request for every emitted current URL. - **Missing URLs:** review include/exclude globs and the normalized input shown in the site summary. ### Responsible use Sitemaps and robots files are public machine-readable endpoints, but they are not blanket permission for every downstream use. Respect site terms, robots policy, rate limits, copyright/database rights, and applicable privacy law. Do not submit credentials, private URLs, signed secrets, or personal data. The Actor rejects localhost, private/link-local/reserved IP space, cloud metadata targets, embedded credentials, unsafe DNS answers, unapproved ports, and unsafe redirects. It ignores environment proxies and does not bypass authentication, CAPTCHA, paywalls, or access controls. ### Development and verification ```bash uv run pytest actors/sitemap-extractor-monitor/tests uv run ruff check . uv run ruff format --check . uv run mypy docker build -t sitemap-extractor-monitor actors/sitemap-extractor-monitor ``` The repository includes deterministic fixtures and economics tests; CI never contacts live third-party sources. # Actor input Schema ## `sites` (type: `array`): One to 100 root websites, robots.txt URLs, direct sitemap files, sitemap indexes, .xml.gz files, or text sitemaps. externalId is echoed for downstream joins. ## `mode` (type: `string`): snapshot extracts the current inventory without state. monitor compares only complete successful inventories in Apify KVS. ## `outputMode` (type: `string`): changesOnly emits URL rows only for real changes; the first baseline and unchanged runs still return one site\_summary. Snapshot mode always returns extracted URLs. ## `discoverFromRobots` (type: `boolean`): For root websites, read every Sitemap: declaration first. If none exist, try /sitemap.xml. Direct sitemap and robots.txt inputs are always respected. ## `checkPageStatus` (type: `boolean`): Optional lightweight checks for emitted current URL rows only. Disabled by default: extracting a million URLs must not make a million page requests. ## `includePatterns` (type: `array`): Optional case-sensitive glob patterns such as *://*/blog/\* or *product*. Empty means include every sitemap URL. Pattern changes use a separate monitor state namespace. ## `excludePatterns` (type: `array`): Optional case-sensitive glob patterns such as */tag/* or *?preview=*. Exclusions run after includes. ## Actor input object example ```json { "sites": [ { "url": "https://www.shopify.com/sitemap.xml", "externalId": "store-demo" } ], "mode": "snapshot", "outputMode": "all", "discoverFromRobots": true, "checkPageStatus": false, "includePatterns": [], "excludePatterns": [] } ``` # Actor output Schema ## `results` (type: `string`): No description ## `runSummary` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "sites": [ { "url": "https://www.shopify.com/sitemap.xml", "externalId": "store-demo" } ] }; // Run the Actor and wait for it to finish const run = await client.actor("automa-flow/sitemap-extractor-monitor").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "sites": [{ "url": "https://www.shopify.com/sitemap.xml", "externalId": "store-demo", }] } # Run the Actor and wait for it to finish run = client.actor("automa-flow/sitemap-extractor-monitor").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "sites": [ { "url": "https://www.shopify.com/sitemap.xml", "externalId": "store-demo" } ] }' | apify call automa-flow/sitemap-extractor-monitor --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,automa-flow/sitemap-extractor-monitor" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/DeQ8XMebYVzamMS6L/builds/EWGfcV12eKxI64ZlJ/openapi.json