Sitemap Extractor & URL Change Monitor

This Actor turns public XML, gzip, text, and nested sitemap trees into a complete URL inventory—and turns later runs into reliable added, lastmod-changed, and confirmed-removed URL events.

Use it for technical SEO audits, site migrations, competitor monitoring, incremental crawlers, link inventories, and RAG ingestion. It is an HTTP sitemap processor, not a browser crawler: no login, proxy, CAPTCHA solving, or external database is required.

Use from AI agents with MCP

Connect the public Actor directly at https://mcp.apify.com?tools=automa-flow/sitemap-extractor-monitor. Ask your agent to inventory a site's sitemap tree or compare later complete runs for added, lastmod-changed, and safely confirmed removed URLs.

Why use this Sitemap Extractor?

• Extract up to 5,000,000 URL entries per input from urlset, sitemapindex, .xml.gz, plain-text sitemaps, or every Sitemap: line in robots.txt.
• Keep URL identity stable without collapsing trailing slashes or query strings.
• Reuse per-sitemap ETag and HTTP Last-Modified validators on scheduled runs.
• Treat a valid empty sitemap as SUCCESS, never as a transport/parser failure.
• Preserve the last good state whenever any child sitemap is incomplete.
• Confirm removals only after two independent, complete missing observations.
• Batch 1–100 websites and keep one bad input isolated from every other result.
• Join results to your CRM or pipeline with the echoed externalId.

Quick start: extract a complete sitemap

The default run is a stateless snapshot and returns every selected URL. Paste a root website, robots.txt, sitemap index, direct sitemap, gzip sitemap, or text sitemap:

{
  "sites": [
    {
      "url": "https://www.shopify.com/sitemap.xml",
      "externalId": "store-demo"
    }
  ]
}

The Dataset contains normalized url_observation records and exactly one site_summary for this input. Download it as JSON, JSONL, CSV, XML, RSS, or Excel from Apify.

Monitor URL changes safely

Save this input in a Task and schedule it daily or weekly:

{
  "sites": [
    {"url": "https://example.com", "externalId": "competitor-a"}
  ],
  "mode": "monitor",
  "outputMode": "changesOnly"
}

The first complete run creates a quiet baseline: it emits a summary but does not pretend every existing URL is new. Later complete runs emit only:

<lastmod> is publisher metadata, not proof that page content changed. If a URL returns before the second missing observation, the pending removal is cleared. A partial or failed run never creates or advances removals and never overwrites the last complete state.

Predictable pay-per-event pricing

Two events, both charged only for work that finished. Apify platform usage is included; the current Pricing tab in Apify is authoritative.

You pay nothing for site summaries, retries, invalid inputs, failed or partial sitemap trees, suppressed baseline/unchanged rows, or equivalent duplicate input aliases. A failed or partial tree costs $0, including its site-check. A snapshot or outputMode: all charges for every URL row returned; a changes-only monitor charges only the change rows returned, so a quiet run costs one site-check and nothing else.

Worked examples: a one-off snapshot of a 50,000-URL site is $5.002. Monitoring 100 sites hourly with changesOnly is $0.20 per quiet run, plus $0.0001 for each URL that actually appears, changes lastmod, or is removed.

If the run reaches its maximum charge, only the affordable URL rows are pushed, the input summary reports PARTIAL with BILLING_LIMIT_REACHED, and monitor state is not advanced. The next run can therefore resume from the last good comparison instead of silently losing changes.

Input reference

Snapshot mode always returns selected URL rows. Pattern changes use a separate monitor-state namespace so incompatible inventories are never compared.

Dataset output

A URL record is history-ready and includes stable IDs, collection time, source sitemap, normalized metadata, change classification, and an optional page status:

{
  "record_type": "url_observation",
  "source": "sitemap",
  "source_id": "e1b2...sha256",
  "source_url": "https://example.com/products-sitemap.xml",
  "scraped_at": "2026-09-02T08:00:00Z",
  "schema_version": 1,
  "fingerprint": "5a1c...sha256",
  "site_id": "173c0b9f...",
  "external_id": "migration-main",
  "status": "SUCCESS",
  "change_type": "URL_LASTMOD_CHANGED",
  "url": "https://example.com/products/red-shoes?size=42",
  "sitemap_url": "https://example.com/products-sitemap.xml",
  "lastmod": "2026-09-02T10:00:00Z",
  "previous_lastmod": "2026-08-28T10:00:00Z",
  "current_lastmod": "2026-09-02T10:00:00Z",
  "changefreq": "weekly",
  "priority": 0.8,
  "http_status": null,
  "error": null
}

Every input also gets one site_summary, including completeness, sitemap and URL counts, changes, bytes, HTTP 304 reuse, duration, and a structured error when needed. Dataset views separate URL changes, URL inventory, and Site summaries.

Run from the API

Synchronous HTTP call:

curl -X POST \
  "https://api.apify.com/v2/acts/automa-flow~sitemap-extractor-monitor/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"sites":[{"url":"https://example.com/sitemap.xml"}]}'

JavaScript:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automa-flow/sitemap-extractor-monitor').call({
  sites: [{ url: 'https://example.com', externalId: 'seo-main' }],
  mode: 'monitor',
  outputMode: 'changesOnly',
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();

Python:

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("automa-flow/sitemap-extractor-monitor").call(
    run_input={
        "sites": [{"url": "https://example.com/sitemap.xml"}],
        "mode": "snapshot",
    }
)
items = client.dataset(run["defaultDatasetId"]).list_items().items

Keep API tokens in environment variables or Apify secrets, never in Actor input.

Scheduling and webhooks

For monitoring, create a Task from a tested input, add an exclusive daily or weekly Schedule, then add an ACTOR.RUN.SUCCEEDED webhook. The webhook run resource exposes defaultDatasetId; filter Dataset rows where

record_type == "url_observation"

For RAG ingestion, seed the corpus with one snapshot. On scheduled runs, enqueue URL_ADDED and URL_LASTMOD_CHANGED, and delete downstream documents only after URL_REMOVED_CONFIRMED.

Completeness and failure semantics

The Actor returns SUCCESS with zero URLs only for a valid, completely parsed empty tree. These conditions return PARTIAL or FAILED and preserve good monitor state:

• malformed XML, DTD/entity declarations, invalid UTF-8, or truncated gzip;
• failed child sitemap, timeout, 403, exhausted 429/5xx retries, or challenge HTML;
• response, decompression, nesting, file-count, or total-entry limit;
• sitemap ancestor cycle or a 304 response whose matching cache is missing;
• Dataset delivery, KVS commit, or PPE charge-limit truncation failure.

Each structurally invalid item in a batch receives its own FAILED summary. NO_RESULTS is never used to disguise a source failure.

Conditional HTTP, memory, and state

Monitor mode stores only the last complete state in a named Apify KVS. It uses a small atomic manifest, 256 logical URL buckets coalesced into byte-bounded KVS records, and per-sitemap parsed caches targeting at most 3 MiB per record. New generation records are written before the manifest commit point; interrupted or partial work leaves the old generation active.

Raw responses and decompressed XML spill from memory to temporary disk above 1 MiB. XML parsing is incremental. Optional page status checks are globally deduplicated and bounded to 20 overall / 2 per host, but can dramatically increase requests—leave them off for large inventories unless you need them.

Supported formats and limits

• Sitemap XML with or without the standard namespace
• Nested sitemapindex trees and globally deduplicated children
• Gzip by payload magic bytes, including .xml.gz
• UTF-8 plain-text sitemaps
• 50,000 entries and 50 MiB uncompressed per sitemap protocol document
• Nesting depth 8, up to 10,000 sitemap files, and 5,000,000 entries per input
• Public HTTP(S) ports 80/443 only; maximum input/URL length 2,048 characters

The Actor preserves path case, trailing slash, query parameters, and query order. It lowercases and IDNA-normalizes hosts, removes default ports and fragments, and does not invent equivalence between distinct public URLs.

Troubleshooting

• Only a summary on the first monitor run: expected with outputMode: changesOnly; use a snapshot to export the initial inventory.
• PARTIAL with a child error: fix or wait for the named sitemap source. The Actor will not compare an incomplete tree.
• No 304 responses: the origin may ignore conditional headers. Correctness is unchanged; the sitemap is downloaded and parsed again.
• Unexpectedly high request count: disable checkPageStatus; it adds a request for every emitted current URL.
• Missing URLs: review include/exclude globs and the normalized input shown in the site summary.

Responsible use

Sitemaps and robots files are public machine-readable endpoints, but they are not blanket permission for every downstream use. Respect site terms, robots policy, rate limits, copyright/database rights, and applicable privacy law. Do not submit credentials, private URLs, signed secrets, or personal data.

The Actor rejects localhost, private/link-local/reserved IP space, cloud metadata targets, embedded credentials, unsafe DNS answers, unapproved ports, and unsafe redirects. It ignores environment proxies and does not bypass authentication, CAPTCHA, paywalls, or access controls.

Development and verification

uv run pytest actors/sitemap-extractor-monitor/tests
uv run ruff check .
uv run ruff format --check .
uv run mypy
docker build -t sitemap-extractor-monitor actors/sitemap-extractor-monitor

The repository includes deterministic fixtures and economics tests; CI never contacts live third-party sources.