Sitemap URL Inventory & Change API

Create a compact, deterministic URL inventory from public sitemap XML, then receive only new_url, removed_from_observed_inventory, lastmod_changed, and sitemap_structure_changed events.

Use explicit sitemap URLs or domains. Domain discovery reads public robots.txt Sitemap declarations and falls back to /sitemap.xml. Sitemap indexes, nested indexes, gzip, loops, duplicates, missing lastmod, and rolling inventories are handled within hard limits.

The first run emits one free summary by default; it does not dump a giant baseline. The complete contracted inventory remains in persistent state. Optional baseline URL rows are capped and warned. “Removed” means absent from the latest complete observed inventory, never proof that a page was deleted.

Every request and redirect is SSRF-checked. No browser, proxy, login, AI, or private endpoints. Hard controls cover targets, documents, depth, URLs, concurrency, retries, redirects, transaction time, wire bytes, and gzip-decoded bytes.

Only committed change events fire sitemap-change. Recommended provisional PPE: $0.001 per change.

Who this Actor is for

Operations, monitoring, research, and data teams that need monitor public sitemap inventories for deterministic url additions, metadata changes and complete-only removals. It is designed for scheduled Apify, API, webhook, integration, and MCP-compatible workflows—not for unsupported professional conclusions.

Input reference

• sitemapUrls: Configure Explicit sitemap URLs.
• domains: Checks robots.txt Sitemap declarations, then the conventional /sitemap.xml fallback.
• monitorKey: Configure Monitor key.
• emitBaselineUrls: Configure Emit capped baseline URL rows.
• maxBaselineRows: Configure Maximum baseline URL rows.
• maxTargets: Configure Maximum targets.
• maxSitemapDocuments: Configure Maximum sitemap documents per target.
• maxDepth: Configure Maximum nested index depth.
• maxUrls: Configure Maximum URLs across the run.
• maxConcurrency: Configure Concurrent targets.
• requestTimeoutSecs: Configure Request timeout.
• maxRequestRetries: Configure Retries.
• maxWireBytes: Configure Maximum wire bytes per document.
• maxDecodedBytes: Configure Maximum decoded bytes per document.
• maxRedirects: Configure Maximum redirects.

Representative input (replace placeholders and use only sources you are authorized to access):

{
  "sitemapUrls": [],
  "domains": [],
  "monitorKey": "default",
  "emitBaselineUrls": false,
  "maxBaselineRows": 100,
  "maxTargets": 50,
  "maxSitemapDocuments": 100,
  "maxDepth": 5,
  "maxUrls": 50000,
  "maxConcurrency": 6,
  "requestTimeoutSecs": 12,
  "maxRequestRetries": 1,
  "maxWireBytes": 5000000,
  "maxDecodedBytes": 20000000,
  "maxRedirects": 5
}

Output reference

• recordType: Structured output field.
• eventId: Structured output field.
• stableId: Structured output field.
• targetId: Structured output field.
• entryUrl: Structured output field.
• urlsInBaseline: Structured output field.
• sitemapsInBaseline: Structured output field.
• baselineRowsEmitted: Structured output field.
• message: Structured output field.
• changeType: Structured output field.
• url: Structured output field.
• lastmod: Structured output field.
• previous: Structured output field.
• current: Structured output field.
• sourceSitemap: Structured output field.
• code: Structured output field.
• retryable: Structured output field.
• observedAt: Structured output field.

The first complete run creates an expected baseline where the product uses state. Later comparable runs emit only the documented observations or changes. Dataset records retain deterministic identity and source evidence where the contract requires it.

Limitations and known semantics

• Only the documented source, input forms, fields, identities, and completeness rules are supported.
• An empty, partial, blocked, timed-out, or malformed upstream response is never silently reported as a successful empty result.
• Absence or removal is reported only with the exact observational wording and completeness evidence documented by this Actor; it is not proof of a real-world deletion or status unless the source explicitly says so.
• Upstream publication schedules, schemas, coverage, terms, and availability can change.

Errors and troubleshooting

Check the run status and OUTPUT summary before consuming Dataset rows. For invalid input, correct the named field. For retryable source failures, preserve the monitor key/state and retry later. For schema or completeness failures, do not reset state or infer changes; report the run ID and non-secret diagnostics to support.

API, integrations, and MCP

Run the Actor through the Apify API, Schedule, webhook, or supported integration. Read structured records from the default Dataset and the run summary from the default Key-value store. MCP/agent consumers must preserve source, status, uncertainty, and limitation fields and must not convert observational outputs into unsupported claims.

Privacy and legal notes

Submit only data necessary for the documented workflow and only sources you are authorized to access. Do not include credentials, sensitive personal data, or private URLs unless the Actor explicitly documents secure support for them. Outputs are factual automation evidence, not legal, financial, safety, employment, regulatory, or compliance advice. You remain responsible for source rights, downstream use, retention, and professional review.

Supported-use boundary

This Actor accepts only public URLs or caller-controlled Apify data that you own or are authorized to process. sourceRightsConfirmed must be true. The Actor does not grant rights to source material, bypass authentication or access controls, or make an otherwise prohibited collection lawful. Submit the minimum data needed and comply with source terms, privacy duties, retention rules, and applicable law.