Small Sitemap Inventory

Build a bounded inventory of URLs declared by small public HTTPS sitemaps. Each dataset row is one unique URL with sitemap provenance. The run receipt tells you whether the inventory is complete, partial, empty or failed.

Use this Actor when you already know a sitemap or sitemap-index URL and want a compact, auditable URL list. It reads sitemap files only. It does not discover sitemaps through robots.txt, open the declared pages, check page status, download page content or cross the input root's exact host.

Quick start

The default input is a bounded documentation sample:

{}

It resolves to https://docs.apify.com/sitemap_base.xml with a maximum of 100 unique URL rows. This is a useful sample, not a promise to inventory every URL on a site. If the sitemap contains more than 100 accepted URLs, the Actor returns the first 100 unique rows and marks the result PARTIAL with an explicit exclusion count.

For a 25-row example that also demonstrates the source limit:

{
  "startUrls": [{"url": "https://docs.apify.com/sitemap.xml"}],
  "maxDepth": 2,
  "maxSources": 2,
  "maxRequests": 4,
  "maxQueuedSources": 20,
  "maxUrls": 25,
  "maxDeclarations": 1000,
  "maxProvenancePerUrl": 5,
  "maxProvenancePerSource": 5,
  "maxDiagnosticsPerSource": 10,
  "maxOutputBytes": 262144,
  "timeoutSeconds": 10,
  "maxRedirects": 1
}

If the index declares more sitemap files than maxSources allows, those sources are not fetched. Their receipts use SKIPPED_SOURCE_LIMIT, and the overall result remains PARTIAL.

Input contract

Provide one to ten public HTTPS sitemap or sitemap-index URLs. Each input root creates an independent exact-host boundary. Child sitemaps, redirects and declared page URLs must stay on that host.

Important defaults and maxima:

Setting	Default	Maximum	Meaning
maxDepth	2	3	Child sitemap depth below each input root
maxSources	20	50	Sitemap files attempted after validation
maxRequests	40	100	HTTP requests, including redirect hops
maxQueuedSources	50	100	Accepted and deduplicated sitemap sources
maxUrls	100	5,000	Unique dataset rows
maxDeclarations	5,000	10,000	XML sitemap and URL declarations examined
maxOutputBytes	2 MiB	8 MiB	Compact bytes represented by eligible rows
timeoutSeconds	10	10	End-to-end deadline for one sitemap source
maxRedirects	2	3	Same-host redirects for one source

Inputs are strict. Unknown fields, booleans in integer fields, extra request-object fields and out-of-range values fail before transport. maxSources cannot exceed maxQueuedSources.

Dataset rows

Each row includes:

• url, preserving a benign query string in its original order;
• declarationCount and declarationCountComplete;
• bounded sourceUrls and sitemap declaration provenance;
• explicit source and declaration provenance omission counters;
• sitemap lastmod, changefreq and priority values when present.

Sitemap lastmod is publisher-provided declaration metadata. It is not a verified page-change timestamp, and this Actor never presents it as one.

Run receipt and completion

Open the OUTPUT record in the default key-value store for source receipts, request traces, aggregate counters, configured-limit exclusions, dataset delivery status and billing evidence.

• COMPLETE: at least one unique URL and no known gap.
• EMPTY: every accepted source was processed cleanly and declared no accepted URL.
• PARTIAL: useful rows or parsed sources exist, but a source, declaration, safety check, configured limit, provenance detail, output row or billing event was excluded.
• FAILED: no source parsed successfully, or dataset delivery failed before a valid result could be confirmed.
• OUTPUT_ERROR: source processing may have completed, but delivery or billing evidence is failed or ambiguous.

Platform run status and product completeness are different. A platform run can succeed while the inventory is correctly marked PARTIAL.

Safety and deliberate limits

• HTTPS only, with certificate and hostname validation.
• Public DNS answers only, with numeric-address socket pinning and original-host TLS SNI.
• Exact-host validation on roots, redirects, child sitemaps and declared page URLs.
• No retries, implicit proxies, cookies, authorization headers or page requests.
• URL credentials, fragments, non-ASCII literals, local names, non-public IPs and common credential or signed-link query keys are rejected.
• Each source has a 2 MiB encoded entity limit and a 2 MiB decompressed limit.
• Each HTTP response has a 2.25 MiB application-reader cap covering initial headers, chunk sizes and extensions, payload and trailers. This is not a physical wire-byte measurement.
• XML DTDs and entities are rejected. XML structure, element count, depth and attributes are bounded before tree construction.
• Inventory bytes, source receipts, diagnostics and provenance are independently bounded. Any pruning makes completeness false.

These limits are intentionally stricter than the sitemap protocol. Large enterprise sitemaps, cross-host arrangements, authenticated sitemaps and workflows that need page fetching are outside this product's scope.

Runtime and billing

The Actor uses 128 MB of memory and a 600-second outer run timeout, with restart-on-error disabled. This outer timeout is separate from the ten-second per-source deadline and is long enough for the advertised 50-source maximum. A caller may choose a shorter platform timeout, but an externally timed-out process may not get the chance to save its final receipt.

The Store price is $0.001 for each emitted unique dataset row through the automatic apify-default-dataset-item event. At the 100-row default, nominal result charges cannot exceed $0.10 before any lower user-selected charge cap. Source receipts, summaries, empty results, rejected declarations and failed sources are not result events. There is no start fee, source fee or custom success event.

Empty and failed runs have no result-event charge. A run can still consume platform resources before it finds no eligible rows or stops with a source error.