Sitemap Extractor - Hardened URL Inventory

Sitemap Extractor - Hardened URL Inventory extracts a bounded, machine-readable inventory of URLs declared in public sitemaps. The Actor accepts explicit sitemap files or site origins, follows same-origin sitemapindex documents, handles strict single-member gzip, and publishes evidence about what it did and did not cover.

It is designed for SEO audits, migration checks, content inventories, indexing diagnostics, URL reconciliation, and agent workflows that need trustworthy structured output instead of an opaque list. The discovery phrase used in Store copy is Secure Sitemap URL Extractor & Coverage API.

This is a source-native HTTPS reader, not a web crawler. It requests only robots.txt and sitemap documents. It never requests page URLs found in <loc>; they are reported as data only and are never fetched.

This Actor is independently published and is not affiliated with, sponsored by, or endorsed by Google, Bing, or the Sitemaps protocol maintainers.

Pricing: $0.0002 per unique canonical URL saved to the default Dataset ($0.20 per 1,000), plus Apify's $0.00005 Actor-start event. The live Monetization panel says User pays platform usage costs: No, so normal run platform usage is included in these event prices. The synthetic default-Dataset-item event is disabled, so receipts and diagnostics are not billed as URL-result events. Post-run Dataset access can still incur normal storage-operation charges under Apify's Store billing rules.

What you get

• Strict flat URL records in the default dataset.
• Run, root, and source-document coverage receipts in the receipts dataset alias.
• Sanitized, stable-code diagnostics in the issues dataset alias.
• A compact OUTPUT record with counts, completion state, limit flags, and dataset IDs.
• Deterministic ordering and a zero-network demo when the input is empty.

Typical uses include comparing sitemap URLs with another index, exporting a site migration inventory, spotting malformed metadata, measuring sitemap-index reach, and feeding a bounded URL list into an AI agent or downstream Actor.

Input

Provide one or both of these root lists:

• sitemapUrls: public HTTPS sitemap XML or .xml.gz URLs to read directly.
• siteUrls: public HTTPS bare origins. For each origin, the Actor reads /robots.txt, accepts only same-origin Sitemap: directives, and otherwise tries /sitemap.xml under the documented fallback rules.

The same canonical URL cannot appear in both lists. Inputs accept only HTTPS on standard port 443. Credentials, fragments, IP literals, raw whitespace, backslashes, private-network destinations, and auth-like query parameters are rejected before source processing.

Fastest first run

Use empty input:

{}

That runs bundled sitemap-index, plain XML, and gzip fixtures through the real decoder, parser, and normalizers. It makes zero source-network requests and returns exactly three rows marked with "demo": true under example.invalid.

Explicit sitemaps

{
  "sitemapUrls": [
    "https://www.example.com/sitemap.xml",
    "https://www.example.com/products.xml.gz"
  ],
  "maxSitemaps": 100,
  "maxUrls": 25000,
  "maxDepth": 3
}

Discover from a site origin

{
  "siteUrls": ["https://www.example.com/"],
  "maxSitemaps": 250,
  "maxUrls": 50000,
  "maxConcurrency": 3,
  "maxRetries": 2,
  "requestTimeoutSeconds": 15,
  "maxRunSeconds": 180,
  "maxIssueRows": 500
}

Limits and defaults

Field	Default	Allowed	Meaning
maxSitemaps	100	1–1,000	Run-wide sitemap scheduling cap
maxUrls	10,000	1–100,000	Requested URL-row cap; fixed 32 MiB retained-output budget may stop earlier
maxDepth	3	0–10	Same-origin sitemap-index recursion depth
maxCompressedBytes	2 MiB	64 KiB–10 MiB	Download cap per robots/sitemap document
maxDecompressedBytes	10 MiB	64 KiB–50 MiB	Decoded body cap per document
requestTimeoutSeconds	15	2–60	Connect-and-read deadline per attempt
maxRunSeconds	120	5–900	Source-processing deadline
maxRetries	2	0–3	Retries for bounded transient failures
maxConcurrency	3	1–10	Concurrent sitemap reads
maxIssueRows	500	0–5,000	Stored diagnostic-row cap

maxDecompressedBytes must be at least maxCompressedBytes. The runtime also requires:

maxConcurrency × (maxCompressedBytes + maxDecompressedBytes) ≤ 64 MiB

This cross-field rule limits simultaneously live compressed and decoded bodies to 64 MiB. Fetches are handled in deterministic windows no larger than maxConcurrency; each window is parsed and committed before the next window starts, so completed bodies and parsed trees never accumulate across a wide index frontier. A separate fixed 32 MiB retained URL-output budget includes a conservative per-row/object allowance and may stop a run before maxUrls when unusually long URLs would otherwise threaten the fixed 512 MiB memory envelope. That stop is reported as OUTPUT_MEMORY_LIMIT, urlLimitReached: true, stopReason: "url_limit", and non-exact coverage. At most 100 canonical input roots are accepted after per-list deduplication.

Output

Default dataset: URL rows only

The default dataset never mixes summaries or errors into URL records. Deduplication is root-scoped: the same page URL found repeatedly under one input root is emitted once, while the same page URL reached from two independent roots is emitted once for each root.

{
  "schemaVersion": "1.1",
  "demo": false,
  "url": "https://www.example.com/products/widget",
  "sourceSitemapUrl": "https://www.example.com/products.xml.gz",
  "rootUrl": "https://www.example.com/sitemap.xml",
  "depth": 1,
  "lastmod": "2026-08-01",
  "changefreq": "weekly",
  "priority": 0.8,
  "capturedAt": "2026-08-10T12:00:00.000Z"
}

lastmod, changefreq, and priority are source hints, not freshness or ranking guarantees. Invalid optional hints become null and produce issue evidence. A page <loc> may be HTTP or HTTPS because it is reported, not requested. Live rows have a UTC capturedAt; deterministic demo rows use null.

receipts dataset: coverage evidence

Receipts use recordType values run_receipt, root_receipt, and sitemap_receipt. Site-origin runs also include a source receipt for the robots.txt request. Terminal receipts are emitted for bounded-frontier documents that cannot be fetched because of a declared depth, sitemap, URL, or runtime limit. When a single index declares more children than the remaining sitemap budget, the parent receipt and aggregated SITEMAP_COUNT_LIMIT issue account for the omitted occurrences without retaining an unbounded receipt or URL string for every excluded child.

{
  "schemaVersion": "1.1",
  "recordType": "sitemap_receipt",
  "coverage": "exact",
  "stopReason": "completed",
  "rootUrl": "https://www.example.com/sitemap.xml",
  "rootKind": "explicit_sitemap",
  "sitemapUrl": "https://www.example.com/products.xml.gz",
  "depth": 1,
  "documentKind": "urlset",
  "robotsStatus": null,
  "httpStatus": 200,
  "attempts": 1,
  "compressedBytes": 512,
  "decompressedBytes": 2048,
  "rootsRequested": 0,
  "sitemapsDiscovered": 0,
  "sitemapsFetched": 1,
  "sitemapsParsed": 1,
  "urlsSeen": 100,
  "urlsAccepted": 99,
  "duplicateUrlCount": 1,
  "duplicateSitemapCount": 0,
  "issueCount": 0,
  "issueRowsEmitted": 0,
  "networkRequests": 1,
  "demo": false,
  "urlLimitReached": false,
  "sitemapLimitReached": false,
  "depthLimitReached": false,
  "runtimeLimitReached": false,
  "startedAt": "2026-08-10T12:00:00.000Z",
  "finishedAt": "2026-08-10T12:00:01.000Z"
}

Logical attempts and physical networkRequests are deliberately separate. A same-origin redirect adds a physical request; retries add attempts and physical requests. Receipt counts, including complete issueCount, remain available when maxIssueRows caps detailed issue rows.

issues dataset: bounded diagnostics

Issue rows have fixed codes and sanitized messages. They never include upstream response bodies, resolved IP lists, exception strings, credentials, or headers.

{
  "schemaVersion": "1.1",
  "issueCode": "CHILD_SITEMAP_CROSS_ORIGIN",
  "stage": "sitemap_index",
  "rootUrl": "https://www.example.com/sitemap.xml",
  "sitemapUrl": "https://www.example.com/sitemap.xml",
  "depth": 0,
  "retryable": false,
  "occurrenceCount": 1,
  "message": "CROSS-ORIGIN CHILD SITEMAP WAS NOT FETCHED",
  "observedAt": "2026-08-10T12:00:00.000Z"
}

Repeated identical issues are aggregated with occurrenceCount. At most maxIssueRows distinct detail records are retained during processing; complete occurrence counts and stop-reason code sets are maintained in bounded per-root and per-sitemap counters even after that detail cap. Common codes distinguish DNS policy failures, DNS rebinding, TLS certificate failures, blocked redirects, terminal rate limiting, network timeouts, document byte limits, the fixed retained-output limit, invalid gzip, prohibited DTD/entity declarations, XML errors, metadata errors, traversal limits, and the run deadline.

OUTPUT summary

OUTPUT links the three datasets and provides run-wide reconciliation:

{
  "schemaVersion": "1.1",
  "demo": false,
  "coverage": "exact",
  "stopReason": "completed",
  "defaultDatasetId": "default-dataset-id",
  "receiptsDatasetId": "receipts-dataset-id",
  "receiptsDatasetAlias": "receipts",
  "issuesDatasetId": "issues-dataset-id",
  "issuesDatasetAlias": "issues",
  "deliveryPlanReused": false,
  "deliveryPlanFingerprint": "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
  "urlsAvailable": 100,
  "urlsDiscoveredThisAttempt": 100,
  "paidEventName": "sitemap-url",
  "pricingModel": "PAY_PER_EVENT",
  "payPerEvent": true,
  "paidEventConfigured": true,
  "paidUnitsExisting": 0,
  "paidUnitsChargedBefore": 0,
  "paidUnitsChargedThisAttempt": 100,
  "paidUnitsChargedRun": 100,
  "paidUnitsRepairedThisAttempt": 0,
  "paidLimitReached": false,
  "paidUnitsOmitted": 0,
  "rootsRequested": 1,
  "sitemapsDiscovered": 3,
  "sitemapsFetched": 3,
  "sitemapsParsed": 3,
  "uniqueUrls": 100,
  "duplicateUrls": 2,
  "duplicateSitemaps": 0,
  "issueCount": 0,
  "issueRowsEmitted": 0,
  "networkRequests": 3,
  "urlLimitReached": false,
  "sitemapLimitReached": false,
  "depthLimitReached": false,
  "runtimeLimitReached": false,
  "startedAt": "2026-08-10T12:00:00.000Z",
  "finishedAt": "2026-08-10T12:00:01.000Z"
}

Use defaultDatasetId, receiptsDatasetId, and issuesDatasetId for API retrieval. receiptsDatasetAlias and issuesDatasetAlias identify the storage aliases configured in the Actor definition; they are not persistent Dataset names.

Restart-safe delivery and pay-per-event accounting

Before publishing any dataset row, the Actor commits an immutable, fingerprinted delivery plan to its default key-value store. That KVS plan does not contain page URLs: it contains one whole-extraction semantic digest, sitemap capture timestamps, the summary, and non-billable receipt/issue chunks. A retry re-fetches the sources and must match that digest before it can reuse the staged metadata or repair delivery. Changed source data therefore fails closed instead of introducing duplicate or stale rows. Each dataset must remain an exact prefix of the verified plan; an unexpected row also fails closed. deliveryPlanReused and deliveryPlanFingerprint make that behavior observable. urlsDiscoveredThisAttempt reports the fresh extraction count on every attempt, while urlsAvailable is the verified plan count.

Under the configured Apify pay-per-event pricing, the only custom billable event is sitemap-url. The Actor checks the remaining spending capacity, saves and reads back only that exact affordable URL prefix, and then charges the exact stored gap through the run API. The charge uses a deterministic SHA-256 idempotency key bound to the run, event, delivery-plan fingerprint, prior charged prefix, and target stored prefix. It does not infer billing from Actor.pushData()'s aggregate result. The synthetic apify-default-dataset-item price must remain disabled, and the sitemap-url price must be positive.

A failed or interrupted charge can therefore leave an exact stored-but-uncharged prefix in the default dataset. The run fails closed and cannot report success in that state. On restart, the Actor verifies the same immutable plan and dataset prefix, charges only the stored gap with the same deterministic transition key, and never republishes those rows. It also retains bounded compatibility with an older charged-before-store run by publishing only an already-charged missing suffix without charging it again. Before success, a fresh live run read-back must prove sitemap-url charged count equals exact stored URL count; unexpected run identity, default-dataset identity, count, prefix, or capacity drift fails closed. Helper receipts and issues are not billable and use exact-prefix restart reconciliation.

paidUnitsChargedThisAttempt reports every newly confirmed sitemap-url charge in this attempt, including a stored-prefix gap repaired after restart. paidUnitsRepairedThisAttempt counts units reconciled in either direction: a stored-but-uncharged prefix charged without republishing, or a legacy charged-but-missing suffix stored without charging again. paidUnitsChargedRun must equal the URL rows stored after reconciliation. paidLimitReached and paidUnitsOmitted explicitly report spending-limit truncation; in that case uniqueUrls is the delivered count and urlsAvailable is the full staged count. Empty-input demo and non-PPE runs publish their deterministic plan without custom-event charges.

Coverage contract

Coverage describes the requested, bounded extraction contract—not all URLs that exist on a website and not search-engine index coverage.

• exact: every sitemap document that came into scope through the configured root and discovery rules completed within the declared limits, with no coverage-affecting issue.
• partial: at least one sitemap parsed, but a source failure, rejected reference, invalid entry, or configured limit prevented complete in-scope processing.
• unavailable: no sitemap document parsed for that receipt scope.

For a site origin, the discovery contract is: read /robots.txt; use valid same-origin Sitemap: directives; if there are no directives, or robots returns 404/410, try the exact same-origin /sitemap.xml fallback. A successful exhausted fallback can be exact for this contract. Other robots failures force partial even when the fallback succeeds, because discovery evidence was unavailable.

Read coverage, stopReason, all four *LimitReached fields, and the issue counts together. urlLimitReached covers either the requested maxUrls count or the fixed 32 MiB retained URL-output budget; OUTPUT_MEMORY_LIMIT distinguishes the latter. Do not infer completeness from the number of URL rows alone. A successful run can validly contain zero URL rows, for example when an exact sitemap index ultimately contains empty urlsets.

Security

The Actor treats every supplied hostname and source byte as untrusted.

• It resolves each hostname twice and rejects private, loopback, link-local, documentation, carrier-grade NAT, NAT64, mapped, reserved, multicast, mixed-public/private, and changing DNS answer sets.
• It pins the chosen public address to a fresh HTTPS connection, checks the connected peer, and keeps normal hostname/SNI certificate verification.
• It permits only same-origin redirects and same-origin child sitemap traversal.
• It does not use cookies, sessions, input headers, credentials, proxies, browser automation, or full-account permissions.
• It caps compressed bytes before decoding, rejects concatenated/trailing/stacked gzip, caps decoded bytes and expansion ratio, rejects DTD/entity declarations, and bounds XML depth, text, events, loc length, metadata length, entries, concurrency, retries, and time.

See ./SECURITY.md for the detailed threat model and reporting process.

Privacy and data handling

Inputs and results are written only through Apify's normal run input, datasets, logs, and default key-value store. The default key-value store contains the whole-plan digest, sitemap capture timestamps, summary, and chunked receipts/issues needed for safe restart reconciliation; it never contains uncharged page URLs or raw source bodies. The Actor does not send page URLs to another analytics service, and it never requests those page URLs. Source responses are processed in memory and are not stored as raw bodies. Dataset and run retention are controlled by your Apify account settings.

Avoid placing secrets in sitemap URLs. Auth-like query keys and URL credentials are rejected, but public sitemap URLs may still be visible in run input, receipts, and output rows.

Troubleshooting

INVALID_INPUT before the run starts

Check that roots use public HTTPS, port 443, no credentials or fragments, and no auth-like query parameters. siteUrls must be bare origins. Remove a URL duplicated across sitemapUrls and siteUrls, or reduce the cross-field memory budget.

DNS_PRIVATE_OR_MIXED

The hostname resolved to a prohibited or mixed address set. This policy is fail-closed; use a public hostname whose DNS answers are all public unicast addresses.

DNS_REBINDING

The two DNS snapshots differed, or the connected peer did not match the pinned address. Stabilize DNS and rerun.

TLS_CERTIFICATE

The certificate chain or hostname check failed. Correct the public endpoint certificate; disabling TLS verification is not supported.

RATE_LIMITED or retryable network issues

Reduce concurrency, allow bounded retries, or run later. HTTP Retry-After is honored up to five seconds; the run deadline always wins.

GZIP_INVALID, DECOMPRESSED_LIMIT, or XML errors

Serve one valid gzip member containing a sitemap XML document. Split oversized sitemaps at the source instead of raising limits indiscriminately.

Coverage is partial

Open the run receipt, then root and sitemap receipts, and inspect stable issue codes. Limit flags identify whether increasing an explicit bound may help.

Non-features

This Actor intentionally does not:

• crawl or render pages;
• scrape page content, emails, social profiles, or contact data;
• test page status codes or canonical tags;
• bypass authentication, robots controls, rate limits, or private-network boundaries;
• accept HTTP sitemap sources, arbitrary ports, custom headers, cookies, proxy settings, or browser sessions;
• execute XSLT, expand XML entities, or fetch external XML resources;
• claim that a sitemap is complete, current, indexed, or authoritative beyond the coverage receipts.

Those omissions keep its authority narrow and make it suitable for Apify AI and MCP-style on-demand execution. The Actor definition explicitly requests LIMITED_PERMISSIONS; it does not require full account access.

Support, security reports, and removal requests

Use this Actor's monitored Issues tab in Apify Store/Console and include the Apify run ID, input shape with secrets removed, expected behavior, and observed issue code. The publisher monitors that channel for support, vulnerability reports, and removal requests involving a domain you control. Do not post credentials, raw authenticated URLs, or private data.

For a suspected security issue, title the report SECURITY and provide a minimal reproduction without targeting third parties. For a removal request, title it DATA REMOVAL, identify the affected public URL or domain, and provide a way to verify control. Retention and deletion of datasets already stored in your own account remain under your Apify account controls.

Artwork status

The canonical current artwork is assets/actor-art.png, rendered from the original brand-neutral assets/actor-icon.svg; its machine-readable vector receipt is assets/actor-icon.provenance.json. The prior ImageGen brief remains at assets/actor-art.prompt.md as immutable historical provenance for the superseded raster. The current 1254×1254, 8-bit RGB PNG and native 76×76 preview were inspected at small size. The current master SHA-256 is 475196170aeb4737c8a062d8d7454ba44f984171b1769650a3f434788a7a9771.

The 512×512 upload derived from this master is attached to Actor 0K4CSTXd3VEUPdIHm; its saved pictureUrl, fetched bytes, and SHA-256 read-back were verified on 2026-08-12. The rendered small Store crop is a high-contrast, brand-neutral sitemap glyph.