Sitemap — Multi-Site URL Inventory

Price: $0.00005 per delivered unique URL. No start or secondary fee.

Expand public sitemap indexes across sites into a deduplicated URL inventory for SEO and data-collection planning. Export publisher modification hints and source sitemap provenance; repeat snapshots through Apify schedules. Independent tool, not affiliated with sites.

Input

sitemapUrls accepts1–5 public HTTP(S) roots/XML/index/XML.gz URLs, default the full OneScales sitemap index. Root inputs discover robots-declared sitemaps or an allowed /sitemap.xml fallback. maxUrls defaults1,000 (1–10,000), maxSitemaps50 (1–50), maxDepth8 (1–8). Optional includeContains matches any case-sensitive substring and excludeContains excludes any; empty arrays inventory all safe URLs within bounds. Large sites can exceed bounds and are explicitly partial. maxSitemaps must accommodate all selected roots, including sitemap roots discovered from robots; a smaller document allowance fails before paid output.

Output

One deduplicated safe published URL row, with page origin, all encountered source sitemap URLs, publisher lastmod/changefreq/priority, duplicate count and coverage flags. These are source hints, not verified page status or a claim that a page changed. No page bodies or status checks are fetched. COVERAGE is an uncharged KVS report of selected roots, traversed/pending documents, unsafe omissions and cap reasons. coverageComplete=true requires every selected traversal to finish without caps or omissions. Breadth-first source ordering prevents one root silently replacing the entire multi-site input. All selected roots validate before any paid output; failures are not paid error rows.

Limits

256MB/300seconds,75HTTPrequests including robots/redirects,50sitemaps/depth8,10,000unique URLs,8MiB wire/16MiB decoded per XML document,64MiB transfer and24MiB staged output. Stream XML/gzip with byte ceilings, clear parsed nodes, reject DTD/entities/decompression bombs/unsupported archives and detect cycles. Sequential minimum1s per origin. At most2 manually checked redirects; each referenced sitemap/redirect host has public IPv4/IPv6 DNS addresses checked/pinned and its robots checked before fetching. Private/reserved/metadata destinations, credentials/secret URLs and nonstandard ports are rejected. Published page URLs are output only. Any required source403/429/timeout/malformed XML/oversize/robots denial fails before paid output; overall empty/no-match is an uncharged error. One$0.00005 url-exported event per unique delivered URL, no start/status/default-dataset/secondary fee.1,000URLs cost$0.05.

Privacy

Only public safe URL strings and sitemap hints. No page bodies, images, contacts/person/customer/account records or secrets. Contact-bearing, token-bearing and private URLs are excluded with explicit uncharged omission counts; no inferred personal data. Preserve source provenance. Website terms apply; robots restrictions and access controls are honored without browser, paid proxy or security bypass.