Sitemap URL Extractor — every page a site declares

List every URL a site publishes, taken from robots.txt and its sitemaps, with the last-modified date each page declares.

Input:   ["https://apify.com"]
Output:  one row per URL — address, last modified, change frequency, which sitemap declared it

Why read a sitemap rather than crawl

robots.txt and sitemap.xml exist so that a machine can ask a site what it publishes. Reading them is the most explicitly invited request on the web — and it is also faster, complete, and carries information a crawler cannot infer:

• lastmod — when the page last changed, so you can re-fetch only what moved.
• Completeness — pages no link points at are still declared.
• Speed — one request per sitemap instead of one per page.

This Actor never crawls, follows links, or guesses at addresses. It reports the declaration and nothing else, and it skips anything robots.txt disallows even though only the declaration is being read.

What it handles

• Sitemap indexes. Large sites split their sitemap into an index of further sitemaps; those are expanded automatically.
• Sitemaps declared in robots.txt, which is where they are supposed to be announced — and the usual locations when a site announces nothing.
• Language alternates. hreflang versions of the same page, on request.
• Pages that are not a sitemap. A site that answers 200 with an error page produces no rows rather than nonsense.

Input

Field	Type	Default	What it does
sites	array of strings	one example	Site URLs, or sitemap URLs directly
pathContains	string	—	Keep only URLs containing any of these words, e.g. /blog, /docs
maxUrls	integer	0 (no limit)	Cap URLs per site
followIndexes	boolean	true	Expand sitemap indexes
includeAlternates	boolean	false	Include hreflang alternates

Examples

Every documentation page on a site:

{ "sites": ["https://docs.apify.com"], "pathContains": "/docs" }

Several sites at once, capped:

{ "sites": ["https://blog.rust-lang.org", "https://github.blog"], "maxUrls": 500 }

Output

Field	Type	Description
site	string	Origin the URL belongs to
url	string	The declared URL
sitemap	string	Which sitemap declared it — useful on sites with many
lastModified	string | null	As declared
changeFrequency	string	As declared
priority	string	As declared
alternates	array	hreflang versions, when requested
fetchedAt	string	When the row was produced

The SUMMARY record adds a breakdown per site: how many sitemaps were read, how many were indexes, whether robots.txt declared them, and a count of URLs by first path segment — the shape of the site at a glance.

Pricing

	Price
Starting a run	$0.00001 — one US cent per 1,000 runs
Each URL returned	$0.0002 — 20 US cents per 1,000 URLs

A site with no sitemap produces no rows, and errors go to the log and SUMMARY rather than the dataset — so a site that cannot be read costs nothing.

Works well with

Pair it with a text extractor: this decides which pages exist and which ones changed, that one reads them. Filter by lastModified and you re-ingest only what moved since your last run.

Notes on data and compliance

• Only robots.txt and the sitemaps it points to are fetched.
• Paths disallowed by robots.txt are skipped, including sitemaps themselves.
• No page content is downloaded, so no personal data is touched.
• The Actor stops after 200 sitemaps per site rather than following an index loop indefinitely.

Limitations — stated up front

• A site with no sitemap returns nothing. That is the honest answer; this will not fall back to crawling.
• Gzipped sitemaps (.xml.gz) are requested but not decompressed yet.
• lastmod is whatever the site declares, which is not always accurate — some CMSs stamp every page with the deploy time.
• Very large sites are capped at 200 sitemaps per run; use pathContains to narrow instead.