Sitemap URL Extractor

Every URL a website's sitemaps list. Give it a website or a sitemap URL; it returns one row per URL, with lastmod, changefreq, priority and the sitemap that listed it. Optionally, it checks each URL's HTTP status.

It finds sitemaps the way search engines do: the Sitemap: lines in robots.txt first, then /sitemap.xml, /sitemap_index.xml and /sitemap-index.xml. It follows sitemap indexes to any depth, reads .xml, .xml.gz and plain-text sitemaps, and streams them, so a 50,000-URL file needs little memory. No browser.

What you get

One row per URL:

Field	Example
url	https://angular.dev/ai
lastmod	2026-10-02T19:38:17.519Z, as the sitemap gives it
changefreq	daily
priority	1.0, a number from 0 to 1, or null
sitemapUrl	https://angular.dev/sitemap.xml, the file that lists it
site	https://angular.dev, the start URL's site
status, redirectTo	301, https://.../about/, when the status check is on
statusError	why a status check got no answer, else null
imageUrls, videoUrls	from Google's image and video sitemap extensions
alternates	[{"hreflang": "et-ee", "href": "..."}], hreflang links
newsTitle, newsPublicationDate	from Google's news sitemap extension

Input

{
  "startUrls": ["https://apify.com", "https://example.com/sitemap.xml"],
  "maxUrls": 1000,
  "urlFilterInclude": "/blog/",
  "urlFilterExclude": "/tag/",
  "checkPageStatus": false
}

• startUrls: websites, sitemap URLs, sitemap indexes, .xml.gz or .txt sitemaps, or a robots.txt URL. A bare example.com works. A sitemap URL that does not answer falls back to finding the site's sitemaps.
• maxUrls: stop after this many URLs across all start URLs. 0 returns every URL. This is your cost control. The form starts at 100, so a first run on a large site does not read all of it: apify.com alone lists 676,119.
• urlFilterInclude, urlFilterExclude: regular expressions, case-insensitive. URLs they drop are not billed.
• checkPageStatus: one HEAD request per URL returned, adding status and redirectTo. Off by default.
• followSitemapIndexes: on by default. Off returns the sitemap files an index lists instead of reading them.
• proxyConfiguration: off by default. Sitemaps are published for crawlers, and most sites serve them to anyone.

The input also takes startUrls as [{"url": "..."}] objects and sitemapUrls, the field names other sitemap Actors use, so an existing input runs unchanged.

Output

{
  "url": "https://www.shopify.com/about",
  "lastmod": "2026-10-02",
  "changefreq": "daily",
  "priority": 0.8,
  "sitemapUrl": "https://www.shopify.com/sitemap.xml",
  "site": "https://www.shopify.com",
  "status": 301,
  "redirectTo": "https://www.shopify.com/about/",
  "statusError": null,
  "imageUrls": [],
  "videoUrls": [],
  "alternates": [],
  "newsTitle": null,
  "newsPublicationDate": null
}

Price

$1.00 per 1,000 URLs ($0.001 each), charged per URL returned. Nothing else is charged: no per-run fee, no charge for sitemaps that fail, and none for duplicates or filtered-out URLs. A URL listed in two sitemaps is returned once. The status check costs nothing extra.

Speed and size

Measured 2026-10-03, outside Apify: all of apify.com's sitemaps, 17 files and 676,119 URLs, read in 15 seconds at a peak of 158 MB. The default 256 MB is enough for a site of that size; for a site with several million URLs, give the run 1 GB.

Limits, honestly

• Sitemaps only. It does not crawl pages or follow links, so a page missing from a site's sitemaps is missing here too.
• robots.txt is respected for the sitemap files themselves. A sitemap that robots.txt disallows is skipped and counted. If robots.txt answers with a server error, that site's sitemaps are not read, as RFC 9309 requires.
• A site with no sitemap returns no rows. The run succeeds and its status message says which places were tried.
• A sitemap cut short (a broken connection, or invalid XML part way through) keeps the URLs read up to that point, and the status message says how many files ended early.
• One host at a time, with a pause between sitemap files, and back-off when a site answers 429 or 503.
• Up to 20,000 sitemap files and six index levels per run.

What it will not do

It reads robots.txt and sitemap files, plus one HEAD per URL when asked. It does not read the pages themselves, sign in, solve CAPTCHAs, or collect personal data.

Support

Open an issue on this Actor's Issues tab. Reported issues are fixed within 14 days.