Sitemap URL Extractor

Give it any website or sitemap URL and get back every page URL. Point it at a site root (e.g. https://example.com) or a direct sitemap.xml, and it reads robots.txt, follows nested sitemap indexes, and returns one tidy JSON row per URL. There is nothing to configure — no API key, no login, and no proxy.

Features

• Accepts either a website root or a direct sitemap URL.
• Reads robots.txt for Sitemap: entries, falling back to /sitemap.xml.
• Follows nested <sitemapindex> files recursively, with a safety cap.
• Emits each URL's last-modified date, change frequency and priority.
• De-duplicates by absolute URL and respects your max URLs limit.
• Skips gracefully over a sitemap that can't be fetched or parsed — it is recorded with an error field so one bad sitemap never stops the run.

Input

Field	Type	Default	Description
startUrls	array	—	A site root (e.g. https://example.com) or a direct sitemap URL (e.g. https://example.com/sitemap.xml).
maxUrls	integer	1000	Stop after emitting this many page URLs across all sitemaps.
maxSitemaps	integer	50	Safety cap on how many sitemap files to fetch (including nested indexes).

Example input

{
  "startUrls": ["https://www.apify.com"],
  "maxUrls": 1000,
  "maxSitemaps": 50
}

You can also point it straight at a sitemap:

{
  "startUrls": ["https://www.sitemaps.org/sitemap.xml"]
}

Output

Each dataset record looks like this:

{
  "url": "https://www.example.com/blog/hello-world",
  "lastmod": "2024-01-31",
  "changefreq": "weekly",
  "priority": 0.8,
  "sourceSitemap": "https://www.example.com/sitemap.xml"
}

Field	Description
url	The page URL (<loc>), resolved to an absolute URL.
lastmod	Last-modified value as published in the sitemap (null if absent).
changefreq	Change frequency hint, e.g. daily, weekly (null if absent).
priority	Crawl priority as a number between 0 and 1 (null if absent).
sourceSitemap	The sitemap file this URL was found in.

If a sitemap can't be fetched or parsed, the row is { "sourceSitemap": "<url>", "error": "<message>" } instead, and it is never charged.

Pricing

This actor uses the pay-per-event model: you are charged once per URL extracted (the url-scraped event). A sitemap that fails to fetch or parse produces an error row and is never charged.

How it works

1. For each start URL: if it looks like a sitemap (path ends in .xml or contains sitemap) it is used directly; otherwise the actor reads <origin>/robots.txt for Sitemap: lines, falling back to <origin>/sitemap.xml.
2. Each sitemap is fetched and parsed. A <sitemapindex> is followed into its child sitemaps (up to maxSitemaps); a <urlset> emits one record per <url>.
3. URLs are de-duplicated by absolute URL, and the run stops once maxUrls is reached.

No authentication is required — robots.txt and sitemaps are public.

License

MIT © Assia Fadli