Sitemap Extractor

This Actor is designed to bridge the gap between discovery and crawling. By traversing a website's sitemap.xml structure, it compiles a comprehensive list of all published pages and verifies their status before you commit resources to a full-scale scrape.

Features

• Recursive Sitemap Discovery: Automatically detects and traverses nested sitemaps (sitemap indexes).
• Efficiency: Uses HTTP HEAD requests for URL validation, which are significantly faster and consume less bandwidth than full GET requests.
• Proxy Support: Integrated with Apify Proxy to prevent rate limiting or blocking during the discovery phase.
• Detailed Output: Provides the final URL, the corresponding HTTP status code, and the date-time of the page's last modification.

How it Works

1. Input: You provide one or more "Start URLs" pointing to the domain name root, sitemaps or sitemap indexes.
2. Extraction: The Actor parses the XML, extracting both page URLs and links to further sitemaps.
3. Validation: For every page URL found, the Actor performs a status check (unless Check page status is disabled, see below).
4. Deduplication: The crawler uses unique keys to ensure that even if a URL appears in multiple sitemaps, it is only checked once.

Listing sitemap URLs without status checks

By default (Check page status enabled), the Actor sends an HTTP HEAD request to every page found in the sitemaps and reports its status code. If you only need the list of URLs contained in the sitemaps and don't care about status codes, disable the Check page status option. In this mode the Actor:

• does not send any request to the individual pages. It only downloads and parses the sitemaps themselves;
• outputs each discovered URL with the lastmod value from the sitemap's <lastmod> tag, whenever the sitemap provides one. Since no pages are requested, the Last-Modified header fallback is not used, so URLs whose sitemap entry has no <lastmod> will have lastmod: null;
• omits the status field from the output.

This is dramatically faster and lighter on large sites, since it replaces one request per page with just a handful of requests for the sitemap files.

Output

For each page URL, the Actor outputs:

Field	Description
url	The page URL from the sitemap.
status	The HTTP status code returned by the HEAD request. Omitted when Check page status is disabled.
lastmod	Best-effort last-modification time (ISO 8601). See the note below.

A note on last-modification data

The lastmod field is a single best-effort timestamp derived from two sources, in this order of preference:

1. The <lastmod> tag declared for the URL in the sitemap.
2. The Last-Modified HTTP header returned by the page (used only when the sitemap has no <lastmod>).

We cannot guarantee that this information is available. Both sources are optional: many sitemaps omit <lastmod> entirely, and a lot of servers don't send a Last-Modified header (this is especially common for dynamically generated pages). When neither source provides a value, lastmod is null. Even when present, the value is self-reported by the site and may not reflect the true last-modification time of the content.

When Check page status is disabled, no page requests are made, so only the first source (the sitemap's <lastmod> tag) is used.

Usage

This Actor is ideal for:

• Pre-crawling filter: Generating a "clean" list of URLs for actors like Website Content Crawler or Web Scraper.
• SEO Audits: Quickly identifying 404 Not Found or 500 Server Error pages listed in your sitemap.
• Site Mapping: Getting a high-level overview of a site's architecture.

Configuration

Field	Description
Start URLs	Just a domain name or a list of sitemap XML URLs to start from.
Proxy configuration	Settings for Apify Proxies.