Sitemap Lastmod Checker

Takes a website URL. Returns a per-sitemap report telling you, for every XML sitemap published by that site, whether it uses <lastmod> tags and what HTTP status it served.

Useful for:

• SEO audits: Google uses <lastmod> as a hint for recrawl priority, so a sitemap without it is leaving a signal on the table.
• RAG pipeline optimization: if the site map supports lastmod you can use the Website Content Crawler's onlyCrawlPagesModifiedSince parameter to crawl only new content

How it works

1. Fetches /robots.txt on the input domain and extracts every Sitemap: directive.
2. Falls back to /sitemap.xml if robots.txt doesn't declare any (or returns 4xx).
3. For each sitemap URL, inspects the XML:
   • If it's a <sitemapindex>, enqueues each child sitemap (one level deep). The index itself is not written to the dataset — only leaf sitemaps are.
   • If it's a <urlset>, checks whether any <url> entry carries a non-empty <lastmod>.
4. Waits for every sitemap to be checked, then writes one dataset row per leaf sitemap in a single batch.

All requests go through a Cheerio-based HTTP crawler — no browser, no rendering. A typical check finishes in a few seconds and costs well below $0.01 in compute units.

Input

Field	Type	Required	Description
websiteUrl	string	yes	Homepage URL to check, e.g. https://www.clay.com/.
maxSitemaps	integer	no	Cap on sitemap files to fetch (default 20, max 1000). Prevents runaway cost on sites with thousands of sub-sitemaps.
proxyConfiguration	object	no	Apify Proxy config. Off by default — rarely needed for sitemap fetching.

Example:

{
    "websiteUrl": "https://www.clay.com/",
    "maxSitemaps": 20
}

Dataset output

One row per leaf sitemap. Three fields:

Field	Type	Description
sitemapUrl	string	The individual sitemap file that was inspected.
httpStatusCode	integer | null	HTTP response status. null if the request failed before a response arrived (DNS, TCP, timeout).
supportsLastmod	boolean	true if at least one <url> in that sitemap carried a non-empty <lastmod>. false otherwise (including for 4xx/5xx/network failures).

Example dataset for clay.com:

[
    {
        "sitemapUrl": "https://www.clay.com/guides/sitemap.xml",
        "httpStatusCode": 200,
        "supportsLastmod": true
    },
    {
        "sitemapUrl": "https://www.clay.com/sitemap.xml",
        "httpStatusCode": 200,
        "supportsLastmod": true
    }
]

A richer summary is also saved to the default key-value store as OUTPUT: total sitemaps reported, how many support lastmod, which sitemap-index files were traversed, and the full list of per-sitemap reports.

Caveats

• Only follows sitemap-index files one level deep. Nested indices (rare) are inspected as plain sitemaps.
• Gzipped sitemaps (.xml.gz) are not decompressed. Most modern CMSs publish an uncompressed version too; if a site publishes only gzipped sitemaps, the entries won't be parsed and the row will show supportsLastmod: false.
• A sitemap containing <url> entries where only some carry <lastmod> is still reported as supportsLastmod: true. The actor treats the question as "does this sitemap use lastmod at all" — not "does every URL have one".
• httpStatusCode is the status of the sitemap fetch. A 404 means the sitemap URL was listed somewhere (robots.txt, a parent index, or the /sitemap.xml fallback) but doesn't exist.

Cost

Pay-per-usage. A single run is one HTTP request to robots.txt plus one per sitemap file, so a typical site costs well under $0.01 in CU. Sites with hundreds of sub-sitemaps hit the maxSitemaps cap, which keeps worst-case cost bounded.