Sitemap URL Status Checker

Discover the XML sitemaps for one or more websites and check each page URL. Every saved row is one URL from a sitemap, with its lastmod when the sitemap published one, and, when status checks are on, the HTTP status, redirect chain, final URL, and response time. 4xx, 5xx, timeouts, and connection failures are marked broken.

A homepage is expanded from Sitemap: lines in robots.txt. When robots.txt does not list a sitemap, the Actor tries /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /sitemap.xml.gz, and /wp-sitemap.xml. A direct sitemap URL is parsed as-is. Sitemap indexes are followed, including .xml.gz files, and a single bad file does not fail the run.

This Actor is unofficial. It is not affiliated with sitemaps.org, Google, or any site you submit.

What you put in

Field	Default	What it does
Website or sitemap URLs	https://www.sitemaps.org/	Homepages, sitemap URLs, or a requestsFromUrl text list.
Maximum URLs	50	Stop after this many unique page URLs. Further child sitemaps are not fetched.
Check HTTP status	on	HEAD each URL, then GET if HEAD is rejected (400, 403, 405, 406, 501).
Respect robots.txt	on	Skip the status check when robots.txt disallows the URL. The URL is still listed.
Max concurrent requests per host	4	In-flight cap for sitemap downloads and status checks.
Minimum delay per host	200 ms	Gap between request starts to the same host.
Request timeout	15 s	Timeouts are broken links and are not retried.
Maximum redirects	5	Hop list and final URL are stored. Hitting the cap is broken.
Include / exclude URL globs	empty	Optional fnmatch patterns, for example https://example.com/blog/*.

The prefilled input is the Sitemaps protocol site. Its robots.txt points at a small sitemap, and maxUrls of 50 keeps the run short.

What you get back

One dataset row per unique page URL. A run summary is also stored on the key-value record OUTPUT.

Real row from the default input (status checks on, 23 October 2026):

{
  "url": "https://www.sitemaps.org/protocol.html",
  "host": "www.sitemaps.org",
  "sourceSitemap": "https://www.sitemaps.org/sitemap.xml",
  "lastmod": "2022-12-15",
  "changefreq": null,
  "priority": null,
  "httpStatus": 200,
  "status": "ok",
  "isBroken": false,
  "finalUrl": "https://www.sitemaps.org/protocol.html",
  "redirectCount": 0,
  "redirectChain": [],
  "responseTimeMs": 492,
  "contentType": "text/html",
  "method": "HEAD",
  "error": null,
  "checkedAt": "2026-10-23T03:17:14.370480+00:00"
}

status is ok, redirect (the final response is still 3xx), client_error, server_error, timeout, error, skipped (disallowed by robots.txt), or unchecked (status checks were off, or the charge limit stopped before the check). isBroken is true for 4xx, 5xx, timeouts, connection failures, and redirect limits.

Image, video, and news extension URLs inside a <url> block are not separate rows. The row is the page <loc>.

Pricing

Pay per event. Configure these two events in Apify Console and do not add a charge on the default dataset item. The Actor already charges for each URL.

Event	When it is charged	Suggested price
apify-actor-start	Once per run, by the platform (memory is capped at 1 GB, so once). The Actor does not call Actor.charge for it.	$0.005
url-found	Once for each unique page URL saved from a sitemap	$0.0005
url-checked	Once for each URL whose HTTP status is requested	$0.001

Suggested primary event: url-checked.

With status checks on, a URL costs about $0.0015 (url-found plus url-checked). With status checks off, only url-found is charged. URLs skipped because of robots.txt are not charged as url-checked. The run stops when a charge reports that the max charge limit has been reached, and it does not save a row it could not pay for.

The default run finds 50 URLs and checks all 50, so it costs about $0.08 ($0.005 start + $0.075 in URL events). Platform usage is not billed to you.

Limits

• Page bodies are not downloaded or stored. The Actor records the status line, headers needed for the redirect chain, and Content-Type.
• It does not render JavaScript or crawl ordinary HTML links. If a site has no XML sitemap, the run fails with the paths it tried.
• At most 100 sitemap files, nested at most 6 levels, each at most 25 MB decompressed.
• At most 50 start URLs, and at most 100,000 page URLs.
• Hosts that resolve only to loopback, private, link-local, or other non-public addresses are refused.
• Crawl-delay in robots.txt is applied per host and capped at 2 seconds so one site cannot stall the run. Your own delay setting is still honored up to 10 seconds.
• Transient 429, 502, 503, and 504 responses are retried with backoff. Timeouts are not retried.
• A 403 from HEAD is requested once more with GET, because many servers reject HEAD. A real 403 from both methods is stored as broken.
• The User-Agent is SitemapUrlStatusChecker/1.0 (+https://apify.com; sitemap and broken-link checker).

Permissions

Publish and run this Actor with limited permissions. It only reads its input and writes the default dataset and key-value store.

Run locally

Python 3.12.

pip install -r requirements.txt
python -m sitemap_checker

With no input file, the Actor uses the same defaults as the input schema (the sitemaps.org homepage, 50 URLs, status checks on). On Apify the Console input is used. Locally, if the default key-value store has no INPUT record, the Actor reads INPUT.json from the working directory:

{
  "startUrls": [{ "url": "https://www.sitemaps.org/" }],
  "maxUrls": 50,
  "checkStatus": true
}

Results are written under storage/datasets/default/. The run summary is storage/key_value_stores/default/OUTPUT.

pip install -r requirements-dev.txt
pytest

Tests mock HTTP. They do not call the network.

Source and terms

Sitemap files follow the Sitemaps XML protocol published at https://www.sitemaps.org/protocol.html. The URL lists and pages belong to the sites that publish them. This Actor stores URLs and HTTP response metadata, not page content.

Only submit sites you are allowed to request. The Actor identifies itself, waits between requests to a host, and honors Disallow rules during status checks. You are responsible for complying with each site's terms and with robots.txt.

Deploy

The Actor is meant to be pushed with the Apify CLI (apify push) from this directory. The Dockerfile uses apify/actor-python:3.12. Do not enable a default dataset-item charge when you set up pay-per-event pricing.