# Sitemap URL Status Checker (`plusonets/sitemap-url-status-checker`) Actor Discover XML sitemaps and check each URL for HTTP status, redirects, response time, and broken links. - **URL**: https://apify.com/plusonets/sitemap-url-status-checker.md - **Developed by:** [Olalekan Johnson](https://apify.com/plusonets) (community) - **Categories:** SEO tools, Developer tools - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $1.00 / 1,000 url checkeds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## Sitemap URL Status Checker Discover the XML sitemaps for one or more websites and check each page URL. Every saved row is one URL from a sitemap, with its `lastmod` when the sitemap published one, and, when status checks are on, the HTTP status, redirect chain, final URL, and response time. `4xx`, `5xx`, timeouts, and connection failures are marked broken. A homepage is expanded from `Sitemap:` lines in `robots.txt`. When `robots.txt` does not list a sitemap, the Actor tries `/sitemap.xml`, `/sitemap_index.xml`, `/sitemap-index.xml`, `/sitemap.xml.gz`, and `/wp-sitemap.xml`. A direct sitemap URL is parsed as-is. Sitemap indexes are followed, including `.xml.gz` files, and a single bad file does not fail the run. This Actor is unofficial. It is not affiliated with [sitemaps.org](https://www.sitemaps.org/), Google, or any site you submit. ### What you put in | Field | Default | What it does | | --- | --- | --- | | **Website or sitemap URLs** | `https://www.sitemaps.org/` | Homepages, sitemap URLs, or a `requestsFromUrl` text list. | | **Maximum URLs** | `50` | Stop after this many unique page URLs. Further child sitemaps are not fetched. | | **Check HTTP status** | on | `HEAD` each URL, then `GET` if `HEAD` is rejected (`400`, `403`, `405`, `406`, `501`). | | **Respect robots.txt** | on | Skip the status check when `robots.txt` disallows the URL. The URL is still listed. | | **Max concurrent requests per host** | `4` | In-flight cap for sitemap downloads and status checks. | | **Minimum delay per host** | `200` ms | Gap between request starts to the same host. | | **Request timeout** | `15` s | Timeouts are broken links and are not retried. | | **Maximum redirects** | `5` | Hop list and final URL are stored. Hitting the cap is broken. | | **Include / exclude URL globs** | empty | Optional `fnmatch` patterns, for example `https://example.com/blog/*`. | The prefilled input is the [Sitemaps protocol site](https://www.sitemaps.org/). Its `robots.txt` points at a small sitemap, and `maxUrls` of 50 keeps the run short. ### What you get back One dataset row per unique page URL. A run summary is also stored on the key-value record `OUTPUT`. Real row from the default input (status checks on, 23 October 2026): ```json { "url": "https://www.sitemaps.org/protocol.html", "host": "www.sitemaps.org", "sourceSitemap": "https://www.sitemaps.org/sitemap.xml", "lastmod": "2022-12-15", "changefreq": null, "priority": null, "httpStatus": 200, "status": "ok", "isBroken": false, "finalUrl": "https://www.sitemaps.org/protocol.html", "redirectCount": 0, "redirectChain": [], "responseTimeMs": 492, "contentType": "text/html", "method": "HEAD", "error": null, "checkedAt": "2026-10-23T03:17:14.370480+00:00" } ``` `status` is `ok`, `redirect` (the final response is still `3xx`), `client_error`, `server_error`, `timeout`, `error`, `skipped` (disallowed by `robots.txt`), or `unchecked` (status checks were off, or the charge limit stopped before the check). `isBroken` is true for `4xx`, `5xx`, timeouts, connection failures, and redirect limits. Image, video, and news extension URLs inside a `` block are not separate rows. The row is the page ``. ### Pricing Pay per event. Configure these two events in Apify Console and do **not** add a charge on the default dataset item. The Actor already charges for each URL. | Event | When it is charged | Suggested price | | --- | --- | --- | | `apify-actor-start` | Once per run, by the platform (memory is capped at 1 GB, so once). The Actor does not call `Actor.charge` for it. | $0.005 | | `url-found` | Once for each unique page URL saved from a sitemap | $0.0005 | | `url-checked` | Once for each URL whose HTTP status is requested | $0.001 | Suggested primary event: `url-checked`. With status checks on, a URL costs about **$0.0015** (`url-found` plus `url-checked`). With status checks off, only `url-found` is charged. URLs skipped because of `robots.txt` are not charged as `url-checked`. The run stops when a charge reports that the max charge limit has been reached, and it does not save a row it could not pay for. The default run finds 50 URLs and checks all 50, so it costs about **$0.08** ($0.005 start + $0.075 in URL events). Platform usage is not billed to you. ### Limits - Page bodies are not downloaded or stored. The Actor records the status line, headers needed for the redirect chain, and `Content-Type`. - It does not render JavaScript or crawl ordinary HTML links. If a site has no XML sitemap, the run fails with the paths it tried. - At most 100 sitemap files, nested at most 6 levels, each at most 25 MB decompressed. - At most 50 start URLs, and at most 100,000 page URLs. - Hosts that resolve only to loopback, private, link-local, or other non-public addresses are refused. - `Crawl-delay` in `robots.txt` is applied per host and capped at 2 seconds so one site cannot stall the run. Your own delay setting is still honored up to 10 seconds. - Transient `429`, `502`, `503`, and `504` responses are retried with backoff. Timeouts are not retried. - A `403` from `HEAD` is requested once more with `GET`, because many servers reject `HEAD`. A real `403` from both methods is stored as broken. - The User-Agent is `SitemapUrlStatusChecker/1.0 (+https://apify.com; sitemap and broken-link checker)`. ### Permissions Publish and run this Actor with **limited permissions**. It only reads its input and writes the default dataset and key-value store. ### Run locally Python 3.12. ```bash pip install -r requirements.txt python -m sitemap_checker ``` With no input file, the Actor uses the same defaults as the input schema (the sitemaps.org homepage, 50 URLs, status checks on). On Apify the Console input is used. Locally, if the default key-value store has no `INPUT` record, the Actor reads `INPUT.json` from the working directory: ```json { "startUrls": [{ "url": "https://www.sitemaps.org/" }], "maxUrls": 50, "checkStatus": true } ``` Results are written under `storage/datasets/default/`. The run summary is `storage/key_value_stores/default/OUTPUT`. ```bash pip install -r requirements-dev.txt pytest ``` Tests mock HTTP. They do not call the network. ### Source and terms Sitemap files follow the Sitemaps XML protocol published at . The URL lists and pages belong to the sites that publish them. This Actor stores URLs and HTTP response metadata, not page content. Only submit sites you are allowed to request. The Actor identifies itself, waits between requests to a host, and honors `Disallow` rules during status checks. You are responsible for complying with each site's terms and with `robots.txt`. ### Deploy The Actor is meant to be pushed with the Apify CLI (`apify push`) from this directory. The Dockerfile uses `apify/actor-python:3.12`. Do not enable a default dataset-item charge when you set up pay-per-event pricing. # Actor input Schema ## `startUrls` (type: `array`): Homepages or direct sitemap URLs. A homepage is expanded from robots.txt Sitemap lines and, when those are missing, from /sitemap.xml and other common sitemap paths. A .xml, .xml.gz, or other sitemap URL is parsed directly, including sitemap indexes. ## `maxUrls` (type: `integer`): Stop after this many unique page URLs. Child sitemaps are not fetched once the cap is reached. The default is small so the standard run finishes quickly. ## `checkStatus` (type: `boolean`): Request each page URL and record the status code, redirect chain, final URL, and response time. Uses HEAD, then GET when HEAD is rejected. Turn this off to only list sitemap URLs. ## `respectRobotsTxt` (type: `boolean`): When status checks are on, skip URLs that robots.txt disallows for this Actor's User-Agent. Sitemap discovery still reads Sitemap lines either way. Skipped URLs are listed and are not charged as URL checks. ## `maxConcurrency` (type: `integer`): How many requests may be in flight to a single host. Applies to sitemap downloads and status checks. ## `minDelayMillis` (type: `integer`): Minimum gap between the start of requests to the same host. A robots.txt Crawl-delay is also applied, capped at 2 seconds so one site cannot stall the run. ## `requestTimeoutSecs` (type: `integer`): Time allowed for one HTTP request, including connection. Timeouts are recorded as broken and are not retried. ## `maxRedirects` (type: `integer`): How many redirects to follow for one URL. The hop list and final URL are stored on the row. Hitting the cap is reported as broken. ## `includeUrlGlobs` (type: `array`): Optional patterns such as https://example.com/blog/\*. When this list is non-empty, a page URL must match at least one pattern. ## `excludeUrlGlobs` (type: `array`): Page URLs matching any of these patterns are dropped. Example: https://example.com/tag/\*. ## Actor input object example ```json { "startUrls": [ { "url": "https://www.sitemaps.org/" } ], "maxUrls": 50, "checkStatus": true, "respectRobotsTxt": true, "maxConcurrency": 4, "minDelayMillis": 200, "requestTimeoutSecs": 15, "maxRedirects": 5, "includeUrlGlobs": [], "excludeUrlGlobs": [] } ``` # Actor output Schema ## `results` (type: `string`): One row per sitemap URL with HTTP status, redirects, and response time. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ { "url": "https://www.sitemaps.org/" } ], "maxUrls": 50, "checkStatus": true, "respectRobotsTxt": true, "maxConcurrency": 4, "minDelayMillis": 200, "requestTimeoutSecs": 15, "maxRedirects": 5, "includeUrlGlobs": [], "excludeUrlGlobs": [] }; // Run the Actor and wait for it to finish const run = await client.actor("plusonets/sitemap-url-status-checker").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": [{ "url": "https://www.sitemaps.org/" }], "maxUrls": 50, "checkStatus": True, "respectRobotsTxt": True, "maxConcurrency": 4, "minDelayMillis": 200, "requestTimeoutSecs": 15, "maxRedirects": 5, "includeUrlGlobs": [], "excludeUrlGlobs": [], } # Run the Actor and wait for it to finish run = client.actor("plusonets/sitemap-url-status-checker").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ { "url": "https://www.sitemaps.org/" } ], "maxUrls": 50, "checkStatus": true, "respectRobotsTxt": true, "maxConcurrency": 4, "minDelayMillis": 200, "requestTimeoutSecs": 15, "maxRedirects": 5, "includeUrlGlobs": [], "excludeUrlGlobs": [] }' | apify call plusonets/sitemap-url-status-checker --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,plusonets/sitemap-url-status-checker" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/T6i6cY2XbWbwKEQlQ/builds/Le5cYtwi4DJcV8Fvl/openapi.json