# Sitemap Extractor & Broken Link / Status Checker (`icyanstudio/sitemap-status-checker`) Actor Extract every URL from a website's sitemaps (robots.txt discovery, sitemap indexes, .xml.gz, hreflang, images) and check each page's HTTP status, redirect chain and response time. Finds broken pages and redirects for SEO audits and site migrations. - **URL**: https://apify.com/icyanstudio/sitemap-status-checker.md - **Developed by:** [iCyan Studio](https://apify.com/icyanstudio) (community) - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.10 / 1,000 url extracteds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## Sitemap Extractor & Broken Link / Status Checker **Get every URL from a website's sitemaps and check each one's HTTP status, redirects and response time in one run.** Enter a domain and the actor finds its sitemaps through `robots.txt` (or `/sitemap.xml` and `/sitemap_index.xml`). It follows sitemap indexes, reads `.xml.gz` and plain-text sitemaps, and keeps `lastmod`, `changefreq`, `priority`, image counts and `hreflang` alternates. It then checks every page: status code, final URL, the full redirect chain, content type, response time and a simple `isBroken` flag. One broken child sitemap never fails the run. It's listed in the run summary with the reason, and you still get every URL from the other sitemaps. Results come as JSON, CSV or Excel, over the API, or from AI agents through the Apify MCP server. ### What it does - **Finds sitemaps for you:** reads `Sitemap:` lines in `robots.txt`, then tries `/sitemap.xml` and `/sitemap_index.xml`. You can also enter sitemap URLs directly. - **Reads every common format:** ``, nested `` files (up to 5 levels deep, 500 files per run), gzip (`.xml.gz`), plain-text sitemaps (one URL per line), the Google image extension and `xhtml:link` hreflang alternates. - **Tolerates broken XML:** an unescaped `&` in a URL doesn't stop the parser. A file that's cut off still gives you every complete entry, and the summary marks that file as `partial`. - **Checks status politely:** HEAD first, then GET when a server rejects HEAD. It follows up to 10 redirects, retries 429 and 5xx with backoff and honours `Retry-After`. It never sends more than 2 requests at once to one host, and its User-Agent names the actor. - **Filters:** include or exclude regex, `lastmod` after a date or a relative age ("30 days"), and a maximum number of URLs. Duplicate URLs across sitemaps are removed. ### Use cases - **SEO audits:** find every 404, 410 and 5xx page listed in your sitemap, plus redirected URLs that shouldn't be in a sitemap at all. - **Site migrations and redirect checks:** run it on the old site's sitemap with `followRedirects` on to see where each old URL ends up, or with it off to see the raw 301/302/308 and its target. - **Broken page monitoring:** schedule a daily run and send `isBroken: true` items to Slack, email or a webhook through Apify integrations. - **Content inventory:** export all URLs with `lastmod` to a spreadsheet, or keep only pages changed in the last 7 days to feed a crawler or an indexing tool. - **International SEO:** check that every page lists its `hreflang` alternates. - **AI agents (Apify MCP, LangChain, OpenAI Agents):** a small input and a flat, documented output. An agent can ask "which pages on example.com are broken?" and get structured answers. ### Input | Field | Type | Default | Description | |---|---|---|---| | `startUrls` | string\[] | `["https://crawlee.dev"]` | Domains (`example.com`) or sitemap URLs (`.xml`, `.xml.gz`, `.txt`). Up to 100. | | `checkStatus` | boolean | `true` | Check each URL's HTTP status. Turn it off to only extract URLs. | | `followRedirects` | boolean | `true` | Follow up to 10 redirects. When off, a redirect is reported as its own 3xx with the target in `redirectChain`. | | `lastmodAfter` | string | none | Keep URLs whose `` is on or after this date: `2025-01-31`, or `7 days`, `2 weeks`, `6 months`. URLs without `` are skipped when this is set. | | `urlPattern` | regex | none | Keep only URLs that match (case-insensitive). | | `excludeUrlPattern` | regex | none | Drop URLs that match (case-insensitive). | | `maxUrls` | integer 1–100,000 | `1000` | Stop after this many URLs (after filters and de-duplication). This also caps the cost. | | `concurrency` | integer 1–20 | `5` | Checks running at the same time across all hosts. One host never gets more than 2 at once. | Example: blog posts changed in the last 30 days, with tag pages excluded: ```json { "startUrls": ["https://example.com"], "urlPattern": "/blog/", "excludeUrlPattern": "/tag/|/page/\\d+", "lastmodAfter": "30 days", "maxUrls": 5000 } ``` Example: extract URLs only (no requests to the pages): ```json { "startUrls": ["https://example.com/sitemap_index.xml"], "checkStatus": false, "maxUrls": 50000 } ``` ### Output One dataset item per URL. This one is from a real run on apify.com, 2026-10-25: ```json { "url": "https://apify.com/alternatives/gmapsextractor-alternative", "sitemapUrl": "https://apify.com/sitemap/pages.xml", "sourceDomain": "apify.com", "lastmod": null, "changefreq": null, "priority": null, "imageCount": 0, "hreflang": [], "statusCode": 200, "finalUrl": "https://apify.com/alternatives/outscraper-alternative", "redirectChain": [ { "url": "https://apify.com/alternatives/gmapsextractor-alternative", "statusCode": 308, "location": "https://apify.com/alternatives/outscraper-alternative" } ], "responseTimeMs": 646, "contentType": "text/html", "isBroken": false, "checkMethod": "HEAD", "error": null, "checkedAt": "2026-10-25T07:05:11.653Z" } ``` With hreflang alternates (from a multilingual site): ```json "hreflang": [ { "lang": "en", "url": "https://www.icyanstudio.net/en" }, { "lang": "ja", "url": "https://www.icyanstudio.net/ja" }, { "lang": "zh-TW", "url": "https://www.icyanstudio.net/zh_tw" }, { "lang": "x-default", "url": "https://www.icyanstudio.net/en" } ] ``` - `statusCode`, `finalUrl`, `redirectChain`, `responseTimeMs`, `contentType`, `isBroken`, `checkMethod`, `error` and `checkedAt` are present only when `checkStatus` is on. - `isBroken` is `true` for status 400 and above, and for requests that failed (DNS, timeout, TLS, too many redirects). `error` says why. - `lastmod` is exactly as written in the sitemap. The dataset has two views: **Status check** and **Sitemap data**. #### Run summary (key-value store `OUTPUT`) Every run saves a summary: the sitemaps found and how each was found (`robots.txt`, default location, sitemap index), the status of each file (`ok`, `partial` or `failed`) with its errors, URL counts at each step, filter counts, counts by HTTP status, broken and redirected totals, and warnings. Example (shortened): ```json { "sitemapsFound": 19, "sitemapsRead": 2, "sitemapsFailed": 0, "sitemaps": [ { "url": "https://apify.com/sitemap.xml", "foundVia": "robots.txt", "status": "ok", "type": "sitemapindex", "childSitemaps": 18, "errors": [] }, { "url": "https://apify.com/sitemap/pages.xml", "foundVia": "sitemap index", "status": "ok", "type": "urlset", "urls": 1605, "errors": [] } ], "urlsReturned": 50, "statusCounts": { "200": 50 }, "brokenCount": 0, "redirectedCount": 1, "stoppedAtMaxUrls": true, "warnings": ["Reached maxUrls (50); 17 more sitemap file(s) were not read."] } ``` ### Pricing Pay per event: you pay only for results. | Event | Price | |---|---| | URL extracted from a sitemap | $0.0005 | | URL status checked (only when `checkStatus` is on, and only for URLs actually requested) | $0.0005 | | Actor start | Apify's default | So 1,000 URLs cost $0.50 extracted only, or $1.00 extracted and checked. Set a maximum cost per run in Apify Console: the actor processes only as many URLs as that limit covers, then stops cleanly. You never pay for URLs that weren't returned. ### Limits and notes - Up to 100,000 URLs per run, 500 sitemap files per run, sitemap indexes nested up to 5 levels, and 50 MB per sitemap file (the sitemaps.org limit). Bigger files are read up to 50 MB and marked `partial`. - One host gets at most 2 requests at a time, so a single-site sitemap is checked 2 URLs at a time. Speed depends on the site: in a test run, 50 apify.com URLs were checked in about 3 seconds. - Some sites block automated requests (for example with Cloudflare bot protection) and return 403 or 429 to any non-browser client. Those URLs show up as broken with the real status code. The actor doesn't try to get around bot protection. - `robots.txt` is used only to find sitemaps. URLs listed in the sitemap are checked whatever `Disallow` rules say, as in other SEO auditing tools. - For safety, the actor only requests public `http` and `https` addresses. URLs and redirects that point at private, loopback or cloud-metadata addresses are refused and reported as broken with the reason. - RSS and Atom feeds used as sitemaps are not supported yet. - Only check sites you own or have permission to audit. Requests go out at a polite rate, but the target site's terms are your responsibility. ### Feedback Found a sitemap it can't read, or need a field? Open an issue on the actor's **Issues** tab with the sitemap URL. # Actor input Schema ## `startUrls` (type: `array`): A domain (example.com or https://example.com) to discover its sitemaps, or a direct sitemap URL (https://example.com/sitemap.xml, .xml.gz or .txt). Up to 100 entries. ## `checkStatus` (type: `boolean`): Request every URL (HEAD, then GET if needed) and add status code, final URL, redirect chain, response time, content type and a broken flag. Turn off to only extract URLs from the sitemaps. ## `followRedirects` (type: `boolean`): Follow up to 10 redirects and report the final status. When off, a redirect is reported as its own 3xx status with the target in the redirect chain. ## `lastmodAfter` (type: `string`): Keep only URLs whose sitemap is on or after this date. Use a date (2025-01-31) or a relative age (7 days, 2 weeks, 6 months). URLs without a are skipped when this is set. ## `urlPattern` (type: `string`): Keep only URLs that match this regular expression (case-insensitive). Example: /blog/ ## `excludeUrlPattern` (type: `string`): Drop URLs that match this regular expression (case-insensitive). Example: .(pdf|jpg)$|/tag/ ## `maxUrls` (type: `integer`): Stop after this many URLs (after filters and de-duplication). Each URL is one result, and the limit also caps the cost. ## `concurrency` (type: `integer`): Status checks running at the same time across all hosts. To be polite, at most 2 requests run at once against any single host, so a one-site sitemap is checked 2 URLs at a time. ## Actor input object example ```json { "startUrls": [ "https://crawlee.dev" ], "checkStatus": true, "followRedirects": true, "maxUrls": 100, "concurrency": 5 } ``` # Actor output Schema ## `urls` (type: `string`): One item per sitemap URL, with status check fields when checking is on. ## `summary` (type: `string`): Sitemaps found and read, errors per sitemap, counts by HTTP status, filters applied. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ "https://crawlee.dev" ], "maxUrls": 100 }; // Run the Actor and wait for it to finish const run = await client.actor("icyanstudio/sitemap-status-checker").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": ["https://crawlee.dev"], "maxUrls": 100, } # Run the Actor and wait for it to finish run = client.actor("icyanstudio/sitemap-status-checker").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ "https://crawlee.dev" ], "maxUrls": 100 }' | apify call icyanstudio/sitemap-status-checker --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,icyanstudio/sitemap-status-checker" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/z9f1RtNTM8y9qcWfl/builds/tKgyd12NDfhXnntIK/openapi.json