# Sitemap URL Extractor & Sitemap Checker (404s, Redirects) (`zahidthani/sitemap-url-extractor-status-checker`) Actor

Get every URL from a website's sitemaps (found automatically via robots.txt, indexes followed), with lastmod dates, and optionally check each one's HTTP status to find the 404s, redirects and server errors listed in your sitemap. About $1 per 1,000 URLs checked.

- **URL**: https://apify.com/zahidthani/sitemap-url-extractor-status-checker.md
- **Developed by:** [Zahid Thani](https://apify.com/zahidthani) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.20 / 1,000 url extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap URL Extractor & Sitemap Checker (404s, Redirects)

**Paste a domain and get every URL in its sitemaps, with last-modified dates. Sitemaps are found automatically
through robots.txt and the usual locations, and sitemap indexes are followed. Turn on the status check to find the
404s, redirects and server errors hiding in your sitemap: the pages you're asking Google to index that don't work.**

**What it costs, in plain terms:** listing 1,000 URLs costs about 20 cents. Listing and checking 1,000 URLs costs about
$1. Sites without a sitemap are free. The form opens pre-filled, so your first run works with one click on **Start**
and costs about 8 cents (list only). To see the checker, turn on **Check each URL's status** before you start.

### Sample output (real run, 9 November 2026)

Status check on the wordpress.org news sitemap, first 150 URLs: **146 OK, 4 redirects, 0 broken.** The 4 redirects
are attachment pages that forward to video files, which shouldn't be in a sitemap. Two of them:

| url | lastmodDate | checkResult | statusCode | finalUrl | responseTimeMs |
|---|---|---|---|---|---|
| wordpress.org/news/toc/ | 2023-06-27 | redirect | 200 | wordpress.org/news/files/2023/06/TOC.mov | 112 |
| wordpress.org/news/showcase-trailer-i2_1/ | 2023-10-18 | redirect | 200 | wordpress.org/news/files/2023/10/showcase-trailer-i2_1.mp4 | 108 |

`statusCode` is the status of the final page after redirects. With **Only output problem URLs** on, these would be the
only rows you get.

**What it checks, honestly:** the URLs *listed in your sitemaps*. It doesn't crawl your pages to follow every link on
them. For a full-site crawl of internal and external links, use a crawler-based broken link checker; use this one to
make sure your sitemap only lists live, final URLs, and to export a site's URL list fast.

### What can this sitemap extractor do?

- 🗺️ **Finds the sitemaps for you**: reads `Sitemap:` lines in robots.txt, then tries `/sitemap.xml`,
  `/sitemap_index.xml`, `/wp-sitemap.xml`, `/sitemap.txt` and more. Or paste a sitemap URL directly.
- 🪆 **Follows sitemap indexes** (up to 4 levels and 1,000 sitemaps per site), including `.xml.gz` files, plain-text
  sitemaps, and RSS/Atom feeds.
- 🧾 **One row per URL** with `lastmod`, `changefreq`, `priority`, image and hreflang counts, and the sitemap it came from.
- 🩺 **Optional status check** for every URL: status code, redirect chain, final URL, response time and `X-Robots-Tag: noindex`.
  Uses lightweight HEAD requests (falls back to GET headers when a server refuses HEAD), so pages aren't downloaded.
- 🧰 **Forgiving parser**: a stray `&` or a missing closing tag doesn't lose the sitemap. URLs are recovered and the
  sitemap is flagged as invalid XML in the run summary.
- 🔍 **Filters**: only URLs changed since a date (find new or updated pages), only URLs containing `/blog/`, skip `/tag/`...
- 🤝 **Gentle by default**: at most 4 requests at a time per website, robots.txt respected for status checks.

### Who uses it

- **SEO audits:** find 404s, redirects and noindexed pages listed in a sitemap, so the sitemap only lists pages you want indexed.
- **Site migrations:** export every old URL before a move, then check that each one redirects to the right place afterwards.
- **Content monitoring:** schedule it weekly with **Only URLs changed on or after** to see a competitor's new pages and posts.
- **Crawling and AI pipelines:** get a clean URL list to feed into a scraper, a search index or a RAG pipeline.

### How to use it

1. Click **Try for free**.
2. Paste domains (`example.com`) or sitemap URLs (`example.com/sitemap.xml`), one per line.
3. Turn on **Check each URL's status** if you want broken links and redirects. Add **Only output problem URLs** to see
   only what needs fixing.
4. Click **Start**. A 500-URL site takes seconds to extract, about a minute or two with status checks.
5. Open the **URLs** table or export CSV/Excel/JSON. Per-site totals are in the **summary** record (Output tab).

### Input example

```json
{
  "websites": ["example.com", "https://www.example.org/sitemap_index.xml"],
  "checkStatus": true,
  "onlyProblems": true,
  "maxUrlsPerSite": 5000
}
```

### Output example (every field)

The values below are illustrative; the sample table at the top is from a real run.

```json
{
  "type": "url",
  "site": "example.com",
  "url": "https://example.com/old-pricing",
  "lastmod": "2025-03-14T09:12:00+00:00",
  "lastmodDate": "2025-03-14",
  "changefreq": "monthly",
  "priority": "0.6",
  "sitemap": "https://example.com/sitemap-pages.xml",
  "statusCode": 200,
  "checkResult": "redirect",
  "redirects": 1,
  "redirectChain": [{"url": "https://example.com/old-pricing", "status": 301}],
  "finalUrl": "https://example.com/pricing",
  "responseTimeMs": 184,
  "contentType": "text/html"
}
```

`checkResult` is one of `ok`, `redirect`, `broken` (404/410), `restricted` (401/403), `client-error`, `server-error`,
`rate-limited`, `redirect-loop`, `timeout`, `unreachable` or `skipped-robots-txt`.

The run's **summary** (key-value store record `OUTPUT`) lists, per site: where the sitemap was found, every sitemap
read, sitemaps that failed (with the reason), URL totals, whether the max was reached, and counts per `checkResult`.

### How much does it cost?

| Event | Price | Per 1,000 |
|---|---|---|
| URL extracted | $0.0002 | **$0.20** |
| URL status checked | $0.0008 | $0.80 |
| Run start | $0.0005 | - |

| Job | Cost |
|---|---|
| 1,000 URLs, list only | about $0.20 |
| 1,000 URLs with status checks | about $1.00 |
| 10,000 URLs with status checks, only problems output (say 300) | about $8.06 |

Sites with no readable sitemap, invalid inputs and robots.txt-skipped checks are free. With **Only output problem URLs**,
every URL is still checked (that's the work) but healthy URLs aren't charged as extracted. Set **Maximum cost per run**
and the Actor stops cleanly at your budget.

**Try it free:** a new Apify account costs nothing, needs no credit card, and includes $5 of usage every month.
That covers about 5,000 URLs checked, or 25,000 URLs listed.

### How it compares

| | This Actor | Google Search Console | Desktop crawlers (Screaming Frog and similar) | Crawler-based broken link checkers |
|---|---|---|---|---|
| Finds the sitemaps for you (robots.txt, indexes, .gz) | **Yes** | You submit them | Yes | Usually not |
| Checks every sitemap URL's status and redirect target | Yes | Partly, with a delay | Yes | Only links they crawl |
| Sites you don't own (competitors, clients) | **Yes** | No, verified sites only | Yes | Yes |
| Finds broken links inside page content | No | No | Yes | **Yes** |
| Runs in the cloud on a schedule, API and MCP | Yes | No | No (desktop app) | Varies |
| Cost | about $1 per 1,000 URLs checked | Free | Free up to 500 URLs, then a yearly licence | Varies |

### Use it with AI agents (MCP) and the API

Connect it to Claude, ChatGPT, Cursor or any MCP client via
`https://mcp.apify.com?tools=zahidthani/sitemap-url-extractor-status-checker` and ask, for example:

> List every blog post on example.com updated since October, and tell me which ones redirect.

```bash
curl -X POST "https://api.apify.com/v2/acts/zahidthani~sitemap-url-extractor-status-checker/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" -H "Content-Type: application/json" \
  -d '{"websites": ["example.com"], "urlContains": ["/blog/"]}'
```

### Limits and tips

- A sitemap lists the pages a site *wants* indexed. Pages missing from the sitemap won't appear. To find every link on
  a site, you need a crawler instead.
- Status checks request each URL once. Sites behind bot protection may answer `403` (`restricted`) to cloud servers even
  when the page works in a browser. Try the proxy setting for those.
- Robots.txt rules apply to status checks: disallowed URLs are output as `skipped-robots-txt` and not charged as checked.
  You can turn this off for your own site.
- **Max URLs per site** is 1,000 by default. For big sites, raise it (up to 200,000). The summary flags a site that hit the limit.

### FAQ

**How do I find a website's sitemap?** Just paste the domain. The Actor reads robots.txt and tries the common sitemap
locations. The summary shows where it was found.

**How do I find broken links in my sitemap?** Turn on **Check each URL's status** and **Only output problem URLs**.
Every 404, redirect and server error is listed with the sitemap it came from.

**What will my first run cost?** The pre-filled run lists up to 200 URLs from each of 2 sites, about 8 cents. With
the status check on, it's about 40 cents. After that: $0.0002 per URL listed, $0.0008 per URL checked, $0.0005 per
run, and no monthly fee. Set **Maximum cost per run** to cap any run.

**Do I need an API key or a credit card?** No. A free Apify account is enough, and it includes $5 of usage a month.

**Can I get only new pages?** Set **Only URLs changed on or after** to a date. It relies on the site's `lastmod`
values, which most CMSs (WordPress, Shopify, Wix...) fill in.

**Is it legal?** Sitemaps are files websites publish so that crawlers can find their pages. The Actor reads them and,
if you ask, makes one lightweight request per URL, respecting robots.txt. No login, no personal data.

**Something broke?** Open an issue on the **Issues** tab.

### Related Actors

- [Website Tech Stack Detector](https://apify.com/zahidthani/website-tech-stack-detector): CMS, e-commerce platform, analytics, hosting and email provider for any list of domains.
- [Domain Expiry, SSL & DMARC Checker](https://apify.com/zahidthani/domain-health-checker): domain and SSL certificate expiry, plus SPF, DMARC and DKIM, for a list of domains.
- [Google Trends Scraper](https://apify.com/zahidthani/google-trends-scraper): search demand for any keyword or brand.

### Want it done for you?

This Actor is built and maintained by [Zahid Gulgee Thani](https://zahidgulgeethani.com/). If you would rather not run it yourself, I can set up scheduled exports into Google Sheets or Excel, a dashboard, or a custom scraper for your use case. Free how-to guides and ways to get in touch are on [zahidgulgeethani.com](https://zahidgulgeethani.com/).

# Actor input Schema

## `websites` (type: `array`):

One per line: a domain (example.com) to find its sitemaps automatically through robots.txt and the usual locations, or a direct sitemap URL (example.com/sitemap.xml, .xml.gz or .txt). Sitemap indexes are followed.

## `checkStatus` (type: `boolean`):

Request every URL (a lightweight HEAD request, no page download) and report its HTTP status, redirect chain and response time. Finds 404s, redirects and server errors listed in the sitemap. Costs extra per URL checked.

## `onlyProblems` (type: `boolean`):

With status checks on: output only URLs that are broken, redirected, erroring, blocked or timing out. Every URL is still checked (and charged as checked), but healthy URLs aren't output (or charged as extracted).

## `maxUrlsPerSite` (type: `integer`):

Stop reading a site's sitemaps after this many URLs (after filters). The run summary says when a site was cut off.

## `lastmodAfter` (type: `string`):

Keep only URLs whose sitemap lastmod is on or after this date (YYYY-MM-DD). Good for finding new or updated pages. URLs without a lastmod are skipped when this is set.

## `urlContains` (type: `array`):

Keep only URLs containing at least one of these texts, like /blog/ or /products/. Not case-sensitive.

## `urlExcludes` (type: `array`):

Drop URLs containing any of these texts, like /tag/ or ?page=.

## `respectRobotsTxt` (type: `boolean`):

Skip status checks for URLs the site's robots.txt disallows (they're output as skipped-robots-txt and not charged as checked). Sitemaps themselves are always read: they're published for crawlers.

## `maxConcurrency` (type: `integer`):

How many URL checks run at once across all sites.

## `maxRequestsPerHost` (type: `integer`):

How many requests one website receives at once. Keep it low to stay gentle on small sites.

## `proxyConfiguration` (type: `object`):

Not needed for most sites. Turn on if a site blocks datacenter traffic.

## Actor input object example

```json
{
  "websites": [
    "apify.com",
    "https://wordpress.org/news/sitemap.xml"
  ],
  "checkStatus": false,
  "onlyProblems": false,
  "maxUrlsPerSite": 200,
  "respectRobotsTxt": true,
  "maxConcurrency": 20,
  "maxRequestsPerHost": 4,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "websites": [
        "apify.com",
        "https://wordpress.org/news/sitemap.xml"
    ],
    "maxUrlsPerSite": 200,
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("zahidthani/sitemap-url-extractor-status-checker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "websites": [
        "apify.com",
        "https://wordpress.org/news/sitemap.xml",
    ],
    "maxUrlsPerSite": 200,
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("zahidthani/sitemap-url-extractor-status-checker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "websites": [
    "apify.com",
    "https://wordpress.org/news/sitemap.xml"
  ],
  "maxUrlsPerSite": 200,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call zahidthani/sitemap-url-extractor-status-checker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,zahidthani/sitemap-url-extractor-status-checker"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Dwdmh3DFIPldvTuAQ/builds/IkVa7nrW5fRT4nm8g/openapi.json
