# Sitemap URL Extractor & Status Checker (`bjgomes/sitemap-url-extractor-status-checker`) Actor

Extract every URL from a website's XML sitemaps, including nested sitemap indexes, gzipped sitemaps and robots.txt discovery. Get lastmod and an optional HTTP status for each URL. If a site blocks the run, you get a row that says why. Export to CSV, JSON or Excel.

- **URL**: https://apify.com/bjgomes/sitemap-url-extractor-status-checker.md
- **Developed by:** [Brian Gomes](https://apify.com/bjgomes) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.35 / 1,000 url extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

Sitemap URL Extractor & Status Checker reads a website's XML sitemaps and returns every URL, with its lastmod date and an optional HTTP status code.

### What it does

Give it a website or a sitemap URL. It finds the sitemaps, follows every nested sitemap index, reads gzipped sitemaps and returns one row per URL. Turn on the status check and each row also gets the HTTP status code and the final URL after redirects.

It looks for sitemaps in three places: the `Sitemap:` lines in robots.txt, then `/sitemap.xml` and `/sitemap_index.xml` if robots.txt allows them.

When something goes wrong, the run doesn't fail silently. A site with no sitemap, a 403, a 429 or a timeout gives you a row that says what happened.

Here's what five rows look like:

| url | lastmod | statusCode | finalUrl |
|---|---|---|---|
| https://example.com/ | 2026-09-20 | 200 | https://example.com/ |
| https://example.com/pricing | 2026-09-18 | 200 | https://example.com/pricing |
| https://example.com/blog/old-post | 2025-11-02 | 301 | https://example.com/blog/new-post |
| https://example.com/careers | 2026-06-30 | 404 | https://example.com/careers |
| https://example.com/docs/start | 2026-09-01 | 200 | https://example.com/docs/start |

(Example data.)

### Who it's for

- **SEO audits.** Get the full list of pages a site says it has, then check which ones return an error.
- **Site migrations.** Pull every URL from the old site before you move it, so you can map redirects and check nothing was lost.
- **Content inventories.** See every page and when it was last changed, in one spreadsheet.
- **Feeding a crawler or an LLM pipeline.** Hand the URL list to the next Actor or your own code through the Apify API.

### Input

| Field | What it does | Default |
|---|---|---|
| Websites or sitemap URLs | One or more homepages (`https://example.com`) or sitemaps (`https://example.com/sitemap.xml`). It works out which one you gave it. | none, required |
| Check HTTP status of each URL | Adds the status code, final URL, number of redirects and response time to each row. It costs more per URL (see Pricing). | off |
| Maximum URLs | Stops after this many unique URLs. Use it to test on a big site first. | 1,000 |
| Maximum concurrent requests | Status checks in flight at once, never more than 4 to one site. | 5 |
| Request timeout (seconds) | How long to wait for any one sitemap or page. | 20 |

Input field names in the API: `startUrls`, `checkStatus`, `maxUrls`, `maxConcurrency`, `requestTimeoutSecs`.

The form starts at 100 URLs, so your first run costs $0.05. Turn on the status check and it's $0.10.

Example input:

```json
{
  "startUrls": [{ "url": "https://example.com" }],
  "checkStatus": true,
  "maxUrls": 1000
}
```

### Output

One row per URL in the run's dataset. Export it from the Console as CSV, JSON, Excel, XML or HTML, or read it through the Apify API.

| Field | What it holds |
|---|---|
| `type` | `url` for a page, `error` for something it couldn't read. |
| `url` | The page URL, as the sitemap lists it. |
| `lastmod` | The last-modified date from the sitemap, if the site gives one. |
| `changefreq` | How often the site says the page changes, if given. |
| `priority` | The priority the site gives the page (0.0 to 1.0), if given. |
| `sourceSitemap` | The sitemap file the URL came from. |
| `startUrl` | The input the row came from. |
| `statusCode` | The HTTP status code. Only with the status check on. |
| `finalUrl` | Where the URL ends up after redirects. Only with the status check on. |
| `redirectCount` | How many redirects it followed. Only with the status check on. |
| `responseTimeMs` | Time to the response, in milliseconds. Only with the status check on. |
| `error` | Empty when the row is fine. Otherwise it says what went wrong, like "no sitemap found" or "403 Forbidden". |

Example row (JSON):

```json
{
  "url": "https://example.com/blog/old-post",
  "lastmod": "2025-11-02",
  "changefreq": "monthly",
  "priority": "0.5",
  "type": "url",
  "sourceSitemap": "https://example.com/post-sitemap.xml.gz",
  "statusCode": 200,
  "finalUrl": "https://example.com/blog/new-post",
  "redirectCount": 1,
  "error": null
}
```

### Pricing

You pay per URL returned. There's no monthly fee.

- **URLs only:** $0.0005 per URL.
- **Status check:** $0.0005 per URL that answers with a status code, on top of the URL price. A URL that robots.txt disallows, or that times out or can't be reached, is listed with the reason and isn't charged for a check.

A worked example. A site has 2,000 pages in its sitemap.

- URLs only: 2,000 × $0.0005 = $1.00.
- With status codes: 2,000 × ($0.0005 + $0.0005) = $2.00.

Apify also charges its standard start fee for each run. Set Maximum URLs, or a maximum cost per run, for a hard ceiling: the run stops cleanly when it reaches your maximum cost.

### How it handles the hard cases

- **No sitemap.** You get one row with the error "no sitemap found" and nothing else is charged. The run finishes normally.
- **Nested sitemap indexes.** It follows each index down to the sitemaps that hold the URLs.
- **Gzipped sitemaps.** It reads `.xml.gz` files directly.
- **403 and 429.** A blocked request or a rate limit is recorded in the `error` field instead of failing the run. It doesn't retry a 429 in this version.
- **Timeouts.** A sitemap that doesn't answer in time is recorded as an error row. The timeout is 20 seconds unless you change it.
- **robots.txt.** It reads robots.txt first and reads the sitemaps it lists. It only tries `/sitemap.xml` and `/sitemap_index.xml` when robots.txt allows them, and it never checks a page that robots.txt disallows for its user agent. If robots.txt itself returns a server error or can't be reached, it doesn't look for sitemaps or check pages on that site, and says so in the row. A sitemap URL you give it directly is always read.
- **Load on the site.** It limits how fast it requests pages from any one host (at most 4 requests at a time, and it honours a `Crawl-delay`) and identifies itself with an honest user agent that names this Actor (`SitemapURLExtractor`).

### Compared with doing it by hand

You can do this yourself. Open `/sitemap.xml` in a browser, open each child sitemap, unzip the gzipped ones and copy the URLs into a spreadsheet. For a small site with one sitemap, that's quick, and a free web tool will do it too.

It gets slow when the site has a sitemap index with dozens of files, or you need status codes for thousands of URLs, or you need the list every week. That's where this Actor fits. It does the whole site in one run, you can schedule it, and it hands the list to your code through the API.

Compared with other sitemap Actors in the store, the difference is how it fails. A site it can't read gives you a row that says why, not a failed run with nothing in it.

### FAQ

**How do I extract all URLs from a sitemap?**
Put the sitemap URL, or just the homepage, in the input and run it. The Actor follows every nested sitemap and returns one row per URL. Export the dataset as CSV, JSON or Excel.

**How do I find a website's sitemap?**
Give the Actor the homepage. It checks the `Sitemap:` lines in robots.txt and the usual locations, `/sitemap.xml` and `/sitemap_index.xml`. The `sourceSitemap` field on each row shows which file the URL came from.

**Why did my run return no URLs?**
The site has no sitemap it could find, or it blocked the request. Check the `error` field. It says "no sitemap found", a status like 403, or a timeout, and lists every place it looked. If the sitemap lives somewhere unusual, put its full URL in the input instead of the homepage.

**Does it respect robots.txt?**
Yes. It reads the sitemaps robots.txt lists, only tries the usual sitemap locations when robots.txt allows them, and never checks a page robots.txt disallows. It also limits its request rate per site.

**Can it check the HTTP status of every URL?**
Yes. Turn on "Check HTTP status". Each row gets the status code (200, 301, 404, 500 and so on) and the final URL after redirects. That's how you find broken pages and redirect chains in a sitemap.

**How do I export a sitemap to CSV?**
Run the Actor, open the run's Dataset tab and pick CSV. You can also download it through the API with `format=csv`.

**Can I run it on a schedule?**
Yes. Use Apify Schedules to run it daily or weekly with the same input. Each run makes a new dataset, so you can compare lists over time.

**Can I call it from my own code or connect it to other tools?**
Yes. Start it and read the results through the Apify API or the Python and JavaScript clients. It also works with Apify's integrations, like webhooks, Make, Zapier and Google Sheets.

### Limits

- It reads XML sitemaps (including gzipped ones). It doesn't crawl the site's links, so a page that isn't in any sitemap won't show up Plain-text (`.txt`) and RSS or Atom sitemaps aren't read in this version.
- It works on public pages. It doesn't log in.
- A sitemap file can hold up to 50,000 URLs under the sitemap protocol. Very large sites take longer It runs in 256 MB by default. A sitemap file over 55 MB uncompressed is refused.
- The status check makes one request per URL, so it takes longer and costs more than URLs alone.

### Integrations

This Actor runs on Apify, so everything the platform does works with it.

- **API.** Start a run and read the results from your own code, in any language. [Apify API](https://docs.apify.com/integrations/api)
- **Schedules.** Run it weekly and sort by lastmod to see which pages changed. [Schedules](https://docs.apify.com/actors/running/schedules)
- **Tasks.** Save your input once as a task and run it again with one click, or on a schedule. [Tasks](https://docs.apify.com/actors/running/tasks)
- **Webhooks.** Get a call to your own URL when a run succeeds or fails. [Webhooks](https://docs.apify.com/integrations/webhooks)
- **Google Sheets and Drive.** Download the results as CSV or Excel and open them in Sheets. Or add a Google Sheets step after this Actor in Make or Zapier. Or send the files to a Google Drive folder. [Datasets and exports](https://docs.apify.com/storage/dataset), [Google Drive](https://docs.apify.com/integrations/drive)
- **Zapier, Make and n8n.** Start a run from another app and send the results on to the next step. [Zapier](https://docs.apify.com/integrations/zapier), [Make](https://docs.apify.com/integrations/make), [n8n](https://docs.apify.com/integrations/n8n)
- **AI agents (MCP).** Claude, ChatGPT and other MCP clients can find and run this Actor through the Apify MCP server. [Apify MCP server](https://docs.apify.com/integrations/mcp)
- **Other Actors.** Pass this Actor's results to another Actor when a run finishes. [Actor-to-Actor](https://docs.apify.com/integrations/actors)

### Works with

- [Site Audit](https://apify.com/bjgomes/site-audit-seo): an SEO score from 0 to 100 for every page, with a fix for each issue.
- [Broken Link Checker from Sitemap](https://apify.com/bjgomes/broken-link-checker-from-sitemap): checks every page in your sitemap for 404s, server errors and redirect chains.
- [Bulk PageSpeed Insights for a Sitemap](https://apify.com/bjgomes/bulk-pagespeed-insights-for-a-sitemap): runs Google PageSpeed Insights on every URL in your sitemap.

### Support

Found a bug or need a feature? Open an issue on this Actor's **Issues** tab. Include the run ID and the URL you gave it.

# Actor input Schema

## `startUrls` (type: `array`):

A website (example.com) to find its sitemaps from robots.txt, /sitemap.xml and /sitemap\_index.xml, or a sitemap URL to read directly. Sitemap indexes and .xml.gz files are followed.

## `maxUrls` (type: `integer`):

Stop after this many unique URLs across all inputs.

## `checkStatus` (type: `boolean`):

Request each URL and add its status code, final URL after redirects, number of redirects and response time. Respects robots.txt: disallowed URLs are listed but not requested.

## `maxConcurrency` (type: `integer`):

Status checks in flight at once. Never more than 4 at a time to any one host, and robots.txt Crawl-delay is honoured.

## `requestTimeoutSecs` (type: `integer`):

How long to wait for any single sitemap, robots.txt or page request.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://apify.com"
    }
  ],
  "maxUrls": 100,
  "checkStatus": false,
  "maxConcurrency": 5,
  "requestTimeoutSecs": 20
}
```

# Actor output Schema

## `results` (type: `string`):

Every URL found in the sitemaps, one row each, plus an error row for any site or sitemap that could not be read.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://apify.com"
        }
    ],
    "maxUrls": 100
};

// Run the Actor and wait for it to finish
const run = await client.actor("bjgomes/sitemap-url-extractor-status-checker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://apify.com" }],
    "maxUrls": 100,
}

# Run the Actor and wait for it to finish
run = client.actor("bjgomes/sitemap-url-extractor-status-checker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://apify.com"
    }
  ],
  "maxUrls": 100
}' |
apify call bjgomes/sitemap-url-extractor-status-checker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,bjgomes/sitemap-url-extractor-status-checker"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/aqYegkW0mhPbPtHhK/builds/geTdgVOwF6ZkI27Ph/openapi.json