# Sitemap URL Extractor & Scraper: All Website URLs (`cyber1nj3ct0r/sitemap-url-extractor`) Actor

Sitemap scraper that extracts all URLs of a website from its XML sitemaps: sitemap.xml, sitemap indexes and .xml.gz files, found through robots.txt. Returns each URL with its last-modified date, and can check each URL's HTTP status to find dead pages and redirects. Export to CSV, Excel or JSON.

- **URL**: https://apify.com/cyber1nj3ct0r/sitemap-url-extractor.md
- **Developed by:** [Sanira Botheju](https://apify.com/cyber1nj3ct0r) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.20 / 1,000 urls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### What does Sitemap URL Extractor do?

Sitemap URL Extractor gives you **every page URL a website lists in its sitemaps**, as a clean table you can download as JSON, CSV or Excel or read through the API.

Enter a website address. The tool finds the sitemaps on its own, follows sitemap indexes down to the last file, and returns each URL with its last-modified date. You do not need to know where the sitemap is.

- **Finds the sitemaps for you**: reads `robots.txt`, then tries the usual locations such as `/sitemap.xml`, `/sitemap_index.xml` and `/wp-sitemap.xml`.
- **Follows sitemap indexes**: an index that points to 500 sitemap files is read to the end, several files at a time.
- **Reads every sitemap format**: XML, gzip-compressed files (`.xml.gz`), RSS and Atom feeds, and plain text lists.
- **Optional status check**: requests each URL and adds its HTTP status, redirect target and response time, so you can find dead pages and redirects in a sitemap.
- **Filters**: keep only URLs that match a pattern, skip others, or keep only pages changed since a date.
- **No duplicates**: a URL listed in several sitemaps is returned once.

### What data do you get?

| Field | Example | Meaning |
|---|---|---|
| `url` | `https://www.example.com/blog/post-1` | The page URL from the sitemap |
| `lastModified` | `2026-09-30T08:00:00.000Z` | The last-modified date the sitemap gives, when it has one |
| `changeFrequency` | `weekly` | The `changefreq` value, when present |
| `priority` | `0.8` | The `priority` value, when present |
| `imageCount` | `2` | How many images the sitemap lists for the page |
| `alternateCount` | `3` | How many language versions (hreflang) the sitemap lists |
| `sitemapUrl` | `https://www.example.com/sitemap-posts.xml` | The sitemap file the URL came from |
| `site` | `https://www.example.com` | The website |
| `httpStatus` | `200` | With the status check on: the HTTP status after redirects |
| `redirectUrl` | `https://www.example.com/new-page` | With the status check on: where the URL redirects to |
| `responseTimeMs` | `184` | With the status check on: how long the request took |
| `error` | `null` | Empty when all went well, otherwise a code and a message |

Turn on **Include image and language entries** to also get the image URLs and the hreflang links themselves.

### How to get all URLs from a sitemap

1. Open the tool and enter one or more websites under **Websites or sitemaps**, for example `https://www.example.com`. A direct sitemap address such as `https://www.example.com/sitemap.xml` works too.
2. Set **Maximum URLs** if you want to cap the size of the run.
3. Press **Start**.
4. Open the **Output** tab and download the result as JSON, CSV, Excel or HTML, or fetch it through the API.

To run it on a schedule, for example every morning to pick up new pages, use the **Schedule** option in Apify Console.

### How much does it cost to extract sitemap URLs?

You pay for results, not for time: **$0.30 per 1,000 URLs** returned. Apify subscribers on the Silver plan pay $0.25 and on the Gold plan $0.20 per 1,000 URLs. Each run also has a start fee of $0.00005.

| URLs returned | Cost |
|---|---|
| 100 | $0.03 |
| 1,000 | $0.30 |
| 10,000 | $3.00 |
| 100,000 | $30.00 |

Apify's free plan includes $5 of credit each month, which covers about 16,000 URLs.

URLs that are filtered out and duplicates are not charged. Set **Maximum URLs**, or a maximum cost for the run in Apify Console, and the tool stops there and keeps what it has found.

### Input

The only required field is **Websites or sitemaps**.

```json
{
    "startUrls": [{ "url": "https://www.example.com" }],
    "maxUrls": 10000,
    "checkStatus": false,
    "includePatterns": ["/blog/"],
    "excludePatterns": ["*.pdf"],
    "modifiedAfter": "30 days"
}
```

Patterns are plain text that may appear anywhere in the URL, with `*` for any characters. For a regular expression, wrap it in slashes, for example `/\/product\/\d+$/`.

### Output example

```json
{
    "url": "https://www.example.com/blog/post-1",
    "lastModified": "2026-09-30T08:00:00.000Z",
    "changeFrequency": "weekly",
    "priority": 0.8,
    "imageCount": 2,
    "alternateCount": 0,
    "sitemapUrl": "https://www.example.com/sitemap-posts.xml",
    "site": "https://www.example.com",
    "error": null
}
```

Each run also stores a `SUMMARY` record with the number of sitemaps read, URLs found, duplicates skipped, and any sitemap file that could not be read and why.

### What can you use it for?

- **SEO audits**: get the full list of pages a site wants indexed, then check them for dead pages and redirects with the status check.
- **Site migrations**: export every URL of the old site before a redesign or domain move, and verify afterwards that nothing was lost.
- **Feeding a crawler or an AI pipeline**: use the URL list as the start list for a content crawler instead of crawling blindly.
- **Content monitoring**: run it daily with **Changed since** set to `1 day` to get only new and updated pages.
- **Competitor research**: see how many pages a site has, how they are organised and how often they change.

### Use it from the API

```bash
curl -X POST "https://api.apify.com/v2/acts/cyber1nj3ct0r~sitemap-url-extractor/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "startUrls": [{ "url": "https://www.example.com" }], "maxUrls": 1000 }'
```

The tool also works as a tool for AI agents through the [Apify MCP server](https://docs.apify.com/mcp).

### FAQ

#### The tool says no sitemap was found. Why?

Some sites have no sitemap, or keep it at an unusual address that is not listed in `robots.txt`. If you know the address, enter it directly. If the site has none, a sitemap cannot be read; you would need a crawler that follows links instead.

#### Does it find pages that are not in the sitemap?

No. It returns what the sitemaps list. Pages a site left out of its sitemap are not found.

#### Can it read very large sitemaps?

Yes. Sitemap files are read piece by piece, sitemap indexes are followed up to eight levels deep, and a site with millions of URLs works as long as **Maximum URLs** allows it. Duplicate removal covers the first 1.5 million URLs of a run.

#### A site blocks the tool. What can I do?

The tool identifies itself honestly in its User-Agent and does not try to get round blocks. If it is your own site, allow the User-Agent in your firewall, or set your own under **More options**.

#### Is it allowed to read sitemaps?

Sitemaps and `robots.txt` are published so that programs can read them. The tool reads only those files and, when you turn on the status check, requests each listed page once. You are responsible for how you use the results.

#### Something does not work

Open an issue on the **Issues** tab with the website you tried. Every report is read and answered.

# Actor input Schema

## `startUrls` (type: `array`):

Add website addresses (for example <code>https://www.example.com</code>) or sitemap files (for example <code>https://www.example.com/sitemap.xml</code>). For a website, the tool reads <code>robots.txt</code> to find the sitemaps and then tries the usual locations.

## `maxUrls` (type: `integer`):

Stop after this many URLs. You pay per URL returned, so this also caps the cost of the run.

## `checkStatus` (type: `boolean`):

Request every URL and add its HTTP status, redirect target and response time to the result. Use it to find dead pages (404) and redirects in a sitemap. Runs take longer.

## `includePatterns` (type: `array`):

Keep a URL only if it matches at least one of these. Plain text matches anywhere in the URL and <code>*</code> stands for any characters, for example <code>/blog/</code> or <code>*/products/\*.html</code>. For a regular expression, wrap it in slashes: <code>//p/\d+$/</code>.

## `excludePatterns` (type: `array`):

Leave out a URL if it matches any of these. Same pattern rules as above, for example <code>/tag/</code> or <code>\*.pdf</code>.

## `modifiedAfter` (type: `string`):

Keep only URLs whose last-modified date in the sitemap is on or after this date. Pick a date or a period such as <code>30 days</code>. URLs without a date in the sitemap are left out when this filter is set.

## `includeExtras` (type: `boolean`):

Add the image addresses and the language versions (hreflang) that the sitemap lists for each URL. Without this, the result only holds how many there are.

## `requestTimeoutSecs` (type: `integer`):

How long to wait for one file or page before giving up.

## `userAgent` (type: `string`):

The name the tool sends with each request. Leave empty to use the tool's own name, which tells site owners what is visiting. Set your own only for sites you run or have permission to test.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://blog.apify.com"
    }
  ],
  "maxUrls": 200,
  "checkStatus": false,
  "includeExtras": false,
  "requestTimeoutSecs": 30
}
```

# Actor output Schema

## `urls` (type: `string`):

One record per URL found in the sitemaps, with its last-modified date and the sitemap it came from.

## `summary` (type: `string`):

Counts for the run: sitemaps read, URLs found, duplicates skipped, sitemaps that could not be read.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://blog.apify.com"
        }
    ],
    "maxUrls": 200
};

// Run the Actor and wait for it to finish
const run = await client.actor("cyber1nj3ct0r/sitemap-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://blog.apify.com" }],
    "maxUrls": 200,
}

# Run the Actor and wait for it to finish
run = client.actor("cyber1nj3ct0r/sitemap-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://blog.apify.com"
    }
  ],
  "maxUrls": 200
}' |
apify call cyber1nj3ct0r/sitemap-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,cyber1nj3ct0r/sitemap-url-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/cttd0fR7kwOi2A59p/builds/HrVxl200cDChHSH1P/openapi.json