# Sitemap Parser & Extractor: Sitemap to CSV, URLs & New Pages (`data-done-right/sitemap-extractor`) Actor

Sitemap parser and sitemap extractor: export every URL of a website to CSV or JSON from robots.txt and all nested sitemaps (gz included), with lastmod dates. Monitor mode returns only pages that are new or updated since your last run, to track competitors' new pages.

- **URL**: https://apify.com/data-done-right/sitemap-extractor.md
- **Developed by:** [Data Done Right](https://apify.com/data-done-right) (community)
- **Categories:** SEO tools, Marketing
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.36 / 1,000 urls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap Parser & Extractor: Sitemap to CSV, URLs & New Pages

This **sitemap parser and extractor** gets **every URL of a website** from its sitemaps in seconds, and **tells you when competitors publish new pages**. Give it a website or a sitemap URL, then export the URL list to CSV, Excel or JSON from the Apify dataset.

### Sitemap to CSV: how to export a sitemap

Paste a website or sitemap URL, run, then download the **sitemap to CSV**, Excel or JSON from the dataset. Each row is one URL with its lastmod date, so you can open the whole sitemap in a spreadsheet.

### What data can you extract from a sitemap

- **Automatic discovery:** finds sitemaps through `robots.txt`, then `/sitemap.xml`, `/sitemap_index.xml`, `/wp-sitemap.xml` and similar.
- **Every sitemap format:** nested indexes, `.xml.gz`, plain-text sitemaps, image counts, and even slightly broken XML.
- **Lastmod, changefreq and priority** for each URL.
- **Monitor mode:** only pages that are **new** (or **updated**) since your last run. Schedule it daily to see a competitor's new product pages, blog posts, landing pages, docs or job pages the day they go live.
- **Filters:** include and exclude patterns (`/blog/`, `re:/p/\d+$`), plus "modified after" a date.
- No duplicates, even when a URL is listed in several sitemaps.

### How to extract all URLs from a sitemap

1. Enter one or more **Websites**, or exact sitemap URLs in **Or exact sitemap URLs**.
2. Optionally filter with **Only URLs containing** / **Skip URLs containing**, or **Only pages modified after** a date.
3. Run it and download the URL list. For monitoring, set **Mode** to "Only new or updated pages since last run" and schedule a daily run.

### Output

```json
{"site": "https://theverge.com", "url": "https://www.theverge.com/tech/1013025/...", "lastmod": "2026-10-18T14:02:11Z",
 "sitemap": "https://www.theverge.com/sitemaps/entries/2026/10", "change": "new"}
```

### Use cases

- **SEO:** full URL inventories, indexation audits, migration checks.
- **Competitive intelligence:** new products, pricing pages and content as they launch.
- **Crawling:** a clean URL list to feed any other scraper or AI crawler.

### How much does it cost

Pay per event: a tiny fee per URL returned. In monitor mode you only pay for the new or changed pages.

### FAQ

#### How does a sitemap extractor work?

It finds a site's sitemaps (from `robots.txt` or common locations such as `/sitemap.xml`), follows nested sitemap indexes, and returns every listed URL with its `lastmod` date.

#### How do I find a website's sitemap?

You do not have to. Enter the website and the Actor checks `robots.txt`, `/sitemap.xml`, `/sitemap_index.xml`, `/wp-sitemap.xml` and similar paths. If you already know the sitemap URL, paste it instead.

#### Does it work with WordPress sitemaps and .xml.gz files?

Yes. WordPress sitemaps (`/wp-sitemap.xml`), nested indexes, compressed `.xml.gz` sitemaps and plain-text sitemaps are supported.

#### Can it find pages that are not in a sitemap?

No. It only returns URLs the site lists in its sitemaps. It does not crawl links.

#### How do I get only new pages from a competitor?

Set **Mode** to "Only new or updated pages since last run" (or "Only new pages") and schedule it. Each run returns the pages that are new or updated since the last run. The first monitored run saves the baseline (see **Return everything on the first monitored run** if you want the full list on that first run).

#### Do I need a login or API key?

No.

### Related tools

- **[Competitor Monitoring](https://apify.com/data-done-right/competitor-monitoring)**: new pages plus new jobs, ads and news per competitor.
- **Company Enrichment API**: includes site size from the sitemap, plus socials and tech stack.
- **Google Sheets Import & Export**: send the URL list to a Google Sheet.

# Actor input Schema

## `websites` (type: `array`):

Domains or URLs. Sitemaps are found via robots.txt and the usual locations. Example: home-assistant.io, https://www.example.com.

## `sitemapUrls` (type: `array`):

Skip discovery and read these sitemaps (indexes, .xml.gz and .txt sitemaps work). Example: https://www.example.com/sitemap_index.xml.

## `mode` (type: `string`):

All URLs, or only pages added/changed since the previous run (the first monitored run saves a baseline).

## `emitOnFirstRun` (type: `boolean`):

By default the first monitored run only saves the baseline.

## `includePatterns` (type: `array`):

e.g. /blog/, /products/. Prefix with re: for a regular expression.

## `excludePatterns` (type: `array`):

e.g. /tag/, /author/.

## `lastmodAfter` (type: `string`):

YYYY-MM-DD, using the sitemap's lastmod.

## `maxUrlsPerSite` (type: `integer`):

Caps very large sites (and cost).

## `proxyConfiguration` (type: `object`):

Usually not needed.

## Actor input object example

```json
{
  "websites": [
    "home-assistant.io"
  ],
  "mode": "all",
  "emitOnFirstRun": false,
  "maxUrlsPerSite": 200,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `results` (type: `string`):

All result items of this run (the default dataset), as JSON, CSV, Excel and more.

## `records` (type: `string`):

Key-value store records: the run summary, notes on any problems, and any files.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "websites": [
        "home-assistant.io"
    ],
    "maxUrlsPerSite": 200
};

// Run the Actor and wait for it to finish
const run = await client.actor("data-done-right/sitemap-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "websites": ["home-assistant.io"],
    "maxUrlsPerSite": 200,
}

# Run the Actor and wait for it to finish
run = client.actor("data-done-right/sitemap-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "websites": [
    "home-assistant.io"
  ],
  "maxUrlsPerSite": 200
}' |
apify call data-done-right/sitemap-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,data-done-right/sitemap-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/pLU0cNenjlCGJIoEo/builds/atE4H4f2VjBCOgOnK/openapi.json