# Sitemap URL Extractor & New Page Monitor (`henri_dewitte/sitemap-url-extractor`) Actor

Extract every URL from a website's sitemaps. Auto-finds sitemaps via robots.txt, opens indexes and .gz files, adds lastmod, hreflang, images and news data. Track new pages daily. $1 per 1,000 URLs.

- **URL**: https://apify.com/henri_dewitte/sitemap-url-extractor.md
- **Developed by:** [Henri De Witte](https://apify.com/henri_dewitte) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$1.00 / 1,000 url results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap URL Extractor & New Page Monitor

Get **every URL of a website** from its XML sitemaps, in seconds. Enter a website and the Actor finds its sitemaps for you (robots.txt, `/sitemap.xml`, `/sitemap_index.xml`, WordPress, Shopify, Yoast...), opens sitemap indexes and `.gz` files, and returns one clean row per page with `lastmod`, `changefreq`, `priority`, hreflang language versions, image count and Google News data.

Turn on **"Only new URLs since the last run"** and schedule it daily: you get a list of every **new page, product or blog post** a website publishes. Perfect for competitor monitoring and SEO.

- ✅ Auto-discovery: robots.txt `Sitemap:` lines + the usual locations
- ✅ Sitemap indexes, nested indexes, gzip (`.xml.gz`), plain-text sitemaps, Google News and image sitemaps
- ✅ `lastmod`, `changefreq`, `priority`, `hreflang` alternates, image count, news title and date
- ✅ Filters: URL contains / excludes, regular expression, modified in the last X days
- ✅ "Only new URLs" mode for daily new-page alerts
- ✅ Fast and cheap: no browser, streams huge sitemaps (50,000 URLs per file) with little memory

### What can you use it for?

- **Competitor monitoring:** get an alert for every new product, landing page or article a competitor publishes.
- **SEO audits:** list all indexable URLs, find pages that were not updated for years, check hreflang setup.
- **Content inventories and migrations:** export the full URL list of a site to Excel or Google Sheets.
- **Feeding other scrapers and AI tools:** use the URL list as input for a content crawler or RAG pipeline.
- **E-commerce:** track new products on Shopify or WooCommerce stores (their sitemaps list every product).

### How to use it

1. Enter one or more **websites** (`https://www.example.com`) or **sitemap URLs** (`https://www.example.com/sitemap_index.xml`).
2. Optional: filter with **URL must contain** (e.g. `/products/`, `/blog/`) or **Modified in the last X days**.
3. Click **Start** and download the URLs as JSON, CSV, Excel, or via the API.
4. For daily new-page alerts: turn on **Only new URLs since the last run** and create a **Schedule**.

### Input example

```json
{
    "startUrls": ["https://www.competitor.com", "https://shop.example.com/sitemap_products_1.xml"],
    "urlContains": ["/products/"],
    "urlExcludes": ["?variant="],
    "modifiedWithinDays": 30,
    "maxSitemaps": 500,
    "maxResults": 10000,
    "onlyNewSinceLastRun": true
}
```

| Field                   | Default | Meaning                                                                       |
| ----------------------- | ------- | ----------------------------------------------------------------------------- |
| `startUrls`             | –       | Websites (we find the sitemaps) or sitemap URLs. Max 200.                     |
| `urlContains`           | `[]`    | Keep URLs that contain any of these texts (not case-sensitive).               |
| `urlExcludes`           | `[]`    | Skip URLs that contain any of these texts.                                    |
| `urlRegex`              | empty   | Keep URLs that match this regular expression.                                 |
| `modifiedWithinDays`    | empty   | Keep pages with a `lastmod` (or news date) in the last X days.                |
| `includeWithoutLastmod` | `false` | With a date filter: also keep pages without a date.                           |
| `maxSitemaps`           | `500`   | Maximum sitemap files to open per website.                                    |
| `maxResults`            | `10000` | Maximum URLs (and maximum cost) per run.                                      |
| `onlyNewSinceLastRun`   | `false` | Only return URLs that an earlier run with the same start URLs did not return. |

### Output example

```json
{
    "url": "https://www.allbirds.com/products/mens-wool-runner-up-mizzle-plus",
    "domain": "www.allbirds.com",
    "lastmod": "2026-10-08T03:52:52.000Z",
    "changefreq": "daily",
    "priority": null,
    "alternates": [],
    "imageCount": 1,
    "newsTitle": null,
    "newsPublishedAt": null,
    "sitemapUrl": "https://www.allbirds.com/sitemap_products_1.xml?from=1878193471557&to=7369944137808",
    "startUrl": "https://www.allbirds.com/",
    "scrapedAt": "2026-10-08T10:40:20.363Z"
}
```

Every row has the same fields. Missing values are `null` (or an empty list). Dates are converted to UTC ISO format. A URL that appears in several sitemaps is returned (and charged) only once per run.

### Pricing

**Pay per result: $1 per 1,000 URLs** ($0.001 per URL). You only pay for URLs in your dataset; duplicates are never charged. Set a **maximum cost per run** and the Actor stops cleanly when it is reached.

**Free Apify plan:** a run returns at most **100 URLs**. Upgrade to a paid Apify plan to get all results.

### FAQ

**Is this legal?** Yes. Sitemaps are files that websites publish specifically so that robots can find their pages. The Actor only reads `robots.txt` and the sitemap files, keeps a polite request rate (max 4 requests per second per site) and does not open the pages themselves. It collects no personal data.

**"No sitemap found"?** Some websites do not list their sitemap in robots.txt and use an unusual location. Find it in your browser (often linked in the footer or in the SEO plugin) and paste the sitemap URL directly.

**Why do I get fewer URLs than the website has?** A sitemap only lists what the owner put in it. Also check `maxSitemaps` and `maxResults`, and look at the SUMMARY record in the key-value store for warnings.

**Some sites return HTTP 403?** A few websites block automated access to their sitemaps. The Actor respects that and reports it in the SUMMARY; it does not try to bypass protections.

**How does "only new" work?** The Actor remembers (per list of start URLs) up to 250,000 URLs it already returned and only returns new ones next time.

### Limitations

- Only URLs listed in sitemaps (the Actor does not crawl links on pages).
- Sitemap files larger than 60 MB are skipped (the official limit is 50 MB).
- RSS/Atom feeds used as sitemaps are not supported (use an RSS Actor for those).

### Support

Found a problem or a sitemap that is not read correctly? Open an issue on the **Issues** tab.

# Actor input Schema

## `startUrls` (type: `array`):

A website (e.g. https://www.example.com, we find its sitemaps via robots.txt) or a sitemap URL (e.g. https://www.example.com/sitemap_index.xml). Max 200.

## `urlContains` (type: `array`):

Keep only URLs that contain at least one of these texts, e.g. "/blog/" or "/products/". Not case-sensitive. Leave empty for all URLs.

## `urlExcludes` (type: `array`):

Skip URLs that contain one of these texts, e.g. "/tag/" or "?page=".

## `urlRegex` (type: `string`):

Optional. Keep only URLs that match this regular expression (not case-sensitive), e.g. ^https://\[^/]+/en/.

## `modifiedWithinDays` (type: `integer`):

Keep only pages whose sitemap lastmod (or news publication date) is within the last X days. Leave empty for all pages.

## `includeWithoutLastmod` (type: `boolean`):

Only used with "Modified in the last X days": also keep pages that have no lastmod in the sitemap.

## `maxSitemaps` (type: `integer`):

Maximum number of sitemap files to open per website (big sitemap indexes can link to thousands).

## `maxResults` (type: `integer`):

Maximum number of URLs this run saves (and you pay for).

## `onlyNewSinceLastRun` (type: `boolean`):

Remember which URLs you already got and only return new ones. Schedule it daily to see every new page, product or article a website publishes (competitor monitoring).

## Actor input object example

```json
{
  "startUrls": [
    "https://apify.com"
  ],
  "urlContains": [],
  "urlExcludes": [],
  "includeWithoutLastmod": false,
  "maxSitemaps": 500,
  "maxResults": 10000,
  "onlyNewSinceLastRun": false
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://apify.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("henri_dewitte/sitemap-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": ["https://apify.com"] }

# Run the Actor and wait for it to finish
run = client.actor("henri_dewitte/sitemap-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://apify.com"
  ]
}' |
apify call henri_dewitte/sitemap-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,henri_dewitte/sitemap-url-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/q56FRa9S9UbGeYPtY/builds/2ne4KQicRZUHMDQIK/openapi.json