# Sitemap URL Extractor & New Page Monitor (`open-data-tools/sitemap-extractor`) Actor

Get every URL from any website's XML sitemaps (robots.txt discovery, sitemap indexes, .gz, text, RSS/Atom) with lastmod, hreflang and images. Optional HTTP status check for 404s and redirects, and a monitor mode that returns only new pages. No browser, no start fee.

- **URL**: https://apify.com/open-data-tools/sitemap-extractor.md
- **Developed by:** [Open Data Tools](https://apify.com/open-data-tools) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap URL Extractor & New Page Monitor

Get every URL a website publishes in its XML sitemaps, as clean rows with `lastmod`, `changefreq`, `priority`, hreflang alternates, images and Google News fields. Give it a domain and it finds the sitemaps for you. Turn on **monitor mode** to get only the pages that are new since the last run, or **status check** to find 404s and redirects listed in sitemaps.

- **Finds sitemaps automatically:** reads `Sitemap:` lines in robots.txt, then tries `/sitemap.xml`, `/sitemap_index.xml`, `/wp-sitemap.xml`, `/sitemap.xml.gz`, `/sitemap.txt` and `/sitemaps.xml`.
- **Reads every format:** sitemap indexes (nested to any depth), gzipped `.xml.gz`, plain-text sitemaps, RSS and Atom feeds, image, video and Google News sitemaps.
- **Light and fast:** plain HTTP requests, no browser. A test run read 1,010 URLs from 3 sites in 3 seconds at 256 MB.
- **Doesn't fail the whole run on one bad file:** each sitemap is retried with backoff. Broken or blocked files are listed in the per-site summary while the rest of the site is still extracted.
- **Pay only for URLs you get.** No start fee, and the monitor's baseline run is free.

### Use cases

- **SEO audits:** a full URL inventory with lastmod dates. Find stale pages, missing hreflang, and 404s or redirects that shouldn't be in the sitemap.
- **Competitor monitoring:** schedule a daily run in monitor mode and get each new product, landing page, blog post or job page a competitor publishes, the day it appears.
- **Crawl seeding:** feed the URL list into Website Content Crawler, a RAG pipeline or your own scraper, so it crawls only real, listed pages.
- **Content migration:** export every URL before a redesign and check the redirects afterwards.
- **News tracking:** read Google News sitemaps (`news:title`, `news:publication_date`) and RSS feeds as one table.

### Input

| Field | What it does | Default |
|---|---|---|
| Websites or sitemap URLs | `example.com`, a homepage, or a direct sitemap/feed URL. One per line, as many sites as you like. | required |
| Maximum URLs per site | Stop reading a site after this many unique URLs. | 10,000 |
| Only URLs matching (regex) | Keep URLs that match any pattern, e.g. `/blog/`, `/products/` | none |
| Skip URLs matching (regex) | Drop URLs that match any pattern, e.g. `/tag/`, `/page/\d+` | none |
| Modified on or after | Keep URLs whose `lastmod` is on or after a date (`2026-09-01`). | none |
| Monitor: return only new URLs | First run saves a baseline (no URLs charged). Later runs return only URLs that weren't there before. | off |
| Check HTTP status of each URL | HEAD request (GET fallback) per URL: status code, redirect, final URL. | off |

#### Example: full URL list of a site

```json
{
  "startUrls": ["apify.com", "https://blog.apify.com/sitemap.xml"],
  "maxUrlsPerSite": 50000
}
```

#### Example: daily "new pages" monitor for competitors

```json
{
  "startUrls": ["competitor-one.com", "competitor-two.com"],
  "onlyNewUrls": true,
  "excludePatterns": ["/tag/", "/author/"]
}
```

Schedule this daily in Apify (Schedules → Create) and connect the dataset to Slack, email, Google Sheets or a webhook. Each run returns only URLs that appeared since the previous run.

#### Example: find broken URLs in your own sitemap

```json
{
  "startUrls": ["https://www.example.com/sitemap_index.xml"],
  "checkStatus": true,
  "statusCheckConcurrency": 5
}
```

### Output

One row per URL. Export it as JSON, CSV, Excel or HTML, or read it through the API.

```json
{
  "url": "https://www.nytimes.com/live/2026/10/08/us/midterms-elections-debates",
  "site": "https://www.nytimes.com",
  "sitemapUrl": "https://www.nytimes.com/sitemaps/new/news.xml.gz",
  "lastmod": "2026-10-08T22:03:22Z",
  "changefreq": null,
  "priority": null,
  "alternates": [],
  "imageCount": 1,
  "images": ["https://static01.nyt.com/images/2026/10/08/multimedia/08midterms-promo6p-lphw/08midterms-promo6p-lphw-articleLarge.jpg"],
  "newsTitle": "Midterm Elections Updates: Senate Debates to Be Held in Michigan, Maine and Georgia",
  "newsPublicationDate": "2026-10-08T12:46:34Z",
  "videoTitles": []
}
```

With status check on, each row also gets `statusCode`, `finalUrl`, `redirected` and `statusError`. In monitor mode, each row has `isNew: true`.

A per-site summary (sitemaps found and read, URLs found, files that failed and why) is saved in the run's key-value store as `SUMMARY`.

### Pricing

Pay per event, with no start fee:

| Event | Price |
|---|---|
| URL returned | $0.0002 (20¢ per 1,000 URLs) |
| HTTP status check (only if you turn it on) | $0.0002 per URL |

A 5,000-page site costs $1.00, or $2.00 with status checks. In monitor mode you pay only for the new URLs each run returns, so a daily check of a site that adds 10 pages a day costs about $0.06 a month. Set **Maximum URLs per site** or a maximum charge per run to cap your spend.

### Tips and limits

- A domain input finds sitemaps for the whole host. To read just one section, give the section's sitemap URL directly.
- If a site has no sitemap at all, the summary says so. This actor doesn't crawl links. Use Website Content Crawler for sites without sitemaps.
- Some sites block automated requests to their sitemaps (HTTP 403). The actor reports this instead of trying to get around it. For your own site, you can set a custom User-Agent that your firewall allows.
- Monitor memory is stored per site in a named key-value store in your account (`sitemap-extractor-state`). Use a different **Monitor state store name** for each separate monitor, and delete the store to reset the baseline.
- If **Maximum URLs per site** is lower than the site's real size, monitor mode only compares that slice. Set the cap above the site's size for reliable new-page detection.

### FAQ

**Is it legal to read sitemaps?** Sitemaps and robots.txt are public files that sites publish so automated tools can find their pages. The actor reads only those files and, if you ask, sends one HEAD request per URL. It doesn't log in, solve CAPTCHAs or collect personal data. You're responsible for how you use the URLs.

**Why did a site return 0 URLs?** Check `SUMMARY` in the key-value store. The usual causes are no sitemap, a 403 block, or filters that removed every URL.

**Can I use it from an AI agent or via API?** Yes. Call it through the Apify API or MCP server with `startUrls`. It returns structured rows and needs no browser.

### More from open-data-tools

- [USAspending Federal Awards Search](https://apify.com/open-data-tools/usaspending-federal-awards): US federal contracts, grants and loans from the official USAspending.gov API.

Questions or a sitemap format that doesn't parse? Open an issue on this actor's Issues tab and include the sitemap URL.

# Actor input Schema

## `startUrls` (type: `array`):

One per line. A domain (example.com) or homepage finds the sitemaps through robots.txt and common locations. A direct sitemap URL (https://example.com/sitemap_index.xml, .xml.gz, .txt, RSS or Atom feed) is read as is.

## `maxUrlsPerSite` (type: `integer`):

Stop reading a site's sitemaps after this many unique URLs. Filters below apply after this cap.

## `includePatterns` (type: `array`):

Keep only URLs matching at least one of these regular expressions (case-insensitive). Example: /blog/ or .pdf$

## `excludePatterns` (type: `array`):

Drop URLs matching any of these regular expressions (case-insensitive). Example: /tag/ or ?

## `lastmodFrom` (type: `string`):

Keep only URLs whose sitemap <lastmod> is on or after this date (YYYY-MM-DD). URLs without a lastmod are dropped when this is set.

## `onlyNewUrls` (type: `boolean`):

Remembers every URL seen per site (in a named key-value store in your account). The first run saves a baseline and returns nothing, so it costs nothing per URL. Later runs return only URLs that were not in the sitemaps before. Schedule it daily to watch competitors' new pages, products or posts.

## `stateStoreName` (type: `string`):

Named key-value store that keeps the monitor's memory. Use different names for separate monitors.

## `checkStatus` (type: `boolean`):

Sends a HEAD request (GET if HEAD is refused) to every output URL and records the status code, redirect and final URL. Finds 404s and redirects listed in sitemaps. Charged as a separate event per URL.

## `statusCheckConcurrency` (type: `integer`):

Parallel status requests. Keep it low for small sites.

## `maxSitemapsPerSite` (type: `integer`):

Safety cap for huge sitemap indexes.

## `requestTimeoutSecs` (type: `integer`):

How long to wait for one sitemap or status request before retrying.

## `userAgent` (type: `string`):

Optional. By default the actor identifies itself as SitemapExtractor. Set your own if you are checking your own site behind a firewall rule.

## Actor input object example

```json
{
  "startUrls": [
    "apify.com",
    "https://blog.apify.com/sitemap.xml"
  ],
  "maxUrlsPerSite": 10000,
  "onlyNewUrls": false,
  "stateStoreName": "sitemap-extractor-state",
  "checkStatus": false,
  "statusCheckConcurrency": 10,
  "maxSitemapsPerSite": 500,
  "requestTimeoutSecs": 30
}
```

# Actor output Schema

## `urls` (type: `string`):

One row per URL found in the sitemaps.

## `summary` (type: `string`):

Sitemaps found, URLs found, errors per site.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "apify.com",
        "https://blog.apify.com/sitemap.xml"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("open-data-tools/sitemap-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [
        "apify.com",
        "https://blog.apify.com/sitemap.xml",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("open-data-tools/sitemap-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "apify.com",
    "https://blog.apify.com/sitemap.xml"
  ]
}' |
apify call open-data-tools/sitemap-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,open-data-tools/sitemap-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/bIhnegHheBPmjabyG/builds/uBMhXAgOnwYFd23DK/openapi.json