# Sitemap URL Extractor: Every URL, lastmod & New Page Monitor (`charlesmarkette/sitemap-url-extractor-new-page-monitor`) Actor

Get every URL a website lists in its sitemaps: robots.txt discovery, nested sitemap indexes, .xml.gz, plain-text and RSS/Atom sitemaps, with lastmod, changefreq, priority, images, news, video and hreflang. Filters, and only new pages since the last run for monitoring.

- **URL**: https://apify.com/charlesmarkette/sitemap-url-extractor-new-page-monitor.md
- **Developed by:** [Charles Markette](https://apify.com/charlesmarkette) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.35 / 1,000 url rows

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap URL Extractor & New Page Monitor: Every URL, lastmod, Only New Pages

**Sitemap URL Extractor & New Page Monitor** reads a website's **XML sitemaps** and returns **every page URL the site publishes**, one row per URL, with its **lastmod** date (ISO 8601, UTC), **changefreq** and **priority**, plus **image, news, video and hreflang** details where the sitemap has them. Give it a plain website URL and the sitemaps are found for you from **robots.txt**; **sitemap index files** are followed level by level, and **.xml.gz**, **plain-text** and **RSS/Atom** sitemaps all work. Turn on **Only new URLs** and schedule it, and each run returns just the pages that appeared since the last run: a cheap monitor for competitors' new pages, products or blog posts.

### Why this Actor

- **Finds the sitemaps for you.** For a website URL it reads `robots.txt` and follows every `Sitemap:` line. If there are none, it tries `/sitemap.xml`, `/sitemap_index.xml` and `/sitemap.xml.gz` (skipping any that robots.txt disallows) and uses the first one that is a real sitemap. You can also give a sitemap URL or a robots.txt URL directly.
- **Every sitemap format.** Sitemap index files (nested up to 5 levels, each sitemap read once), gzipped sitemaps (detected from the file's bytes, so a `.gz` served under any name or content type works), XML with or without namespace prefixes, files that start with a byte-order mark or blank lines, plain-text sitemaps (one URL per line) and RSS 2.0 / RSS 1.0 / Atom feeds submitted as sitemaps.
- **Extension data included.** Google's image (`images`), news (`newsTitle`, `newsPublicationDate`), video (`videoTitle`) and `xhtml:link` hreflang (`alternates`) extensions become columns when a URL has them.
- **Clean, de-duplicated output.** A URL listed in several sitemaps of the same site is output once. Dates are converted to UTC; `priority` is a number; invalid `lastmod` or `priority` values become `null` instead of breaking the row.
- **Only new URLs.** It remembers the URLs of each site (up to 1,000,000 per site) and, on later runs, outputs only the ones it hasn't seen. You pay only for the new rows.
- **Filters before you pay.** Include and exclude patterns (globs, plain text or regular expressions), *Changed since* on `lastmod`, and same-domain only. URLs left out by the filters are free.
- **Safe and polite.** XML is parsed with [defusedxml](https://github.com/tiran/defusedxml): entity declarations (billion-laughs bombs, XXE) are refused. Each sitemap is capped at 50 MB uncompressed, the sitemaps.org limit, and decompression is bounded. At most 2 requests at a time per host, retries with backoff on 429 and 5xx (honouring `Retry-After`), and a user agent that names the tool. URLs that point at private, local or cloud-metadata addresses are refused, including after a redirect.
- **One broken site never stops the run.** A site with no readable sitemap gets one free row with the reason in `error`; a site that could only partly be read gets one free row with a `warning`.

### Use cases

- **SEO audits and site migrations:** list every URL a site declares, with `lastmod`, to compare old and new sites, build redirect maps, or find sections that are missing from the sitemap.
- **Feed a crawler or a checker:** send the URL list to [Bulk URL Status Checker](https://apify.com/charlesmarkette/bulk-url-status-checker-redirects-ssl) to find 404s and redirect chains among the URLs a site tells Google about, or to [Broken Link Checker](https://apify.com/charlesmarkette/broken-link-checker-crawl-website-404) or your own scraper as start URLs.
- **Monitor competitors' new pages:** schedule it daily on competitors' sites with *Only new URLs* (and, for example, `*/blog/*` or `*/products/*`) and get an alert with each new page.
- **Content and news tracking:** news sitemaps give `newsTitle` and `newsPublicationDate` for each recent article; schedule a run with *Only new URLs* to collect new articles as they appear.
- **RAG and search indexing lists:** get a site's full, de-duplicated URL list, or only the pages changed in the last 7 days, before you fetch and embed the content.

### How to use it

1. Add **Start URLs**, one per line: a website (`https://crawlee.dev`), a sitemap (`https://example.com/sitemap.xml`, `…/sitemap.xml.gz`, `…/sitemap.txt`) or a robots.txt URL. Any other page URL is tried as a sitemap first; if it isn't one, that site's sitemaps are used.
2. Optional: set **Include URLs matching**, **Exclude URLs matching** or **Changed since**. *Only URLs on the input's domain* is on by default.
3. For monitoring, turn on **Only new URLs**, save the input as a task and schedule it.
4. **Max output rows** is prefilled at 100 so your first try is cheap. Clear it to get every URL (large sites list tens of thousands).
5. Run it. Export the URLs as CSV, Excel or JSON, or pass them to another Actor (see Related Actors). A run summary per site is in the key-value store as `OUTPUT`.

**Input example**

```json
{
  "startUrls": ["https://crawlee.dev", "https://www.djangoproject.com/sitemap.xml"],
  "includePatterns": ["*/blog/*", "re:/news/20\\d\\d/"],
  "excludePatterns": ["*.pdf"],
  "lastmodSince": "30 days",
  "sameDomainOnly": true,
  "onlyNew": false,
  "maxSitemaps": 500
}
```

Patterns: a glob like `*/blog/*` is matched against the whole URL, ignoring case; text without `*`, `?` or `[` (like `/products/`) means "the URL contains this", ignoring case; a pattern starting with `re:` is a regular expression searched in the URL, case-sensitive unless you add `(?i)`.

**Dataset input:** pick a **Dataset of websites** (for example company websites from another scraper) and set **URL fields** (default `url`, `website`, `sitemapUrl`, `domain`; dotted paths work). With *Keep the item's original fields* on, each URL row also carries the item's fields.

**Output example**

```json
{
  "url": "https://www.example.com/blog/launch",
  "lastmod": "2026-10-15T06:30:00Z",
  "changefreq": "weekly",
  "priority": 0.8,
  "sitemapUrl": "https://www.example.com/sitemap-posts.xml.gz",
  "site": "https://www.example.com/",
  "images": ["https://www.example.com/img/launch.png"],
  "newsTitle": "We launched",
  "newsPublicationDate": "2026-10-16T09:00:00Z",
  "videoTitle": "Launch video",
  "alternates": [{"hreflang": "en", "url": "https://www.example.com/blog/launch"},
                 {"hreflang": "de", "url": "https://www.example.com/de/blog/launch"}]
}
```

`url`, `lastmod`, `changefreq`, `priority`, `sitemapUrl` (the sitemap file that listed it) and `site` (your input) are on every row, `null` when the sitemap doesn't give them. `images`, `newsTitle` / `newsPublicationDate`, `videoTitle` and `alternates` appear only on URLs that have them. A site that couldn't be read gives one row with `url: null` and an `error` such as *"No sitemap found: robots.txt lists no Sitemap: lines, and /sitemap.xml (HTTP 404 Not Found), … didn't work"*; a site read only in part gives one row with a `warning` (for example a child sitemap that returned 404, or a sitemap over 50 MB).

The **`OUTPUT`** record summarises the run, with a line per site:

```json
{
  "sitesRequested": 2, "sitesRead": 2, "sitesFailed": 0,
  "sitemapsRead": 14, "sitemapsFailed": 1,
  "urlsFound": 12480, "urlsAfterFilters": 12000, "urlsNew": null,
  "urlRows": 12000, "errorRows": 0, "warningRows": 1, "stoppedBy": null,
  "sites": [{"site": "https://www.example.com/", "status": "partial", "discovery": "robots.txt",
             "sitemapsRead": 13, "sitemapsFailed": 1,
             "failedSitemaps": [{"url": "https://www.example.com/sitemap-old.xml", "error": "HTTP 404 Not Found"}],
             "urlsFound": 11900, "afterFilters": 11420, "output": 11420}]
}
```

### How "Only new URLs" works

- The **first run** for a site outputs all its current URLs (after your filters) and remembers them. Turn on **Skip existing URLs on the first run** to only remember them, so you pay only for pages added after you start monitoring.
- Later runs output only URLs that weren't remembered. URLs are remembered once they are output or left out by your filters. URLs not output because the run hit *Max output rows* or your cost limit are **not** remembered, so they come in the next run.
- The memory is in the key-value store **`sitemap-extractor-state`** in your account: up to **1,000,000 URLs per site** (as compact 8-byte fingerprints). URLs still in the sitemaps are kept the longest; past the limit, the URLs gone from the sitemaps the longest are forgotten first. On a site with more than 1,000,000 URLs, the URLs beyond that limit come back as new on each run.
- A site whose sitemaps can't be read at all on a run keeps its memory unchanged. If some of its sitemaps fail, URLs from those sitemaps stay remembered.

**What resets the memory (starts a fresh one):**

- Running from a different saved task, or running without a task (all runs outside a task share one memory per filter set).
- Changing *Include URLs matching*, *Exclude URLs matching*, *Changed since* or *Only URLs on the input's domain* (unless you set a **Memory name**).
- Setting a new **Memory name** (runs with the same memory name share one memory, whatever the task or filters).
- Writing the site differently in *Start URLs*: `https://example.com` and `https://example.com/sitemap.xml` are two sites with separate memories (`example.com` and `https://example.com/` are the same).
- Deleting the `sitemap-extractor-state` key-value store or its records.

Turning *Only new URLs* off and on again doesn't reset anything: runs without it neither read nor change the memory.

### How much does it cost?

Pay per event: **$0.0005 per URL row** ($0.50 per 1,000 URLs), plus Apify's run start fee of **$0.005** (one start event per GB of run memory; the default 512 MB run is charged one, so $0.005 a run). Apify platform usage is included. These are free: error and warning rows, URLs left out by your filters or by *Only URLs on the input's domain*, URLs already seen (*Only new URLs*), repeats of a URL listed in several sitemaps, and URLs remembered without output by *Skip existing URLs on the first run*. The run start applies to every run, including a monitoring run that finds no new URLs.

**Apify Store discounts:** the price above is for Apify's Free plan. Paid plans pay less per 1,000 URLs: $0.45 on Starter, $0.40 on Scale, $0.35 on Business. The "from" price at the top of the Store page is the Business price.

**Worked examples:**

*Examples use the Free-plan rate. Where an example doesn't add it, each run also has its run-start charge, counted per GB of Actor memory (minimum one; one at the default memory); Apify Proxy (where used) is billed separately.*

- A full export of a 12,000-URL site: 12,000 × $0.0005 = **$6.00**.
- The same site with *Include URLs matching* `*/blog/*`, which keeps 800 blog posts: 800 × $0.0005 = **$0.40**.
- A daily monitor of a competitor with *Only new URLs* that finds about 20 new URLs a day: 20 × $0.0005 = **$0.01 per day** in URL charges plus the $0.005 start, about $0.45 a month. With *Skip existing URLs on the first run*, the first run costs only the start.

If you set a maximum cost per run, the Actor stops cleanly at that limit, and with *Only new URLs* the URLs it didn't output come in the next run.

### FAQ and limits

#### Is this a crawler?

No. It returns what the site publishes in its sitemaps, and nothing else: pages missing from the sitemap aren't found, and URLs in the sitemap aren't checked to still work. To find pages by following links, use [Broken Link Checker](https://apify.com/charlesmarkette/broken-link-checker-crawl-website-404); to check the URLs, use [Bulk URL Status Checker](https://apify.com/charlesmarkette/bulk-url-status-checker-redirects-ssl).

#### Can I trust `lastmod`?

It's the site's own claim. Many sites set it correctly; some set every page to the build date, or leave it out. With *Changed since*, URLs without a `lastmod` are left out.

#### Why did a site return "No sitemap found"?

Its robots.txt lists no sitemap and none of `/sitemap.xml`, `/sitemap_index.xml` and `/sitemap.xml.gz` exists (or robots.txt disallows them). If you know the sitemap's address (some sites use `/post-sitemap.xml`, `/sitemap-index.xml.gz` and so on), add it directly. If robots.txt itself answers with a server error, no paths are guessed, as RFC 9309 asks; a sitemap URL you give directly is still read.

#### How is robots.txt used?

Sitemaps listed in robots.txt, sitemaps you give directly, and child sitemaps listed in a sitemap index are read (the site published them for crawlers). The guessed paths are tried only when robots.txt allows them for user agent `SitemapExtractor` (or `*`).

#### A site gives "HTTP 403" or "HTTP 429" but works in my browser.

Some sites block automated or datacenter traffic. This Actor doesn't use proxies, log in, or send cookies, so sitemaps behind a login, a paywall or bot protection can't be read.

#### Why are some URLs missing with *Only URLs on the input's domain*?

URLs on another registered domain are left out (subdomains are kept: `blog.example.com` counts for `example.com`, and so does the domain the input redirects to). Turn it off to keep every URL, for example for a site whose sitemaps list a CDN or a sister domain.

#### Why is a URL "Blocked"?

URLs that resolve to private, local, link-local or cloud-metadata addresses (for example 127.0.0.1, 10.x.x.x or 169.254.169.254) are refused for safety, including when a redirect leads there.

#### Other limits

- Sitemaps over 50 MB uncompressed are read up to 50 MB, with a warning row. Up to *Max sitemaps per site* (default 500) files are read per site, and sitemap indexes are followed 5 levels deep.
- Sitemaps that use XML entity declarations are refused (no real sitemap needs them).
- Up to 1,000 images and 500 hreflang alternates are kept per URL; only the first video title is kept.
- Very large sites need memory for de-duplication (about 70 bytes per URL): the default 512 MB is fine for a few million URLs per run; read very large sites one at a time (*Parallel sites* = 1) or give the run 1–2 GB.
- Don't run two "only new" runs with the same memory at the same time; the one that finishes last wins.

### Related Actors

| Actor | Use it to |
|---|---|
| [Bulk URL Status Checker](https://apify.com/charlesmarkette/bulk-url-status-checker-redirects-ssl) | Check the status code, redirects and SSL of every URL from the sitemaps. |
| [Broken Link Checker](https://apify.com/charlesmarkette/broken-link-checker-crawl-website-404) | Crawl the site itself and find broken links, including pages missing from the sitemap. |
| [PageSpeed & Lighthouse Audit](https://apify.com/charlesmarkette/pagespeed-lighthouse-audit-core-web-vitals) | Run Lighthouse on the pages you found: Core Web Vitals, SEO and accessibility scores. |
| [RSS & Atom Feed Reader](https://apify.com/charlesmarkette/rss-atom-feed-reader-only-new-items) | Follow the site's blog or news feed, with titles and summaries, only new items. |
| [Merge & Deduplicate Datasets](https://apify.com/charlesmarkette/merge-deduplicate-datasets-only-new-items) | Combine URL lists from several runs or sites and remove duplicates. |
| [Send Email Report & Slack, Discord, Teams, Telegram Alerts](https://apify.com/charlesmarkette/send-email-report-slack-discord-teams-telegram-alerts) | Get the new URLs by email or chat after each scheduled run, only when there are some. |
| [Google Sheets Sync](https://apify.com/charlesmarkette/google-sheets-sync) | Append the URLs to a Google Sheet. |
| [Dataset Delivery: Export to SFTP, FTP, Dropbox & Google Drive](https://apify.com/charlesmarkette/dataset-delivery-export-to-sftp-ftp-dropbox-google-drive) | Deliver the URL list as a CSV, JSON or Excel file. |

Sitemap format: [sitemaps.org protocol](https://www.sitemaps.org/protocol.html). robots.txt: [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309).

# Actor input Schema

## `startUrls` (type: `array`):

One per line. A website (https://example.com: its sitemaps are found from robots.txt, or at /sitemap.xml, /sitemap_index.xml or /sitemap.xml.gz), a sitemap URL (.xml, .xml.gz, .txt, or an RSS/Atom feed used as a sitemap), or a robots.txt URL. Sitemap index files are followed.

## `datasetId` (type: `string`):

Optional. A dataset from another run (company lists, leads, search results…). For each item, the first field below that holds a website or sitemap URL is read, and the item's fields are added to each of its URL rows. Use {{resource.defaultDatasetId}} in an integration.

## `urlFields` (type: `array`):

Item fields to take the website or sitemap URL from, in order of preference. Dotted paths work (company.website). Default: url, website, sitemapUrl, domain.

## `keepItemFields` (type: `boolean`):

On: each URL row is the original item plus the sitemap columns (the sitemap's url wins over an item field of the same name). Off: only the sitemap columns.

## `includePatterns` (type: `array`):

Optional. Keep only URLs that match at least one of these. A glob like */blog/* (matched against the whole URL, ignoring case), plain text like /products/ (the URL contains it, ignoring case), or a regular expression starting with re: (for example re:/20(25|26)/).

## `excludePatterns` (type: `array`):

Optional. Leave out URLs that match any of these (same pattern rules as above), for example \*.pdf or /tag/.

## `lastmodSince` (type: `string`):

Optional. Keep only URLs whose sitemap lastmod is on or after this: a date like 2026-10-01 or 2026-10-01T08:00:00Z (UTC), or a period like '7 days', '24 hours', '2 weeks'. URLs without a lastmod are left out when this is set. lastmod is the site's own claim.

## `sameDomainOnly` (type: `boolean`):

Leave out URLs on other registered domains than the input's (or the domain it redirects to). Subdomains count as the same domain: blog.example.com is kept for example.com.

## `onlyNew` (type: `boolean`):

Output only URLs that weren't in this site's sitemaps on an earlier run with this setting. The first run outputs every URL (turn on the next option to skip them). Memory: up to 1,000,000 URLs per site, in the key-value store 'sitemap-extractor-state'.

## `skipExistingOnFirstRun` (type: `boolean`):

With 'Only new URLs': the first run for a site only remembers its current URLs and outputs nothing (and costs nothing per URL), so later runs output only pages added after you started monitoring.

## `stateKey` (type: `string`):

Optional. Runs with the same memory name share their 'only new' memory. By default each saved task has its own memory, and changing the filters starts a fresh one. Letters, digits, '-', '\_' and '.'.

## `maxSitemaps` (type: `integer`):

Stop reading a site's sitemaps after this many files (sitemap indexes count). Sitemap indexes are followed up to 5 levels deep.

## `concurrency` (type: `integer`):

How many sites to read at the same time (1–10). Each host still gets at most 2 requests at a time.

## `maxResults` (type: `integer`):

Optional cap on output rows for the whole run (URL rows plus any free error or warning rows). Prefilled at 100 so a first try costs about $0.06; clear it to get every URL.

## Actor input object example

```json
{
  "startUrls": [
    "https://crawlee.dev"
  ],
  "keepItemFields": true,
  "sameDomainOnly": true,
  "onlyNew": false,
  "skipExistingOnFirstRun": false,
  "maxSitemaps": 500,
  "concurrency": 3,
  "maxResults": 100
}
```

# Actor output Schema

## `items` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://crawlee.dev"
    ],
    "maxResults": 100
};

// Run the Actor and wait for it to finish
const run = await client.actor("charlesmarkette/sitemap-url-extractor-new-page-monitor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://crawlee.dev"],
    "maxResults": 100,
}

# Run the Actor and wait for it to finish
run = client.actor("charlesmarkette/sitemap-url-extractor-new-page-monitor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://crawlee.dev"
  ],
  "maxResults": 100
}' |
apify call charlesmarkette/sitemap-url-extractor-new-page-monitor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,charlesmarkette/sitemap-url-extractor-new-page-monitor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/s7Kwd7FOGlIiKMAM3/builds/djpBagzTwXTqADSdG/openapi.json