# Sitemap URL Extractor: Every Page, Lastmod & New URLs (`lanternlane-data/sitemap-url-extractor`) Actor

Extract every URL from any website's XML sitemaps. Auto-discovers sitemaps via robots.txt, follows sitemap indexes, reads .gz, text and news/image/hreflang sitemaps. Returns lastmod, priority and images, filters by pattern or date, and flags new URLs since the last run.

- **URL**: https://apify.com/lanternlane-data/sitemap-url-extractor.md
- **Developed by:** [Lantern Lane Data](https://apify.com/lanternlane-data) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.30 / 1,000 urls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap URL Extractor: every page URL, lastmod & new pages from any website

### 🗺️ What is the Sitemap URL Extractor?

Get **the complete list of pages of any website** from its XML sitemaps, in seconds. Enter a domain and the Actor **finds the sitemaps automatically** (robots.txt plus the usual locations), **follows sitemap indexes recursively**, reads **gzip, plain-text and RSS/Atom** sitemaps, and returns one clean row per URL with `lastmod`, `changefreq`, `priority`, **hreflang alternates, images and Google News data**.

Schedule it to **monitor sites for new pages**: it remembers what it saw and flags `isNew`, or outputs **only new URLs** since the last run.

- ✅ Auto-discovery from `robots.txt` `Sitemap:` lines, `/sitemap.xml`, `/sitemap_index.xml`, `/wp-sitemap.xml` ...
- ✅ Sitemap indexes (any depth), `.xml.gz`, `.txt` and RSS/Atom feed sitemaps
- ✅ `lastmod`, `changefreq`, `priority`, hreflang `alternates`, `images`, `newsTitle` / `newsPublishedAt`
- ✅ Filters: include / exclude regex, *modified since*, same host only
- ✅ New-URL tracking and *only new URLs* mode for monitoring
- ✅ Handles big sites: hundreds of sitemap files, hundreds of thousands of URLs, streamed parsing
- ✅ Respects robots.txt, polite rate limiting, no browser and no proxy needed
- ✅ Error report per site in `RUN_SUMMARY` (which sitemaps failed and why)

### 🎯 What can I use it for?

- **SEO audits**: list every indexable page, find stale pages by `lastmod`, check hreflang coverage.
- **Competitor monitoring**: get alerted when a competitor publishes new pages, products, landing pages or blog posts.
- **Crawl seeding**: feed the URL list into a scraper, Website Content Crawler, RAG pipeline or a link checker.
- **Content inventory & migrations**: export every URL to CSV/Excel before a redesign or CMS migration.
- **News & blog tracking**: Google News sitemaps give title and publication time of fresh articles.
- **AI agents**: a fast "what pages does this site have?" tool via the Apify API or MCP.

### ⬇️ Input

| Field | Description |
|---|---|
| `startUrls` (required) | Websites (`apify.com`) or sitemap URLs (`https://site.com/sitemap_index.xml`). |
| `maxUrls` | Total cap (default 10,000; `0` = no limit). Also caps your cost. |
| `maxUrlsPerSite` | Per-site cap (`0` = none). |
| `includePatterns` / `excludePatterns` | Regular expressions on the URL, e.g. `/blog/`, `/tag/`. |
| `lastmodSince` | Only URLs modified since `2026-10-01` or `7 days`. |
| `keepUrlsWithoutLastmod` | With a date filter, keep URLs that have no `lastmod` (default on). |
| `sameHostOnly` | Drop URLs on other hosts. |
| `includeExtensions` | hreflang, images and news fields (default on). |
| `trackNewUrls` / `onlyNewUrls` | Track URLs between runs (default on) / output only new ones. |
| `stateStoreName` | Named store for tracking; one per monitor. |
| `maxSitemapsPerSite` | Safety limit for huge indexes (default 1,000). |
| `respectRobotsTxt` | Skip sitemap files disallowed by robots.txt (default on). |

#### Example input

```json
{
  "startUrls": ["apify.com", "https://blog.cloudflare.com/sitemap.xml"],
  "includePatterns": ["/blog/"],
  "lastmodSince": "30 days",
  "maxUrls": 5000
}
```

### ⬆️ Output

```json
{
  "url": "https://blog.cloudflare.com/zh-cn/agentic-web",
  "lastmod": "2026-10-08T07:40:32.088000Z",
  "changefreq": null,
  "priority": null,
  "site": "https://blog.cloudflare.com",
  "sitemapUrl": "https://blog.cloudflare.com/sitemap-posts.xml",
  "alternates": [{"hreflang": "pt-br", "url": "https://blog.cloudflare.com/pt-br/agentic-web"}],
  "images": [],
  "imageCount": 0,
  "newsTitle": null,
  "newsPublishedAt": null,
  "isNew": true,
  "firstSeenAt": "2026-10-08T08:10:00Z",
  "scrapedAt": "2026-10-08T08:10:00Z"
}
```

`RUN_SUMMARY` (default key-value store) lists, per site: how sitemaps were discovered, sitemap files read, URLs found / output, new URLs and errors.

### 💰 How much does it cost?

**Free until 22 October 2026.** From then on it uses **pay-per-event** pricing: **$0.30 per 1,000 URLs** (`url-result`, $0.0003 each) plus Apify's standard $0.00005 Actor start fee. Platform usage is included. Use `maxUrls` or the run's maximum cost setting to cap spending.

### 🔌 Use it via API, Python or integrations

Run it from the Apify Console, on a schedule, or from code with the Apify API, the Python or JavaScript `apify-client`, or MCP. Connect results to Google Sheets, Slack, Zapier, Make, n8n or a webhook from the **Integrations** tab.

```python
from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("lanternlane-data/sitemap-url-extractor").call(run_input={"startUrls": ["apify.com"], "maxUrls": 10000})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)
```

### ❓ FAQ

**The site has no sitemap.** The run reports it in `RUN_SUMMARY`. Most CMSs (WordPress, Shopify, Wix, Webflow, Ghost ...) publish one; try the sitemap URL directly if it's in an unusual place.

**How do I get only new pages every day?** Turn on `onlyNewUrls` and schedule the Actor. Connect Slack, e-mail or a webhook integration for alerts.

**Does it crawl pages?** No. It only reads the sitemap files the site publishes for crawlers, which is why it's so fast and cheap. Use the URL list as input for a crawler if you need page content.

**Something broke?** Open an issue and we'll fix it quickly.

### 🏮 More from Lantern Lane Data

**Jobs & hiring data** is our focus: job postings straight from employers' applicant tracking systems and official public feeds, in one consistent schema. More ATS platforms and hiring-signal tools are on the way.

- 💼 [ATS Jobs Scraper](https://apify.com/lanternlane-data/careers-page-jobs-aggregator): Every open job from 7 ATS platforms (Greenhouse, Lever, Ashby, Workday, SmartRecruiters, Workable, Recruitee) in one schema, with new and closed job alerts.
- 🌍 [Remote Jobs Scraper](https://apify.com/lanternlane-data/remote-jobs-aggregator): Remote jobs from 7 boards via their official public feeds, deduplicated, with salary and location filters.

**Other tools**

- 🛍️ [Shopify Products Scraper](https://apify.com/lanternlane-data/shopify-store-products-scraper): Every product, price, variant and stock level from any Shopify store, with price-change tracking.
- 🔦 [AI Brand Visibility Tracker](https://apify.com/lanternlane-data/ai-brand-visibility-tracker): How often ChatGPT, Perplexity, Gemini and Claude mention, rank and cite your brand vs competitors.

***

Made by **Lantern Lane Data**.

# Changelog

This Actor's version history is a separate document: https://apify.com/lanternlane-data/sitemap-url-extractor/changelog.md

# Actor input Schema

## `startUrls` (type: `array`):

One per line. A website (<code>apify.com</code>, <code>https://blog.example.com</code>) is auto-discovered via robots.txt and the usual sitemap locations. A sitemap URL (<code>https://example.com/sitemap_index.xml</code>, <code>.xml.gz</code>, <code>.txt</code>) is read directly.

## `maxUrls` (type: `integer`):

Overall cap on output URLs (0 = no limit). Also caps your cost.

## `maxUrlsPerSite` (type: `integer`):

0 = no per-site limit.

## `includePatterns` (type: `array`):

Keep only URLs matching at least one regular expression, e.g. <code>/blog/</code> or <code>/products/.\*</code>.

## `excludePatterns` (type: `array`):

Drop URLs matching any of these, e.g. <code>/tag/</code>, <code>?page=</code>.

## `lastmodSince` (type: `string`):

Keep URLs whose lastmod is on/after this date: <code>2026-10-01</code> or <code>7 days</code>.

## `keepUrlsWithoutLastmod` (type: `boolean`):

When 'Modified since' is set, also keep URLs that have no lastmod.

## `sameHostOnly` (type: `boolean`):

Drop URLs on other hosts (sitemaps sometimes list CDN or partner URLs).

## `includeExtensions` (type: `boolean`):

Adds alternates (hreflang), images and Google News title/date when the sitemap has them.

## `trackNewUrls` (type: `boolean`):

Remembers URLs per site and sets isNew / firstSeenAt.

## `onlyNewUrls` (type: `boolean`):

Output only URLs that appeared since the previous run. Ideal for scheduled content / competitor monitoring.

## `stateStoreName` (type: `string`):

Named key-value store for tracking. Use different names for separate monitors.

## `maxSitemapsPerSite` (type: `integer`):

Safety limit for huge sitemap indexes.

## `respectRobotsTxt` (type: `boolean`):

Skip sitemap files that the site's robots.txt disallows.

## `proxyConfiguration` (type: `object`):

Not needed for most sites. Use it if a site blocks or rate-limits you.

## Actor input object example

```json
{
  "startUrls": [
    "apify.com",
    "https://blog.cloudflare.com/sitemap.xml"
  ],
  "maxUrls": 10000,
  "maxUrlsPerSite": 0,
  "keepUrlsWithoutLastmod": true,
  "sameHostOnly": false,
  "includeExtensions": true,
  "trackNewUrls": true,
  "onlyNewUrls": false,
  "stateStoreName": "sitemap-url-tracker",
  "maxSitemapsPerSite": 1000,
  "respectRobotsTxt": true
}
```

# Actor output Schema

## `urls` (type: `string`):

One record per URL.

## `summary` (type: `string`):

Sitemaps found, URL counts and errors per site.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "apify.com",
        "https://blog.cloudflare.com/sitemap.xml"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("lanternlane-data/sitemap-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [
        "apify.com",
        "https://blog.cloudflare.com/sitemap.xml",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("lanternlane-data/sitemap-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "apify.com",
    "https://blog.cloudflare.com/sitemap.xml"
  ]
}' |
apify call lanternlane-data/sitemap-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,lanternlane-data/sitemap-url-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/RXZM97VWfcSXouFgu/builds/Fz3D58kt8jPZCKTXy/openapi.json