# Sitemap URL Extractor & Monitor — Recursive (`hipersoft/sitemap-extractor`) Actor

Extract every URL from any website's sitemaps: loc, lastmod, changefreq and priority. Follows sitemap indexes recursively, discovers sitemaps from robots.txt, handles gzipped sitemaps, and can return only URLs changed since a date. No key.

- **URL**: https://apify.com/hipersoft/sitemap-extractor.md
- **Developed by:** [hiper soft](https://apify.com/hipersoft) (community)
- **Categories:** Developer tools, Other
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.0002 / url extracted

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Sitemap URL Extractor & Monitor — Recursive, lastmod, changefreq & priority

Extract **every URL** from any website's sitemaps — with **lastmod, changefreq and priority** — into clean JSON, CSV or Excel. Follows **sitemap indexes recursively**, discovers sitemaps from **robots.txt**, handles **gzipped** sitemaps, and can return **only URLs changed since a date** so you can monitor a site for new and updated pages. No key, no login. Fast and reliable.

![Sitemap URL Extractor input — start URLs, max URLs and changedSince in the Apify Console](https://api.apify.com/v2/key-value-stores/SUfvnaFLd9z9eBCtV/records/sitemap-extractor-input.png?v=2)

Great for **site audits, crawl seeding, SEO monitoring, content-change alerts and migrations**.

### What does Sitemap URL Extractor do?

It reads a site's sitemaps and returns one clean record per URL — the page location plus its `lastmod`, `changefreq`, `priority` and the source sitemap it came from. It follows sitemap-index files recursively, discovers sitemaps from `robots.txt`, reads gzipped sitemaps, and can filter to only URLs changed since a date.

### What you get

![Sitemap URL Extractor output example — one structured JSON row per URL](https://api.apify.com/v2/key-value-stores/SUfvnaFLd9z9eBCtV/records/sitemap-extractor-output.png?v=2)

- 🔗 **All URLs** — every `<loc>` with `lastmod`, `changefreq`, `priority` and its source sitemap
- 🌲 **Recursive** — automatically follows sitemap-index → child sitemaps
- 🤖 **Auto-discovery** — finds sitemaps from `robots.txt`; falls back to `/sitemap.xml`
- 🗜️ **Gzip** — reads `.xml.gz` sitemaps
- 🕒 **Monitor mode** — `changedSince` returns only URLs modified on/after a date
- 📥 **Scales** — walk indexes with millions of URLs, capped by your limits

#### Output schema

| Field | Type | Description |
|-------|------|-------------|
| `url` | string (URL) | The page URL from the sitemap `<loc>`. |
| `lastmod` | string | Last-modified date reported for the URL. |
| `changefreq` | string | Suggested crawl frequency (e.g. weekly, daily). |
| `priority` | string | Relative priority of the URL (0.0–1.0). |
| `sourceSitemap` | string (URL) | The sitemap file the URL was found in. |

### Use cases

- **Site audits** — get a full inventory of a site's indexable URLs.
- **Crawl seeding** — feed the URL list into a crawler or scraper.
- **SEO monitoring** — track which pages change and how often.
- **Content-change alerts** — use `changedSince` to catch new or updated pages.
- **Migrations** — compare URL sets before and after a site move.
- **Coverage checks** — verify every important page is in the sitemap.

### How to extract sitemap URLs

1. Add the **Sitemap URL Extractor & Monitor** to your Apify account.
2. Add a site URL (sitemaps discovered automatically) or a direct sitemap URL to `startUrls`.
3. Set `maxUrls`, `maxSitemaps` and, optionally, `changedSince`.
4. Click **Run**.
5. Export the results as JSON, CSV or Excel, or pull them via the Apify API.

### Input

```json
{
  "startUrls": ["https://www.apify.com"],
  "maxUrls": 20000,
  "changedSince": "2026-07-01"
}
```

Pass a **site URL** (sitemaps discovered automatically) or a **direct sitemap** URL (`.xml` or `.xml.gz`).

| Field | Description |
|-------|-------------|
| `startUrls` | Site URLs or direct sitemap URLs. |
| `maxUrls` | Max URLs to extract. |
| `maxSitemaps` | Max sitemap files to fetch. |
| `changedSince` | Only return URLs with `lastmod` on/after this date. |

### Output

```json
{
  "url": "https://www.apify.com/pricing",
  "lastmod": "2026-07-18",
  "changefreq": "weekly",
  "priority": 0.8,
  "sourceSitemap": "https://www.apify.com/sitemap.xml"
}
```

### Need more site data?

- [Website Content Crawler Pro](https://apify.com/hipersoft/web-content-crawler-pro) — turn pages into clean text
- [Domain Inspector](https://apify.com/hipersoft/domain-inspector) — DNS & domain intelligence
- [URL Checker](https://apify.com/hipersoft/urlchecker) — status codes & redirects

### FAQ

**Do I need to know the sitemap URL?**
No — pass the site's homepage and the Actor finds sitemaps via `robots.txt` and `/sitemap.xml`.

**How many URLs can I extract?**
As many as the site publishes; set `maxUrls` and `maxSitemaps` to cap the run.

**Does it follow sitemap index files?**
Yes, recursively, up to your `maxSitemaps` limit.

**Can I monitor a site for changes?**
Yes — set `changedSince` to only get URLs modified since that date.

**What export formats are supported?**
JSON, CSV, Excel and XML, plus the Apify API.

**Can I integrate it with other tools?**
The Sitemap URL Extractor & Monitor can be connected with almost any cloud service or web app thanks to [integrations on the Apify platform](https://apify.com/integrations). It works with [Make](https://apify.com/integrations/make), [Zapier](https://apify.com/integrations/zapier), [Slack](https://docs.apify.com/platform/integrations/slack), [Airbyte](https://docs.apify.com/platform/integrations/airbyte), [GitHub](https://docs.apify.com/platform/integrations/github), [Google Drive](https://docs.apify.com/platform/integrations/drive) and [many more](https://apify.com/integrations), plus the [Apify API](https://docs.apify.com/api/v2), JavaScript/Python clients and MCP. Or use [webhooks](https://docs.apify.com/platform/integrations/webhooks) to trigger an action whenever a run finishes.

**How does billing work?**
You pay only per URL extracted — cheap per item, cheapest in its class.

### Related Actors

- [Website Content Crawler Pro](https://apify.com/hipersoft/web-content-crawler-pro) — turn pages into clean text
- [Domain Inspector](https://apify.com/hipersoft/domain-inspector) — DNS & domain intelligence
- [URL Checker](https://apify.com/hipersoft/urlchecker) — status codes & redirects

### Notes

Original clean-room implementation. Reads public sitemaps and robots.txt only. You are responsible for how you use the data.

# Actor input Schema

## `startUrls` (type: `array`):

A website URL (its robots.txt / sitemap.xml is discovered automatically) or a direct sitemap .xml / .xml.gz URL.

## `maxUrls` (type: `integer`):

Maximum URLs to extract in total.

## `maxSitemaps` (type: `integer`):

Maximum sitemap files to fetch (guards huge indexes).

## `changedSince` (type: `string`):

Optional ISO date (e.g. 2026-07-01). Only return URLs whose lastmod is on/after this date.

## Actor input object example

```json
{
  "startUrls": [
    "https://example.com/sitemap.xml"
  ],
  "maxUrls": 10000,
  "maxSitemaps": 500,
  "changedSince": ""
}
```

# Actor output Schema

## `results` (type: `string`):

The scraped results as dataset items.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://www.apify.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("hipersoft/sitemap-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": ["https://www.apify.com"] }

# Run the Actor and wait for it to finish
run = client.actor("hipersoft/sitemap-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://www.apify.com"
  ]
}' |
apify call hipersoft/sitemap-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,hipersoft/sitemap-extractor"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Bj6F9jE91tw1cEm2C/builds/hiig8X4v5M4I17c1V/openapi.json