# Sitemap URL Extractor: sitemap.xml, Get All Website URLs (`sysco32/sitemap-url-extractor`) Actor

Sitemap.xml parser: extract every page URL of a website from its XML sitemaps, with robots.txt auto-discovery, nested sitemap indexes, gzip and text sitemaps, lastmod, hreflang and images. URL pattern and date filters plus a new-pages monitor. $0.20 per 1,000 URLs.

- **URL**: https://apify.com/sysco32/sitemap-url-extractor.md
- **Developed by:** [Gabor Sziladi](https://apify.com/sysco32) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.20 / 1,000 url results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap URL Extractor: Get All URLs of a Website & Monitor New Pages

Extract **every page URL of a website** from its **XML sitemaps**. Enter a homepage and the Actor finds the sitemaps through **robots.txt** and common paths, follows **nested sitemap indexes**, reads **gzip (.xml.gz)**, text and RSS sitemaps, and returns each URL with `lastmod`, `changefreq`, `priority`, **hreflang alternates**, **image URLs** and **Google News** title and date. Filter by URL pattern or modification date, or switch on **Only new URLs** to get pages added since the last run. It's fast, with no browser and no crawling.

- ✅ Sitemap **auto-discovery** (robots.txt `Sitemap:` lines, `/sitemap.xml`, `/sitemap_index.xml`, `/wp-sitemap.xml` and more), or give a sitemap URL directly
- ✅ Nested **sitemap indexes**, `.xml.gz`, text and RSS/Atom sitemaps, and a broken-XML tolerant parser
- ✅ **Include/exclude regex patterns** (such as `/blog/` or `/product/`) and a **modified after** date filter
- ✅ Follow only matching child sitemaps (such as `post` or `product`) to save time on huge sites
- ✅ hreflang alternates, images, Google News and video flags
- ✅ **Only new URLs** monitoring: track competitors' new pages, products and posts. You pay only for new URLs.

### Input examples

First 100 URLs of a website (default):

```json
{
  "startUrls": [
    "https://apify.com"
  ],
  "maxUrlsPerSite": 100
}
```

All blog posts changed since 1 October 2026:

```json
{
  "startUrls": [
    "https://www.theverge.com"
  ],
  "includePatterns": [
    "/tech/",
    "/news/"
  ],
  "lastmodAfter": "2026-10-01",
  "maxUrlsPerSite": 5000
}
```

Monitor new product pages of competitors (schedule it daily):

```json
{
  "startUrls": [
    "https://shop-one.example",
    "https://shop-two.example/sitemap_products_1.xml"
  ],
  "includePatterns": [
    "/products?/"
  ],
  "maxUrlsPerSite": 100000,
  "onlyNewUrls": true,
  "stateStoreName": "competitor-products"
}
```

### Output example (real run)

```json
{
  "site": "theverge.com",
  "url": "https://www.theverge.com/tech/1017066/meta-muse-sharing-back-on-artifiacts",
  "lastmod": "2026-10-22T20:24:04Z",
  "changefreq": null,
  "priority": null,
  "sitemap_url": "https://www.theverge.com/sitemaps/entries/2026/10",
  "path": "/tech/1017066/meta-muse-sharing-back-on-artifiacts",
  "alternates": [],
  "images": [
    "https://platform.theverge.com/wp-content/uploads/sites/2/2026/09/STKB394_MUSE_AI_CVIRGINIA_C.png"
  ],
  "news_title": null,
  "news_publication_date": null,
  "has_video": false,
  "scraped_at": "2026-10-25T19:06:40Z"
}
```

Use the **URLs** or **URLs with hreflang, images & news** views, or export to JSON, CSV or Excel. The `OUTPUT` record has per-site counts and errors (for example, when a site has no sitemap).

### Pricing

Pay per event, no subscription:

| Event | Price |
|---|---|
| URL | **$0.0002** ($0.20 per 1,000) |
| Actor start (per run) | $0.002 |

The default run (100 URLs) costs about $0.022. 100,000 URLs cost $20. Filtered and already-seen URLs are never charged.

### Use cases

- **SEO audits**: the full list of indexable URLs, lastmod dates and hreflang coverage.
- **Competitor monitoring**: new products, landing pages, blog posts and job pages as soon as they reach the sitemap.
- **Crawl seeding**: feed the URL list to a web scraper, a link checker or an AI crawler instead of crawling blindly.
- **Content inventory and migrations**: export every URL of a site to a spreadsheet.
- **News tracking**: Google News sitemaps with titles and publication dates.

### FAQ

**What if a site has no sitemap?** The site is reported in `OUTPUT.errors` with "No sitemap found", and the other sites still run. You can also enter a sitemap URL directly when it's in a non-standard place.

**How does Only new URLs work?** Seen URLs are stored per site in a named key-value store (`stateStoreName`). In this mode the Actor reads all sitemap files, up to **Max sitemap files per site**, so it remembers every URL even when it outputs fewer, and the next run returns only URLs that are actually new.

**Does it visit the pages?** No. It only reads sitemap files, so it's very fast and light on the website. To scrape page content, pass the `url` field to a web scraper.

**Why is `lastmod` null?** Many sitemaps don't include it. The **Modified after** filter drops URLs without a lastmod.

### Related actors

More Actors by the same author:

- [RSS Feed Reader & Monitor (sysco32/rss-feed-reader)](https://apify.com/sysco32/rss-feed-reader): RSS, Atom and JSON Feed items as clean JSON, with only-new-item monitoring.
- [Bulk Domain WHOIS/RDAP, DNS & Email Security Checker (sysco32/domain-rdap-dns-lookup)](https://apify.com/sysco32/domain-rdap-dns-lookup): registrar and expiry dates, DNS records, mail provider and SPF/DMARC checks for many domains.
- [Substack Scraper (sysco32/substack-newsletter-scraper)](https://apify.com/sysco32/substack-newsletter-scraper): posts, likes, comments and full text of free posts from any public Substack newsletter.

### Need a custom scraper or data pipeline?

If you need page content or metadata from the URLs you found, extra fields, or the results delivered to Google Sheets, a database or an n8n workflow on a schedule, I take custom projects on Fiverr: [Python scraping scripts and n8n automation workflows](https://www.fiverr.com/sysco32/build-python-scraping-scripts-and-n8n-automation-workflows). Public data only; message me with the details first and I'll confirm whether it's doable.

# Actor input Schema

## `startUrls` (type: `array`):

A homepage such as <code>https://example.com</code> (sitemaps are discovered from robots.txt and common paths) or a direct sitemap URL (<code>.xml</code>, <code>.xml.gz</code>, <code>.txt</code>, sitemap index).

## `maxUrlsPerSite` (type: `integer`):

Upper limit of page URLs returned per website. Set high (e.g. 100000) for a full crawl list.

## `includePatterns` (type: `array`):

Keep only URLs matching at least one pattern, e.g. <code>/blog/</code>, <code>/products?/</code>, <code>.html$</code>.

## `excludePatterns` (type: `array`):

Drop URLs matching any pattern, e.g. <code>/tag/</code>, <code>/page/\d+</code>.

## `lastmodAfter` (type: `string`):

Keep only URLs whose <code>lastmod</code> is on or after this date, e.g. <code>2026-10-01</code>. URLs without lastmod are dropped when set.

## `sitemapFilter` (type: `array`):

In sitemap indexes, follow only child sitemaps whose URL contains one of these strings, e.g. <code>post</code>, <code>product</code>, <code>news</code>.

## `maxSitemapsPerSite` (type: `integer`):

Safety limit for huge sitemap indexes.

## `includeExtras` (type: `boolean`):

Add hreflang alternates, image URLs, Google News title and publication date and a video flag when the sitemap has them.

## `onlyNewUrls` (type: `boolean`):

Remember the URLs already seen per site and return only URLs that appeared since the last run: track competitors' new pages, products, blog posts or landing pages.

## `stateStoreName` (type: `string`):

Named key-value store for seen URLs. Use a different name for each independent monitor.

## Actor input object example

```json
{
  "startUrls": [
    "https://apify.com"
  ],
  "maxUrlsPerSite": 100,
  "maxSitemapsPerSite": 200,
  "includeExtras": true,
  "onlyNewUrls": false,
  "stateStoreName": "sitemap-url-extractor-seen"
}
```

# Actor output Schema

## `urls` (type: `string`):

URLs with lastmod

## `detailed` (type: `string`):

URLs with hreflang, images and news data

## `summary` (type: `string`):

Records per task, failed inputs with error messages.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://apify.com"
    ],
    "maxUrlsPerSite": 100
};

// Run the Actor and wait for it to finish
const run = await client.actor("sysco32/sitemap-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://apify.com"],
    "maxUrlsPerSite": 100,
}

# Run the Actor and wait for it to finish
run = client.actor("sysco32/sitemap-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://apify.com"
  ],
  "maxUrlsPerSite": 100
}' |
apify call sysco32/sitemap-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,sysco32/sitemap-url-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/lvUl24W0t9IFnUTkF/builds/6oYpYQnZKbUnmYEcx/openapi.json