# Sitemap URL Extractor (`pipwort/sitemap-url-extractor`) Actor

Get every page URL a website lists in its sitemaps. Finds sitemaps via robots.txt and common paths, follows nested indexes, reads .gz, text, news and RSS sitemaps, and filters by date or pattern.

- **URL**: https://apify.com/pipwort/sitemap-url-extractor.md
- **Developed by:** [Pipwort Data](https://apify.com/pipwort) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap URL Extractor

Get every page URL a website publishes in its sitemaps, as a clean table you can export to CSV, Excel or JSON.

Give it a home page or a sitemap link. It reads `robots.txt`, checks the usual sitemap locations, follows nested sitemap indexes, and returns one row per page with its last-modified date.

### What it handles

- **Discovery:** `Sitemap:` lines in `robots.txt`, then `/sitemap.xml`, `/sitemap_index.xml`, `/wp-sitemap.xml`, `/sitemap.txt` and similar.
- **Formats:** XML sitemaps and sitemap indexes (nested up to 10 levels), `.gz` files, plain-text sitemaps, and RSS or Atom feeds used as sitemaps.
- **Extras:** image sitemaps (`image:image`), hreflang alternates (`xhtml:link`) and Google News sitemaps (title and publication date).
- **Filters:** last-modified date range, include and exclude patterns (regular expressions), same-domain only, and per-site or total limits.
- **Reliability:** retries with backoff on timeouts, 429 and 5xx responses, a hard time limit per request, and safe XML parsing. A broken sitemap is logged and skipped, and the rest of the run carries on.

### Common uses

- Building a crawl list for SEO audits, content inventories or site migrations
- Feeding page lists to an AI or RAG pipeline
- Tracking new or updated pages on a site over time (use the date filter on a schedule)

### Input

| Field | What it does |
|---|---|
| `startUrls` | Home pages or direct sitemap URLs (required) |
| `lastmodAfter` / `lastmodBefore` | Keep only pages modified in a date range |
| `includePatterns` / `excludePatterns` | Regular expressions, for example `/blog/` |
| `maxUrlsPerSite` / `maxUrlsTotal` | Stop after this many URLs (0 means no limit) |
| `includeImages`, `includeAlternates` | Add image URLs and language alternates |
| `sameDomainOnly` | Drop URLs on other domains (on by default) |

### Output

```json
{
  "url": "https://www.canada.ca/en/impact-plus-chairs/program-details.html",
  "lastmod": "2026-09-28T15:05:23.890-04:00",
  "changefreq": null,
  "priority": null,
  "sitemapUrl": "https://www.canada.ca/en/impact-plus-chairs.sitemap.xml",
  "site": "https://www.canada.ca"
}
```

A `SUMMARY` record in the run's key-value store lists, for each site, the sitemaps found, any that failed and why, and how many URLs were returned. If a site publishes no sitemap, the run says so plainly and you are not charged for any results.

### Pricing

Pay per result: **US$0.50 per 1,000 URLs**, plus a small start fee of US$0.005 per run. You can cap the cost of any run with the "Maximum cost per run" setting. The Actor stops cleanly when it reaches your limit.

### Good to know

- The Actor reads only `robots.txt` and the sitemaps a site publishes for automated tools. It does not load page content.
- Some sites block automated requests to their sitemaps (you'll see `HTTP 403` in the summary). The Actor does not try to get around these blocks.
- `lastmod` is only as accurate as the website makes it. Many sites leave it out. Use `keepUrlsWithoutLastmod` to decide what to do with those URLs.

### Support

Found a sitemap that doesn't parse? Open an issue on the Actor's Issues tab with the URL, and we'll look at it.

# Actor input Schema

## `startUrls` (type: `array`):

A website home page (for example https://example.com) or a direct sitemap URL. For a home page, the Actor looks in robots.txt first, then at common sitemap locations.

## `discoverFromRobots` (type: `boolean`):

Use the Sitemap: lines in robots.txt.

## `tryCommonPaths` (type: `boolean`):

If robots.txt lists nothing, try /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml and similar.

## `maxUrlsPerSite` (type: `integer`):

0 means no limit.

## `maxUrlsTotal` (type: `integer`):

0 means no limit. You can also cap spending with the run's maximum cost setting.

## `lastmodAfter` (type: `string`):

Keep URLs whose lastmod is on or after this date (YYYY-MM-DD).

## `lastmodBefore` (type: `string`):

Keep URLs whose lastmod is on or before this date (YYYY-MM-DD).

## `keepUrlsWithoutLastmod` (type: `boolean`):

Many sitemaps omit lastmod. Turn off to drop those URLs when a date filter is set.

## `includePatterns` (type: `array`):

Keep only URLs matching at least one pattern, for example /blog/ or .html$

## `excludePatterns` (type: `array`):

Drop URLs matching any of these patterns.

## `sameDomainOnly` (type: `boolean`):

Drop URLs on other domains (subdomains are kept).

## `includeImages` (type: `boolean`):

Add image:image locations listed for each page.

## `includeAlternates` (type: `boolean`):

Add xhtml:link language alternates for each page.

## `maxDepth` (type: `integer`):

How many levels of nested sitemap indexes to follow.

## `maxConcurrency` (type: `integer`):

Keep this low to be polite to the website.

## `requestTimeoutSecs` (type: `integer`):

Per request.

## `maxRetries` (type: `integer`):

Retries on timeouts, 429 and 5xx responses, with backoff.

## `userAgent` (type: `string`):

Optional custom User-Agent header.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://www.python.org"
    }
  ],
  "discoverFromRobots": true,
  "tryCommonPaths": true,
  "maxUrlsPerSite": 0,
  "maxUrlsTotal": 0,
  "keepUrlsWithoutLastmod": true,
  "sameDomainOnly": true,
  "includeImages": false,
  "includeAlternates": false,
  "maxDepth": 5,
  "maxConcurrency": 5,
  "requestTimeoutSecs": 30,
  "maxRetries": 3
}
```

# Actor output Schema

## `results` (type: `string`):

One row per page URL found in the sitemaps.

## `summary` (type: `string`):

Sitemaps found, sitemap errors and URL counts for each input site.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://www.python.org"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("pipwort/sitemap-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://www.python.org" }] }

# Run the Actor and wait for it to finish
run = client.actor("pipwort/sitemap-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://www.python.org"
    }
  ]
}' |
apify call pipwort/sitemap-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,pipwort/sitemap-url-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/4seu0iJzkEh4DY1vk/builds/4xEwPMMRoT68neOFd/openapi.json