# Sitemap URL Extractor (`appreciative_duct/sitemap-url-extractor`) Actor

Extract all URLs from website sitemaps. Finds sitemaps in robots.txt, follows sitemap indexes, and reads gzip, news, image, and video sitemaps.

- **URL**: https://apify.com/appreciative_duct/sitemap-url-extractor.md
- **Developed by:** [Ross AI](https://apify.com/appreciative_duct) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap URL Extractor

Sitemap URL Extractor pulls every page URL out of one or many websites. Give it a homepage or a sitemap file. It reads `robots.txt`, checks the sitemap paths sites actually use, follows sitemap indexes, and decompresses `.gz` files. Each result has the URL, `lastmod`, `changefreq`, `priority`, and the source sitemap.

It uses plain HTTP. It does not start a browser, so a full site map stays fast and cheap enough to run ahead of an SEO audit or a crawl.

### Why teams use a sitemap URL extractor

A sitemap is the site's own list of canonical URLs. Crawling every link misses pages that are not linked, and it spends money on pages you did not need. Starting from the sitemap is the usual first step for:

- SEO crawls and content audits. Feed the URLs into an [SEO audit](https://apify.com/appreciative_duct/seo-audit-ai-search-readiness) and check titles, canonicals, and indexation.
- AI knowledge bases and RAG. Collect the public URL list before you extract page text.
- Change monitoring. Keep only URLs whose `lastmod` is newer than a date.
- Migration and QA. Diff the URL list before and after a launch.
- Hreflang and media reviews. Read alternate language links, image URLs, video URLs, and Google News fields from the sitemap extensions.

### How sitemap discovery works

For a website such as `https://example.com`, the Actor:

1. Requests `/robots.txt` and collects every `Sitemap:` directive.
2. Requests common files the sitemap protocol and popular CMSs use, including `/sitemap.xml`, `/sitemap_index.xml`, `/wp-sitemap.xml`, news, image, and video sitemap paths, plus `.xml.gz` variants.
3. If those steps find nothing, it looks for `<link rel="sitemap">` on the homepage.
4. Follows sitemap indexes, including nested indexes, and remembers files it has already downloaded.
5. Reads XML sitemaps and plain-text URL lists. Gzip responses and `.gz` files are decompressed, including a file that was gzipped twice.

A direct sitemap URL such as `https://example.com/sitemap.xml` is downloaded as-is and is not mixed with other files unless it is an index.

Timeouts, connection errors, HTTP 429, and HTTP 5xx are retried with backoff. A file that still fails is listed in the run summary. The other sitemaps are kept. The run fails only when every sitemap that was found fails, so a silent empty dataset is not treated as success.

### Input

```json
{
  "startUrls": [
    { "url": "https://www.python.org" },
    { "url": "https://apify.com/sitemap.xml" }
  ],
  "maxUrls": 1000,
  "includeUrlPatterns": ["/downloads/"],
  "excludeUrlPatterns": ["\\\\?"],
  "minLastmod": "2024-01-01",
  "deduplicate": true
}
```

| Field | Default | What it does |
| --- | --- | --- |
| `startUrls` | required | Website homepages or sitemap URLs. `example.com` is accepted and treated as HTTPS. |
| `maxUrls` | 20000 | Stop after this many page URLs are saved. |
| `includeUrlPatterns` | empty | Case-insensitive JavaScript regular expressions. Empty keeps every URL. |
| `excludeUrlPatterns` | empty | Drop URLs that match any of these regular expressions. |
| `minLastmod` / `maxLastmod` | empty | ISO dates. Index entries outside the window are not downloaded. |
| `includeUndated` | false | When a date filter is set, also keep URLs that have no `lastmod`. |
| `deduplicate` | true | Keep the first copy of a URL. |
| `includeImages` / `includeVideos` / `includeNews` / `includeAlternates` | true | Sitemap extension fields. |
| `useRobotsTxt` | true | Read `Sitemap:` directives. |
| `probeCommonPaths` | true | Check well-known sitemap paths. |
| `maxIndexDepth` | 5 | Nested sitemap indexes to follow. |
| `maxConcurrency` | 8 | Parallel downloads. |
| `maxRetries` | 3 | Retries for timeouts and temporary HTTP errors. |
| `requestTimeoutSecs` | 30 | Per-request timeout. |
| `maxSitemapBytes` | 20000000 | Refuse a single response larger than this. |

The form opens with `maxUrls` set to 1000 so a console test stays small. Raise it when you want the whole site.

### Output

One dataset item per page URL:

```json
{
  "url": "https://www.python.org/downloads/",
  "lastmod": "2024-10-01",
  "changefreq": "weekly",
  "priority": 0.8,
  "sourceSitemap": "https://www.python.org/sitemap.xml",
  "inputUrl": "https://www.python.org",
  "imageCount": 1,
  "images": [{ "url": "https://www.python.org/static/img/python-logo.png", "title": "Python logo" }],
  "videoCount": 0,
  "newsTitle": null,
  "alternateCount": 2,
  "alternates": [
    { "hreflang": "en", "href": "https://www.python.org/downloads/" },
    { "hreflang": "es", "href": "https://www.python.org/es/downloads/" }
  ]
}
```

The key-value record `SUMMARY` lists every sitemap that was fetched, how it was discovered (`robots.txt`, `common-path`, `direct`, `sitemap-index`, or `homepage-link`), and the error for any file that failed.

### API

```bash
curl "https://api.apify.com/v2/acts/appreciative_duct~sitemap-url-extractor/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://apify.com"}],"maxUrls":100}'
```

Synchronous calls are capped by Apify at about 5 minutes. For a large site, start a run and read the dataset when it finishes.

### Pricing

The Actor is free. You pay only the Apify platform usage for compute, storage, and data transfer. It defaults to 256 MB of memory and does not use a browser or a proxy.

The code charges one `url-extracted` event per saved URL when pay-per-event pricing is turned on. It is not turned on yet.

### Tips for large sites

- Set `maxUrls` while you confirm the right sitemap, then raise it.
- Set `minLastmod` to skip old sitemap shards. Index files that publish `lastmod` are not downloaded when they fall outside the window.
- Exclude tag, search, and attachment URLs with a regular expression instead of downloading them into the dataset.
- Turn off image, video, or hreflang fields if you only need the URL list.

### FAQ

#### Where does the sitemap URL extractor look for sitemaps?

It reads `Sitemap:` lines in `robots.txt`, then requests the common XML and gzip paths used by custom sites, WordPress, news, image, and video sitemaps. If both are empty, it checks the homepage for `link rel="sitemap"`.

#### Does it handle sitemap indexes and gzip?

Yes. Indexes are followed up to `maxIndexDepth` (default 5). Gzip transport compression and `.gz` files are both decompressed. A gzip file that was compressed twice is decompressed twice.

#### Can I extract only URLs updated after a date?

Yes. Set `minLastmod` to an ISO date such as `2024-01-01`. URLs without `lastmod` are skipped unless you enable `includeUndated`. Child sitemaps whose own `lastmod` is older are not downloaded.

#### Why did a website return no URLs?

Open the run summary. Typical causes are: the site has no sitemap, `robots.txt` and the common paths all returned HTML or 404, or the host blocked the request. The summary names the URL and the HTTP status. Private and link-local addresses are refused so the Actor cannot be pointed at a cloud metadata service.

#### Does the sitemap extractor run a browser?

No. It is plain HTTP with retries and timeouts. Pages that only appear after JavaScript runs will not be invented. If they are absent from the sitemap, they are absent from the output.

#### What does a failed sitemap do to the rest of the run?

That file is recorded in `SUMMARY` with the status code or parse error. URLs from the other files are still saved. The run is marked failed only when every discovered sitemap fails.

#### How do I audit the URLs this finds?

Run the [SEO audit and AI search readiness Actor](https://apify.com/appreciative_duct/seo-audit-ai-search-readiness) on the URLs, or pass the same website there and let it read the sitemap itself.

### Notes

The Actor sends a normal browser `User-Agent` and follows a limited number of redirects. It does not execute JavaScript, solve CAPTCHAs, or use a proxy. Respect the site's terms and rate limits. `maxConcurrency` defaults to 8 so a single site is not flooded.

# Changelog

This Actor's version history is a separate document: https://apify.com/appreciative_duct/sitemap-url-extractor/changelog.md

# Actor input Schema

## `startUrls` (type: `array`):

One or more website addresses (https://example.com) or direct sitemap URLs (https://example.com/sitemap.xml). A website is expanded through robots.txt, common sitemap paths, and a homepage sitemap link.

## `maxUrls` (type: `integer`):

Stop after this many page URLs have been saved. The default is high enough for a normal site. Lower it for a sample.

## `includeUrlPatterns` (type: `array`):

Optional JavaScript regular expressions, case-insensitive. A URL is kept when it matches at least one pattern. Leave empty to keep every URL.

## `excludeUrlPatterns` (type: `array`):

Optional JavaScript regular expressions, case-insensitive. A URL that matches any pattern is dropped.

## `minLastmod` (type: `string`):

Optional ISO date, for example 2024-01-01. URLs whose lastmod is older are skipped. Child sitemaps with an older lastmod are not downloaded.

## `maxLastmod` (type: `string`):

Optional ISO date. URLs whose lastmod is newer are skipped.

## `includeUndated` (type: `boolean`):

Used only when a modified-after or modified-before date is set. Off means undated URLs are skipped.

## `deduplicate` (type: `boolean`):

Keep the first copy of a URL when it appears in more than one sitemap. The fragment is ignored and the hostname is compared case-insensitively.

## `includeImages` (type: `boolean`):

Save image locations, titles, and captions from Google image sitemaps.

## `includeVideos` (type: `boolean`):

Save video title, description, thumbnail, and content or player URLs.

## `includeNews` (type: `boolean`):

Save the news title, publication name, language, and publication date.

## `includeAlternates` (type: `boolean`):

Save xhtml:link alternate URLs, including the language code.

## `useRobotsTxt` (type: `boolean`):

Read Sitemap directives from the website's robots.txt.

## `probeCommonPaths` (type: `boolean`):

Also request paths such as /sitemap.xml, /sitemap_index.xml, and /wp-sitemap.xml. This catches sitemaps that robots.txt forgot.

## `maxIndexDepth` (type: `integer`):

How many levels of sitemap indexes to follow. 1 is a normal index of URL files. The default 5 covers nested indexes.

## `maxConcurrency` (type: `integer`):

How many sitemaps to download at once.

## `maxRetries` (type: `integer`):

Retries for timeouts, connection errors, HTTP 429, and HTTP 5xx. Waits with exponential backoff.

## `requestTimeoutSecs` (type: `integer`):

Time limit for one HTTP request, including its redirects.

## `maxSitemapBytes` (type: `integer`):

Refuse a single sitemap larger than this. The sitemap protocol limit is 50 MB uncompressed.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://apify.com"
    }
  ],
  "maxUrls": 1000,
  "includeUrlPatterns": [],
  "excludeUrlPatterns": [],
  "includeUndated": false,
  "deduplicate": true,
  "includeImages": true,
  "includeVideos": true,
  "includeNews": true,
  "includeAlternates": true,
  "useRobotsTxt": true,
  "probeCommonPaths": true,
  "maxIndexDepth": 5,
  "maxConcurrency": 8,
  "maxRetries": 3,
  "requestTimeoutSecs": 30,
  "maxSitemapBytes": 20000000
}
```

# Actor output Schema

## `urls` (type: `string`):

One dataset item per URL found in the sitemaps. Fields include url, lastmod, changefreq, priority, and sourceSitemap.

## `summary` (type: `string`):

Counts, the sitemaps that were fetched, and errors for files that failed.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://apify.com"
        }
    ],
    "maxUrls": 1000,
    "includeUrlPatterns": [],
    "excludeUrlPatterns": []
};

// Run the Actor and wait for it to finish
const run = await client.actor("appreciative_duct/sitemap-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://apify.com" }],
    "maxUrls": 1000,
    "includeUrlPatterns": [],
    "excludeUrlPatterns": [],
}

# Run the Actor and wait for it to finish
run = client.actor("appreciative_duct/sitemap-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://apify.com"
    }
  ],
  "maxUrls": 1000,
  "includeUrlPatterns": [],
  "excludeUrlPatterns": []
}' |
apify call appreciative_duct/sitemap-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,appreciative_duct/sitemap-url-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/FrqRm3fbvkyZYmySr/builds/BngiQDudAnZiGg25z/openapi.json