# Sitemap URL Extractor Pro (`steadyutil/sitemap-url-extractor-pro`) Actor

Extract every URL from any website's sitemaps: auto-discovery via robots.txt, sitemap indexes, gzipped .xml.gz, TXT and RSS sitemaps, lastmod, hreflang, images, robots.txt conflicts and optional HTTP status codes.

- **URL**: https://apify.com/steadyutil/sitemap-url-extractor-pro.md
- **Developed by:** [SteadyUtil](https://apify.com/steadyutil) (community)
- **Categories:** Developer tools, SEO tools
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.50 / 1,000 url extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap URL Extractor Pro: get every URL of any website from its sitemaps

**Sitemap URL Extractor Pro** turns any domain or sitemap link into a clean list of every page URL the site publishes, with `lastmod`, `changefreq`, `priority`, hreflang alternates, image URLs and (optionally) the live HTTP status of each page. Give it `example.com` and it finds the sitemaps by itself: robots.txt, sitemap indexes nested several levels deep, gzipped `.xml.gz` files, plain-text `.txt` sitemaps and RSS/Atom feeds.

It's built for SEO audits, content inventories, migrations, competitor research and as a fast pre-step for crawlers and AI/RAG pipelines. You get the URLs in seconds without crawling the whole site.

### Why use it?

- **Auto-discovery that actually finds sitemaps.** It reads `Sitemap:` lines in robots.txt, falls back to common locations (`/sitemap.xml`, `/sitemap_index.xml`, `/wp-sitemap.xml`, `/sitemap.xml.gz`, `/sitemap.txt` and more), and retries `http://` sitemaps over `https://`, which is a common robots.txt mistake on big news sites.
- **Every format.** Sitemap indexes (any depth), urlsets, `.xml.gz` (with or without gzip headers), TXT sitemaps, RSS and Atom feeds, namespaced XML and CDATA.
- **Rich output.** `lastmod`, `changefreq`, `priority`, the sitemap each URL came from, image URLs, hreflang alternates, and Google News and video titles.
- **SEO checks built in.** It flags sitemap URLs that **robots.txt blocks** (a classic SEO bug), and with one switch checks the **HTTP status, redirects and final URL** of every page.
- **Filters that cut cost.** Include/exclude regex patterns, "modified after" date, same-host only, and per-site and total URL limits. You only pay for URLs you keep.
- **A bad site never breaks your run.** Dead sitemaps, HTML soft-404 pages, DNS errors and timeouts are recorded in a per-site `SUMMARY` and the run still succeeds. Retries use exponential backoff and honour `Retry-After`.
- **Fast and cheap.** Plain HTTP only (no browser). Over 1,000,000 URLs from apify.com in about 2 minutes on 512 MB in our tests.

### How to use it

1. Add one or more **websites or sitemap URLs** (`apify.com`, `https://example.com/sitemap_index.xml`, `https://site.com/sitemap.xml.gz`).
2. Optionally set **Max URLs** (your cost cap), filters, or turn on **Check HTTP status**.
3. Click **Start** and download the results as JSON, CSV, Excel or HTML, or use the API.

#### Input example

```json
{
  "startUrls": [{ "url": "https://apify.com" }, { "url": "https://www.nytimes.com" }],
  "maxUrls": 5000,
  "maxUrlsPerSite": 2500,
  "includeUrlPatterns": ["/blog/", "/2026/"],
  "excludeUrlPatterns": ["\\.pdf$"],
  "lastmodAfter": "2026-01-01",
  "checkStatus": false,
  "respectRobotsTxt": true
}
```

#### Output example

```json
{
  "url": "https://www.ebay.com/itm/800940000815",
  "lastmod": null,
  "changefreq": "daily",
  "priority": 1,
  "sitemapUrl": "https://www.ebay.com/lst/GTC-0_10.xml.gz",
  "sitemapType": "urlset",
  "site": "https://www.ebay.com/lst/VIS-0-index.xml",
  "host": "www.ebay.com",
  "blockedByRobotsTxt": false,
  "imageCount": 1,
  "imageUrls": ["https://secureir.ebaystatic.com/pictures/aw/pics/stockimage1.jpg"],
  "alternates": [],
  "news": null,
  "videoTitle": null
}
```

With **Check HTTP status** on, each item also has `statusCode`, `finalUrl` and `redirected`.

The run also stores a `SUMMARY` record in the key-value store. For each site it lists the sitemaps processed, the sitemaps that failed and why, whether robots.txt was found, the URL counts and the duration.

| Field | Description |
| --- | --- |
| `url` | Page URL listed in the sitemap |
| `lastmod` | Last modification date (ISO 8601) if the sitemap provides it |
| `changefreq`, `priority` | Sitemap hints, if present |
| `sitemapUrl`, `sitemapType` | The sitemap file the URL came from (`urlset`, `feed`, `text`) |
| `site`, `host` | Which of your inputs it belongs to and its hostname |
| `blockedByRobotsTxt` | `true` if robots.txt disallows the URL even though the sitemap lists it |
| `imageUrls`, `imageCount` | Images attached to the URL in an image sitemap |
| `alternates` | hreflang alternates `{ hreflang, url }` |
| `news`, `videoTitle` | Google News and video sitemap data |
| `statusCode`, `finalUrl`, `redirected` | Only with **Check HTTP status** |

### Run it from the API

Replace `YOUR_API_TOKEN` with your [Apify API token](https://console.apify.com/account/integrations). The call returns when the run finishes; then read the dataset.

#### cURL

```bash
curl "https://api.apify.com/v2/acts/steadyutil~sitemap-url-extractor-pro/runs?waitForFinish=300" \
  -X POST \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "startUrls": [{ "url": "https://apify.com" }],
    "maxUrls": 1000,
    "checkStatus": false
  }'
```

Fetch results (use the `defaultDatasetId` from the run):

```bash
curl "https://api.apify.com/v2/datasets/DATASET_ID/items?format=json" \
  -H "Authorization: Bearer YOUR_API_TOKEN"
```

#### JavaScript

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_API_TOKEN' });

const run = await client.actor('steadyutil/sitemap-url-extractor-pro').call({
    startUrls: [{ url: 'https://apify.com' }],
    maxUrls: 1000,
    checkStatus: false,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items.map((item) => item.url));
```

#### Python

```python
from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")

run = client.actor("steadyutil/sitemap-url-extractor-pro").call(run_input={
    "startUrls": [{"url": "https://apify.com"}],
    "maxUrls": 1000,
    "checkStatus": False,
})

items = client.dataset(run["defaultDatasetId"]).list_items().items
print([item["url"] for item in items])
```

The Actor is also available through the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp), so AI agents can discover and call it from the Store page.

### Related Actors

These SteadyUtil Actors combine well with this one:

- [Website Screenshot Pro](https://apify.com/steadyutil/website-screenshot-pro) — after you have the URL list, capture desktop and mobile screenshots (or PDFs) of every page, or of only the new and changed ones.
- [Website Tech Stack Detector & SEO Audit](https://apify.com/steadyutil/website-tech-stack-seo) — take the unique domains from the sitemap and detect each site's CMS, ecommerce platform, analytics and on-page SEO score.

### Automation recipes (Make, n8n, Zapier)

Each recipe starts the same way: in Apify Console, save this Actor as a Task with your input, then connect that Task from Make, n8n or Zapier (search for the Apify app). You can also call the API examples above from an HTTP module.

#### Weekly sitemap diff to Google Sheets

1. Create a Task with your domain, a `lastmodAfter` of seven days ago (or no date filter if you want the full list), and **Max URLs** as a cost cap.
2. In Apify Console, add a **Schedule** that runs the Task every Monday.
3. In Make, n8n or Zapier, trigger on a finished Apify run (or poll the dataset), then upsert each `url` into a Google Sheet. Keep `lastmod`, `statusCode` and `blockedByRobotsTxt` as columns.
4. Optional: add a filter so the sheet only gets URLs with `statusCode` 404 or `blockedByRobotsTxt` true, and email yourself the new rows.

#### New blog posts to Slack or email

Use the example Task **Extract blog URLs from a sitemap** (`includeUrlPatterns` such as `/blog/`), schedule it daily, and post any URL whose `lastmod` is after yesterday to Slack, email or a CMS draft queue.

#### Feed a crawler or screenshot run

Pass the dataset's `url` field into [Website Screenshot Pro](https://apify.com/steadyutil/website-screenshot-pro) or your own crawler. In n8n, this is two Apify nodes in a row: extract URLs, then screenshot (or audit) them.

### Pricing

This Actor uses **pay-per-event** pricing, so you pay only for results and never for compute time:

| Event | Price |
| --- | --- |
| `url-extracted`: one URL in your dataset | **$0.50 per 1,000 URLs** |
| `url-status-checked`: HTTP status check of one URL (optional) | **$1.00 per 1,000 URLs** |
| `residential-proxy-mb`: only if you turn on Apify **residential** proxy, per started MB downloaded through it | **$0.012 per MB** |

Sites without a sitemap, failed sitemaps and filtered-out URLs cost nothing. Without a proxy or with the default datacenter proxy there are no other charges. Residential proxy traffic costs us about $8/GB, so it's passed on at $0.012/MB, and a typical gzipped sitemap is 0.2–2 MB. Use **Max URLs** to cap the bill, for example `maxUrls: 10000` costs at most $5. Apify Store discounts for higher subscription plans apply automatically.

### FAQ

**How do I get all URLs of a website from its sitemap in 30 seconds?**
Paste the domain (for example `apify.com`) into **Websites or sitemap URLs**, set **Max URLs** if you want a cap, and click **Start**. The Actor finds robots.txt and the sitemaps, then writes every URL to the dataset. Download JSON, CSV or Excel, or use the API examples above. You don't need the exact sitemap URL.

**How do I find 404s and broken pages in a sitemap?**
Turn on **Check HTTP status** and, if you like, set **Max URLs**. Each result gets `statusCode`, `finalUrl` and `redirected`. Filter the dataset (or the **SEO issues** view) for 404, 5xx or unexpected redirects. Try the public example Task **Find 404s in your sitemap**.

**The site has no sitemap. What happens?**
The site is listed in the `SUMMARY` record with status `no-sitemap` and the reason. You aren't charged and the run doesn't fail. Some sites, such as python.org and docs.github.com, genuinely publish no sitemap.

**Does it crawl the website pages?**
No. It only downloads robots.txt and sitemap files, unless you enable **Check HTTP status**, which sends a lightweight HEAD request per URL.

**Can it handle huge sites?**
Yes. Nested indexes and millions of URLs are processed as a stream with de-duplication. Use **Max URLs** or **Max URLs per site** when you only need a sample.

**Do I need a proxy?**
Usually not, because sitemaps are meant for bots. If a site rate-limits datacenter IPs, enable Apify Proxy in the input.

**Can I get only new or changed pages?**
Yes. Set **Only URLs modified after** to a date and schedule the Actor daily or weekly. Pair this with the Google Sheets recipe above.

**Can I use it through the API or with AI agents?**
Yes. It works with the Apify API, the JS and Python clients, Make, Zapier, n8n and the Apify MCP server. Copy the examples in **Run it from the API**.

### Support and feedback

Sitemap URL Extractor Pro is built and maintained by **SteadyUtil**, a small publisher of reliable, low-cost web utilities on Apify. Found a site that doesn't work, or need a field we don't output yet? Open an issue in the **Issues** tab of this Actor (fastest, usually answered within 1–2 business days) or email **support@steadyutil.com**.

Other SteadyUtil Actors: [Website Screenshot Pro](https://apify.com/steadyutil/website-screenshot-pro) and [Website Tech Stack Detector & SEO Audit](https://apify.com/steadyutil/website-tech-stack-seo).

### Legal note

This Actor only reads robots.txt and sitemap files, which websites publish specifically so that automated agents can discover their public pages. It collects URLs and sitemap metadata only, never personal data. By default it respects robots.txt for the sitemap files it downloads. You are responsible for using the results in line with the target website's terms and applicable law.

# Actor input Schema

## `startUrls` (type: `array`):

Domains (e.g. apify.com) or direct sitemap URLs (sitemap.xml, sitemap_index.xml, .xml.gz, .txt, RSS/Atom). For a domain, sitemaps are discovered from robots.txt and common locations.

## `maxUrls` (type: `integer`):

Stop after this many URLs across all sites. 0 = no limit. You pay per URL, so this is your cost cap.

## `maxUrlsPerSite` (type: `integer`):

Stop after this many URLs for each start URL. 0 = no limit.

## `includeUrlPatterns` (type: `array`):

Optional regular expressions (or plain text). A URL is kept if it matches at least one. Example: /blog/

## `excludeUrlPatterns` (type: `array`):

Optional regular expressions (or plain text). Matching URLs are dropped. Example: .pdf$

## `lastmodAfter` (type: `string`):

Keep only URLs whose sitemap lastmod is on or after this date (YYYY-MM-DD). URLs without lastmod are dropped when this is set.

## `sameHostOnly` (type: `boolean`):

Drop URLs whose host differs from the start URL's host (www. is ignored).

## `includeExtensions` (type: `boolean`):

Adds image URLs, hreflang alternates and news/video titles found in the sitemap entry.

## `checkStatus` (type: `boolean`):

Sends a lightweight HEAD request to each URL and adds statusCode, finalUrl and redirected. Slower and billed as an extra event per URL. Great for finding 404s and redirects in your sitemap.

## `discoverSitemaps` (type: `boolean`):

If robots.txt lists no sitemap (or they are dead), try common locations such as /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml, /sitemap.xml.gz and /sitemap.txt.

## `respectRobotsTxt` (type: `boolean`):

Do not download sitemap files that robots.txt disallows.

## `flagRobotsBlockedUrls` (type: `boolean`):

Adds blockedByRobotsTxt=true to sitemap URLs that robots.txt disallows (a common SEO mistake).

## `maxSitemapDepth` (type: `integer`):

How many nested sitemap index levels to follow. 0 = unlimited.

## `maxSitemapsPerSite` (type: `integer`):

Safety limit on the number of sitemap files downloaded per site.

## `maxConcurrency` (type: `integer`):

Parallel sitemap downloads per site.

## `statusConcurrency` (type: `integer`):

Parallel HEAD requests when status checking is on.

## `requestTimeoutSecs` (type: `integer`):

Timeout for each HTTP request.

## `maxRetries` (type: `integer`):

Retries per request on network errors, 429 and 5xx (exponential backoff, honours Retry-After).

## `proxyConfiguration` (type: `object`):

Optional. Use Apify Proxy if a site blocks or rate-limits datacenter requests. Not needed for most sites. Residential proxy traffic is charged at $0.012 per MB (event residential-proxy-mb); the datacenter proxy has no extra charge.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://apify.com"
    },
    {
      "url": "https://crawlee.dev"
    }
  ],
  "maxUrls": 1000,
  "maxUrlsPerSite": 0,
  "includeUrlPatterns": [],
  "excludeUrlPatterns": [],
  "sameHostOnly": false,
  "includeExtensions": true,
  "checkStatus": false,
  "discoverSitemaps": true,
  "respectRobotsTxt": true,
  "flagRobotsBlockedUrls": true,
  "maxSitemapDepth": 5,
  "maxSitemapsPerSite": 1000,
  "maxConcurrency": 5,
  "statusConcurrency": 10,
  "requestTimeoutSecs": 30,
  "maxRetries": 3,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `urls` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://apify.com"
        },
        {
            "url": "https://crawlee.dev"
        }
    ],
    "maxUrls": 1000,
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("steadyutil/sitemap-url-extractor-pro").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [
        { "url": "https://apify.com" },
        { "url": "https://crawlee.dev" },
    ],
    "maxUrls": 1000,
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("steadyutil/sitemap-url-extractor-pro").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://apify.com"
    },
    {
      "url": "https://crawlee.dev"
    }
  ],
  "maxUrls": 1000,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call steadyutil/sitemap-url-extractor-pro --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,steadyutil/sitemap-url-extractor-pro"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Brz5YOrvLf5icVSUL/builds/bB4Z7M4JWZwdi61GE/openapi.json
