# Sitemap URL Extractor (`jadeite_geometry/sitemap-url-extractor`) Actor

Get every URL from a website's sitemap, with last-modified date, change frequency and priority. Follows nested sitemap indexes and gzip files. For SEO audits, crawl planning and content inventories.

- **URL**: https://apify.com/jadeite_geometry/sitemap-url-extractor.md
- **Developed by:** [Niklesh Kumar](https://apify.com/jadeite_geometry) (community)
- **Categories:** SEO tools, Developer tools
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.50 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap URL Extractor

Every URL listed in a website's sitemap files, with last-modified date, change frequency and priority, following nested sitemap indexes, for SEO audits, crawl planning and content inventories.

### What does Sitemap URL Extractor do?

A sitemap is a file that a website publishes to tell search engines which pages it has. Big sites split theirs into many files and list those files in a "sitemap index". This Actor takes a website address (or the address of a sitemap file), finds the sitemap files, reads all of them, and gives you one flat list of page addresses.

It reads plain XML sitemaps, sitemap indexes inside sitemap indexes, gzip-compressed files (`.xml.gz`) and plain-text sitemaps with one address per line. It reads only the sitemap files, never the pages themselves. It is built for SEO consultants, developers planning a crawl, and anyone who needs a content inventory of a site.

### What data does it return?

One row per page address. For an address that can't be read, one row that says why.

| Field | Example | Meaning |
|---|---|---|
| `inputUrl` | `https://www.gov.uk/` | The website or sitemap address from your input that this row belongs to. |
| `status` | `ok` | `ok` for a page address. Otherwise `no-sitemap`, `not-found`, `blocked-by-robots` or `error`, explained below. |
| `url` | `https://www.gov.uk/government/news/workshops-for-creative-industries-professionals` | The page address listed in the sitemap. Empty when the row explains a problem. |
| `host` | `www.gov.uk` | The host name of the page address (of your input, when the row explains a problem). |
| `lastmod` | `2026-06-01T12:58:18+01:00` | When the page was last modified, as the sitemap states it (ISO 8601). Empty when the sitemap gives no valid date. |
| `changefreq` | `weekly` | How often the page changes, as the sitemap states it: `always`, `hourly`, `daily`, `weekly`, `monthly`, `yearly` or `never`. Often empty. |
| `priority` | `0.29` | The page's priority from 0.0 to 1.0, as the sitemap states it. Often empty. |
| `sitemapUrl` | `https://www.gov.uk/sitemaps/sitemap_1.xml` | The sitemap file the address was listed in. |
| `message` | `HTTP 503: https://example.org/sitemap.xml` | Empty for a page address. Otherwise what went wrong, in plain English. |

The `status` of a row that explains a problem is one of these:

- `no-sitemap`: none of the usual places has a sitemap for the website.
- `not-found`: the sitemap file you gave doesn't exist.
- `blocked-by-robots`: the site's robots.txt doesn't allow reading it, or couldn't be read.
- `error`: anything else, for example a server error, a file that isn't a sitemap, or an address the Actor doesn't read.

A row that explains a problem counts as one result, like any other row. If part of a site's sitemap couldn't be read, you still get the URLs that were read, plus one such row at the end of that site, so a short list is never mistaken for a full one.

### How to use it

1. Open the Input tab and add the websites or sitemap addresses. The prefilled values work as they are.
2. Click Start.
3. Download the results as JSON, CSV or Excel, or read them through the Apify API.

| Input | Meaning |
|---|---|
| `urls` | Websites such as `https://example.com`, or sitemap files such as `https://example.com/sitemap.xml` or `.xml.gz`. For a website the Actor reads the `Sitemap:` lines in its robots.txt and, if there are none, tries `/sitemap.xml`, `/sitemap_index.xml` and `/sitemap.xml.gz`. A page address such as `https://example.com/blog/` is treated as its website; use `includePatterns` to keep one section. |
| `includePatterns` | Keep a URL only if it matches at least one of these patterns. Write `*` for "anything", for example `*/blog/*`. Not case sensitive, and the pattern can match any part of the URL. |
| `excludePatterns` | Drop a URL if it matches any of these patterns, for example `*.pdf` or `*/tag/*`. Applied after `includePatterns`. |
| `lastmodAfter` | Keep only URLs that changed on or after this date, written `YYYY-MM-DD`. URLs without a valid last-modified date are left out when you set it. A date written as only a year or a month counts as its first day. |
| `maxDepth` | How many levels of sitemap files to follow below the first one, from 1 to 3 (default 3). |
| `maxUrlsPerSite` | Stop reading a website after this many URLs (default 10,000). |
| `maxItems` | Stop after this many results in total (default 1,000), up to 100,000 per run. |

### How much does it cost?

You pay **$0.50 per 1,000 results**, with no monthly fee. New Apify accounts get free monthly credit, so you can try it at no cost.

### Example output

A page address from the sitemap of a large website:

```json
{
    "inputUrl": "https://www.gov.uk/",
    "status": "ok",
    "url": "https://www.gov.uk/government/news/workshops-for-creative-industries-professionals",
    "host": "www.gov.uk",
    "lastmod": "2026-06-01T12:58:18+01:00",
    "changefreq": null,
    "priority": 0.29,
    "sitemapUrl": "https://www.gov.uk/sitemaps/sitemap_1.xml",
    "message": null
}
```

A website with no sitemap:

```json
{
    "inputUrl": "https://example.org/",
    "status": "no-sitemap",
    "url": null,
    "host": "example.org",
    "lastmod": null,
    "changefreq": null,
    "priority": null,
    "sitemapUrl": null,
    "message": "No sitemap was found in robots.txt or at /sitemap.xml, /sitemap_index.xml or /sitemap.xml.gz."
}
```

### Data source and limits

- **Source:** the sitemap files that each website you list publishes for search engines, following the [Sitemaps protocol](https://www.sitemaps.org/protocol.html). Nothing else is read from the site except its robots.txt.
- **Robots.txt:** the Actor reads each host's robots.txt first and skips every address that it doesn't allow for this Actor or for crawlers in general. If robots.txt can't be read (it answers 401, 403 or a server error), nothing is read from that host. A `Crawl-delay` is kept, up to 30 seconds; a site that asks for more is skipped.
- **Rate:** about one request per second in total, and a retry with a pause after a 429 or server error.
- **Fresh:** the Actor reads the live files every time it runs.
- **Size limits:** a sitemap file may be up to 50 MB once unzipped (the protocol's limit), a website up to 500 sitemap files, and the Actor stops reading a website's remaining sitemap files after 5 of them fail.
- **Not read:** addresses with a username or password, ports other than 80, 443, 8080 and 8443, addresses on private or local networks, and social networks such as Facebook, LinkedIn, Instagram and TikTok.
- **No contact details:** a URL that contains an email address or a phone number is left out, so the output holds no personal contact data.

The data is public. This Actor doesn't collect personal data. It is an independent tool and isn't affiliated with, endorsed by or sponsored by any website you read or by any search engine.

### FAQ

**Can I run it on a schedule?** Yes. Use Apify Schedules to run it daily or hourly and send the results to Google Sheets, Slack, a webhook or your own API. Set `lastmodAfter` to find pages that changed since the last run.

**Why do I get fewer URLs than the site has pages?** A sitemap lists only what the site's owner chooses to list. Some sites leave out pages, and some publish no sitemap at all.

**Why are `lastmod`, `changefreq` and `priority` empty?** They are optional in the Sitemaps protocol and many sites don't publish them. The Actor shows exactly what the sitemap says and doesn't guess.

**What if a site blocks it?** You get one row with the status `blocked-by-robots` and the reason. The Actor respects robots.txt and doesn't try to get around it.

**Can I get more than 100,000 results in one run?** No, `maxItems` stops at 100,000. For a bigger site, run it several times with `includePatterns` or `lastmodAfter` to split the list.

**Is it legal?** It reads only public sitemap files, which sites publish so that crawlers can read them, at a polite rate and only where robots.txt allows it. It collects no personal data. Check that your own use of the URLs fits the site's terms.

# Actor input Schema

## `urls` (type: `array`):

Websites to read the sitemap of, such as https://example.com, or the address of a sitemap file, such as https://example.com/sitemap.xml or https://example.com/sitemap.xml.gz. For a website the Actor reads the Sitemap lines in its robots.txt and, if there are none, tries /sitemap.xml, /sitemap_index.xml and /sitemap.xml.gz.

## `includePatterns` (type: `array`):

Keep a URL only if it matches at least one of these patterns. Write \* for 'anything', for example */blog/* or https://example.com/shop/\*. Not case sensitive; the pattern can match any part of the URL. Leave empty to keep every URL.

## `excludePatterns` (type: `array`):

Drop a URL if it matches any of these patterns, with the same \* rule as above, for example \*.pdf or */tag/*. Applied after the keep-only patterns. Leave empty to drop nothing.

## `lastmodAfter` (type: `string`):

Keep only URLs whose sitemap entry says they changed on or after this date, written as YYYY-MM-DD, for example 2026-01-31. URLs without a valid last-modified date are left out when you set this. Leave empty to keep every URL.

## `maxDepth` (type: `integer`):

How many levels of sitemap files to follow below the first one. 1 reads the sitemap you give and the sitemap files it lists; 2 also reads the files those list; 3 is the most.

## `maxUrlsPerSite` (type: `integer`):

Stop reading a website after this many URLs. Useful to take a sample of a very large site, or to cap the cost when you list several sites.

## `maxItems` (type: `integer`):

Stop after saving this many results. You pay per saved result.

## Actor input object example

```json
{
  "urls": [
    "https://www.djangoproject.com/sitemap.xml",
    "https://www.gov.uk/",
    "https://www.nasa.gov/"
  ],
  "includePatterns": [],
  "excludePatterns": [],
  "lastmodAfter": "",
  "maxDepth": 3,
  "maxUrlsPerSite": 20,
  "maxItems": 100
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://www.djangoproject.com/sitemap.xml",
        "https://www.gov.uk/",
        "https://www.nasa.gov/"
    ],
    "maxDepth": 3,
    "maxUrlsPerSite": 20,
    "maxItems": 100
};

// Run the Actor and wait for it to finish
const run = await client.actor("jadeite_geometry/sitemap-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "urls": [
        "https://www.djangoproject.com/sitemap.xml",
        "https://www.gov.uk/",
        "https://www.nasa.gov/",
    ],
    "maxDepth": 3,
    "maxUrlsPerSite": 20,
    "maxItems": 100,
}

# Run the Actor and wait for it to finish
run = client.actor("jadeite_geometry/sitemap-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://www.djangoproject.com/sitemap.xml",
    "https://www.gov.uk/",
    "https://www.nasa.gov/"
  ],
  "maxDepth": 3,
  "maxUrlsPerSite": 20,
  "maxItems": 100
}' |
apify call jadeite_geometry/sitemap-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,jadeite_geometry/sitemap-url-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/UMLzNATiYJr6aa831/builds/6oaQVkyhbsorpaPLW/openapi.json