# Sitemap URL Extractor: Every Page of a Website, with Lastmod (`nebulacolors/sitemap-urls`) Actor

Lists every URL of a website from its sitemaps: robots.txt Sitemap lines or /sitemap.xml, nested sitemap indexes, gzip and text sitemaps, with lastmod, changefreq and priority. Filter by URL pattern or date, or crawl links when a site has no sitemap. $0.0003 per URL.

- **URL**: https://apify.com/nebulacolors/sitemap-urls.md
- **Developed by:** [Nebula Colors LLC](https://apify.com/nebulacolors) (community)
- **Categories:** SEO tools, Developer tools, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.21 / 1,000 urls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap URL Extractor: Every Page of a Website, with Lastmod

Give it a website and get back every URL its sitemaps list, with when each page last changed. It finds the sitemaps the way search engines do: the Sitemap lines in robots.txt, or /sitemap.xml and /sitemap_index.xml when there are none. It follows sitemap indexes down to the last file and reads gzipped and plain-text sitemaps. Filter by URL pattern or by date, or turn on the link crawl for a site with no sitemap. $0.0003 per URL, and failed sitemaps are free. It works for people and for AI agents through Apify's MCP server.

### What it's for

- Get the list of pages to feed a scraper, a crawler or a RAG index, without crawling the whole site.
- Watch a competitor or a news site: list only the pages changed in the last day or week.
- Check your own site: which pages your sitemaps list, which have no lastmod, which sitemap files are broken.
- Give an AI agent a site's page list so it can pick what to read.

### Input

| Field | What to put |
|---|---|
| Websites or sitemaps | One per line. Required. A website like `https://example.com` is read from the sitemaps its robots.txt lists. A sitemap address like `https://example.com/sitemap.xml` is read directly, with any sitemaps it points to. |
| Max URLs | Stop after this many URLs. 100 if you leave it out, in the form, through the API or from an AI agent. Raise it for a big site. |
| Only URLs matching | Optional patterns, like `https://example.com/blog/**`. `*` matches within one part of the path, `**` across parts. A pattern matches the whole URL, so write https:// and the host the way the sitemap does (with or without www), or start with `**` to match any host, like `**/blog/**`. |
| Skip URLs matching | Optional patterns to leave out, like `https://example.com/tag/**`. |
| Only changed since | Optional. A date like `2026-10-01`, or a span like `24 hours`, `7 days`, `2 weeks` or `1 month`. Only URLs with a lastmod that recent are kept. |
| Crawl links too | Off by default. On: after the sitemaps, it follows links on the site's pages to find pages the sitemaps miss, or to list a site that has no sitemap. |
| Max pages to crawl | With the crawl on: the most pages fetched, across all sites in the run. Default 100. Only fetched pages are listed, so this is also the most URLs the crawl adds. Raise it for a site with no sitemap. |
| Proxy | Off by default. Turn on Apify Proxy for a site that blocks cloud servers, or give your own proxy URLs. Residential proxies aren't available in this tool. |

Example:

```json
{
  "startUrls": ["https://techcrunch.com/news-sitemap.xml"],
  "newerThan": "24 hours"
}
```

### Output

One row per URL. This one came from a test run on October 16, 2026, on TechCrunch's news sitemap:

```json
{
  "type": "url",
  "url": "https://techcrunch.com/2026/10/15/tesla-reveals-spacex-powered-roadster-that-flies-and-falls-with-style/",
  "lastmod": "2026-10-16T01:17:43.000Z",
  "changefreq": null,
  "priority": null,
  "source": "sitemap",
  "sitemapUrl": "https://techcrunch.com/news-sitemap.xml",
  "listedAt": "2026-10-16T06:03:01.718Z"
}
```

- `lastmod`, `changefreq` and `priority` are what the sitemap says, or null when it doesn't say. Dates are ISO 8601 in UTC. In a news sitemap, `lastmod` is the article's publication date.
- `source` is `sitemap`, or `crawl` for a page found by following links.
- `sitemapUrl` is the sitemap file that listed the page, handy for finding a broken one.
- A sitemap that failed to load has a row with `type: "sitemap"` and an `error`. A website with no sitemap at all has a row with `type: "site"`. Error rows are free.

The run's `SUMMARY` record says how many URLs were listed, how many sitemaps were read and failed, how many URLs the filters left out, and why the run stopped.

### Price

- $0.0003 per URL listed, so $0.30 per 1,000 and $3 per 10,000.
- Free: duplicate URLs, URLs your filters leave out, and sitemaps that fail.
- You can set a maximum charge for a run. It stops when it's reached and keeps what it already listed.

### Good to know

- **Big sites.** Some sites list millions of URLs. Max URLs is there so a run never costs more than you expect; set it to what you need.
- **Only changed since** needs a lastmod in the sitemap. Sites that don't give one have all their URLs left out when it's set; leave it empty for those. When a sitemap index dates its sitemaps, the ones older than your date aren't read at all, which keeps the run short. To watch a news site, give its news sitemap directly (often listed in robots.txt, like `https://techcrunch.com/news-sitemap.xml`): it holds only the last two days. On a big site whose index dates every sitemap today, the date filter still has to read them all, which can take minutes; use the news sitemap, or give the run a longer timeout.
- **The link crawl** reads robots.txt and skips what it disallows, stays on the same site, and skips links to files like images and PDFs. Crawled pages have no lastmod.
- **Deep indexes.** Sitemap indexes are followed 5 levels deep, which covers every site we've seen.
- **Polite.** It paces requests to the same site and reads at most 4 files at a time.

### Use it from an AI agent

The Actor works through Apify's MCP server like any Store Actor. An agent can find it, read its input and call it. The MCP call waits 30 seconds by default (45 at most), and a site with a few hundred URLs usually finishes in that time. A bigger run takes longer, so the call returns the run's status and dataset ID, and the agent reads the rows with `get-dataset-items`.

### Questions or problems

Open an issue on this Actor's Issues tab. I read them.

# Actor input Schema

## `startUrls` (type: `array`):

One per line. A website, like https://example.com, is read from the sitemaps its robots.txt lists (or /sitemap.xml). A sitemap address, like https://example.com/sitemap.xml, is read directly, with any sitemaps it points to.

## `maxUrls` (type: `integer`):

Stop after this many URLs are listed. Each listed URL is one charged event. 100 if you leave it out, through the API or an AI agent too.

## `includeUrlGlobs` (type: `array`):

Optional. Keep only URLs that match one of these patterns, like https://example.com/blog/\*\*. \* matches within one part of the path, \*\* across parts. URLs left out aren't charged.

## `excludeUrlGlobs` (type: `array`):

Optional. Leave out URLs that match these patterns, like https://example.com/tag/\*\*. URLs left out aren't charged.

## `newerThan` (type: `string`):

Optional. A date like 2026-10-01, or a span like 7 days or 24 hours: only URLs whose sitemap lastmod is that recent. URLs without a lastmod are left out when this is set.

## `crawlLinks` (type: `boolean`):

Off: sitemaps only. On: after the sitemaps, also follow links on the site's pages to find pages the sitemaps miss, or a site with no sitemap. Crawled pages have no lastmod.

## `maxCrawlPages` (type: `integer`):

With Crawl links on: the most pages fetched to find links, across all sites in the run.

## `proxyConfiguration` (type: `object`):

Off by default. Turn on Apify Proxy for a site that blocks cloud servers.

## Actor input object example

```json
{
  "startUrls": [
    "https://blog.apify.com"
  ],
  "maxUrls": 100,
  "crawlLinks": false,
  "maxCrawlPages": 100,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `urls` (type: `string`):

All URL rows as JSON: url, lastmod, changefreq, priority, source (sitemap or crawl) and sitemapUrl. A sitemap that failed, or a site with no sitemap, has an error row instead.

## `summary` (type: `string`):

URLs listed, sitemaps read and failed, URLs from the crawl, URLs filtered out, and why the run stopped.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://blog.apify.com"
    ],
    "maxUrls": 100
};

// Run the Actor and wait for it to finish
const run = await client.actor("nebulacolors/sitemap-urls").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://blog.apify.com"],
    "maxUrls": 100,
}

# Run the Actor and wait for it to finish
run = client.actor("nebulacolors/sitemap-urls").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://blog.apify.com"
  ],
  "maxUrls": 100
}' |
apify call nebulacolors/sitemap-urls --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,nebulacolors/sitemap-urls"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/myw2CzvS2DaVMMEt6/builds/itngA08KDPkNkPyzi/openapi.json