# Sitemap URL Extractor – All URLs & Lastmod Filter (`martinvarelaa/sitemap-url-extractor`) Actor

Get every URL from a website's sitemap.xml, including indexes, gzip and robots.txt. Filter by lastmod and pay only for the URLs you keep: $1 per 1,000 URLs.

- **URL**: https://apify.com/martinvarelaa/sitemap-url-extractor.md
- **Developed by:** [Martin Varela](https://apify.com/martinvarelaa) (community)
- **Categories:** SEO tools, Developer tools, Agents
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.70 / 1,000 urls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap URL Extractor

Get every URL from a website's sitemap.xml at $1 per 1,000 URLs. Paste a domain, a sitemap URL or a robots.txt URL and get one dataset row per URL. Filter by last modified date and pay only for the URLs you keep.

The run finishes even when some inputs have no sitemap or refuse the request. Those inputs are explained in the key-value record `SITEMAPS_SUMMARY`, which is not charged.

### Who it is for

- **SEO teams** who need the full URL list of a site before an audit or a migration.
- **Scraping pipelines** that feed a crawler only the pages changed since the last run, using the `lastmod` filter.
- **Competitor monitoring**: see which pages a site published or updated this week.

### What you get

- Discover sitemaps from robots.txt, then from common paths such as `/sitemap.xml` and `/wp-sitemap.xml`.
- Read sitemap indexes, gzipped files (including gzip that is not named `.gz`), and plain-text URL lists.
- Keep `lastmod`, change frequency, and priority. Images and hreflang links are optional.
- Stop at `maxUrls`.
- Pass `lastmodSince` to keep only URLs modified on or after that day. Older URLs are not returned and are not charged. URLs with no `lastmod` stay in the result unless you also enable "Drop URLs without lastmod".

### Sample output

A real row from a run on oracle.com:

```json
{
  "url": "https://www.oracle.com/ai-data-platform/",
  "lastmod": "2026-09-04",
  "changefreq": "monthly",
  "priority": 0.9,
  "sitemapUrl": "https://www.oracle.com/sitemap-core.xml",
  "sitemapDepth": 1,
  "discoveredVia": "robots-txt",
  "sourceDomain": "oracle.com",
  "inputRef": "https://www.oracle.com",
  "scrapedAt": "2026-09-23T20:04:37Z"
}
```

`SITEMAPS_SUMMARY` lists, per input, whether the result was `ok`, `empty`, `no-sitemap`, `blocked`, `not-a-sitemap`, or `error`.

### How to use

1. Paste a domain, a sitemap.xml URL, or a robots.txt URL.
2. Leave Maximum URLs at 20 for a trial. The default cap is 1,000.
3. Run and export the dataset. Inputs with no sitemap are listed in `SITEMAPS_SUMMARY` and are not charged.

```json
{
  "startUrls": [{ "url": "https://apify.com" }],
  "maxUrls": 20
}
```

### Input

| Field | Default | What it does |
|---|---|---|
| `startUrls` | `https://apify.com` | Domains, sitemap files, or robots.txt files. |
| `maxUrls` | 1000 (prefill 20) | Stop after this many unique URLs. You pay per URL returned. |
| `discoverViaRobotsTxt` | true | Read Sitemap lines from robots.txt first. |
| `tryCommonPaths` | true | Also try `/sitemap.xml` and `/wp-sitemap.xml`. |
| `includeUrlPatterns` | empty | Keep URLs matching at least one regex. |
| `excludeUrlPatterns` | empty | Drop URLs matching any regex. |
| `includeImages` | false | Add image URLs from the image sitemap extension. |
| `includeAlternates` | false | Add hreflang alternates. |
| `lastmodSince` | empty | Drop URLs modified before this day. They are not charged. |
| `dropUrlsWithoutLastmod` | false | With a date filter, also drop URLs that have no `lastmod`. |
| `maxSitemapDepth` | 5 | How many sitemap-index levels to follow. |

### Pricing

You pay for URLs written to the dataset. Failed inputs and `SITEMAPS_SUMMARY` are free. The platform start event stays at $0.00005.

| Plan | Per URL | Per 1,000 URLs |
|---|---|---|
| Free | $0.001 | $1.00 |
| Bronze | $0.0009 | $0.90 |
| Silver | $0.0008 | $0.80 |
| Gold, Platinum, Diamond | $0.0007 | $0.70 |

### Use with AI agents (MCP)

Call the Actor `martinvarelaa/sitemap-url-extractor`. Pass `startUrls` and, when you only need recent pages, `lastmodSince`. Read the dataset for URLs and `SITEMAPS_SUMMARY` for inputs that produced nothing.

### Use via API

```bash
curl -X POST "https://api.apify.com/v2/acts/martinvarelaa~sitemap-url-extractor/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://apify.com"}],"maxUrls":20}'
```

### FAQ

#### What happens if a site has no sitemap?

The run still finishes. The summary says `no-sitemap` for that input and nothing is charged.

#### Can I get only the pages that changed recently?

Yes. Set `lastmodSince` to a date. URLs older than that are omitted and not charged. URLs with no `lastmod` stay unless you also enable "Drop URLs without lastmod".

#### Does it handle sitemap indexes and gzip?

Yes. It follows sitemap indexes up to `maxSitemapDepth` levels and reads gzipped files, even when the file name does not end in `.gz`.

#### Is the data legal to use?

This Actor extracts publicly available, non-personal data. You are responsible for using the data in compliance with applicable laws and with the target website's terms where applicable.

### Other actors

- [SEO Page Audit](https://apify.com/martinvarelaa/seo-page-audit) scores pages and checks sitemap URLs and AI-crawler rules.
- [Wayback Machine Scraper](https://apify.com/martinvarelaa/wayback-machine-snapshots) lists Wayback Machine snapshots and archived pages.
- [Clutch.co Scraper](https://apify.com/martinvarelaa/clutch-agency-scraper) builds agency lead lists with rates, team size and published contacts.

### Changelog

- 0.1 (2026-09-23): sitemap discovery from robots.txt and common paths, indexes and gzip, and the `lastmodSince` filter.

# Changelog

This Actor's version history is a separate document: https://apify.com/martinvarelaa/sitemap-url-extractor/changelog.md

# Actor input Schema

## `startUrls` (type: `array`):

Domains (https://example.com), specific sitemap URLs (https://example.com/sitemap.xml) or robots.txt URLs. For domains the Actor discovers sitemaps automatically.

## `maxUrls` (type: `integer`):

Stop after this many unique URLs across all inputs. You pay per URL returned.

## `discoverViaRobotsTxt` (type: `boolean`):

Read Sitemap: lines from robots.txt before trying common paths.

## `tryCommonPaths` (type: `boolean`):

Also check /sitemap.xml, /sitemap\_index.xml, /wp-sitemap.xml and similar when robots.txt has no Sitemap line.

## `includeUrlPatterns` (type: `array`):

Keep only URLs matching at least one pattern, e.g. "/blog/".

## `excludeUrlPatterns` (type: `array`):

Drop URLs matching any of these patterns.

## `includeImages` (type: `boolean`):

Add an images array when the sitemap lists images for a URL.

## `includeAlternates` (type: `boolean`):

Add an alternates array with hreflang links when the sitemap has them.

## `lastmodSince` (type: `string`):

URLs older than this date are not returned and are not charged. URLs without lastmod are kept unless 'Drop URLs without lastmod' is checked.

## `dropUrlsWithoutLastmod` (type: `boolean`):

Use together with the date filter. Undated URLs are omitted and not charged.

## `maxSitemapDepth` (type: `integer`):

How many levels of sitemap index to follow. 1 reads the root sitemap only.

## `proxyConfiguration` (type: `object`):

Not needed for most sites. Residential proxies are not used.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://apify.com"
    }
  ],
  "maxUrls": 20,
  "discoverViaRobotsTxt": true,
  "tryCommonPaths": true,
  "includeUrlPatterns": [],
  "excludeUrlPatterns": [],
  "includeImages": false,
  "includeAlternates": false,
  "dropUrlsWithoutLastmod": false,
  "maxSitemapDepth": 5,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `urls` (type: `string`):

One dataset item per URL found in the sitemaps.

## `summary` (type: `string`):

Per-input result: ok, empty, no sitemap, blocked, or error. This record is not charged.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://apify.com"
        }
    ],
    "maxUrls": 20
};

// Run the Actor and wait for it to finish
const run = await client.actor("martinvarelaa/sitemap-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://apify.com" }],
    "maxUrls": 20,
}

# Run the Actor and wait for it to finish
run = client.actor("martinvarelaa/sitemap-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://apify.com"
    }
  ],
  "maxUrls": 20
}' |
apify call martinvarelaa/sitemap-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,martinvarelaa/sitemap-url-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/jF5sbrOajWhVNOXc9/builds/HK7PVDJxpMUbLN3m8/openapi.json