# Sitemap URL Extractor (all pages of a website, with lastmod) (`larkmoor/sitemap-url-extractor`) Actor

Get every URL a website publishes in its XML sitemaps: finds sitemaps via robots.txt or common paths, follows sitemap indexes and gzipped files, and returns each URL with lastmod, changefreq, priority, image/video counts and source sitemap. Filter by path keyword or modification date.

- **URL**: https://apify.com/larkmoor/sitemap-url-extractor.md
- **Developed by:** [Larkmoor Data](https://apify.com/larkmoor) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.26 / 1,000 urls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap URL Extractor (all pages of a website, with lastmod)

Get every URL a website publishes in its XML sitemaps: finds the sitemaps from robots.txt or common paths, follows sitemap indexes and gzipped files, and returns each URL with lastmod, changefreq, priority, image/video counts and the sitemap it came from. Filter by path keyword or modification date. The fastest way to list a site's pages or products for SEO audits, migrations, content inventories and as input for other scrapers.

### What you can use it for

- SEO audits and migrations: the complete list of a site's pages with last-modified dates
- Content inventories and competitor research: how many products, posts or docs a site has and what changed since a date
- Input for other scrapers: feed thousands of product or article URLs straight into a crawler
- Monitoring: run weekly and diff the URL lists to catch new or removed pages

### Why this one

- Finds the sitemaps itself: robots.txt entries, sitemap.xml, sitemap_index.xml, wp-sitemap.xml and other common paths; follows nested indexes and .gz files
- Keeps lastmod, changefreq, priority, image/video counts and the source sitemap for every URL
- Path keyword and modified-since filters, so a 2-million-URL site can be narrowed to this month's product pages
- Pay only for results: **$0.30 per 1,000 URLs**. No subscription.

### Input

- **Websites or sitemap URLs** (`websites`): A domain (sitemaps are found via robots.txt and common paths) or a direct sitemap URL.
- **URL must contain (any)** (`includeKeywords`): e.g. /blog/, /product/
- **URL must not contain (any)** (`excludeKeywords`)
- **Modified since (YYYY-MM-DD)** (`modifiedSince`): Keep only URLs whose lastmod is on or after this date (URLs without lastmod are kept).
- **Max URLs per site** (`maxUrlsPerSite`) Default: `100000`.
- **Max URLs total** (`maxResults`) Default: `100000`.

### Output

One row per url in the dataset (JSON, CSV, Excel, or through the API). Sample:

```json
[
 {
  "site": "apify.com",
  "url": "https://apify.com/",
  "lastmod": null,
  "changefreq": null,
  "priority": null,
  "images": null,
  "videos": null,
  "news": null,
  "alternates": null,
  "path": "/",
  "sitemap": "https://apify.com/sitemap/pages.xml"
 },
 {
  "site": "apify.com",
  "url": "https://apify.com/about",
  "lastmod": null,
  "changefreq": null,
  "priority": null,
  "images": null,
  "videos": null,
  "news": null,
  "alternates": null,
  "path": "/about",
  "sitemap": "https://apify.com/sitemap/pages.xml"
 }
]
```

### Tips

- Give a direct sitemap URL to skip discovery (e.g. https://example.com/sitemap-products.xml).
- Sites with hundreds of sitemap files can take a few minutes; set maxUrlsPerSite to stop early.

### Integrations and API

Run it on a schedule, trigger it from the API or a webhook, and send results to Google Sheets, Zapier, Make, Slack, a database or your own code with the platform's standard integrations.

### Data and fair use

This tool reads only public sources (official APIs, public feeds or pages the publisher makes available to everyone) and respects robots.txt and rate limits. You are responsible for using the data in line with the source's terms and applicable law.

### How much it costs

You pay per result (pay-per-event), nothing per month and nothing when a run returns nothing. The price is **$0.30 per 1,000 URLs** on the Free plan, with automatic discounts on paid plans (Bronze −5%, Silver −10%, Gold −15%, Platinum −20%).

Worked examples:

- 100 URLs → **$0.03**
- 1,000 URLs → **$0.30**
- 10,000 URLs → **$3.00**
- Plus $0.001 per run start.

Platform usage (compute, storage) is included in the price: the actor runs at a small memory setting by default, so a typical run of 1,000 URLs finishes in well under a minute.

### FAQ

**What if a site has no sitemap?**

The actor checks robots.txt and the usual sitemap paths. If none exists (or it is empty or broken), it falls back to a polite link crawl from the homepage: up to `crawlMaxPages` pages, same site only, robots.txt respected. Those rows are marked `source: "crawl"` and carry the HTTP `status`. Turn `crawlIfNoSitemap` off to get sitemap URLs only.

**Why are some fields empty?**

A field is `null` when the source does not publish it for that row. The output keeps every column so your spreadsheet or integration always sees the same shape.

**Can I run it on a schedule?**

Yes. Use Apify Schedules (daily, hourly, cron) and connect the run to Make, Zapier, n8n, Google Sheets, a webhook or the API. Each run writes a dataset you can download as JSON, CSV or Excel.

**Is there a free trial?**

The Apify Free plan includes $5 of monthly platform credit, enough to try this actor on a few hundred results at no cost. There is no subscription for the actor itself.

**Can an AI agent use this?**

Yes. The actor is available through Apify's MCP server and the API, so LLM agents (Claude, ChatGPT, LangChain, CrewAI) can call it as a tool with the same input.

**What happens if something fails?**

Runs log every skipped item with a reason. If the source is down, the actor stops cleanly and you are only charged for the results it returned.

### Changelog

- **2026-10-21** — No sitemap? No problem: when a site has no usable sitemap, the actor now finds its pages by following links from the homepage (same site only, robots.txt and nofollow respected, `crawlIfNoSitemap`, `crawlMaxPages`). Those rows carry `source: "crawl"` and the HTTP `status`. URLs listed in several sitemaps are now returned once.
- **2026-10-18** — Lower price with Bronze/Silver/Gold discounts; smaller default memory (cheaper runs); clean table view of results; FAQ and cost examples; default limits lowered to 10,000 per site so a first run never writes 100,000 rows by surprise.

### Support

Found a problem or need a field added? Open an issue on the Issues tab; it is read and answered.

# Actor input Schema

## `websites` (type: `array`):

A domain (sitemaps are found via robots.txt and common paths) or a direct sitemap URL.

## `includeKeywords` (type: `array`):

e.g. /blog/, /product/

## `excludeKeywords` (type: `array`):

URL must not contain (any).

## `modifiedSince` (type: `string`):

Keep only URLs whose lastmod is on or after this date (URLs without lastmod are kept).

## `maxUrlsPerSite` (type: `integer`):

Stop after this many URLs from one site. Default: 10000.

## `maxResults` (type: `integer`):

Hard cap on URLs for the whole run (you pay per URL). Default: 10000.

## `crawlIfNoSitemap` (type: `boolean`):

When a site has no usable sitemap, find its pages by following links from the homepage instead (same site only, robots.txt respected). Rows get source = 'crawl' and the HTTP status.

## `crawlMaxPages` (type: `integer`):

Page limit per site for the link-crawl fallback.

## Actor input object example

```json
{
  "websites": [
    "apify.com"
  ],
  "maxUrlsPerSite": 10000,
  "maxResults": 10000,
  "crawlIfNoSitemap": true,
  "crawlMaxPages": 200
}
```

# Actor output Schema

## `results` (type: `string`):

All results as JSON items in the run's default dataset (also downloadable as CSV/Excel from the dataset).

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "websites": [
        "apify.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("larkmoor/sitemap-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "websites": ["apify.com"] }

# Run the Actor and wait for it to finish
run = client.actor("larkmoor/sitemap-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "websites": [
    "apify.com"
  ]
}' |
apify call larkmoor/sitemap-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,larkmoor/sitemap-url-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/ldZOyuUzgS8wdkQVa/builds/hE7jjSitr5AYpjhHP/openapi.json