# Sitemap Extractor: All URLs of a Website with Lastmod (`tidelane/sitemap-extractor`) Actor

Get every URL of a website from its XML sitemaps: finds them via robots.txt and common paths, follows sitemap indexes, unpacks .gz, reads RSS/Atom feeds. Returns lastmod, changefreq, priority, images and hreflang alternates; filter by pattern or date and optionally check each URL's HTTP status.

- **URL**: https://apify.com/tidelane/sitemap-extractor.md
- **Developed by:** [Tidelane Data](https://apify.com/tidelane) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.14 / 1,000 urls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap Extractor: All URLs of a Website with Lastmod

Get **every URL of a website** from its sitemaps in seconds, without crawling the site. Enter a domain and the Actor finds the sitemaps itself (robots.txt and the usual paths), follows sitemap indexes, unpacks `.xml.gz` files and returns each URL with its **last-modified date**, change frequency, priority, **images** and **hreflang language versions**.

- ✅ **Automatic discovery**: `Sitemap:` lines in robots.txt, then `/sitemap.xml`, `/sitemap_index.xml`, `/wp-sitemap.xml`, `/sitemap.xml.gz`, `/sitemap.txt`
- ✅ **Every format**: sitemap indexes (nested), urlsets, gzip, image and news sitemaps, hreflang alternates, RSS and Atom feeds, plain-text URL lists
- ✅ **Find new or changed pages**: keep only URLs changed on or after a date
- ✅ **Filters**: include or exclude URLs by regular expression (only `/blog/`, no `/tag/` …)
- ✅ **Optional status check**: HTTP status, redirect target and result (`ok`, `redirected`, `broken`, `blocked`) for every URL
- ✅ **Fast**: 35,000 URLs from 3 sites in 8 seconds in our test, de-duplicated across sitemaps
- ✅ **Fair billing**: you pay per URL returned; a site without sitemap costs nothing

Typical uses: SEO audits and migrations, feeding URL lists into scrapers or AI pipelines, watching competitors for new product or blog pages, checking that every sitemap URL still works.

### Input example

```json
{
    "urls": ["https://www.ikea.com/sitemaps/sitemap.xml", "https://wordpress.org/news/", "https://www.heise.de/rss/heise-atom.xml"],
    "includePattern": "",
    "excludePattern": "/tag/|/author/",
    "lastmodAfter": "2026-10-01",
    "checkStatus": false,
    "maxUrls": 100000
}
```

`maxUrls` applies to each website or sitemap in the list.

### Output example

Real output from a test run (`images` and `alternates` shortened):

```json
{
    "url": "https://www.ikea.com/ee/et/p/yxstaby-vitriinkarp-maengupuldile-valge-70620011/",
    "lastmod": "2026-10-20T10:48:56.790Z",
    "changefreq": "weekly",
    "priority": 1,
    "imageCount": 4,
    "images": ["https://www.ikea.com/ee/et/images/products/yxstaby-vitriinkarp-maengupuldile-valge__1505544_pe1008573_s5.jpg"],
    "alternates": [
        { "hreflang": "en-ee", "url": "https://www.ikea.com/ee/en/p/yxstaby-gaming-controller-display-box-white-70620011/" },
        { "hreflang": "ru-ee", "url": "https://www.ikea.com/ee/ru/p/yxstaby-vitrina-dlya-geympada-belyy-70620011/" }
    ],
    "newsTitle": null,
    "sitemap": "https://www.ikea.com/sitemaps/sitemap.xml",
    "site": "https://www.ikea.com/sitemaps/sitemap.xml"
}
```

With **Check HTTP status** on, each item also gets `status`, `category`, `finalUrl` and `statusError`. For feeds, `newsTitle` holds the article title and `lastmod` the publish or update date.

The `OUTPUT` record lists per input which sitemap files were read, how many URLs they gave and any problems (for example "No sitemap found" or "HTTP 500").

### Pricing

A small fixed price per URL returned, plus a price per URL when the status check is on.

Paid Apify plans pay less: 10% off on Starter, 20% on Scale and 30% on Business and higher plans.

### FAQ

**The site has no sitemap.** Then the run ends for that site with a note in `OUTPUT` and nothing is billed. Use our Broken Link Checker to crawl such sites.

**Why are there fewer URLs than in the sitemap?** URLs that appear in several sitemaps are returned once, and your filters or `maxUrls` may drop some. The `OUTPUT` record shows the details.

**Does `lastmodAfter` keep URLs without a date?** No. With a date filter only URLs that state a last-modified date on or after it are kept.

### More tools from Tidelane Data

- [Broken Link Checker](https://apify.com/tidelane/broken-link-checker): find 404s, redirects and dead links
- [SEO Audit](https://apify.com/tidelane/seo-audit): on-page SEO checks for any list of URLs
- [Website to Markdown](https://apify.com/tidelane/website-to-markdown): clean Markdown of web pages for AI, LLMs and RAG
- [Wayback Machine Scraper](https://apify.com/tidelane/wayback-machine): snapshots, archived URLs and site history

All our tools: [apify.com/tidelane](https://apify.com/tidelane)

***

#### Impressum

Levi Griesinger, Tidelane Data\
Besigheimer Straße 18\
74369, Löchgau, Deutschland\
E-Mail: LeviGriesinger@gmail.com

# Actor input Schema

## `urls` (type: `array`):

A website (https://example.com) to find its sitemaps automatically, or a direct sitemap, sitemap index, .xml.gz, RSS/Atom feed or text sitemap URL.

## `includePattern` (type: `string`):

Keep only URLs that match this regular expression, for example /blog/ or .pdf$.

## `excludePattern` (type: `string`):

Drop URLs that match this regular expression, for example /tag/|/author/.

## `lastmodAfter` (type: `string`):

Keep only URLs whose lastmod is on or after this date (YYYY-MM-DD). Great for finding new or updated pages. URLs without lastmod are dropped when this is set.

## `checkStatus` (type: `boolean`):

Send a light request to every URL and add status, result (ok, redirected, broken, blocked, error) and redirect target. Billed per checked URL.

## `maxUrls` (type: `integer`):

Stop after this many URLs for each website or sitemap in the input.

## `maxSitemaps` (type: `integer`):

Safety limit for huge sitemap indexes.

## `respectRobotsTxt` (type: `boolean`):

Do not read sitemaps or check URLs that robots.txt forbids.

## `timeoutSecs` (type: `integer`):

How long to wait for one sitemap file or status check.

## Actor input object example

```json
{
  "urls": [
    "https://apify.com"
  ],
  "checkStatus": false,
  "maxUrls": 100000,
  "maxSitemaps": 1000,
  "respectRobotsTxt": true,
  "timeoutSecs": 60
}
```

# Actor output Schema

## `results` (type: `string`):

All results of the run as dataset items.

## `summary` (type: `string`):

Summary per input with status (ok, error and the reason).

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://apify.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("tidelane/sitemap-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": ["https://apify.com"] }

# Run the Actor and wait for it to finish
run = client.actor("tidelane/sitemap-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://apify.com"
  ]
}' |
apify call tidelane/sitemap-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,tidelane/sitemap-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/8DuWvS5CQ65RxqfaH/builds/49a3vIszFS6AAhbUG/openapi.json