# Sitemap URL Extractor: All Pages, Lastmod & Status (`tidyharvest/sitemap-urls-extractor`) Actor

Get every page URL from a website's sitemaps with lastmod dates, hreflang alternates, Google News titles and optional HTTP status checks. Auto-discovers sitemaps via robots.txt, follows indexes and .gz files. From $0.08 per 1,000 URLs.

- **URL**: https://apify.com/tidyharvest/sitemap-urls-extractor.md
- **Developed by:** [TidyHarvest](https://apify.com/tidyharvest) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 42.9% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.08 / 1,000 url extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap URL Extractor: All Pages, Lastmod & Status

Get **every page URL a website lists in its sitemaps**, with `lastmod` dates, change frequency, priority, **hreflang alternates**, image counts and **Google News** titles. Just type a domain: sitemaps are found automatically from **robots.txt** and common paths, and **sitemap indexes** (including `.xml.gz`) are followed all the way down.

Fast and cheap: a 3,000-URL site takes about a second.

### What you can do with it

- **Crawl planning:** feed a clean, de-duplicated URL list into a content scraper instead of crawling blindly.
- **Change monitoring:** schedule it with *Changed on or after* to get only new or updated pages (new blog posts, products, job pages, news articles).
- **SEO audits:** find broken links and redirects listed in the sitemap (*Check HTTP status*), pages with missing `lastmod`, and hreflang coverage.
- **Competitor research:** see how many product, blog or landing pages a competitor has, and how fast they publish.
- **News tracking:** read news sitemaps for headline + publication time, minutes after publishing.

### Input

| Field | What it does |
|---|---|
| `urls` | Domains, homepages, section URLs (`https://www.bbc.com/news`) or sitemap URLs. One per line. |
| `onlyUnderPath` | For a section URL, return only pages under that path (on by default). |
| `includeUrlPatterns` / `excludeUrlPatterns` | Keep or drop URLs containing these strings. |
| `lastmodAfter` | Only pages changed on or after this date. |
| `checkStatus` | Adds `statusCode`, `finalUrl` and `httpLastModified` per URL (broken links, redirects, server-side change date). |
| `maxUrlsPerSite`, `maxItems` | Caps to control cost. `maxItems` is shared fairly: every site gets an equal share, and sites with fewer URLs pass the rest on. |

Example:

```json
{ "urls": ["apify.com", "https://www.bbc.com/sitemaps/https-index-com-news.xml"], "includeUrlPatterns": ["/news/"], "lastmodAfter": "2026-10-01" }
```

### Output

One item per URL:

```json
{
  "url": "https://wordpress.org/news/2026/09/some-post/", "lastmod": "2026-09-01T12:37:54Z",
  "changefreq": "weekly", "priority": 0.8, "alternates": [{"hreflang": "de", "url": "https://..."}],
  "imageCount": 2, "newsTitle": null, "newsPublishedAt": null,
  "statusCode": 200, "finalUrl": "https://...", "httpLastModified": "Tue, 01 Sep 2026 12:37:54 GMT",
  "site": "https://wordpress.org", "sitemap": "https://wordpress.org/news/wp-sitemap-posts-post-1.xml"
}
```

`statusCode`/`finalUrl`/`httpLastModified` appear only with *Check HTTP status*. A per-site summary (URLs returned, sitemaps read, errors, and whether *Max URLs in total* cut a site short) is saved to the run's key-value store as `SUMMARY`.

### Pricing

Pay per URL, with lower prices on higher Apify plans: **$0.40 per 1,000 URLs** on the free plan, down to **$0.08 per 1,000** on Gold and above. Filters and caps apply before you are charged.
A run where no sitemap can be found fails clearly instead of silently returning an empty dataset. A filtered run with no new pages succeeds with 0 results.

### Limits and notes

- Reads only what a site publishes in its sitemaps. Pages not listed there won't appear (it doesn't crawl links).
- Up to 300,000 URLs and 1,000 sitemap files per site, and 60 MB per sitemap file (the sitemap spec allows 50 MB). Memory use stays flat whatever the site size.
- Section URLs: if a site lists nothing under the path in 100 sitemap files in a row, the search for that section stops there (noted in `SUMMARY`).
- With *Changed on or after*, URLs without a `lastmod` date are skipped.
- Public data only. Status checks send one light request per URL; use caps for very large sites.

# Actor input Schema

## `urls` (type: `array`):

One per line. A domain or homepage (`apify.com`, `https://www.bbc.com`) finds its sitemaps automatically; a section URL (`https://www.bbc.com/news`, `https://site.com/blog/`) returns only the pages under that path; a sitemap URL (`.../sitemap.xml`, `.xml.gz`, `.txt`, or a sitemap index) is read directly. Sitemap indexes are followed to every child sitemap.

## `onlyUnderPath` (type: `boolean`):

When you enter a section URL such as `https://www.bbc.com/news` or `https://site.com/blog/`, return only URLs under that path. Turn off to get the whole site for any URL you enter.

## `includeUrlPatterns` (type: `array`):

Keep a URL only if it contains one of these strings (e.g. `/blog/`, `/products/`). Empty = all.

## `excludeUrlPatterns` (type: `array`):

Drop URLs containing any of these (e.g. `/tag/`, `?page=`).

## `lastmodAfter` (type: `string`):

Only URLs whose sitemap `lastmod` is on or after this date: new or updated pages since your last run. URLs without a lastmod are skipped when this is set.

## `checkStatus` (type: `boolean`):

Request each URL (HEAD, GET fallback) and add `statusCode`, `finalUrl` and `httpLastModified` (the server's Last-Modified header), to find broken links and redirects listed in the sitemap. Slower on big sites.

## `maxUrlsPerSite` (type: `integer`):

0 or empty = no limit.

## `maxItems` (type: `integer`):

Hard cap on results for the whole run (also caps your cost), shared fairly between your sites: each gets an equal share and sites with fewer URLs pass the rest on. 0 or empty = no limit. Big sites list hundreds of thousands of URLs (apify.com ~300k), so set this for a first look.

## Actor input object example

```json
{
  "urls": [
    "apify.com",
    "https://wordpress.org/news/"
  ],
  "onlyUnderPath": true,
  "checkStatus": false,
  "maxItems": 200
}
```

# Actor output Schema

## `urls` (type: `string`):

One item per url.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "apify.com",
        "https://wordpress.org/news/"
    ],
    "maxItems": 200
};

// Run the Actor and wait for it to finish
const run = await client.actor("tidyharvest/sitemap-urls-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "urls": [
        "apify.com",
        "https://wordpress.org/news/",
    ],
    "maxItems": 200,
}

# Run the Actor and wait for it to finish
run = client.actor("tidyharvest/sitemap-urls-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "apify.com",
    "https://wordpress.org/news/"
  ],
  "maxItems": 200
}' |
apify call tidyharvest/sitemap-urls-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,tidyharvest/sitemap-urls-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/HcyADw96qcjmgc5To/builds/1HOSsBVSJv2V2F7q4/openapi.json