# Sitemap Extractor & Audit - All URLs, Lastmod, Broken Links (`donrulezz/sitemap-extractor-audit`) Actor

Get every URL from a site's sitemaps (robots.txt discovery, indexes, .gz, text & RSS sitemaps) with lastmod, hreflang, images. Optional HTTP status check finds broken and redirected URLs; per-site health report. $0.30 per 1,000 URLs.

- **URL**: https://apify.com/donrulezz/sitemap-extractor-audit.md
- **Developed by:** [Ashish Agarwal](https://apify.com/donrulezz) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.22 / 1,000 url extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap Extractor & Audit – All URLs, Lastmod, Broken Links

Get **every URL a website lists in its sitemaps**, in one table: give a homepage and the actor finds the sitemaps itself (robots.txt `Sitemap:` lines, then the usual locations), follows sitemap indexes, unpacks `.xml.gz` files and also reads text and RSS/Atom sitemaps. Each URL comes with **lastmod, change frequency, priority, hreflang alternates and image/video/news counts**.

Turn on **Check HTTP status** and it also requests every URL, so you see which pages in your sitemap are **broken (4xx/5xx)** or **redirected**. Every site gets a free **health summary** of what's wrong with its sitemaps.

### What you get

**One row per URL**

| Field | Description |
|---|---|
| `url`, `sitemapUrl` | The page, and the sitemap file it was listed in |
| `lastmod`, `changefreq`, `priority` | As declared in the sitemap (lastmod normalised to ISO 8601 UTC) |
| `alternates` | hreflang versions of the page (`xhtml:link`) |
| `images`, `videos`, `news` | Image, video and Google News entries for the page |
| `statusCode`, `redirectsTo` | With status checks on: HTTP status and redirect target |

**One summary per site** (free): sitemaps read and failed, URL counts and a list of issues with an example each:

- sitemap missing, unreachable or not valid XML (URLs are still recovered from `<loc>` tags)
- URLs in the sitemap that are broken, redirect or don't answer
- URLs on another host, a mix of `http://` and `https://`
- duplicate URLs across sitemaps, `lastmod` dates in the future
- files over the protocol limits (50,000 URLs or 50 MB)

### Use cases

- **SEO audits:** find broken and redirected URLs you are asking Google to crawl, and stale or future lastmod dates.
- **Content inventories and migrations:** export every page of a site, with dates, before a redesign or CMS move.
- **Competitor research:** see how many product, blog or category pages a competitor has and what changed recently (filter by lastmod).
- **Feeding crawlers and AI pipelines:** give a scraper or RAG pipeline the exact list of pages to fetch, without crawling.
- **Monitoring:** schedule it and compare URL counts and issues over time.

### Input examples

Whole sites, with status checks:

```json
{
  "startUrls": ["https://www.sitemaps.org", "https://apify.com"],
  "checkStatus": true
}
```

Only blog posts changed since a date:

```json
{
  "startUrls": ["https://example.com"],
  "includePattern": "/blog/",
  "lastmodAfter": "2026-01-01"
}
```

### Output example

From a real run:

```json
{
  "type": "url",
  "url": "https://www.sitemaps.org/",
  "site": "https://www.sitemaps.org",
  "sitemapUrl": "https://www.sitemaps.org/sitemap.xml",
  "lastmod": "2016-11-21T00:00:00Z",
  "changefreq": null,
  "priority": null,
  "images": 0,
  "videos": 0,
  "news": false,
  "alternates": []
}
```

With *Check HTTP status* on, a redirected URL looks like this (real run, 26 Oct 2026: 27 of the first 80 URLs in
wordpress.org's sitemaps redirect):

```json
{
  "url": "https://wordpress.org/embed-image/",
  "statusCode": 301,
  "redirectsTo": "https://wordpress.org/files/2022/08/embed-image.png"
}
```

and the site summary lists `{"issue": "URL in sitemap redirects", "count": 27, "example": "https://wordpress.org/embed-image/"}`.

### Pricing

Pay per event: **$0.30 per 1,000 URLs extracted**, plus **$0.50 per 1,000 URLs** when *Check HTTP status* is on. Site summaries are free. Set a maximum spend per run in the run options; the actor stops cleanly when it is reached. Use *Max URLs per site* or the filters to keep large sites cheap.

### Tips

- Give the homepage, not a page deep in the site: discovery starts at `/robots.txt`.
- Huge sites (millions of URLs): set *Max URLs per site* or filter by path or lastmod.
- Status checks hit every URL: lower *Status checks in parallel* for small servers, or use a proxy for sites that block data-centre traffic.

### FAQ

**Does it crawl the site?** No. It reads only what the site publishes in its sitemaps, which is fast and polite. Pages missing from the sitemap won't appear.

**My sitemap is broken XML. Will it work?** Usually yes: the actor recovers URLs from `<loc>` tags and flags the file in the summary.

**Something wrong?** Open an issue on the actor's Issues tab and it will be fixed.

# Actor input Schema

## `startUrls` (type: `array`):

A website (https://example.com: sitemaps are found through robots.txt and the usual locations) or a direct sitemap URL (.xml, .xml.gz, .txt, RSS).

## `checkStatus` (type: `boolean`):

Request each URL (HEAD, GET fallback) to find broken (4xx/5xx) and redirected URLs listed in the sitemap. Charged separately per URL checked.

## `maxUrlsPerSite` (type: `integer`):

Stop after this many URLs for each site (0 = all).

## `includePattern` (type: `string`):

Optional. Keep only URLs matching this regular expression, e.g. /blog/ or .html$

## `excludePattern` (type: `string`):

Optional. Drop URLs matching this regular expression, e.g. /tag/|/author/

## `lastmodAfter` (type: `string`):

Optional date (YYYY-MM-DD). URLs without a lastmod are dropped when this is set.

## `sameDomainOnly` (type: `boolean`):

Drop URLs on other hosts (they are still reported in the site summary).

## `maxSitemapsPerSite` (type: `integer`):

Safety cap on how many sitemap files to read per site (indexes are followed up to 5 levels).

## `statusConcurrency` (type: `integer`):

Parallel HTTP status checks per site (1-30). Lower it for small servers.

## `maxConcurrency` (type: `integer`):

How many websites to process at the same time (1-20).

## `proxyConfiguration` (type: `object`):

Optional. Use Apify Proxy for sites that block data-centre traffic.

## Actor input object example

```json
{
  "startUrls": [
    "https://www.sitemaps.org",
    "https://apify.com/sitemap.xml"
  ],
  "checkStatus": false,
  "maxUrlsPerSite": 0,
  "sameDomainOnly": false,
  "maxSitemapsPerSite": 1000,
  "statusConcurrency": 10,
  "maxConcurrency": 5,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `urls` (type: `string`):

Every URL with lastmod, change frequency, priority, images/videos and (if checked) HTTP status.

## `summary` (type: `string`):

One row per site: sitemaps read, URL counts and issues.

## `results` (type: `string`):

Every item, including hreflang alternates and the sitemap list.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://www.sitemaps.org"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("donrulezz/sitemap-extractor-audit").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": ["https://www.sitemaps.org"] }

# Run the Actor and wait for it to finish
run = client.actor("donrulezz/sitemap-extractor-audit").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://www.sitemaps.org"
  ]
}' |
apify call donrulezz/sitemap-extractor-audit --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,donrulezz/sitemap-extractor-audit"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/XatsVTDhhO8bFrcIB/builds/aHEh26DKmZzeVCz8Q/openapi.json