# Sitemap URL Extractor & XML Sitemap Checker (`somepineapple/sitemap-extractor-checker-tool`) Actor

Get every URL from a website's XML sitemaps and spot problems with them. Finds the sitemap from robots.txt or /sitemap.xml, follows sitemap indexes and .gz files, returns lastmod, changefreq and priority, and flags duplicates, bad dates and broken sitemaps.

- **URL**: https://apify.com/somepineapple/sitemap-extractor-checker-tool.md
- **Developed by:** [Ollie Wood](https://apify.com/somepineapple) (community)
- **Categories:** SEO tools, Developer tools, Marketing
- **Stats:** 1 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$1.60 / 1,000 site analyseds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap URL Extractor & XML Sitemap Checker

Give it a website or a sitemap URL. It finds the XML sitemaps, follows index files and compressed .gz sitemaps, and lists every page URL with its lastmod, changefreq and priority.

It also does the checks you'd do by hand in an SEO audit: duplicate URLs, missing or badly formatted lastmod dates, URLs pointing at another domain, files over the 50,000 URL limit, sitemaps that won't load, and robots.txt not mentioning the sitemap. Each site gets a pass, warn or fail so you can see the problem ones straight away.

### What it does

- Finds the sitemap from robots.txt, falls back to /sitemap.xml, or takes a sitemap link directly
- Follows sitemap index files and opens .gz sitemaps
- Lists every URL with its lastmod, changefreq and priority, plus the oldest and newest dates
- Flags duplicate URLs, bad or missing dates, URLs on other domains and sitemaps that won't load
- Gives each site a pass, warn or fail
- One bad address doesn't stop the rest of the list
- Lets you cap how many URLs come back while still counting them all
- Refuses private and internal addresses

### Use cases

- Get a full list of a site's pages to crawl, monitor or migrate.
- Technical SEO audits: broken sitemaps, duplicate URLs, stale or invalid dates.
- See how big a competitor's site is and what they've published lately.
- Feed the URL list into a scraper, link checker or your own pipeline.
- Compare sitemaps before and after a site migration.

### How to use

1. Add website addresses or direct sitemap links to URLs.
2. Optionally set maxUrlsPerSite to limit how many URLs come back per site (all of them are still counted).
3. Press Start. You get one row per site with its verdict, issues and URLs.

You can also call it from the Apify API, run it on a schedule, or hook it up to Make, Zapier or n8n. AI agents can use it through Apify's MCP server.

### Input

```json
{
  "urls": [
    "https://docs.python.org"
  ],
  "maxUrlsPerSite": 20
}
```

### Output

One dataset item per input with fields: `url`, `source`, `sitemapsCrawled`, `isIndex`, `totalUrls`, `uniqueUrls`, `duplicateUrls`, `urlsWithLastmod`, `oldestLastmod`, `newestLastmod`, `sitemaps`, `urls`, `urlsTruncated`, `verdict`, `issues`, `error`.

Here's a real result:

```json
{
  "url": "https://docs.python.org",
  "source": "robots.txt",
  "sitemapsCrawled": 1,
  "isIndex": false,
  "totalUrls": 7,
  "uniqueUrls": 7,
  "duplicateUrls": 0,
  "urlsWithLastmod": 0,
  "oldestLastmod": null,
  "newestLastmod": null,
  "sitemaps": [
    {
      "url": "https://docs.python.org/sitemap.xml",
      "type": "urlset",
      "entries": 7,
      "status": 200,
      "error": null
    }
  ],
  "urls": [
    {
      "loc": "https://docs.python.org/3.11/",
      "lastmod": null,
      "changefreq": "yearly",
      "priority": null,
      "sitemap": "https://docs.python.org/sitemap.xml"
    },
    {
      "loc": "https://docs.python.org/3.12/",
      "lastmod": null,
      "changefreq": "yearly",
      "priority": null,
      "sitemap": "https://docs.python.org/sitemap.xml"
    }
  ],
  "urlsTruncated": true,
  "verdict": "warn",
  "issues": [
    "7 URLs without <lastmod>"
  ],
  "error": null
}
```

### Pricing

Pay per event: **$0.0016 per site analysed** (successful results only;
failed items are returned with an `error` field and are not charged). Set a maximum charge per run in the run options.

So 100 sites cost about $0.16, and 1,000 cost about $1.60.

### FAQ

**Does it handle sitemap index files and .gz sitemaps?**

Yes, it follows indexes and unzips .gz files on its own.

**Why aren't image sitemap entries counted as duplicates?**

Image, video and news sitemaps are supposed to list pages that are already in the main sitemap, so flagging them would be wrong.

**What about really big sites?**

Each site gets about 20 seconds. On very large sites you get a summary of the sitemaps it got through in that time, and the result says it stopped early. Raise the run timeout if you need everything.

**Do I pay for sites with no sitemap?**

No. You get an error row for that site and no charge.

### Limitations

- Results reflect what the target returns at run time; they are informational and not a certification of any kind.
- Only public `http(s)` targets are processed; private and internal addresses are refused.
- Large inputs are capped per run (see input field descriptions).

### Other tools you might like

- [Website Tech Stack Detector](https://apify.com/somepineapple/website-tech-detector-tool): what a website is built with: CMS, frameworks, analytics, hosting.
- [Steam Reviews Analyzer](https://apify.com/somepineapple/steam-review-insights): why players like (or drop) a game, from its Steam reviews.

### Support

This Actor is built and maintained by an automated system run by its publisher. If something's wrong, open an
issue and it'll be looked at within 7 days. Please don't put passwords or personal details in an issue.

# Actor input Schema

## `urls` (type: `array`):

Site URLs (https://example.com) or direct sitemap URLs (https://example.com/sitemap.xml). Max 200 per run.

## `maxUrlsPerSite` (type: `integer`):

How many extracted URLs to include per site (0 = summary and audit only). All URLs are still counted.

## `maxSitemaps` (type: `integer`):

Upper limit of sitemap files fetched per site when following sitemap indexes.

## Actor input object example

```json
{
  "urls": [
    "https://docs.python.org"
  ],
  "maxUrlsPerSite": 100,
  "maxSitemaps": 20
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://docs.python.org"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("somepineapple/sitemap-extractor-checker-tool").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": ["https://docs.python.org"] }

# Run the Actor and wait for it to finish
run = client.actor("somepineapple/sitemap-extractor-checker-tool").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://docs.python.org"
  ]
}' |
apify call somepineapple/sitemap-extractor-checker-tool --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,somepineapple/sitemap-extractor-checker-tool"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/bfCPw7VBKn4f09GCL/builds/Je2uRxymO7G52jxFr/openapi.json