# Sitemap Extractor & Validator (`riprap-tools/sitemap-validator`) Actor

Extract URLs from XML sitemaps, gzip sitemaps and sitemap indexes, with lastmod, changefreq and priority.
Flags invalid or future dates, duplicates, bad priorities and over-long URLs.
Start from a sitemap or a site root. No page crawling, no browser, no AI. Pay only per URL returned.

- **URL**: https://apify.com/riprap-tools/sitemap-validator.md
- **Developed by:** [Riprap Tools](https://apify.com/riprap-tools) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.15 / 1,000 url results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap Extractor & Validator

### What it does

Sitemap Extractor & Validator by Riprap Tools turns XML sitemaps, gzip sitemaps and nested sitemap indexes into a distinct URL list with declared metadata and validation issues. Built for SEO specialists, migration and QA engineers, and developers building crawlers or monitors, it reads sitemaps—not destination pages.

### What you can use it for

- Build a pre-migration inventory of URLs declared in your sitemaps.
- Audit sitemap hygiene: duplicates, fragments, invalid dates and metadata values.
- Feed normalized URLs into your own crawler or monitoring pipeline.
- Inspect declared `lastmod` dates for freshness analysis and future-date errors.
- Identify cross-host entries by turning off `sameHostOnly`.

### How to use it

1. In Apify Console, open the Actor's Input tab and enter sitemap URLs or site roots. For a root, it reads robots.txt `Sitemap:` lines, falling back to `/sitemap.xml`.
2. Set URL and sitemap limits, then start the run. For API use, send the same input JSON through Apify's Actor run API; use the Actor's API tab for integration details.
3. Read the default dataset for URL records and the key-value store's `SUMMARY` record for counts, skips and truncation. Increase the defaults when you need more than a small sample.

### Input

| Field | Type | Default | What it does |
|---|---|---|---|
| `startUrls` | array of strings | `["https://www.gov.uk/sitemaps/sitemap_1.xml"]` | Required. Sitemap, index or site-root URLs; 1–100 HTTP(S) URLs. |
| `maxUrls` | integer | `10` | Maximum distinct URLs; range 1–100,000. |
| `maxSitemaps` | integer | `2` | Sitemap-file attempt limit, including index files, nested children and failed files; range 1–1,000. |
| `sameHostOnly` | boolean | `true` | Skip sitemap and result URLs whose hostname differs from the starting URL, except www-only differences (allowed for both files and rows). |
| `includeUrlParameters` | boolean | `false` | Include query-string result URLs when enabled; otherwise skip them. |

### Output

One dataset item per normalized URL. Normalization lowercases scheme and hostname, removes default ports and fragments, and preserves query order. Duplicates encountered within the processed entries are not emitted again; the retained item gets `duplicate`. Missing metadata is `null`; invalid metadata is retained for diagnosis.

| Field | Meaning |
|---|---|
| `url` | Original sitemap URL entry (trimmed). |
| `normalizedUrl` | URL used for deduplication. |
| `sourceSitemap` | Sitemap containing the entry. |
| `sitemapDepth` | Nesting depth; starting sitemap is 0. |
| `lastmod` | Declared modification date, or null. |
| `lastmodValid` | True/false for date validity; null when absent or empty. |
| `changefreq` | Declared change frequency, or null. |
| `priority` | Declared priority as text, or null. |
| `issues` | Array of validation codes; empty when none found. |

```json
{
  "url": "https://www.gov.uk/",
  "normalizedUrl": "https://www.gov.uk/",
  "sourceSitemap": "https://www.gov.uk/sitemaps/sitemap_1.xml",
  "sitemapDepth": 0,
  "lastmod": null,
  "lastmodValid": null,
  "changefreq": null,
  "priority": "0.5",
  "issues": []
}
```

#### Issue codes

| Code | Meaning |
|---|---|
| `invalid_lastmod` | Nonempty date could not be parsed in the accepted date format. |
| `future_lastmod` | Parsed date is later than the current time. |
| `duplicate` | Another processed entry has the same normalized URL. |
| `priority_out_of_range` | Nonempty priority is not numeric within 0–1. |
| `url_too_long` | Original URL exceeds 2,048 characters. |
| `has_fragment` | Original URL contains a fragment. |
| `invalid_changefreq` | Nonempty value is not always, hourly, daily, weekly, monthly, yearly or never. |
| `cross_host` | Hostname differs from the starting URL; only delivered when `sameHostOnly` is false. |
| `suspicious_lastmod` | Valid date belongs to a file with suspicious date patterns; heuristic, not proof. |
| `http_in_https_sitemap` | HTTP entry in an HTTPS sitemap file. |
| `www_mismatch` | Entry host and sitemap-file host differ only by the www prefix; retained even with `sameHostOnly: true`. |

#### Run summary

`SUMMARY` contains `issueCounts`, `sitemapsFetched` (successfully parsed sitemap/index files with a supported root), `sitemapsAttempted` (fetch attempts, including failures and indexes), `urlsEmitted`, `skipped` (counts by reason), `truncated`, `warnings`, `failedSitemaps`, `sitemapFindings` and `outcome`. A spending stop additionally sets `billingLimitReached`.

`urlsEmitted` counts only deliveries whose dataset push and charge both completed. A pushed row whose charge fails may be visible in the dataset but is not counted here; charges are never retried. `outcome` is `no_results` when no deliveries completed, otherwise `partial` when truncated or any sitemap failed, otherwise `complete`. Complete means the selected inventory was processed, not that every entry is valid: inspect skips and findings too.

`truncated: true` signals a URL, sitemap, depth, per-file or billing limit—not a complete inventory. Sitemap-budget warnings name `maxSitemaps` as the input to raise. Skips include `email_url`, `invalid_url`, `url_parameters`, `cross_host`, `cycle`, `depth_limit`, `sitemap_limit` and `file_url_limit`. Stable failure codes include `robots_denied`, `robots_unavailable`, `crawl_delay_limit`, `http_<status>`, `redirect_limit`, `invalid_root`, `html_instead_of_xml`, `doctype_forbidden`, `response_size_limit`, `decompressed_size_limit` and `unsafe_destination`; unknown fetch/parse errors use `fetch_or_parse_failed`.

`failedSitemaps` and `sitemapFindings` are lists of `{ "url": "https://example.org/sitemap.xml", "code": "invalid_root" }` objects, capped at 100 each. Report URLs omit queries/fragments, redact credentials and apparent email-like URLs, and are capped at 2,048 characters. Failure totals remain in `skipped` even when the report cap is reached.

Per-file findings: `invalid_namespace` (missing/wrong sitemaps.org namespace), `html_instead_of_xml` (body begins with HTML/HTML doctype; response headers are unavailable), `file_url_limit` (overflow truncation: the first 50,000 entries are still processed and delivered rows are charged; its skip count counts affected files, not omitted URLs), and `file_size_limit` (over the 50 MiB uncompressed safety cap, rejected). Existing DTD and size rejection remains in effect. This is body sniffing, not a Content-Type audit.

Lastmod findings use valid declared dates per file: at least 5 dates all identical (`lastmod_all_identical`) or all equal to the UTC run date (`lastmod_all_today`); at least 20 dates with at most 10% distinct calendar days (`lastmod_low_day_diversity`). Only valid-date rows in those files receive `suspicious_lastmod`; missing/invalid dates do not. These patterns do not establish actual page modification history. File findings inspect the parsed file, while row findings cover only processed entries.

### Pricing

Pay per event: one `result` per distinct URL delivered. Per 1,000 URLs: **$0.30 no discount / $0.25 Bronze / $0.20 Silver / $0.15 Gold**. No start fee. No charge for failed, skipped or duplicate URLs. A per-run spending cap stops delivery cleanly. All plans get the same features.

**10,000 URLs ≈ $3.00 at no-discount pricing.**

### Limits and what it does not do

- Reads sitemap XML only, plus robots.txt for discovery and access rules. No page crawling, JavaScript rendering, login or proxy/CAPTCHA bypass.
- Follows robots.txt for sitemap requests; unavailable robots.txt denies access. Crawl delays above 10 seconds are refused.
- Sequential requests, minimum one-second delay, 10-second socket timeout and at most five redirects. Redirect destinations are checked too.
- Private/internal and other non-public network destinations are refused; URL credentials are not accepted.
- Compressed responses and decompressed XML are each capped at 50 MiB. XML DTDs/entities are rejected.
- Processes at most 50,000 URL entries per file and sitemap depths 0–10. Run maxima: 100 starting URLs, 100,000 distinct results and 1,000 sitemap attempts.
- Query-string result URLs are skipped unless enabled. Extraction and deduplication use memory, not streaming output.

### Data and responsibility

You must have the right to process the sites you submit and comply with their terms and applicable data-protection rules. Sitemaps can contain URLs with people's names; names are not detected or removed. Apparent email-like URLs, including percent-encoded forms, are dropped. Output is limited to what the sitemap declares.

### FAQ

**Does it check that each URL is live?** No. It does not request destination pages or check their HTTP status.

**Does a valid `lastmod` prove the page changed?** No. It validates the declared date, not the page's actual modification history.

**Why did I get only 10 URLs or no child sitemap results?** Defaults are 10 URLs and two sitemap attempts. An index itself consumes an attempt, leaving one for a child; deeper indexes or additional children need a larger budget. Raise the limits and inspect `SUMMARY`.

### Support

Report reproducible problems in the Actor's Issues tab. Response within 14 days.

Contains public sector information licensed under the [Open Government Licence v3.0](https://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/). Source: GOV.UK; no affiliation or endorsement implied.

# Actor input Schema

## `startUrls` (type: `array`):

Sitemap XML URLs (plain or .gz), sitemap index URLs, or site root URLs. For a site root the Actor reads robots.txt "Sitemap:" lines and falls back to /sitemap.xml.

## `maxUrls` (type: `integer`):

Stop after this many distinct URLs. You are charged once per URL returned.

## `maxSitemaps` (type: `integer`):

Maximum sitemap fetch attempts: index files, their children and failed files each count. Default 2 permits an index and one child; raise this for larger inventories.

## `sameHostOnly` (type: `boolean`):

Skip sitemap entries and nested sitemaps on a different host from the starting URL. Hosts that differ only by a leading www. are kept, not skipped; a kept row is flagged www_mismatch when its host differs from its own sitemap file's host only by www.

## `includeUrlParameters` (type: `boolean`):

When off, URLs containing a query string are skipped and counted in the run summary.

## Actor input object example

```json
{
  "startUrls": [
    "https://www.gov.uk/sitemaps/sitemap_1.xml"
  ],
  "maxUrls": 10,
  "maxSitemaps": 2,
  "sameHostOnly": true,
  "includeUrlParameters": false
}
```

# Actor output Schema

## `results` (type: `string`):

Sitemap URLs with validation issues

## `summary` (type: `string`):

SUMMARY: issueCounts, sitemapsFetched, sitemapsAttempted, urlsEmitted, skipped, truncated, optional billingLimitReached, warnings, failedSitemaps (url/code), sitemapFindings (url/code), outcome (complete/partial/no_results). File report lists capped at 100; report URLs omit queries and fragments.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://www.gov.uk/sitemaps/sitemap_1.xml"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("riprap-tools/sitemap-validator").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": ["https://www.gov.uk/sitemaps/sitemap_1.xml"] }

# Run the Actor and wait for it to finish
run = client.actor("riprap-tools/sitemap-validator").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://www.gov.uk/sitemaps/sitemap_1.xml"
  ]
}' |
apify call riprap-tools/sitemap-validator --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,riprap-tools/sitemap-validator"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/amZRyVUXslEp2BB2S/builds/n80oUhOjfkUVFgZXT/openapi.json