# Sitemap URL Extractor & Status Checker (`pinkish_armada/sitemap-url-extractor`) Actor

Get every URL from a website's sitemaps: robots.txt auto-discovery, nested sitemap indexes, .gz, text, RSS and Atom sitemaps, lastmod/priority, hreflang. Optional HTTP status, redirect and noindex check.

- **URL**: https://apify.com/pinkish_armada/sitemap-url-extractor.md
- **Developed by:** [Phuc Tiet](https://apify.com/pinkish_armada) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.30 / 1,000 url extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap URL Extractor & Status Checker

Get **every URL a website publishes in its sitemaps** as a clean, deduplicated list — with `lastmod`, `changefreq`, `priority`, image counts and hreflang alternates. Optionally **check every URL** for HTTP status, redirects and `noindex` headers.

Enter a website (`example.com`) or a sitemap URL. The Actor finds the sitemaps for you, follows nested sitemap indexes, and handles gzipped, plain-text, RSS and Atom sitemaps.

### What you can use it for

- **SEO audits** – find URLs in your sitemap that return 404, redirect or are marked `noindex`.
- **Site migrations** – get before/after URL inventories and compare them.
- **Content inventories** – list every blog post or product page with its last-modified date.
- **Crawl seeding** – feed a complete URL list into another scraper, link checker or AI agent.
- **Monitoring** – schedule a run to see which pages were added or updated.

### Features

- **Auto-discovery** from `robots.txt` `Sitemap:` lines, then common locations (`/sitemap.xml`, `/sitemap_index.xml`, `/wp-sitemap.xml`, `/sitemap.xml.gz`, `/sitemap.txt`).
- **Nested sitemap indexes** followed automatically, with a safety cap on the number of files.
- **All sitemap formats**: XML `urlset`/`sitemapindex`, `.xml.gz`, plain-text, RSS 2.0 and Atom.
- **Image and hreflang extensions** parsed (`imageCount`, optional `images`, `alternates`).
- **Filters**: include/exclude by substring or `/regex/`. Entering `example.com/blog/` keeps only URLs under `/blog/`.
- **Optional status check**: one HEAD request per URL (GET fallback if HEAD is rejected), with status code, full redirect chain, final URL, `X-Robots-Tag` and response time, plus `issues` tags (`http-404`, `redirect`, `redirect-chain`, `noindex-header`, `unreachable`).
- **Robust**: retries with backoff (honours `Retry-After`), timeouts, gzip-bomb and size limits, and clear per-site error reports. One broken site never stops the others.

### Input

| Field | Default | Description |
|---|---|---|
| `startUrls` | – | Websites or sitemap URLs (required). |
| `maxUrlsPerSite` | 10000 | Stop after this many saved URLs per site (0 = no limit). Also caps your cost. |
| `includePatterns` / `excludePatterns` | \[] | Substrings or `/regex/`. |
| `checkStatus` | false | Request each URL and record status/redirects/noindex. |
| `restrictToInputPath` | true | For `example.com/blog/`, keep only URLs under `/blog/`. |
| `maxSitemapFiles` | 500 | Safety cap on sitemap files per site. |
| `includeImages` | false | Add image URLs to each row. |

Example:

```json
{
  "startUrls": ["https://www.sitemaps.org", "https://crawlee.dev/sitemap.xml"],
  "maxUrlsPerSite": 5000,
  "excludePatterns": ["/tag/"],
  "checkStatus": false
}
```

### Output

One row per unique URL:

```json
{
  "site": "https://www.sitemaps.org",
  "url": "https://www.sitemaps.org/protocol.html",
  "lastmod": "2022-12-15",
  "changefreq": null,
  "priority": null,
  "imageCount": 0,
  "sitemapUrl": "https://www.sitemaps.org/sitemap.xml",
  "sitemapType": "urlset"
}
```

With `checkStatus: true` each row also has `status`, `ok`, `finalUrl`, `redirectCount`, `redirectChain`, `responseTimeMs`, `xRobotsTag`, `error` and `issues`.

A run summary (sitemaps found, failed sitemap files with reasons, truncation) is saved as `OUTPUT` in the run's key-value store. You can download results as JSON, CSV, Excel or HTML, or read them through the Apify API.

### How much does it cost to extract sitemap URLs?

Pay per event. You only pay for URLs saved to your dataset, plus a tiny Actor start fee ($0.00005 per GB of memory):

- **URL extracted**: $0.0003 per URL ($0.30 per 1,000) when the status check is off.
- **URL checked**: $0.001 per URL ($1.00 per 1,000) when the status check is on.

Sites without a readable sitemap, failed sitemap files and filtered-out URLs are **not charged**. Platform usage is included in the price. To cap spending, use `maxUrlsPerSite` or the run's maximum cost setting; when the cap is reached, the run stops cleanly.

### Limitations and fair use

- It reads only sitemap files, which websites publish for machines to read. It does not crawl pages or render JavaScript. With `checkStatus` on, it sends one lightweight request per URL.
- Sites behind aggressive bot protection may block sitemap downloads. Such sites are reported in `OUTPUT` and not charged.
- Some sites list URLs in sitemaps that no longer exist. That is exactly what `checkStatus` is for.
- Please use status checks on sites you own or manage, or at a gentle concurrency.

### FAQ

**The site has no sitemap. What happens?** The run reports it in `OUTPUT` and the status message, and costs nothing beyond the Actor start.

**Can I pass several sites?** Yes, up to 1,000 per run. Each site is deduplicated separately.

**Does it support `sitemap.xml.gz`?** Yes. Gzip is detected by content, not file name.

### Use it from AI agents (MCP), the API, Make, Zapier and n8n

- **AI agents and MCP clients** (Claude, ChatGPT, Cursor, VS Code): add `https://mcp.apify.com?tools=pinkish_armada/sitemap-url-extractor` as an MCP server, and your agent can list every URL of a website from its sitemaps and check for 404s as a tool call.
- **API**: run it and get the results in one request:

```bash
curl -X POST "https://api.apify.com/v2/acts/pinkish_armada~sitemap-url-extractor/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":["https://www.sitemaps.org"],"maxUrlsPerSite":1000}'
```

- **No-code**: use the official Apify apps for **Make**, **Zapier** and **n8n** ("Run an Actor"), then send results to Google Sheets, Slack, Airtable or your database. You can also schedule runs in Apify Console and add a webhook or email integration.

### Related tools

- [Broken Link Checker](https://apify.com/pinkish_armada/broken-link-checker): crawl a website and report broken links, 404s and redirects
- [PDF to Markdown](https://apify.com/pinkish_armada/pdf-to-markdown): turn PDFs (including scans, via OCR) into Markdown, text and tables
- [Office Docs to Markdown](https://apify.com/pinkish_armada/office-to-markdown): turn DOCX, PPTX, XLSX, HTML and EPUB into clean Markdown for LLMs and RAG
- [Domain Lookup](https://apify.com/pinkish_armada/domain-intel): bulk RDAP WHOIS, DNS, SPF/DMARC and SSL-expiry checks for domains

# Actor input Schema

## `startUrls` (type: `array`):

Website roots (e.g. <code>example.com</code>) or direct sitemap URLs (<code>sitemap.xml</code>, sitemap index, <code>.xml.gz</code>, <code>.txt</code>, RSS/Atom). For a website root, sitemaps are discovered from robots.txt and common locations.

## `maxUrlsPerSite` (type: `integer`):

Stop after saving this many URLs for one site. 0 = no limit. You are charged per saved URL, so this also caps cost.

## `includePatterns` (type: `array`):

Keep only URLs that match at least one pattern. Plain text = case-insensitive substring (e.g. <code>/blog/</code>); wrap in slashes for a regular expression (e.g. <code>//product/\d+/</code>).

## `excludePatterns` (type: `array`):

Drop URLs that match any pattern (same syntax as above). Excluded URLs are not charged.

## `checkStatus` (type: `boolean`):

Request every extracted URL (HEAD, falling back to GET) and record status code, redirect chain, final URL, X-Robots-Tag and response time. Slower and priced as a 'checked URL' instead of an 'extracted URL'.

## `statusConcurrency` (type: `integer`):

Parallel requests when checking URL status. Keep it low for small sites.

## `maxSitemapFiles` (type: `integer`):

Safety cap on how many sitemap files (index + child sitemaps) are downloaded per site.

## `discoverFromRobotsTxt` (type: `boolean`):

For website roots, read <code>Sitemap:</code> lines from robots.txt before trying common paths.

## `restrictToInputPath` (type: `boolean`):

If you enter a website URL with a path (e.g. <code>example.com/blog/</code>), keep only URLs under that path. robots.txt sitemaps usually cover the whole host.

## `includeImages` (type: `boolean`):

Add the list of image URLs from image sitemap extensions to each row (the count is always included).

## `requestTimeoutSecs` (type: `integer`):

Timeout for each sitemap download or status request. Failed requests are retried with backoff.

## Actor input object example

```json
{
  "startUrls": [
    "https://www.sitemaps.org"
  ],
  "maxUrlsPerSite": 10000,
  "includePatterns": [],
  "excludePatterns": [],
  "checkStatus": false,
  "statusConcurrency": 10,
  "maxSitemapFiles": 500,
  "discoverFromRobotsTxt": true,
  "restrictToInputPath": true,
  "includeImages": false,
  "requestTimeoutSecs": 30
}
```

# Actor output Schema

## `results` (type: `string`):

All extracted URLs with lastmod, changefreq, priority, image count, source sitemap and (optionally) status/redirect/noindex check.

## `summary` (type: `string`):

Per-site summary: how sitemaps were discovered, sitemap files processed/failed with reasons, URLs found/saved, truncation.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://www.sitemaps.org"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("pinkish_armada/sitemap-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": ["https://www.sitemaps.org"] }

# Run the Actor and wait for it to finish
run = client.actor("pinkish_armada/sitemap-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://www.sitemaps.org"
  ]
}' |
apify call pinkish_armada/sitemap-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,pinkish_armada/sitemap-url-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/KMIMbda1mTf2Pzxej/builds/GMBDM12BQreqBijfn/openapi.json