# Sitemap Scraper - All URLs from XML Sitemaps + Status Check (`tinyrex/sitemap-scraper`) Actor

Every URL from a website's XML sitemaps (robots.txt discovery, indexes, .gz, text sitemaps) with lastmod, images, hreflang and news tags. Optional HTTP status/redirect check. Monitoring of new URLs.

- **URL**: https://apify.com/tinyrex/sitemap-scraper.md
- **Developed by:** [TinyRex](https://apify.com/tinyrex) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.16 / 1,000 urls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap Scraper: extract all URLs from any XML sitemap (and find broken links)

**Get every URL a website lists in its XML sitemaps** as clean JSON or CSV — page URL, `lastmod`, change frequency, priority, images, videos, hreflang alternates and Google News tags. Paste a domain: sitemaps are discovered from `robots.txt` and common paths, indexes are followed, `.xml.gz` and text sitemaps are supported. Optionally **check the HTTP status of every URL** to catch broken pages and redirects, or schedule runs to return **only new URLs**. **$0.20 per 1,000 URLs** (status checks $0.50/1k); sites without a sitemap cost nothing.

### What you get from the Sitemap Scraper

- **Complete URL inventory** — every `<loc>` from sitemap indexes (nested up to 5 levels), gzip and plain-text sitemaps, with lastmod, changefreq, priority, image/video extensions, hreflang and news tags.
- **Broken link & redirect audit (optional)** — status code, final URL after redirects, redirect chain, response time and content type for each URL, while respecting robots.txt.
- **New-URL monitoring** — turn on *Only new URLs* and each scheduled run returns only pages that were not in the sitemap before (new blog posts, products, job pages, competitor landings).

### Sample output (JSON)

```json
{
  "site": "crawlee.dev",
  "url": "https://crawlee.dev/python/blog",
  "lastmod": null,
  "changefreq": "weekly",
  "priority": 0.5,
  "sitemapUrl": "https://crawlee.dev/python/sitemap.xml",
  "imageCount": 0,
  "statusCode": 200,
  "finalStatusCode": 200,
  "finalUrl": "https://crawlee.dev/python/blog",
  "redirectChain": [],
  "responseTimeMs": 581,
  "contentType": "text/html"
}
```

Per-run reports: **`SITES`** (robots.txt found?, sitemaps read/failed, URLs saved, broken counts) and **`SUMMARY`** (totals). Download as **CSV, Excel or JSON**, or send to Sheets / Make / Zapier / n8n / webhooks / API.

### How to configure the Sitemap Scraper

1. Add **websites, sitemap URLs or robots.txt URLs**, one per line (`apify.com`, `https://crawlee.dev/sitemap.xml`).
2. Optional: include/exclude URL patterns (plain text or `/regex/`), *last modified within N days*, *max URLs per site*.
3. Optional: enable **Check HTTP status of every URL** for a broken-link / redirect audit.
4. Run.

#### Example input

```json
{
  "startUrls": ["https://apify.com", "https://crawlee.dev"],
  "maxUrlsPerSite": 300,
  "includeUrlPatterns": [],
  "checkStatus": true
}
```

| Setting | Default | Notes |
|---|---|---|
| startUrls | (required) | Domains, sitemap URLs or robots.txt URLs |
| maxUrlsPerSite | 500 | Cap per site (raise for full inventories) |
| checkStatus | false | HEAD→GET status check + redirect chain |
| includeUrlPatterns / excludeUrlPatterns | \[] | e.g. `/blog/`, `/regex/` |
| lastModifiedWithinDays | 0 | 0 = no lastmod filter |
| onlyNewUrls | false | Diff against previous run for monitoring |

### Common use cases

- **SEO audits** — all indexable URLs with lastmod, plus 4xx/5xx and unexpected redirects still listed in the sitemap.
- **Crawl seeding** — a complete URL list for a content or product scraper instead of link-crawling.
- **Competitor monitoring** — new product, blog or landing pages daily with *Only new URLs*.
- **Migrations** — export old and new sitemaps to build redirect maps.
- **AI / RAG** — the list of documentation pages to index.

### Pricing (failed = free)

Pay per event:

- **URL** (`url`): one charge per URL saved (~$0.20 / 1,000).
- **URL status check** (`url-status-check`): one charge per checked URL when the option is on (~$0.50 / 1,000).
- **Free:** sites without a sitemap, failed sitemaps, filtered-out, duplicate and already-seen URLs.

Set a max cost per run; the Actor stops gracefully when it is reached. See the *Pricing* tab for Bronze/Silver/Gold discounts.

### Limitations

- Only URLs the site lists in its sitemaps; it does not crawl links. Sites with no sitemap are reported in `SITES` (and are free).
- Datacenter IP blocks may need a proxy.
- Status checks use HEAD (fallback GET); some servers answer HEAD differently from browsers.

### FAQ — Sitemap Scraper / XML sitemap URL extractor

**How do I extract all URLs from a website's sitemap?**
Paste the domain (or the sitemap URL). The Actor reads `robots.txt` `Sitemap:` lines, then common paths (`/sitemap.xml`, `/sitemap_index.xml`, `/wp-sitemap.xml`, …), follows indexes, and writes every URL to the dataset.

**Where is sitemap.xml if it is not at /sitemap.xml?**
Many sites only list it in robots.txt, or use `/sitemap_index.xml`, `/wp-sitemap.xml` or a `.gz` file. Paste the bare domain and auto-discovery covers the usual locations; if the path is unusual, paste the sitemap URL directly.

**How can I find broken links in a sitemap?**
Turn on **Check HTTP status of every URL**. Each row gets `statusCode`, `finalStatusCode`, `finalUrl` and `redirectChain` so you can filter 4xx/5xx and redirect chains that should not be indexed.

**Does it support gzipped and sitemap indexes?**
Yes — nested indexes up to 5 levels, `.xml.gz`, plain-text sitemaps, plus image, video, hreflang and Google News extensions.

**How many URLs can it handle?**
Hundreds of thousands per site; files are streamed and capped by `maxUrlsPerSite`.

**Is scraping sitemaps legal?**
Sitemaps and robots.txt are published so automated tools can discover pages. The Actor reads those files (and, if enabled, one polite request per URL while respecting robots.txt). It does not log in or bypass protection. Use results in line with each site's terms.

**Does it work with AI agents / MCP?**
Yes. Connect to `https://mcp.apify.com?tools=tinyrex/sitemap-scraper` and ask e.g. *"List all blog URLs on crawlee.dev updated in the last 30 days"* or *"Find broken URLs in the sitemap of example.com"*.

📘 **Step-by-step guide (Python, JS, curl, MCP):** [How to extract all URLs from a website's sitemap (and find broken links)](https://emirmrkaljevic.github.io/tinyrex-data-tools/extract-all-urls-from-sitemap/)

### Related actors

- [Tech Stack Detector](https://apify.com/tinyrex/tech-stack-detector) — CMS, ecommerce, analytics and email provider at scale
- [Contact Details Extractor](https://apify.com/tinyrex/contact-details-extractor) — emails, phones and socials from domain lists
- [RSS Feed Scraper](https://apify.com/tinyrex/rss-feed-scraper) — RSS/Atom/JSON feeds as clean JSON/CSV
- [Shopify Products Scraper](https://apify.com/tinyrex/shopify-products-scraper) — products, prices, variants and stock

Built by **TinyRex**. Questions or feature requests? Open an issue on the Actor page.

# Actor input Schema

## `startUrls` (type: `array`):

Website URLs or domains (sitemaps are discovered from robots.txt and common paths like /sitemap.xml), direct sitemap URLs (XML, .xml.gz, text) or robots.txt URLs. Many sites per run.

## `maxUrlsPerSite` (type: `integer`):

Maximum URLs saved per website. 0 = no limit.

## `includeUrlPatterns` (type: `array`):

Keep only URLs containing one of these strings, or matching a /regex/ (e.g. /blog/ or //products//). Empty = all.

## `excludeUrlPatterns` (type: `array`):

Drop URLs containing one of these strings or matching a /regex/.

## `lastModifiedWithinDays` (type: `integer`):

Only URLs whose sitemap lastmod is within the last N days (URLs without lastmod are dropped). 0 = any.

## `includeImages` (type: `boolean`):

Include image and video sitemap entries for each URL.

## `includeAlternates` (type: `boolean`):

Include hreflang language alternates for each URL.

## `checkStatus` (type: `boolean`):

Request every URL (HEAD, falls back to GET) and add status code, final URL, redirect chain, response time and content type. Finds broken (4xx/5xx) and redirected sitemap URLs. Charged separately per checked URL.

## `respectRobotsTxt` (type: `boolean`):

Do not request URLs that robots.txt disallows (they are reported, not checked).

## `statusConcurrency` (type: `integer`):

Parallel requests per site during status checks. Keep it low to be polite.

## `onlyNewUrls` (type: `boolean`):

Return only URLs not returned by previous runs with the same state key. Schedule it to get new pages of a site.

## `stateKey` (type: `string`):

Name of the saved state for Only new URLs. Use different keys for different monitors.

## `maxSitemapsPerSite` (type: `integer`):

Safety cap on how many sitemap files are read per site.

## `maxConcurrency` (type: `integer`):

How many websites are processed in parallel.

## `proxyConfiguration` (type: `object`):

Optional proxy, only needed for sites that block datacenter IPs.

## Actor input object example

```json
{
  "startUrls": [
    "https://apify.com"
  ],
  "maxUrlsPerSite": 500,
  "includeUrlPatterns": [],
  "excludeUrlPatterns": [],
  "lastModifiedWithinDays": 0,
  "includeImages": true,
  "includeAlternates": true,
  "checkStatus": false,
  "respectRobotsTxt": true,
  "statusConcurrency": 3,
  "onlyNewUrls": false,
  "stateKey": "default",
  "maxSitemapsPerSite": 1000,
  "maxConcurrency": 3,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `urls` (type: `string`):

All URLs found in the sitemaps.

## `sites` (type: `string`):

Per site: sitemaps found and read, URLs, broken URLs, errors.

## `summary` (type: `string`):

Totals for the run.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://apify.com"
    ],
    "maxUrlsPerSite": 500
};

// Run the Actor and wait for it to finish
const run = await client.actor("tinyrex/sitemap-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://apify.com"],
    "maxUrlsPerSite": 500,
}

# Run the Actor and wait for it to finish
run = client.actor("tinyrex/sitemap-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://apify.com"
  ],
  "maxUrlsPerSite": 500
}' |
apify call tinyrex/sitemap-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,tinyrex/sitemap-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/fCceXLt4MN67bcAo0/builds/3htDckFLYdLdvA4qO/openapi.json