# Sitemap URL Extractor & 404 Checker — XML Sitemap Scraper (`factpipe/sitemap-url-extractor`) Actor

Extract every URL from any website's XML sitemaps: robots.txt discovery, nested sitemap indexes, lastmod and priority. Optional HTTP status check finds 404s and redirects. $0.30 per 1,000 URLs.

- **URL**: https://apify.com/factpipe/sitemap-url-extractor.md
- **Developed by:** [Neo B](https://apify.com/factpipe) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$0.30 / 1,000 url results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Sitemap URL Extractor & 404 Checker — XML Sitemap Scraper

Extract **every URL from a website's sitemaps** with one input: the site root. Discovers sitemaps via robots.txt and common paths, walks sitemap indexes, and returns flat records with `lastmod`, `changefreq`, and `priority` — optionally HEAD-checking each URL's HTTP status to find broken pages. Sitemaps are built for machines; this is the clean, reliable way to read them.

### Quick start

1. Click **Start** with the prefilled site `https://www.sitemaps.org`. It finishes in under a minute.
2. You get up to 200 URLs with lastmod and priority (tick **Check HTTP status** to also flag 404s and redirects).
3. That first run costs at most $0.06, well within Apify's free monthly credit. Then swap in your own input or schedule it.

### What you get

```json
{
  "site": "https://www.example.com",
  "found": true,
  "sitemap_url": "https://www.example.com/sitemap.xml",
  "url": "https://www.example.com/blog/post-42",
  "lastmod": "2026-08-30",
  "changefreq": "weekly",
  "priority": 0.8,
  "http_status": 200,
  "ok": true,
  "source_url": "https://www.example.com/sitemap.xml",
  "fetched_at": "2026-09-12T12:00:00.000Z"
}
```

### Use cases

- **SEO audits**: full URL inventory with freshness signals; find 404s and redirects in the sitemap (`checkStatus: true`).
- **Site migrations**: before/after URL inventories to verify nothing was lost.
- **Crawl seeding**: feed complete URL lists into other scrapers and agents.
- **Monitoring**: schedule it to detect new/removed pages on any site you track.

### Switching from another sitemap extractor

Same input idea (website URLs in, URL records out), plus: robots.txt discovery, sitemap-index traversal, `lastmod/changefreq/priority` preserved, optional live status checks, and `source_url`/`fetched_at` on every record. **Charged per URL row delivered — sites without sitemaps are free.**

### Input

| Field | Type | Notes |
|---|---|---|
| `websiteUrls` | string\[], required | Site roots or direct sitemap.xml URLs |
| `checkStatus` | boolean | HEAD-check each URL (default false) |
| `maxUrlsPerSite` | integer | Cap on charged rows per site (default 2000) |

### Pricing (pay per event)

| Event | Price | Meaning |
|---|---|---|
| `url-result` | **$0.30 per 1,000** ($0.0003 each) | One URL record delivered. **Sites with no discoverable sitemap and empty runs are never charged.** |

Example: 10,000 URLs cost **$3.00**. You only pay for delivered results.

### factpipe Website Audit Toolkit

Bulk technical checks for agencies, SEO teams and deliverability owners, all pay-per-result. Feed a list of sites in, get one flat row per page or domain out.

| Workflow | Actor | What it does | Price |
|---|---|---|---|
| Website audit | **Sitemap URL Extractor & 404 Checker** (this Actor) | Extract every URL from XML sitemaps and flag 404s and broken entries | $0.30/1k |
| Website audit | [Lighthouse Auditor](https://apify.com/factpipe/lighthouse-auditor) | Lighthouse scores and Core Web Vitals for many pages, mobile or desktop | $10/1k |
| Website audit | [Email Security Checker](https://apify.com/factpipe/email-security-checker) | SPF, DKIM, DMARC and MX audit for any list of domains | $3/1k |

**Use it together:**

- **Full-site audit:** extract URLs here, send the live pages to [Lighthouse Auditor](https://apify.com/factpipe/lighthouse-auditor) for Core Web Vitals, and check the domain's mail setup with [Email Security Checker](https://apify.com/factpipe/email-security-checker).

### FAQ

**How do I find 404 URLs listed in a sitemap?**
Enter the site root and set `checkStatus: true`. The Actor extracts every URL from the sitemaps and HEAD-checks each one, so 404s, other errors and redirects show up in the status fields of each record.

**How do I get all URLs of a website?**
Enter the site root. The Actor discovers sitemaps from robots.txt and common paths, follows nested sitemap indexes and returns one record per URL with lastmod, changefreq and priority.

**Can it find broken links (404s) in my sitemap?**
Yes. Set `checkStatus: true` and every URL gets an HTTP status and an `ok` flag, so 404s and redirects in the sitemap are easy to filter.

**What if a site has no sitemap?**
It is reported as not found and never charged.

**Can I call it from Python, JavaScript, Make, Zapier or an AI agent?**
Yes. Run it through the Apify API or official Python/JavaScript clients, connect it to Make, Zapier, n8n, Slack or Google Sheets via Apify integrations, or expose it to AI agents through the Apify MCP server. Input is small and output is deterministic flat JSON.

### Reliability

Deterministic XML parsing, retries with backoff, polite rate limiting, structured failure reporting, daily health checks and issue triage.

# Actor input Schema

## `websiteUrls` (type: `array`):

Site roots (sitemap discovered via robots.txt and common paths) or direct sitemap.xml URLs.

## `checkStatus` (type: `boolean`):

HEAD-checks every URL (slower). Status and ok fields stay null when off.

## `maxUrlsPerSite` (type: `integer`):

Cap on charged URL records per site.

## Actor input object example

```json
{
  "websiteUrls": [
    "https://www.sitemaps.org"
  ],
  "checkStatus": false,
  "maxUrlsPerSite": 200
}
```

# Actor output Schema

## `resultsDatasetUrl` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "websiteUrls": [
        "https://www.sitemaps.org"
    ],
    "maxUrlsPerSite": 200
};

// Run the Actor and wait for it to finish
const run = await client.actor("factpipe/sitemap-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "websiteUrls": ["https://www.sitemaps.org"],
    "maxUrlsPerSite": 200,
}

# Run the Actor and wait for it to finish
run = client.actor("factpipe/sitemap-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "websiteUrls": [
    "https://www.sitemaps.org"
  ],
  "maxUrlsPerSite": 200
}' |
apify call factpipe/sitemap-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,factpipe/sitemap-url-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/s1dbHjTJTSoeYEefz/builds/5diAuLtlXmn7KCdRd/openapi.json