# Sitemap Crawler & Broken Link Checker (`winterlock/sitemap-link-checker`) Actor

Discover every URL a site publishes through robots.txt and its sitemaps, check that each one still responds, and find broken links inside its pages. Handles sitemap indexes, gzipped sitemaps and servers that reject HEAD requests.

- **URL**: https://apify.com/winterlock/sitemap-link-checker.md
- **Developed by:** [Alex Burton](https://apify.com/winterlock) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.50 / 1,000 sitemap url listed, or skipped under robots.txts

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap Crawler & Broken Link Checker

Find every URL a site publishes, confirm each one still responds, and catch broken links inside its pages. No API key, no account, no third-party service.

| Mode | What you get |
|---|---|
| **List sitemap URLs** | Every URL from the site's sitemaps, with last-modified date, change frequency and priority |
| **Check URL health** | HTTP status, redirect chain, final URL, content type, size and response time for each URL |
| **Find broken links on pages** | Every link on each page, with its status and which page it was found on |

### Why use this one

- **HEAD first, GET on rejection.** Many servers answer a HEAD request with 405, 501 or 403 while serving the same URL fine over GET. Checkers that only send HEAD report those pages as broken. This one retries with GET and tells you which method produced the answer, so a clean report means a clean site. It is the single most common bug in this category, and it is covered by tests here.
- **It finds the sitemap the way a search engine does.** robots.txt first for its `Sitemap:` lines, then `/sitemap.xml` and `/sitemap_index.xml`. A sitemap index is followed into its children, and gzipped sitemaps are unpacked.
- **Redirects are reported, not hidden.** A URL that quietly 301s somewhere else is not broken, but you usually want to know. You get `redirected`, `redirectCount` and `finalUrl` on every row.
- **Links are de-duplicated across pages.** A link in your site-wide footer is checked once per run, not once per page, which keeps both the cost and the load on the target site down.
- **robots.txt is honoured**, using Python's own robots parser. Disallowed URLs come back marked `skipped`, never silently dropped and never counted as broken.

### How to use it

1. Choose **What to do**.
2. Enter a **Site** (a domain is enough) or paste exact **URLs to check**.
3. Click **Start**, then download as JSON, CSV, Excel or HTML.

#### Example input: list everything a site publishes

```json
{
  "mode": "sitemapUrls",
  "sites": ["example.com"],
  "maxUrlsPerSite": 5000
}
```

#### Example input: health-check every page in the sitemap

```json
{
  "mode": "checkUrls",
  "sites": ["example.com"],
  "maxUrlsPerSite": 1000
}
```

#### Example input: find broken links on specific pages

```json
{
  "mode": "brokenLinks",
  "urls": ["https://example.com/", "https://example.com/blog"],
  "maxLinksPerPage": 200,
  "internalLinksOnly": false
}
```

To find broken links across a whole site, give `sites` instead of `urls` and the Actor will scan the first `maxPagesToScan` pages from the sitemap (25 unless you raise it).

### Output

**Sitemap URL**

```json
{
  "type": "sitemapUrl",
  "site": "https://example.com",
  "url": "https://example.com/pricing",
  "lastModified": "2026-09-01T00:00:00Z",
  "changeFrequency": "weekly",
  "priority": 0.8,
  "foundIn": "https://example.com/sitemap.xml"
}
```

**URL check**

```json
{
  "type": "urlCheck",
  "url": "https://example.com/old-page",
  "status": 301,
  "ok": true,
  "broken": false,
  "redirected": true,
  "redirectCount": 1,
  "finalUrl": "https://example.com/new-page",
  "contentType": "text/html",
  "responseMs": 184,
  "method": "HEAD",
  "error": null
}
```

**Broken link** adds `foundOn` and `internal`:

```json
{
  "type": "link",
  "url": "https://partner.example/gone",
  "foundOn": "https://example.com/blog",
  "internal": false,
  "status": 404,
  "broken": true,
  "method": "GET"
}
```

Filter on `broken: true` to get just the problems.

### Robots and identity

Requests carry a normal browser User-Agent, and `robots.txt` is evaluated against
the `*` rules. If you run this against a site you do not own, check its rules
first — the Actor honours `Disallow` but it does not announce itself as a bot.

### Pricing

| Result | Price |
|---|---|
| Sitemap URL listed, or a URL skipped under robots.txt | $0.50 per 1,000 |
| URL checked | $2.00 per 1,000 |
| Link on a page checked | $2.00 per 1,000 |

Checks cost more than listings because each one is a live request, sometimes two when a server rejects HEAD. Set a maximum spend per run in the run options and the Actor stops cleanly when it is reached.

### Limits to know about

- **This Actor sends real traffic to a real website.** Default concurrency is 4 on purpose. Please raise it only against a site you own, and use the URL cap on large sites.
- **Sitemap indexes are followed three levels deep** and up to 50 sitemap files per run, which covers any normal site and stops a misconfigured loop.
- **`lastmod` is only as good as the site makes it.** Many content systems write today's date on every URL, so treat it as a hint.
- **A 403 can mean a bot block, not a broken page.** Sites behind an aggressive firewall may reject datacenter traffic; turn on Apify Proxy if a whole site returns 403.
- **Links are read from the HTML as served.** Links a page builds in JavaScript after loading are not visible to this Actor.
- **Response times include network latency** from Apify's servers and are useful for comparison between URLs, not as a benchmark of your server.

### FAQ

**Do I need to own the site?** No for checking public URLs, but be considerate: this is live traffic. For broken-link scans across a large site, run it against your own property.

**Why is a URL marked skipped?** The site's robots.txt disallows it. Turn off *Respect robots.txt* only if the site is yours.

**Why does `method` say GET on some rows?** That URL's server rejected the HEAD request, so the Actor retried with GET to get a trustworthy answer.

**Can I schedule this?** Yes. Schedule it in Apify Console and connect the dataset to Slack or email, filtered to `broken: true`, for an alert when something breaks.

**Is this legal?** The Actor reads public pages and honours robots.txt by default. Keep concurrency reasonable and do not point it at sites you have no relationship with.

### Feedback

Found a bug or need a field that's missing? Open an issue on the Actor's **Issues** tab.

# Actor input Schema

## `mode` (type: `string`):

List the URLs a site publishes, check that those URLs respond, or scan pages for broken links inside them.

## `sites` (type: `array`):

One per line, as a domain or any URL on it. The Actor reads robots.txt for Sitemap: lines, then falls back to /sitemap.xml and /sitemap_index.xml.

## `urls` (type: `array`):

Check these exact URLs instead of discovering them. In broken-link mode these are the pages to scan. Leave empty to use the sitemaps from Sites above.

## `maxUrlsPerSite` (type: `integer`):

For List sitemap URLs and Check URL health. The main control on run size: large sites publish tens of thousands of URLs.

## `maxPagesToScan` (type: `integer`):

For Find broken links with a site: how many pages from its sitemap to open and scan. Each page can hold up to Max links per page results, so this sets the run's size.

## `maxLinksPerPage` (type: `integer`):

For broken-link mode. A navigation-heavy page can hold hundreds of links; each unique link is checked once per run and reused across pages.

## `internalLinksOnly` (type: `boolean`):

Ignore links pointing to other domains. Useful when you only care about your own broken pages.

## `respectRobotsTxt` (type: `boolean`):

Skip URLs the site's robots.txt disallows, and report them as skipped rather than broken. Leave this on unless you own the site and know what you are doing.

## `maxItems` (type: `integer`):

Stop the run after saving this many items. Leave empty for no limit.

## `maxConcurrency` (type: `integer`):

This Actor sends traffic to a real website. Four at a time is polite; raise it only against a site you own.

## `proxyConfiguration` (type: `object`):

Not needed for ordinary runs. Turn on Apify Proxy if a site blocks datacenter traffic.

## Actor input object example

```json
{
  "mode": "sitemapUrls",
  "sites": [
    "crawlee.dev"
  ],
  "maxUrlsPerSite": 1000,
  "maxPagesToScan": 25,
  "maxLinksPerPage": 100,
  "internalLinksOnly": false,
  "respectRobotsTxt": true,
  "maxConcurrency": 4,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "sites": [
        "crawlee.dev"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("winterlock/sitemap-link-checker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "sites": ["crawlee.dev"] }

# Run the Actor and wait for it to finish
run = client.actor("winterlock/sitemap-link-checker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "sites": [
    "crawlee.dev"
  ]
}' |
apify call winterlock/sitemap-link-checker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,winterlock/sitemap-link-checker"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/oweDiWutlEX60qEc7/builds/tukCqQdNVTGVBCVL0/openapi.json