# Sitemap Link Checker: broken links, redirects and noindex (`reflexdev/sitemap-link-checker`) Actor

Extract every URL from your sitemaps (indexes, .gz, robots.txt discovery) and check status codes, redirect chains, noindex, canonicals and broken links. $0.50 per 1,000 URLs, half the price of the most-used broken-link checker.

- **URL**: https://apify.com/reflexdev/sitemap-link-checker.md
- **Developed by:** [Rishad M](https://apify.com/reflexdev) (community)
- **Categories:** Developer tools, SEO tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.50 / 1,000 url checkeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap Link Checker: broken links, redirect chains and noindex pages

Give it a **sitemap** (or just your homepage) and get one row per URL with the **HTTP status code**, the full **redirect chain**, **response time**, **noindex** (meta robots and `X-Robots-Tag`), **canonical** and the sitemap's **lastmod**. Turn on the link crawl to also check every link on those pages and find **broken links** inside your content.

- **Reads any sitemap:** sitemap indexes (nested), `.xml.gz`, plain-text sitemaps, and `Sitemap:` lines in `robots.txt`. Give a homepage and it finds the sitemap for you.
- **Every redirect hop recorded:** see 301 → 302 → 200 chains, redirect loops, and redirects that end in a 404.
- **SEO checks in the same pass:** noindex pages that are still in your sitemap, and pages whose canonical points somewhere else.
- **Broken link crawl (one level):** checks the links on each page, internal only or outbound too, and tells you which page each broken link was found on.
- **Fast and light:** plain HTTP requests, no browser, so it runs in 256–512 MB.
- **Polite by default:** respects `robots.txt` (including `Crawl-delay`), sends at most 2 requests per second per site by default, backs off automatically when a site answers 429 or 503, and identifies itself with a clear user agent.

### What can you use it for?

| Use case | How |
| --- | --- |
| Find broken pages listed in your sitemap | Add the sitemap, filter `isBroken = true` |
| Clean up redirect chains after a migration | Look at `redirectCount` > 1 in the **Redirect chains and errors** view |
| Find noindex pages that shouldn't be in the sitemap | Filter `noindex = true` |
| Check canonicals | Filter `canonicalIsSelf = false` |
| Find broken links in your content | Turn on **Also check links found on pages** |
| Monitor a site | Schedule a daily or weekly run and get the `SUMMARY` record by webhook |
| Feed other tools | Export the 200 OK URLs to the Lighthouse Bulk Auditor or your own pipeline |

### How to use it

1. Paste one or more sitemap URLs, for example `https://www.example.com/sitemap.xml`, **or** add your homepage under **Start URLs**.
2. Set **Max URLs to check**. The default is 50 so a first run is quick; raise it for a full site.
3. Optional: turn on **Also check links found on pages** to find broken links.
4. Click **Start**. Download the results as JSON, CSV, Excel or HTML, or read them with the API.

### Input

| Field | Description | Default |
| --- | --- | --- |
| `sitemapUrls` | Sitemaps, sitemap indexes, `.gz` sitemaps or `robots.txt` URLs | `["https://apify.com/sitemap.xml"]` |
| `startUrls` | Pages or homepages to check directly | – |
| `discoverSitemaps` | Read the sitemaps listed in each start URL's `robots.txt` (falls back to `/sitemap.xml`) | `true` |
| `maxUrls` | Cap on URLs from start URLs and sitemaps (shared fairly between several sitemaps) | 50 |
| `crawlLinks` | Also check `<a href>` links found on checked pages, one level deep | `false` |
| `sameDomainOnly` | Only check links on the same host (`www.` ignored) | `true` |
| `maxLinkUrls` | With the link crawl on: cap on extra on-page link URLs, on top of `maxUrls` | 500 |
| `maxConcurrency` | Requests in flight at once, across all sites | 10 |
| `maxRequestsPerSecondPerHost` | Per-site rate limit; slows down automatically on 429/503 | 2 |
| `requestTimeoutSecs` | Timeout per request (each redirect hop) | 15 |
| `maxRedirects` | Longer chains are reported as an error | 10 |
| `maxRetries` | Retries for timeouts, 429 and 503 (honours `Retry-After`) | 2 |

Example: check a whole site and every internal link on it.

```json
{
  "startUrls": [{ "url": "https://www.example.com/" }],
  "maxUrls": 5000,
  "crawlLinks": true,
  "maxLinkUrls": 5000,
  "sameDomainOnly": true
}
```

### Output

One dataset row per URL. Real example from a test run against apify.com:

```json
{
  "url": "https://apify.com/alternatives/gmapsextractor-alternative",
  "source": "sitemap",
  "foundOn": null,
  "sitemapUrl": "https://apify.com/sitemap/pages.xml",
  "lastmod": null,
  "robotsBlocked": false,
  "statusCode": 308,
  "finalStatusCode": 200,
  "statusClass": "2xx",
  "isBroken": false,
  "finalUrl": "https://apify.com/alternatives/outscraper-alternative",
  "redirectCount": 1,
  "redirectChain": [
    { "url": "https://apify.com/alternatives/gmapsextractor-alternative", "statusCode": 308, "location": "https://apify.com/alternatives/outscraper-alternative" }
  ],
  "contentType": "text/html; charset=utf-8",
  "noindex": false,
  "noindexSource": null,
  "metaRobots": "robots: index,follow",
  "xRobotsTag": null,
  "canonical": "https://apify.com/alternatives/outscraper-alternative",
  "canonicalIsSelf": true,
  "linksFound": null,
  "error": null
}
```

| Field | Meaning |
| --- | --- |
| `source` | `sitemap`, `startUrl`, or `page` (a link found on a checked page) |
| `foundOn` | For links: the page that links to this URL |
| `statusCode` / `finalStatusCode` | First response (e.g. 301) and the status after all redirects |
| `isBroken` | Final status is 4xx/5xx, or the request failed (DNS, timeout, TLS, redirect loop) |
| `redirectChain` | Every hop: URL, status, and where it pointed |
| `responseTimeMs` | Network time until the final response headers, summed over all hops |
| `noindex`, `noindexSource` | `meta`, `header` (`X-Robots-Tag`), or `both` |
| `canonical`, `canonicalIsSelf` | From `<link rel="canonical">` or the `Link` header |
| `robotsBlocked` | `robots.txt` disallows this URL, so it was not requested |
| `error` | Why the check failed, or a note such as "Rate limited by the site (429)". A final 429 is not counted as broken |

There is also a **`SUMMARY`** record in the key-value store with counts of 2xx/3xx/4xx/5xx/errors (by first and by final status), the number of broken and rate-limited URLs and redirect chains with more than one hop, noindex and non-self canonical counts, average response time, the sitemaps that were read (with any errors), and lists of the broken URLs and long redirect chains (up to 200 each).

### How much does it cost?

**$0.50 per 1,000 URLs checked** ($0.0005 per URL), plus a start fee of $0.00005 per run at the default memory (Apify's standard Actor-start event). Platform usage is included, so that is the whole bill: checking a 10,000-URL sitemap costs about $5.

You only pay for URLs that returned an HTTP response, whatever the status (a 404 is a useful result). URLs skipped because of `robots.txt` and requests that got no HTTP response at all (DNS failure, connection refused, timeout) are free. Set **Maximum cost per run** in the run options to cap your spend: the Actor checks as many URLs as that allows and stops cleanly. See the **Pricing** tab for the current price.

### Good to know

- **robots.txt is always respected.** If your own site blocks bots, allow the `SitemapLinkChecker` user agent for the paths you want checked. A `robots.txt` that returns a 5xx error means "don't crawl" (RFC 9309), so that site's URLs are skipped and marked `robotsBlocked`.
- **Pages are fetched with GET**, because some servers answer HEAD wrongly. Only HTML pages are read (up to 3 MB) to look for meta robots, canonical and links; other files are closed after the headers.
- **No JavaScript rendering.** Links added by JavaScript after the page loads aren't seen. Use a browser-based crawler for those.
- **The link crawl is one level deep.** Links found on pages are checked but not crawled further. Pages from your sitemaps and start URLs are checked first, then the links found on them, up to `maxLinkUrls`.
- **No personal data is collected.** The Actor stores URLs, status codes and SEO headers only, never page text. `mailto:` and `tel:` links are ignored.
- Only check sites you own or have permission to test, and keep the rate limit gentle on sites that aren't yours.

**Further reading:** [How to find broken links from your sitemap.xml (and check every URL in it)](https://dev.to/reflexdev/how-to-find-broken-links-from-your-sitemapxml-and-check-every-url-in-it-2f4j), a step-by-step walkthrough on dev.to.

### FAQ

**Can it check a sitemap with 50,000 URLs?** Yes. Set `maxUrls` high enough. At the default 2 requests per second per site, 50,000 URLs on one site take about 7 hours; raise `maxRequestsPerSecondPerHost` on your own site to go faster.

**Why is a URL marked broken with no status code?** The request failed before any response: DNS lookup, connection refused, TLS certificate problem, timeout, or a redirect loop. The `error` field says which.

**Does it follow redirects to other domains?** Yes, every hop is followed and recorded, up to `maxRedirects`.

**Can I run it on a schedule?** Yes. Create a task with your input, add a schedule, and use a webhook or integration to get the `SUMMARY` record after each run.

Questions or a site that doesn't work? Open an issue on the **Issues** tab.

# Actor input Schema

## `sitemapUrls` (type: `array`):

Sitemaps to read. Sitemap indexes, nested indexes, .xml.gz files, plain-text sitemaps and robots.txt URLs (their Sitemap: lines are used) all work.

## `startUrls` (type: `array`):

Pages to check directly. With 'Discover sitemaps' on, the sitemaps listed in each site's robots.txt (or /sitemap.xml) are read too, so a homepage alone is enough.

## `discoverSitemaps` (type: `boolean`):

For each start URL's site, read the Sitemap: lines in robots.txt (falls back to /sitemap.xml) and check those URLs too. Skipped for sites you already gave a sitemap for.

## `maxUrls` (type: `integer`):

Cap on URLs taken from start URLs and sitemaps. When you give several sitemaps, the budget is shared fairly between them. On-page links have their own cap below. Raise it for full-site checks.

## `crawlLinks` (type: `boolean`):

Extract <a href> links from each checked HTML page and check them as well (they are not crawled further). This is how you find broken links inside your content.

## `sameDomainOnly` (type: `boolean`):

When on, only links to the same host (www. ignored) are checked. Turn off to check outbound links to other sites too.

## `maxLinkUrls` (type: `integer`):

With the link crawl on: cap on extra link URLs checked, on top of 'Max URLs to check'. Links are deduplicated across the whole run.

## `maxConcurrency` (type: `integer`):

Total requests in flight at once, across all hosts.

## `maxRequestsPerSecondPerHost` (type: `integer`):

Gentle per-site rate limit. It slows down automatically if a site answers 429 or 503, and a robots.txt Crawl-delay (up to 10 s) is honoured if it is slower. Raise it for your own site.

## `requestTimeoutSecs` (type: `integer`):

Per request (each redirect hop has its own timeout).

## `maxRedirects` (type: `integer`):

Longer chains are reported as an error.

## `maxRetries` (type: `integer`):

Retries transient failures with back-off (Retry-After is honoured, up to 60 s). 404s and DNS failures are never retried.

## Actor input object example

```json
{
  "sitemapUrls": [
    "https://apify.com/sitemap.xml"
  ],
  "discoverSitemaps": true,
  "maxUrls": 50,
  "crawlLinks": false,
  "sameDomainOnly": true,
  "maxLinkUrls": 500,
  "maxConcurrency": 10,
  "maxRequestsPerSecondPerHost": 2,
  "requestTimeoutSecs": 15,
  "maxRedirects": 10,
  "maxRetries": 2
}
```

# Actor output Schema

## `overview` (type: `string`):

No description

## `redirects` (type: `string`):

No description

## `summary` (type: `string`):

No description

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "sitemapUrls": [
        "https://apify.com/sitemap.xml"
    ],
    "maxUrls": 50
};

// Run the Actor and wait for it to finish
const run = await client.actor("reflexdev/sitemap-link-checker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "sitemapUrls": ["https://apify.com/sitemap.xml"],
    "maxUrls": 50,
}

# Run the Actor and wait for it to finish
run = client.actor("reflexdev/sitemap-link-checker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "sitemapUrls": [
    "https://apify.com/sitemap.xml"
  ],
  "maxUrls": 50
}' |
apify call reflexdev/sitemap-link-checker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,reflexdev/sitemap-link-checker"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/uZLEFs4RZQRX7nPrq/builds/rN6sUw77ySeVOgaLe/openapi.json