# Sitemap Extractor, URL Status and Broken Link Checker (`vindr/sitemap-url-checker`) Actor

Get every URL from a website's sitemaps and check them: status codes, redirect chains, soft 404s, noindex, canonical and response time. Also finds broken links on your pages. Handles sitemap indexes, .gz, text and RSS sitemaps. $1 per 1,000 URLs checked.

- **URL**: https://apify.com/vindr/sitemap-url-checker.md
- **Developed by:** [VindR](https://apify.com/vindr) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 urls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### What does Sitemap Extractor, URL Status and Broken Link Checker do?

It gets every URL from a website's sitemaps and tells you which ones work. Give it a website or a sitemap, and for every URL you get the sitemap data (last modified, priority, images, language versions) and a check: status code, redirect chain, response time, page title, canonical URL and noindex. Turn on **Find broken links** and it also checks the links on every page and tells you where each broken link is.

Use it to:

- **Audit a sitemap.** Find URLs in your sitemap that are broken, redirect, are marked noindex or point their canonical to another page. Search engines treat those as mistakes.
- **Find broken links.** See every link on your site that leads to a 404, a dead domain or a server error, with the pages and link texts to fix.
- **Get a full list of a site's pages** in seconds, for a migration, a crawl or a content inventory.
- **Check a list of URLs in bulk** for status codes and redirects, without a sitemap.

### Which sitemaps does it read?

- The sitemaps listed in the site's **robots.txt**, or the usual places when there are none: `/sitemap.xml`, `/sitemap_index.xml`, `/sitemap-index.xml`, `/wp-sitemap.xml` and `/sitemap.txt`.
- **Sitemap indexes** (sitemaps that list other sitemaps), followed all the way down.
- **Gzipped sitemaps** (`.xml.gz`), **plain-text sitemaps** (one URL per line) and **RSS or Atom feeds** used as sitemaps.
- Image, video and hreflang (language version) entries.

It works with WordPress, Shopify, Wix, Squarespace, Webflow and any other system that writes standard sitemaps.

### What data do you get?

| Field | Example |
|---|---|
| URL and where it was found | sitemap, your list, or a link on a page |
| Sitemap data | last modified, change frequency, priority, image and video count, hreflang versions |
| Status code | 200, 301, 404, 503 |
| Redirects | the full chain, like `307 > 308 > 200`, and where it ends |
| Response time | in milliseconds |
| Page info | title, canonical URL, noindex (from the robots meta tag or the X-Robots-Tag header) |
| Issues | `broken`, `soft_404`, `blocked`, `redirect`, `redirect_chain`, `noindex`, `canonical_elsewhere`, `slow`, `not_checked_robots_txt` |
| For links | the pages that link to it, with the link text |

#### What do the issues mean?

- **broken**: the URL doesn't work: 404, 410, a 5xx server error, a timeout or a domain that doesn't exist.
- **soft\_404**: the server answers 200 OK, but the page says it doesn't exist ("Page not found", "Siden finnes ikke" and the like). Search engines report these as soft 404s.
- **blocked**: the server refused an automated visitor (401, 403 or 429). The page may work fine in a browser, so it isn't counted as broken.
- **redirect** / **redirect\_chain**: the URL sends you somewhere else, once or several times. Links and sitemaps should point to the final URL.
- **noindex**: a sitemap URL that asks search engines not to index it.
- **canonical\_elsewhere**: a sitemap URL whose canonical tag points to another page.
- **slow**: the answer took more than 5 seconds.
- **not\_checked\_robots\_txt**: the site's robots.txt asks automated tools not to visit this URL, so it was not checked.

### How to use it

1. Click **Try for free** at the top of this page.
2. Add a website (`example.com`) or a sitemap URL under **Websites or sitemaps**. Or paste URLs under **URLs to check**.
3. Choose what to check. For a quick list of URLs, turn off **Check each URL**. For a broken link report, turn on **Find broken links on the pages**.
4. Optional: save only URLs with issues, or only URLs that contain `/blog/`.
5. Click **Start**, then download the results as Excel, CSV or JSON, or use the API.

#### Input examples

Sitemap audit, saving only problems:

```json
{
    "startUrls": ["https://www.example.com"],
    "onlyIssues": true
}
```

Broken link check of the blog, links within the site only:

```json
{
    "startUrls": ["https://www.example.com/sitemap.xml"],
    "includePatterns": ["/blog/"],
    "findBrokenLinks": true,
    "checkExternalLinks": false
}
```

All URLs in the sitemaps, without checking them:

```json
{
    "startUrls": ["https://www.example.com"],
    "checkUrls": false
}
```

### Output example

A sitemap URL that redirects:

```json
{
    "url": "https://vipps.no/nyheter",
    "source": "sitemap",
    "sitemap": "https://vipps.no/sitemap.xml",
    "lastmod": "2026-07-21T06:08:42.000Z",
    "changefreq": null,
    "priority": null,
    "imageCount": 0,
    "videoCount": 0,
    "hreflang": [],
    "checked": true,
    "statusCode": 200,
    "ok": true,
    "finalUrl": "https://vipps.no/news",
    "redirectCount": 1,
    "redirectChain": [
        { "url": "https://vipps.no/nyheter", "status": 307 },
        { "url": "https://vipps.no/news", "status": 200 }
    ],
    "responseTimeMs": 175,
    "contentType": "text/html",
    "title": "Nå kan du betale med Vipps i Spond | Vipps",
    "canonical": null,
    "noindex": false,
    "error": null,
    "issues": ["redirect"],
    "foundOn": null,
    "checkedAt": "2026-09-25T12:00:00.000Z"
}
```

A broken link found on a page (shortened):

```json
{
    "url": "https://help.vippsmobilepay.com/nb-NO/categories/children-profile",
    "source": "link",
    "statusCode": 404,
    "issues": ["broken"],
    "foundOn": [
        { "page": "https://vipps.no/under-15", "anchor": "Se ofte stilte spørsmål" }
    ]
}
```

A `RUN_SUMMARY` record in the key-value store shows the sitemaps found and read, the number of URLs checked, the count of each issue, and the number of links checked and broken.

### How much does it cost?

You pay **$1 per 1,000 URLs** checked. Server time is included.

- A site with 500 URLs: $0.50.
- A site with 20,000 URLs: $20.
- Listing the URLs in the sitemaps without checking them costs the same per URL.
- With **Find broken links**, every link that is checked counts as one URL too. A 200-page site whose pages link to 800 other URLs costs $1.
- URLs that robots.txt asks us not to visit are not checked and not charged.

**Only save URLs with issues** and **Links to save** only filter what you get back, so your results stay short. They don't change the price. The $5 in free monthly credit on the Apify free plan covers about 5,000 URLs a month. Set **Max URLs**, the URL filters or a maximum cost per run for a hard budget: the run stops before it goes over it.

### FAQ

#### Is it safe to run on my website?

Yes. By default it sends at most 4 requests at a time to the same website, always respects a Crawl-delay in robots.txt, and visits each URL once. You can lower **Max requests at a time per website** to 1 for small servers.

#### Does it respect robots.txt?

Yes. URLs that robots.txt asks automated tools to avoid are not visited and are marked `not_checked_robots_txt`. It identifies itself honestly as VindR-SitemapChecker, so you can allow or block it in your robots.txt.

#### Why are some pages "blocked" and not "broken"?

Some websites answer automated visitors with 403 or 429, often through bot protection like Cloudflare, even though the page works in a browser. The Actor doesn't try to get around that, and marks those URLs as `blocked` so you don't chase problems that aren't there.

#### Does it find links added by JavaScript?

No. It reads the HTML the server sends, like a search engine's first pass does. Links that only appear after JavaScript runs in a browser are not found. For most websites, the navigation and content links are in the HTML.

#### What if a website has no sitemap?

Then you'll see a warning in the log, and the site is listed under `sitesWithoutSitemap` in the run summary. You can still check URLs by pasting them under **URLs to check**.

#### How big a site can it handle?

Sitemaps with hundreds of thousands of URLs are fine. Checking takes as long as the website needs to answer: with 4 requests at a time and a fast site, around 1,000 URLs a minute. Some platforms, Shopify for example, only allow a few automated requests a second. When a site answers "too many requests" (HTTP 429), the Actor waits as long as the site asks, slows down for that site and tries again, so you get real results instead of errors. On such sites, expect around 1,000 URLs in 20 minutes. Use the URL filters or **Max URLs** to check part of a large site.

#### Can I use it from my own code?

Yes. Every Apify Actor has an API. See the **API** tab for examples in Python, JavaScript and plain HTTP. It also works with Make, n8n, Zapier and Google Sheets, and you can schedule it to run every week.

#### Something missing or not working?

Open an issue on the **Issues** tab with the website you tried.

### More tools from VindR

- [Website Tech Stack Detector](https://apify.com/vindr/website-tech-stack-detector): what a website is built with.
- [Bulk WHOIS Lookup](https://apify.com/vindr/bulk-whois-lookup): domain registration, expiry and SSL for many domains.
- [PDF Text Extractor](https://apify.com/vindr/pdf-text-extractor): the text of the PDFs a website publishes.

# Actor input Schema

## `startUrls` (type: `array`):

Websites (<code>example.com</code>) or sitemap URLs (<code>https://example.com/sitemap.xml</code>), one per line. For a website, the sitemaps are found in its robots.txt or at the usual places like /sitemap.xml. Sitemap indexes are followed.

## `urls` (type: `array`):

Or check a list of URLs directly, one per line, without a sitemap.

## `checkUrls` (type: `boolean`):

Visit every URL and save its status code, redirect chain, response time, page title, canonical URL and noindex. Turn off to only list the URLs in the sitemaps, which is much faster.

## `findBrokenLinks` (type: `boolean`):

Also collect the links on every checked page and check them. Each link is checked once, and the result shows which pages link to it.

## `checkExternalLinks` (type: `boolean`):

Only for broken links: also check links that go to other websites. Turn off to check only links within the same site.

## `linkResults` (type: `string`):

Only for broken links: which of the checked links to save. Every link that is checked counts as one URL for pricing, whether it is saved or not.

## `onlyIssues` (type: `boolean`):

Save only URLs that are broken, redirect, are blocked, marked noindex, look like an error page (soft 404), point their canonical elsewhere, are slow, or could not be checked because of robots.txt. This only filters the results: every checked URL counts for pricing.

## `includePatterns` (type: `array`):

Keep only URLs that contain any of these texts, for example <code>/blog/</code>. Not case sensitive.

## `excludePatterns` (type: `array`):

Leave out URLs that contain any of these texts, for example <code>/tag/</code> or <code>?page=</code>.

## `maxUrls` (type: `integer`):

Stop after this many URLs from the sitemaps and your list. Use 0 for no limit.

## `maxLinks` (type: `integer`):

Only for broken links: check at most this many links found on the pages.

## `perHostConcurrency` (type: `integer`):

How many requests go to the same website at the same time. Keep it low to be kind to the site. A Crawl-delay in the site's robots.txt is always respected.

## `maxConcurrency` (type: `integer`):

How many URLs are checked at the same time across all websites.

## Actor input object example

```json
{
  "startUrls": [
    "https://www.vipps.no"
  ],
  "checkUrls": true,
  "findBrokenLinks": false,
  "checkExternalLinks": true,
  "linkResults": "broken",
  "onlyIssues": false,
  "maxUrls": 100,
  "maxLinks": 10000,
  "perHostConcurrency": 4,
  "maxConcurrency": 20
}
```

# Actor output Schema

## `urls` (type: `string`):

One item per URL: where it was found, its sitemap data and the check results with any issues.

## `runSummary` (type: `string`):

Sitemaps found and read, URLs checked, issues by type, links checked and broken links.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://www.vipps.no"
    ],
    "maxUrls": 100
};

// Run the Actor and wait for it to finish
const run = await client.actor("vindr/sitemap-url-checker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://www.vipps.no"],
    "maxUrls": 100,
}

# Run the Actor and wait for it to finish
run = client.actor("vindr/sitemap-url-checker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://www.vipps.no"
  ],
  "maxUrls": 100
}' |
apify call vindr/sitemap-url-checker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,vindr/sitemap-url-checker"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Yiem0pEmuwNoitFX3/builds/axqG871GBZNNsccBy/openapi.json