# Sitemap URL Extractor & Broken Link Checker (`grit-77/sitemap-url-extractor-broken-link-checker`) Actor

Sitemap URL extractor for public sites. Export URLs and lastmod values as JSON with robots.txt discovery; optionally include HTTP status, redirects and request errors.

- **URL**: https://apify.com/grit-77/sitemap-url-extractor-broken-link-checker.md
- **Developed by:** [Grit](https://apify.com/grit-77) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$0.08 / 1,000 page urls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap URL Extractor & Broken Link Checker

Sitemap links scraper returns page URLs and lastmod values from public sitemaps and robots.txt discovery. Sitemap links API results can also include HTTP status, redirect, and request error details when status checking is enabled.

### Input example

```json
{
  "startUrls": [
    "https://www.sitemaps.org/"
  ],
  "checkStatus": false,
  "maxItems": 20
}
```

The input keys and allowed values follow [input_schema.json](.actor/input_schema.json).

### Output fields

| Field | Type | Illustrative example |
|---|---|---|
| `rowType` | string | `"url"` |
| `url` | string | `"https://www.sitemaps.org/protocol.html"` |
| `lastmod` | string | `"2022-12-15"` |
| `sitemapUrl` | string | `"https://www.sitemaps.org/sitemap.xml"` |
| `inputUrl` | string | `"https://www.sitemaps.org/"` |
| `checkStatus` | string | `"not_checked"` |
| `statusCode` | integer or null | `null` |
| `isBroken` | boolean or null | `null` |
| `finalUrl` | string or null | `null` |
| `redirectChain` | array | `[]` |
| `requestMethod` | string or null | `null` |
| `headStatusCode` | integer or null | `null` |
| `responseTimeMs` | number or null | `null` |
| `checkedAt` | string or null | `null` |
| `errors` | array | `[]` |

### Output example

No `sample_output.json` is present in this actor folder. The following illustrative row is preserved from the previous README and is not a recorded actor run.

```json
{
  "rowType": "url",
  "url": "https://www.sitemaps.org/protocol.html",
  "lastmod": "2022-12-15",
  "sitemapUrl": "https://www.sitemaps.org/sitemap.xml",
  "inputUrl": "https://www.sitemaps.org/",
  "checkStatus": "not_checked",
  "statusCode": null
}
```

### Limits

| Setting | Default | Accepted range |
|---|---:|---:|
| maxItems (all output rows) | 1000 | 1–100000 |
| concurrency | 10 | 1–50 |
| requestTimeoutSecs (each network operation) | 20 | 1–120 |
| maxRetries (extra attempts per hop) | 2 | 0–5 |
| maxSitemaps (global sitemap fetches) | 1000 | 1–10000 |
| maxDepth (initial source = zero) | 10 | 0–50 |
| maxRedirects (followed hops) | 10 | 0–30 |

A sitemap document is limited to 10 MiB before and after gzip decompression. Larger files, DTD/entity declarations, malformed XML, private IP targets and non-HTTP URLs are rejected. robots.txt requests are additional to maxSitemaps. maxItems includes free diagnostics; errors can therefore reduce the number of extracted URLs. At most 1000 starting URLs are accepted.

Retrievable sitemap entries determine coverage. Pages omitted from sitemaps are not discovered. The actor does not create sitemaps, crawl HTML links, render JavaScript, authenticate, solve CAPTCHAs or verify content-level soft 404s. robots.txt Sitemap directives are used for discovery; Disallow rules are not evaluated for page checks. Use it on sites you are authorized to check.

### FAQ

**What are the output limits?** `maxItems` caps extracted URLs and diagnostics. The actor rejects malformed, oversized, non-public or non-HTTP sitemap targets.

**Do I need a proxy or login?** Public sitemaps need no login or proxy. Private network targets are rejected.

**How is this priced?** Proposed pay per event: $0.00008 per unique extracted `url` row, including one with a failed optional status check. Discovery diagnostics and duplicates are uncharged. These are proposals in [PRICING.md](PRICING.md); the owner must configure Store billing.

**How are rate limits handled?** `concurrency` limits HTTP requests; selected transient failures are retried with backoff and `Retry-After`.

**How fresh is the data?** Sitemaps contain only URLs the publisher exposes; this actor does not crawl page links or render JavaScript. `lastmod` is the publisher’s value, not an independently verified update date. Check only sites you are authorized to query.

### Use with AI agents / MCP

After the actor is published, connect your agent to the [Apify MCP server](https://docs.apify.com/integrations/mcp) with an Apify API token. Use `call-actor` with the published actor identifier and the JSON input shown above. Read the returned dataset with `get-dataset-items`; the run response itself contains status and storage IDs, not the result rows. Alternatively, use the [Apify API](https://docs.apify.com/api/v2) to start a run and read its default dataset. Keep the actor input bounded with `maxItems`, and inspect source links and any error fields before using results in an answer.

# Actor input Schema

## `startUrls` (type: `array`):

Public HTTP(S) sites, robots.txt files, or XML/gzip sitemap URLs. Site inputs use origin robots.txt and fall back to /sitemap.xml. Bare domains use HTTPS. Duplicates are removed. Maximum 1000 starting URLs.

## `checkStatus` (type: `boolean`):

Check each extracted URL. Disabled: extract metadata without requesting page URLs.

## `checkMethod` (type: `string`):

HEAD requests headers only. GET also closes after headers, without reading page bodies.

## `fallbackToGet` (type: `boolean`):

Retry a HEAD check using GET when the final response is 405 or 501. The redirectChain then describes the GET request.

## `maxItems` (type: `integer`):

Hard cap on all dataset rows, including uncharged discovery diagnostics.

## `concurrency` (type: `integer`):

Maximum simultaneous HTTP requests and URL checker tasks.

## `requestTimeoutSecs` (type: `integer`):

Timeout for each network operation and DNS lookup; retries and redirects can increase total time.

## `maxRetries` (type: `integer`):

Extra attempts for network errors and HTTP 408, 425, 429, 500, 502, 503 and 504. Exponential backoff; Retry-After is capped at 30 seconds.

## `maxSitemaps` (type: `integer`):

Global cap on fetched sitemap documents. robots.txt requests are additional.

## `maxDepth` (type: `integer`):

Initial sitemap is depth zero. A deeper source produces an uncharged diagnostic.

## `maxRedirects` (type: `integer`):

Limit per sitemap fetch or URL status check. Redirect loops and limits remain visible.

## Actor input object example

```json
{
  "startUrls": [
    "https://www.sitemaps.org/"
  ],
  "checkStatus": false,
  "checkMethod": "HEAD",
  "fallbackToGet": true,
  "maxItems": 1000,
  "concurrency": 10,
  "requestTimeoutSecs": 20,
  "maxRetries": 2,
  "maxSitemaps": 1000,
  "maxDepth": 10,
  "maxRedirects": 10
}
```

# Actor output Schema

## `dataset` (type: `string`):

Open the extracted URLs and diagnostic rows in the default dataset.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://www.sitemaps.org/"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("grit-77/sitemap-url-extractor-broken-link-checker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": ["https://www.sitemaps.org/"] }

# Run the Actor and wait for it to finish
run = client.actor("grit-77/sitemap-url-extractor-broken-link-checker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://www.sitemaps.org/"
  ]
}' |
apify call grit-77/sitemap-url-extractor-broken-link-checker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,grit-77/sitemap-url-extractor-broken-link-checker"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/ftJ2vhGLn3q8Z9s9S/builds/OKCFc5gHpwEogR8LG/openapi.json