# Sitemap Checker & Redirect Chain Auditor (`velveteen_driver/sitemap-redirect-auditor`) Actor

Audit your own website from its sitemap: status codes, redirect chains and loops, canonical mismatches, noindex pages in the sitemap, URLs robots.txt blocks, and optional broken internal links. Polite, robots.txt-respecting, no browser.

- **URL**: https://apify.com/velveteen_driver/sitemap-redirect-auditor.md
- **Developed by:** [Elliott](https://apify.com/velveteen_driver) (community)
- **Categories:** SEO tools, Developer tools
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$1.50 / 1,000 url checkeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap & Redirect Auditor

Audit your own website from its XML sitemap in one run. For every URL in the sitemap (and any extra URLs you add) you get
the status code, the full redirect chain, redirect loops, the canonical tag compared with the URL, noindex from the meta
robots tag or the `X-Robots-Tag` header, and whether robots.txt blocks it. Optionally it also checks the same-website
links found on those pages, so you see broken internal links and links that point at redirects.

Built for SEO specialists, agencies and developers checking sites they run or have been asked to audit, for example
after a migration or a redesign. Plain HTTP, no browser, so it is quick and cheap.

### What it checks

- **robots.txt**: read first for every host, following RFC 9309 (our own user-agent group if there is one, otherwise `*`;
  longest match wins; `*` and `$` wildcards). A 4xx answer means no rules; a 5xx or an unreachable robots.txt means
  everything is treated as disallowed, and nothing on that host is fetched.
- **Sitemaps**: the sitemap URL you give, or the `Sitemap:` lines in robots.txt, or `/sitemap.xml`. Sitemap index files
  are followed up to 3 levels and 50 sitemap files. Gzipped sitemaps are supported. Sitemaps on another website are not read.
- **Each URL** (one GET, redirects followed by hand up to 10 hops on the same website):
  - error pages (4xx/5xx) and URLs that don't answer;
  - redirects: hops, the chain with each status, loops, temporary redirects (302/303/307) to live pages;
  - canonical tag and whether it points to the URL itself;
  - noindex (meta robots or `X-Robots-Tag`);
  - sitemap problems: in the sitemap but redirecting, noindex, canonicalised elsewhere, or blocked by robots.txt.
- **Internal links** (optional): same-website links on the listed pages (`rel="nofollow"` links skipped), checked with a
  HEAD request (GET if the server refuses HEAD). One level only: links are never followed further.

### Input example

```json
{
  "websites": ["https://www.example.com"],
  "extraUrls": ["https://www.example.com/old-pricing"],
  "maxUrlsPerSite": 500,
  "checkInternalLinks": true,
  "maxLinksPerPage": 100
}
```

You can give a homepage or a sitemap URL, and up to 20 websites per run (they are audited one after another).
`extraUrls` are used only when exactly one website is given. When internal links are on, the listed pages take up to
half of `maxUrlsPerSite` and links the rest.

### Output

One row per URL (`type` = `url` or `link`) and one `summary` row per website (also saved as `SUMMARY` in the run's
key-value store).

| Field | Meaning |
|---|---|
| `url`, `source` | The URL and where it came from: `sitemap`, `input` or `link` |
| `inSitemap` | Whether the URL is listed in the sitemap |
| `statusCode`, `responseMs` | First response and its time to headers |
| `redirectHops`, `redirectChain`, `redirectLoop` | Each hop as `{url, status, to}` |
| `finalUrl`, `finalStatusCode` | Where the chain ends |
| `canonical`, `canonicalMatches` | Canonical tag (resolved) and whether it equals the URL |
| `robotsMeta`, `xRobotsTag`, `noindex` | Indexing directives |
| `robotsAllowed` | `false` when robots.txt disallows the URL (it is then not fetched) |
| `foundOn` | For links: up to 3 pages that link to it |
| `issues` | Plain-English list of problems found |
| `note`, `error` | Why something was not followed or fetched |

The summary row lists the robots.txt status, the sitemaps read (with entry counts), sitemap errors, URL counts, whether
the URL cap was reached, `stoppedEarly` if the website asked us to stop, and a count per issue.

Only technical fields are saved. Page text, emails, phone numbers and names are never extracted or stored.

### Limits and politeness

- At most 2 requests at a time to one website (a host and its `www.` twin share the limit) and at least 0.5 s between
  request starts, or the site's `Crawl-delay` up to 30 s. A site whose robots.txt asks for a longer Crawl-delay is not
  audited (the summary says why).
- Timeouts: 10 s to connect, 20 s per read, 45 s in total per request.
- Back-off: a 429 or 503 answer doubles the gap for that website (up to 30 s) and pauses it until its `Retry-After`
  (seconds or a date), then the request is tried once more. A `Retry-After` over 60 s, or three 429/503 answers in a
  row, stops the audit of that website (`stoppedEarly` in the summary). A 429 on robots.txt is treated like an
  unreachable robots.txt: nothing on that website is fetched.
- robots.txt is read once per host and read again if the run is still going 24 hours later.
- Page bodies are read up to 2 MB (only HTML, only for canonical/noindex/links), robots.txt up to 500 KB, sitemaps up to
  50 MB (also after unzipping).
- Up to 10,000 URLs per website (default 500).
- No cookies (none are accepted or sent), logins, proxies or browser. Pages behind a login, captcha or bot check are reported with their status code
  and never bypassed.
- Only public web addresses on ports 80/443; private and reserved network addresses are refused. Platforms whose terms
  forbid automated access (for example Google, Amazon, LinkedIn, Facebook and other Meta sites, Instagram, X, TikTok,
  Yelp, Trustpilot, Indeed, Glassdoor, Zillow, Booking, Expedia, Tripadvisor, ticketing sites) are refused on every
  country domain.
- The user-agent is `Mozilla/5.0 (compatible; SitemapRedirectAuditor/1.0; +https://apify.com/velveteen_driver/sitemap-redirect-auditor)`,
  and robots.txt rules for `SitemapRedirectAuditor` are honoured.

### Pricing

$0.0015 per URL checked (event `url-checked`, so $1.50 per 1,000 URLs). Only URLs that were actually fetched are charged:
URLs blocked by robots.txt, on another website or refused, and the summary rows, are free. Example: a 500-page sitemap
with internal links off = $0.75. Set a maximum charge per run in the run options if you like; the run stops when it is
reached and the status message says so.

### Data source and responsible use

- The only data source is the website you supply. Use it on websites you own or are authorised to audit.
- robots.txt is respected for every request, including sitemaps and internal links. robots.txt is not permission to
  access a site, so make sure you are allowed to audit it.
- Not affiliated with any search engine. "Noindex" and "canonical" findings describe the page markup only; they don't
  report how any search engine has actually indexed the site.

# Actor input Schema

## `websites` (type: `array`):

Websites you own or are authorised to audit (homepage or sitemap URL), up to 20 per run. For a homepage, the sitemaps listed in robots.txt are used, else /sitemap.xml.

## `extraUrls` (type: `array`):

Optional. More URLs on the same website to check as well as the sitemap (used only when exactly one website is given).

## `maxUrlsPerSite` (type: `integer`):

Upper limit of URLs checked per website (sitemap URLs first, then internal links). Each fetched URL is one charged event.

## `checkInternalLinks` (type: `boolean`):

Also check the same-website links found on the listed pages (one level only, never followed further). Counts towards the max URLs: when this is on, the listed pages take up to half of it and links the rest.

## `maxLinksPerPage` (type: `integer`):

When checking internal links, at most this many links are taken from each page.

## Actor input object example

```json
{
  "websites": [
    "https://www.example.com"
  ],
  "maxUrlsPerSite": 10,
  "checkInternalLinks": false,
  "maxLinksPerPage": 100
}
```

# Actor output Schema

## `results` (type: `string`):

All results in the default dataset, shown in the overview table.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "websites": [
        "https://www.example.com"
    ],
    "maxUrlsPerSite": 10
};

// Run the Actor and wait for it to finish
const run = await client.actor("velveteen_driver/sitemap-redirect-auditor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "websites": ["https://www.example.com"],
    "maxUrlsPerSite": 10,
}

# Run the Actor and wait for it to finish
run = client.actor("velveteen_driver/sitemap-redirect-auditor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "websites": [
    "https://www.example.com"
  ],
  "maxUrlsPerSite": 10
}' |
apify call velveteen_driver/sitemap-redirect-auditor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,velveteen_driver/sitemap-redirect-auditor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/87LrPlQhSy6yD09h1/builds/jhhbOVX7uJdA8QCgj/openapi.json