# Sitemap Link Inventory: All URLs + Broken-Link Check, $1/1k (`transparent_meteorite/sitemap-link-inventory`) Actor

Find every sitemap for a domain (robots.txt, nested indexes, .gz, .txt, common paths) and return all URLs with lastmod, priority, images, hreflang, path groups and duplicates. Optional broken-link, redirect, noindex checks. Partial results never fail the run. $0.001 per sitemap file / link checked.

- **URL**: https://apify.com/transparent_meteorite/sitemap-link-inventory.md
- **Developed by:** [Detroit Web Agent - Open Data](https://apify.com/transparent_meteorite) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap Link Inventory: every sitemap URL, with optional broken-link, redirect and noindex checks

Give it a domain (or a list of them). It finds the sitemaps through robots.txt and common paths, follows nested sitemap indexes, reads plain XML, `.gz` and `.txt` files as a stream, and returns **every URL** with `lastmod`, `changefreq`, `priority`, the sitemap file it came from, image, video and hreflang entries when the sitemap has them, URL pattern groups (counts per first path segment) and a duplicate report. Switch on check mode and it also requests the URLs (HEAD first, GET as fallback) and flags 404s, 5xx errors, timeouts, redirects, `noindex` pages, robots-disallowed URLs and, optionally, canonical mismatches.

![Sitemap Link Inventory output preview: real run results as a table](https://open-data-actors.pages.dev/img/sitemap-link-inventory.png)

The design goal is boring reliability: one broken child sitemap, one truncated `.gz`, one slow server or one HTML "soft 404" never fails the run. You get the URLs that could be read, plus a plain explanation of what could not.

- No login, no proxies, no browser. Plain HTTP against public files.
- Honours robots.txt for every request it makes on your behalf (more on exactly what that means below).
- Refuses private, loopback and link-local addresses (SSRF guard, with DNS resolution checks).
- Pay per event: **$0.001 per sitemap file that delivered URLs**, **$0.001 per link actually checked**, and the platform's tiny per-run start fee. Check mode is off by default, so an inventory-only run of a 50,000-URL site split over five files costs about half a cent.

### What does it do

For each input row (a domain such as `example.com`, a site URL, or a direct sitemap URL) the actor:

1. **Normalises and vets the input.** A missing `https://` is added. Anything that is not a public http(s) host is skipped and reported in the run's `OUTPUT` record.
2. **Reads robots.txt.** It collects every `Sitemap:` line (absolute or relative), the disallow rules for its own user agent token, and any `Crawl-delay` (capped at 10 seconds).
3. **Discovers sitemaps.** The `Sitemap:` lines come first. If robots.txt yields nothing that parses, it probes common locations: `/sitemap.xml`, `/sitemap_index.xml`, `/sitemap-index.xml`, `/sitemap.xml.gz`, `/sitemap.txt`, `/wp-sitemap.xml`, `/sitemaps.xml`, `/sitemap/sitemap.xml`, `/sitemap1.xml`. If your input already looks like a sitemap (`.xml`, `.xml.gz`, `.txt`, or "sitemap" in the path) it is fetched directly.
4. **Streams each sitemap.** XML is parsed as it arrives, chunk by chunk, so a 400 MB file does not need 400 MB of memory. Gzip is detected from the magic bytes, not the file name or the content type, so a `.gz` served as `application/octet-stream` still works. Sitemap index files are followed up to six levels deep, with a cap on how many files are fetched.
5. **Emits one row per URL** and one **site summary** row per input, with discovery details, per-file results, pattern groups, duplicates and counts.
6. **Optionally checks links** (see "Check modes"), billing only the requests it makes.

#### What each URL row contains

| Field | Meaning |
|---|---|
| `url`, `site` | The sitemap URL (entities and CDATA decoded) and the input host |
| `lastmod`, `changefreq`, `priority` | As written in the sitemap (`priority` parsed to a number; `null` when absent or not numeric) |
| `sourceSitemap`, `sitemapDepth` | The file the URL came from and how deep in the index tree it sat |
| `pathGroup`, `pathDepth`, `hasQuery` | First path segment (`/blog`, or `/` for the home page), number of segments, query-string flag |
| `images`, `imageCount` | Up to 10 `image:loc` values and the true count |
| `videos`, `videoCount` | Up to 3 entries with title, content URL, player URL, thumbnail, duration |
| `hreflang`, `hreflangCount` | `xhtml:link rel="alternate"` entries as `{lang, href}` |
| `robotsDisallowed` | `true` when the site's own robots.txt disallows this URL for the actor's user agent; `null` for other hosts |
| `isDuplicate` | Only `true` when you turned on duplicate rows |
| `flags` | Plain-language flags, see the list below |
| `checked` plus check fields | `statusCode`, `finalUrl`, `redirects`, `redirectChain`, `responseMs`, `method` (HEAD or GET), `error`, `noindex`, `noindexSource`, `canonicalUrl`, `canonicalMismatch`, `isBroken` |

#### Flags

`robots-disallowed` (listed in the sitemap but blocked in robots.txt), `off-host` (the URL's host differs from the input host), `not-found` (404 or 410), `server-error` (5xx), `unreachable` (timeout, DNS or connection error, redirect loop), `client-error` (other 4xx), `blocked` (401, 403, 429: the server refused us, which is not the same as the page being broken), `redirect` (one or more redirects before the final response), `noindex` (via `X-Robots-Tag` or a meta robots tag) and `canonical-mismatch`. `isBroken` is `true` for not-found, 5xx and unreachable; it is deliberately `false` for 401/403/429.

### Why use it

- **It finishes.** Each failure mode becomes data instead of a crash: a 404 child sitemap is recorded with `error: "http-404"`, an HTML page pretending to be a sitemap is rejected as `not-a-sitemap-html`, a truncated `.gz` keeps every URL read before the break and is marked `partial`, a stalled connection gives up after 30 seconds of silence, and an expired time budget returns a summary with status `partial-time-budget`.
- **It scales without exploding.** URLs are pushed to the dataset in batches while the file is still downloading. The `maxUrlsPerSite` cap aborts the download as soon as it is reached, so you do not wait for the other 1.9 million URLs you did not ask for.
- **It is predictable to pay for.** Sitemap index files and failed fetches are free. The cost ceiling for check mode is your own `maxLinksToCheck` value.
- **It sees what a crawler cannot see quickly.** Pattern groups tell you in one row that 62% of a site lives under `/js` and 37% under `/blog`; the duplicate report tells you which URLs are listed twice and in which sitemap files.
- **It finds sitemap hygiene problems.** URLs that 404, redirect, carry `noindex`, are blocked by robots.txt or point to a different canonical are the classic "sitemap contains junk" findings, and they are one dataset view away.

### How to use

1. Open the actor and put one or more domains or sitemap URLs into **Domains or sitemap URLs**.
2. Leave **Link check mode** on *Off* for a pure inventory, or choose *Check sitemap URLs* or *Check internal links on pages*.
3. Set **Max URLs per site** and, if you use check mode, **Max links to check** (this is your spend ceiling).
4. Run it. Open the **URL inventory**, **Flagged URLs** or **Site summaries** views, or download JSON, CSV or Excel.
5. Schedule it weekly if you want to diff sitemap health over time; the dataset is stable and sortable by `url`.

#### Quick start input

```json
{
  "startUrls": ["https://crawlee.dev"],
  "maxUrlsPerSite": 100,
  "checkMode": "sitemapUrls",
  "maxLinksToCheck": 10
}
```

#### All inputs

| Input | Default | Notes |
|---|---|---|
| `startUrls` | (required) | Domains, site URLs or direct sitemap URLs, one per row |
| `maxUrlsPerSite` | 1000 | 1 to 500,000 unique URLs per site; the download is aborted once reached |
| `maxSitemapFiles` | 50 | Files fetched per site, index files included (max 1000) |
| `includeDuplicates` | false | Emit a row for every repeat occurrence (`isDuplicate: true`); duplicates are always counted in the summary |
| `checkMode` | off | `off`, `sitemapUrls` or `pageLinks` |
| `pageUrls` | empty | Pages to scan in `pageLinks` mode (default: the home page of each start URL) |
| `maxLinksToCheck` | 100 | `sitemapUrls`: per site, evenly spaced sample. `pageLinks`: per page |
| `checkIndexability` | true | GET (first 256 KB) to read meta robots, plus the `X-Robots-Tag` header |
| `checkCanonical` | false | Compare `rel=canonical` with the sitemap URL |
| `checkConcurrency` | 5 | 1 to 20 parallel checks; per-host spacing still applies |
| `requestDelayMs` | 300 | Minimum gap between requests to one host (robots Crawl-delay overrides when larger, up to 10 s) |
| `requestTimeoutSecs` | 20 | Time to first response headers |
| `timeBudgetSecs` | 540 | Soft budget for the whole run; partial results are returned when it passes |
| `userAgent` | SitemapLinkInventoryBot/1.0 | Also the token matched against robots.txt |

### Check modes

**Off (default).** Inventory only. No requests are made to the URLs themselves, only to robots.txt and the sitemap files. The `robots-disallowed` and `off-host` flags still work because they are computed from the robots.txt that was already downloaded.

**Check sitemap URLs (`sitemapUrls`).** Requests up to `maxLinksToCheck` of the URLs per site. When the sitemap has more URLs than that, the actor takes an **evenly spaced sample across the whole inventory**, not the first N rows, so you see how the whole site behaves rather than one alphabetical corner. A warning in the site summary says how many were sampled. URLs that robots.txt disallows are not requested (they are flagged instead, for free). For each checked URL the actor records the status, the redirect chain and final URL, the response time and whether it used HEAD or GET. With `checkIndexability` on (the default) it uses GET and reads the first 256 KB, which is what makes the meta-robots `noindex` and canonical checks possible; with it off it sends HEAD and falls back to GET only when the server refuses HEAD (400, 403, 405, 406, 501) or answers with a 5xx or a network error.

**Check internal links on pages (`pageLinks`).** For each page in `pageUrls` (or each start URL's home page) the actor downloads the HTML, extracts the internal links (same site, `www.` ignored, fragments removed, duplicates collapsed), and checks up to `maxLinksToCheck` of them per page. Output rows have `recordType: "link"` with `sourcePage`, the link URL, anchor text, `nofollow`, and the same status fields. A link already checked from an earlier page in the run is reused (`cached: true`) and is not billed twice. The page download itself is not billed; every link actually requested is.

Concurrency is limited (default 5) and requests to one host are spaced by `requestDelayMs`, so checking a site does not hammer it. 100 URLs on one host at the default 300 ms spacing take roughly 30 to 60 seconds.

### Sample output

These rows come from a real platform run of the Quick start input above (run `eLV2GOV7aZpXOXpWp`, 100 sitemap rows plus 1 summary, 10 URLs checked, succeeded in about 6 seconds). The sitemap at crawlee.dev carries `changefreq` and `priority` but no `lastmod`, so `lastmod` is `null`; sites that publish it return it as written.

| url | changefreq | priority | pathGroup | source sitemap | HTTP | ms | method | flags |
|---|---|---|---|---|---|---|---|---|
| https://crawlee.dev/blog | weekly | 0.5 | /blog | crawlee.dev/sitemap.xml | 200 | 272 | GET | none |
| https://crawlee.dev/blog/crawlee-v3-16 | weekly | 0.5 | /blog | crawlee.dev/sitemap.xml | 200 | 599 | GET | none |
| https://crawlee.dev/blog/playwright-stealth-vs-fingerprint-spoofing | weekly | 0.5 | /blog | crawlee.dev/sitemap.xml | 200 | 871 | GET | none |
| https://crawlee.dev/blog/scrapy-vs-crawlee | weekly | 0.5 | /blog | crawlee.dev/sitemap.xml | 200 | 1168 | GET | none |
| https://crawlee.dev/js/api/3.10/basic-crawler | weekly | 0.5 | /js | crawlee.dev/sitemap.xml | 200 | 1503 | GET | none |

One URL row, in full:

```json
{
  "recordType": "url",
  "site": "crawlee.dev",
  "url": "https://crawlee.dev/blog",
  "lastmod": null,
  "changefreq": "weekly",
  "priority": 0.5,
  "sourceSitemap": "https://crawlee.dev/sitemap.xml",
  "sitemapDepth": 0,
  "pathGroup": "/blog",
  "pathDepth": 1,
  "hasQuery": false,
  "imageCount": 0,
  "images": [],
  "videoCount": 0,
  "videos": [],
  "hreflangCount": 0,
  "hreflang": [],
  "robotsDisallowed": false,
  "isDuplicate": false,
  "flags": [],
  "checked": true,
  "statusCode": 200,
  "finalUrl": "https://crawlee.dev/blog",
  "redirects": 0,
  "redirectChain": [],
  "responseMs": 272,
  "method": "GET",
  "error": null,
  "noindex": false,
  "noindexSource": null,
  "contentType": "text/markdown",
  "isBroken": false
}
```

The site summary row from the same run (abridged; the real row also lists the file-by-file results in `sitemapFiles`):

```json
{
  "recordType": "site-summary",
  "site": "crawlee.dev",
  "status": "ok",
  "discoveredBy": "robots.txt",
  "sitemapsDeclaredInRobots": ["https://crawlee.dev/sitemap.xml", "https://crawlee.dev/python/sitemap.xml"],
  "sitemapFilesParsed": 1,
  "totalUrls": 101,
  "uniqueUrls": 100,
  "duplicateUrls": 0,
  "truncatedByMaxUrls": true,
  "patternGroups": [
    { "pattern": "/js", "count": 62, "share": 62 },
    { "pattern": "/blog", "count": 37, "share": 37 },
    { "pattern": "/search", "count": 1, "share": 1 }
  ],
  "linksChecked": 10,
  "brokenLinks": 0,
  "chargedFiles": 1,
  "chargedChecks": 10,
  "warnings": [
    "Checked an evenly spaced sample of 10 of 100 checkable URLs (maxLinksToCheck).",
    "maxUrlsPerSite (100) reached; the sitemap has more URLs than were returned."
  ]
}
```

Notice that the cap warning is not hidden: the summary says plainly that more URLs exist than were returned, and the second sitemap declared in robots.txt (`/python/sitemap.xml`) was not fetched because the URL cap was reached first. Raise `maxUrlsPerSite` to read it.

The flag-heavy cases (404, redirect chains, noindex via header and via meta tag, canonical mismatch, redirect loops, timeouts, robots-disallowed entries, HEAD refused with GET fallback) are exercised by the actor's own offline test suite against a local mock site, not shown here as if they were production rows.

### What you get

- **A flat URL list** you can load straight into Sheets, a database, BigQuery or a diff tool. Columns are stable between runs.
- **A health table**: which URLs in the sitemap redirect, 404, error, carry noindex or are blocked by robots.txt. Filter the *Flagged URLs* view.
- **Structure at a glance**: pattern groups with counts and shares per first path segment, and the number of segments per URL.
- **Freshness signals**: newest and oldest `lastmod`, and how many URLs have no `lastmod` at all.
- **Duplicate evidence**: URLs listed more than once, with how many times and in which files (top 100 in the summary).
- **An honest audit trail**: every sitemap file with its HTTP status, type, bytes, whether it was gzip, whether it was partial, and whether it was billed.
- **Status per site**: `ok`, `no-sitemap-found`, `invalid-input`, `blocked-non-public`, `partial-time-budget`, `partial-error` or `charge-limit-reached`, always with warnings in plain English.

### Monitor mode: only new or updated URLs

Set `monitorMode` to `true` to get a change feed of a site's sitemap. The first run is a **baseline** and returns every URL. Later runs return only URL rows that are **new** or whose sitemap **`lastmod` changed**, each with `changeType` = `new` or `changed`. The `site-summary` row is returned on every run so you can see what was read.

- Baseline lives in the key-value store `sitemap-link-inventory-monitor` under `MONITOR_<monitorName>`; use one `monitorName` per set of sites.
- Pricing is per sitemap file read and per link checked, so monitor mode reduces the rows you handle, not the file reads. It is designed for `checkMode` = `off`; with link checks on, checks still run and are charged for every URL they reach.
- Sites that publish no `lastmod` can only report new URLs, not updated ones. URLs removed from a sitemap are not reported.
- The baseline holds up to 200,000 URLs. Beyond that, extra URLs keep being returned as new on every run, and the actor says so in the run summary.

**Schedule it.** Open the actor in Apify Console, click **Save as a new task** with `monitorMode` on and a fixed `monitorName`, then open **Schedules** > **Create new schedule** and add that task on a cron such as `0 7 * * *` (daily at 07:00). Add a webhook or an integration (Slack, email, Make, Zapier, n8n) to the task's **Run succeeded** event so each run's delta is delivered where you need it. A run that finds no changes returns zero rows and incurs no per-result charges.

#### Removed records (`reportRemoved`, opt-in)

Turn on `reportRemoved` (together with `monitorMode`) and a run also emits URLs that were in your saved baseline but are absent now, once each, with `changeType: "removed"`, `lastSeenOn` and a `removedNote`. **Removed rows are never charged**; new and changed rows are priced exactly as before. Nothing changes if you leave it off.

Honesty guards: a URL is reported removed only for a site whose sitemap files were all read completely (no `maxUrlsPerSite` truncation, no sitemap file cap, no failed or HTTP-error sitemap file). The row is skipped (and the reason is logged and in `OUTPUT.monitor.removedSkipped`) when the fetch was incomplete or hit an error, when you changed the filters since the baseline was built, or when the baseline was created before this option existed (run once with it on to build a scope-tagged baseline). A removed row means "not returned any more", which can also mean the source retired or re-filtered the record.

### Pricing

Pay per event. You are charged for three things only.

| Event | Price | When it is charged |
|---|---|---|
| Actor start | $0.00005 | Once per run (platform standard) |
| Sitemap file parsed | $0.001 | Once per sitemap file (XML, `.gz` or `.txt`) that was fetched and from which at least one URL was delivered |
| Link checked | $0.001 | Once per URL that check mode actually requested |

What is **not** charged: sitemap index files (they only contain pointers), sitemap files that 404 or time out, HTML pages that are not sitemaps, files with no URLs, robots.txt, URLs skipped because robots.txt disallows them, hosts refused by the SSRF guard, the page downloads in `pageLinks` mode, and URL rows themselves. Delivering a million URLs from one file costs the same $0.001 as delivering ten, which is why the title says "$1/1k": it is the price of a thousand checked links, the part that scales with your input.

Worked examples (platform compute is a few hundredths of a cent on top; the last example shows a measured run):

| Scenario | Billed events | Cost |
|---|---|---|
| Inventory of a 50,000-URL site split over 5 sitemap files, check mode off | 5 files | $0.00505 |
| Inventory of a small site with one sitemap, 100 URLs, check mode off | 1 file | $0.00105 |
| Same 100-URL site, check mode on, every URL checked (`maxLinksToCheck` 100) | 1 file + 100 links | $0.10105 |
| Large site, 20,000 URLs in 20 files, weekly health sample of 200 URLs | 20 files + 200 links | $0.22005 |
| 10 domains, each with one sitemap, inventory only | 10 files | $0.01005 |
| `pageLinks` on 3 pages with 40 unique internal links each, 20 shared across pages | up to 100 links (shared ones billed once) | about $0.10 |
| Measured: Quick start input above (1 file, 10 links checked) | 1 file + 10 links | $0.01105 plus compute |

You can cap spend two ways: set `maxLinksToCheck` (the real ceiling in check mode) and use the platform's maximum-charge-per-run setting. When the charge limit is reached the actor stops cleanly, returns what it has and reports `charge-limit-reached` in the summary.

### Integrations: API, n8n, Make, Zapier

**Run it from the API (JavaScript):**

```js
import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('transparent_meteorite/sitemap-link-inventory').call({
  startUrls: ['example.com'],
  maxUrlsPerSite: 5000,
  checkMode: 'sitemapUrls',
  maxLinksToCheck: 100,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
const broken = items.filter((r) => r.recordType === 'url' && r.isBroken);
```

**Python:**

```python
from apify_client import ApifyClient
client = ApifyClient("<APIFY_TOKEN>")
run = client.actor("transparent_meteorite/sitemap-link-inventory").call(run_input={
    "startUrls": ["example.com"],
    "checkMode": "sitemapUrls",
    "maxLinksToCheck": 50,
})
for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    if row.get("recordType") == "site-summary":
        print(row["site"], row["status"], row["uniqueUrls"], row["brokenLinks"])
```

**n8n.** Add the *Apify* node (or an *HTTP Request* node), choose *Run actor*, set the actor to `transparent_meteorite/sitemap-link-inventory`, paste the input JSON, and enable "wait for finish". A second node reads the default dataset. Filter on `recordType == "site-summary"` and `brokenLinks > 0` to alert only when something is wrong.

**Make.** Use the *Apify* app, module *Run an Actor*, then *Get Dataset Items*. Map `startUrls` from a Google Sheet column, and use a filter on `flags` containing `not-found` to feed a Slack or email module.

**Zapier.** Use the *Apify* app's *Run Actor* action with a schedule trigger, then *Get Dataset Items*. Send flagged rows to Sheets, Airtable or a ticketing tool. For a run that may take longer than Zapier's polling window, use the Apify webhook (`ACTOR.RUN.SUCCEEDED`) instead of waiting.

**Scheduling and diffing.** Schedule the actor in Apify (daily or weekly) and compare datasets between runs on `url` to detect added, removed or newly broken pages.

### Use with AI agents and MCP

The actor works through the Apify MCP server at `https://mcp.apify.com` (add `?tools=transparent_meteorite/sitemap-link-inventory` to load only this actor), so Claude, ChatGPT, Cursor and other MCP clients can call it as a tool. The actor id is `JZZL7210O5lLZZ8b2` (name `transparent_meteorite/sitemap-link-inventory`).

Example prompts:

- "Use sitemap-link-inventory on example.com with checkMode sitemapUrls and maxLinksToCheck 50. List every flagged URL and the sitemap file it came from."
- "Inventory the sitemaps of these five competitor domains with checkMode off and compare their pattern groups."
- "Which URLs does example.com list in its sitemap that robots.txt disallows?"

Keep `maxLinksToCheck` small in agent prompts; the agent pays per link checked.

### Use it from code

Small real input: it returns only a couple of rows, so a test run costs very little (see Pricing). Set the `APIFY_TOKEN` environment variable to your token from [Apify Console > Settings > API & Integrations](https://console.apify.com/settings/integrations).

**Python** (`pip install apify-client`)

```python
import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run_input = {'startUrls': ['https://crawlee.dev'], 'maxUrlsPerSite': 20, 'maxSitemapFiles': 2, 'checkMode': 'off'}
run = client.actor("transparent_meteorite/sitemap-link-inventory").call(run_input=run_input)
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)
```

**JavaScript / Node.js** (`npm install apify-client`)

```js
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const input = {
  "startUrls": [
    "https://crawlee.dev"
  ],
  "maxUrlsPerSite": 20,
  "maxSitemapFiles": 2,
  "checkMode": "off"
};
const run = await client.actor('transparent_meteorite/sitemap-link-inventory').call(input);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);
```

**curl** (runs the actor and returns the dataset items in one call)

```bash
curl -X POST "https://api.apify.com/v2/acts/transparent_meteorite~sitemap-link-inventory/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": ["https://crawlee.dev"], "maxUrlsPerSite": 20, "maxSitemapFiles": 2, "checkMode": "off"}'
```

### Send results to Google Sheets, Zapier, Make or n8n

- **Google Sheets:** every run's dataset has a CSV export link: `https://api.apify.com/v2/datasets/<DATASET_ID>/items?format=csv&clean=true` (the dataset ID is on the run's Storage tab). In Sheets, `=IMPORTDATA("<that link>")` loads it. Anyone with the link can read that dataset, so share it carefully. To export to Google Drive automatically, use the Google Drive integration on the actor's Integrations tab.
- **Zapier:** in the Apify app, use the trigger **Finished Actor run** (or **Run Actor** as an action) with `transparent_meteorite/sitemap-link-inventory`, then **Fetch dataset items** and pass the rows to Google Sheets, a CRM or Slack.
- **Make:** the Apify app has **Watch Actor Runs**, **Run an Actor** and **Get Dataset Items** modules; connect them to any Make app.
- **n8n:** install the Apify node (`@apify/n8n-nodes-apify`) and use **Run an Actor and get dataset**, or the **On new Apify event** trigger.
- **Schedule and webhooks:** create a Task with your input, add a Schedule in Apify Console, and add a webhook if you want a call when a run finishes.

See the [Apify integrations docs](https://docs.apify.com/platform/integrations) for setup steps.

### Use cases

- **Technical SEO audits.** Find sitemap entries that redirect, 404, carry noindex or are blocked by robots.txt before they cost crawl budget.
- **Migrations and redesigns.** Snapshot the full URL inventory before launch, check it again after, and diff.
- **Competitor research.** See how a competitor structures content (pattern groups), how fresh it is (`lastmod`) and how large each section is, using only what they publish.
- **Content and link-building prospecting.** Pull all `/blog` or `/resources` URLs of a domain as a list to work through.
- **Seeding crawlers.** Feed the URL list to a crawler or to the sister actor below for per-page audits, instead of crawling to discover URLs.
- **International SEO.** The hreflang entries in sitemaps come out as structured `{lang, href}` pairs per URL.
- **Image and video SEO.** Image and video sitemap entries are returned with their URLs and metadata.
- **Monitoring.** A scheduled run with a small sample check is a cheap alarm for "the sitemap started returning errors".
- **Agencies.** One input row per client domain gives a comparable summary row each, with status and warnings.

### Frequently asked questions

#### Does it respect robots.txt?

Yes. The actor identifies itself as `SitemapLinkInventoryBot` and applies the matching robots.txt group (falling back to `*`) with longest-match, allow-wins-ties rules and the `*` and `$` wildcards. It will not request a URL that robots.txt disallows for it, and it honours `Crawl-delay` up to 10 seconds. One deliberate exception: sitemap files that robots.txt itself declares with a `Sitemap:` line are fetched, because publishing them is an invitation to read them. Common-path probes (`/sitemap.xml` and friends) are only attempted when robots.txt allows them.

#### What happens when a sitemap is broken, huge or slow?

Broken pieces are recorded, not fatal. A missing child file shows up in `sitemapFiles` with `error: "http-404"`; an HTML page returned in place of a sitemap is rejected; a truncated `.gz` keeps every URL read before the cut and is marked `partial`; a connection that goes silent for 30 seconds is abandoned; the run has a soft time budget after which it returns partial results. Huge files are streamed and stop at `maxUrlsPerSite`. The run itself exits cleanly in all of these cases.

#### Why did I get `no-sitemap-found`?

The site has no sitemap in robots.txt and none at the common paths, the sitemap is behind a login or bot wall, or it lives at an unusual URL. Pass the sitemap URL directly as an input row. If the site returns an HTML page for every path, the actor recognises that as "not a sitemap" rather than parsing it.

#### Does it follow nested sitemap indexes and read gzip files?

Yes to both. Index files are followed up to six levels deep, bounded by `maxSitemapFiles`. A child that appears twice is fetched once. Gzip is detected from the first two bytes of the body, so it works whatever the file is called and however the server labels it. Decompressed size is capped at 300 MB per file as a guard against decompression bombs.

#### How is a duplicate defined?

Two sitemap entries with exactly the same URL string (fragment removed). `http://` versus `https://`, trailing slash versus none, and different query-string order count as different URLs because the sitemap lists them as different strings. Duplicates are counted in the site summary (`duplicateUrls`, `duplicateOccurrences`), with the top 100 listed with their occurrence counts and source files. Turn on `includeDuplicates` to also get a row per repeat.

#### Why is a URL flagged `blocked` rather than `not-found` or broken?

A 401, 403 or 429 means the server refused our request, not that the page is missing. Treating those as broken links produces false alarms, particularly on sites behind bot protection, so they get the `blocked` flag and `isBroken: false`. Check them in a browser before acting.

#### How is this different from the technical-seo-page-audit actor?

This actor works at the site-inventory level: which URLs are in the sitemaps, how they group, and whether they respond. [technical-seo-page-audit](https://apify.com/transparent_meteorite/technical-seo-page-audit) goes deep on single pages: title, meta description, headings, canonical, hreflang, structured data types, images without alt text, link counts and a scored issues list. A common workflow is to run this actor first, then feed a sample of its URLs to the page audit.

#### Can it check pages that need JavaScript to render?

No. It makes plain HTTP requests and reads the HTML as the server sends it. Sitemaps are static files so inventory is unaffected, but internal links that only appear after client-side rendering will not be found in `pageLinks` mode, and a meta robots tag injected by JavaScript will not be seen.

#### Can I run it on private or intranet sites?

No. Hosts that are loopback, private (RFC 1918), link-local, carrier-grade NAT or that resolve to such addresses are refused, as are non-http(s) schemes. This is a deliberate safeguard, not a bug.

#### Does it use proxies or log in anywhere?

No proxies, no cookies, no login. If a site blocks Apify's datacenter IP ranges, the request fails and you will see it as `unreachable` or `blocked`.

### More questions

**How do I get every URL from a website's sitemap?**
Enter the domain in `startUrls`. The actor finds the sitemaps through robots.txt and common paths, follows nested sitemap indexes, and returns one row per URL with `lastmod`, the source sitemap file and flags.

**How do I find broken links and redirects in a sitemap?**
Set `checkMode` to `sitemapUrls`. It requests up to `maxLinksToCheck` URLs per site (an evenly spaced sample across the whole inventory) and reports status codes, redirects and optional noindex and canonical checks.

**Can it read sitemap index files and .gz sitemaps?**
Yes. It follows nested indexes up to `maxSitemapFiles` and reads plain XML, `.gz` and `.txt` sitemap files.

**Does it respect robots.txt?**
Yes. It applies the robots.txt group for its own user agent and will not request a URL robots.txt disallows. Sitemap files that robots.txt itself declares are still fetched.

**How much does it cost to export a sitemap?**
$0.001 per sitemap file that delivered URLs and $0.001 per link checked, plus the platform start fee. Check mode is off by default, so a 50,000-URL site split over five files costs about half a cent.

**Can it crawl pages that need JavaScript?**
No. It reads public sitemap files and plain HTML over HTTP, with no browser. Pages that only exist after JavaScript runs are not discovered.

### How this compares

Desktop crawlers and SEO suites can also read sitemaps and often render JavaScript. This actor is API-first: no browser or install, plain HTTP, robots.txt honoured, one dataset you can schedule or call from code, and a pay-per-file price. It does not render JavaScript or crawl beyond what the sitemaps and the links you ask it to check contain.

### Limitations

- **Plain HTTP only.** No JavaScript rendering. Soft-404 detection is limited to recognising HTML where a sitemap should be; for pages, a 200 response is a 200.
- **Noindex and canonical need HTML.** The meta robots and canonical checks read the first 256 KB of an HTML response. The `X-Robots-Tag` header works for any response. In the sample run above crawlee.dev returned `text/markdown` for the checked pages, so no meta-tag inspection was possible there; the field `contentType` tells you what was received.
- **The 100-URL check cap is per site by default.** In `sitemapUrls` mode you get a sample, not a full census, unless you raise `maxLinksToCheck` (up to 5,000 per site). Checking a million URLs is not the intended use of this actor.
- **Politeness is slow by design.** With the default 300 ms spacing, one host yields roughly 3 requests per second at most. Raise `checkConcurrency` only if the target can take it; it does not bypass the per-host spacing.
- **The "off-host" robots check is skipped for other hosts.** The `robotsDisallowed` field is computed only for URLs on the input host (ignoring `www.`). Off-host URLs get `null` for it. When an off-host URL is checked, that host's own robots.txt is fetched and respected for the request.
- **Cross-run state is not kept.** The actor does not diff against earlier runs by itself; compare datasets on `url`.
- **Dates are passed through.** `lastmod` is returned as the sitemap wrote it. Summary-level newest/oldest values are parsed where the date is valid and ignored where it is not.
- **DNS checks are not atomic.** The SSRF guard resolves a hostname before connecting; an attacker-controlled DNS server that changes its answer between the check and the connection (rebinding) is outside what plain `fetch` allows us to prevent. Running on the Apify platform limits exposure, but do not treat the guard as a security boundary for untrusted input at scale.
- **Sites can still block you.** Some sites rate-limit or blocklist cloud IPs. Raise `requestDelayMs` and reduce concurrency if you see 429s.
- **Very large inventories need memory.** Duplicate detection keeps the set of seen URLs in memory, which is why the cap is 500,000 unique URLs per site and the default run size is 512 MB.

### Related actors

- [technical-seo-page-audit](https://apify.com/transparent_meteorite/technical-seo-page-audit): per-page SEO audit (title, meta, headings, canonical, hreflang, JSON-LD types, images without alt, broken links, redirect chain, TTFB, scored issues).
- [domain-intelligence-lookup](https://apify.com/transparent_meteorite/domain-intelligence-lookup): WHOIS/RDAP, DNS records with SPF and DMARC, certificate-transparency subdomains and SSL expiry for a list of domains.
- [shopify-store-leads](https://apify.com/transparent_meteorite/shopify-store-leads): detect Shopify stores and summarise their public catalog.

### Support and feedback

Open an issue on the actor's page with the input you used and the run id. If a sitemap parses wrongly, include its URL: real-world sitemaps are the test suite that matters, and each report becomes a regression test.

### Reliability

Last 30 days, measured from Apify run records (all runs, including our own daily health checks): **33 of 33 finished runs succeeded (100.0%)**, median run time **2.6 s** (90th percentile 6.3 s). Aborted runs (0) are not counted.

Updated 2026-10-11 automatically every week from the run history. Report any failure on the Issues tab with the run URL.

### More resources

- Sample output and example input on GitHub (masked sample CSV): [github.com/Mamoo85/apify-actor-samples/sitemap-link-inventory](https://github.com/Mamoo85/apify-actor-samples/tree/main/sitemap-link-inventory)

### Changelog

- 2026-10-09: First release.
- 2026-10-09: Input sections with a tiny prefill, run status messages, friendly input errors, examples, API snippets, integrations how-tos, FAQ.
- 2026-10-09: Branding: maintained by Detroit Web Agent - Open Data.
- 2026-10-11: Closing log line points to the Issues tab; Reliability section added to this README.

### Works well with

Other actors from Detroit Web Agent - Open Data that pair with this one:

- [Technical SEO Page Audit](https://apify.com/transparent_meteorite/technical-seo-page-audit): audit individual pages from the URL list you just exported (titles, canonicals, headings, issues and score).
- [Domain Intelligence Lookup](https://apify.com/transparent_meteorite/domain-intelligence-lookup): check DNS, SSL expiry and security headers for the domains you inventory.
- [Website Tech Stack Detector](https://apify.com/transparent_meteorite/website-tech-stack-detector): see which CMS, frameworks and analytics each site uses.
- [Shopify Store Leads](https://apify.com/transparent_meteorite/shopify-store-leads): summarise the public catalog of a Shopify store whose sitemap you just read.

### What the free plan gets you

Free plan: about 5,000 sitemap files parsed or links checked from your $5 monthly Apify credit (at $0.001 each, today's price); default settings return about 50 URLs from one sitemap file, for well under 1 cent. You are charged per sitemap file parsed and per link checked, not per URL row. Need more? Upgrade your Apify plan or set a higher result limit in the input.

### Need more than the data?

This actor is built by [Detroit Web Agent](https://www.detroitwebagent.com/?utm_source=apify\&utm_medium=readme\&utm_campaign=sitemap-link-inventory), a small Detroit studio that builds websites, lead pages and simple data tools for small businesses. If you want these results turned into a working website, a landing page or a CRM import, or you need a field this actor does not cover yet, open an issue on the Issues tab or visit [detroitwebagent.com](https://www.detroitwebagent.com/?utm_source=apify\&utm_medium=readme\&utm_campaign=sitemap-link-inventory).

***

Built and maintained by **Detroit Web Agent - Open Data** (Detroit, Michigan). Questions, bugs or a field you need: open an issue on this Actor's Issues tab and we will reply there.

# Actor input Schema

## `startUrls` (type: `array`):

One per row. A bare domain or site URL (https:// is added if missing) triggers discovery: robots.txt Sitemap lines first, then common paths such as /sitemap.xml. A URL that looks like a sitemap (.xml, .xml.gz, .txt, or 'sitemap' in the path) is fetched directly. Public hosts only; private IPs and localhost are refused.

## `maxUrlsPerSite` (type: `integer`):

Hard cap on unique URLs returned per site. Reading stops (and the download is aborted) once it is reached, so a 2-million-URL sitemap does not stall the run. The site summary says when the cap truncated results.

## `checkMode` (type: `string`):

off: inventory only. sitemapUrls: request the sitemap URLs (HEAD, GET fallback) and flag 404, 5xx, unreachable, redirects and noindex. pageLinks: request the internal links found on the pages listed below (or the home page of each start URL). Each request made is a billable 'link checked' event.

## `maxSitemapFiles` (type: `integer`):

Cap on sitemap files fetched per site, counting index files. Nested indexes are followed up to 6 levels deep.

## `includeDuplicates` (type: `boolean`):

Duplicates are always counted and listed in the site summary. Turn this on to also emit a row (isDuplicate = true) for every repeat occurrence.

## `pageUrls` (type: `array`):

Pages whose internal links are checked when Link check mode is 'pageLinks'. Leave empty to use the home page of each start URL. The page fetch itself is not billed; each link checked is.

## `maxLinksToCheck` (type: `integer`):

sitemapUrls mode: per site, an evenly spaced sample across the whole inventory (not just the first rows). pageLinks mode: per page. This is the cost ceiling for check mode.

## `checkIndexability` (type: `boolean`):

Uses a GET for each checked URL (first 256 KB) to read meta robots, plus the X-Robots-Tag header, so noindex pages in the sitemap are flagged. Turn off for HEAD-only, lighter checks.

## `checkCanonical` (type: `boolean`):

Compare each checked page's rel=canonical with its sitemap URL (trailing slash ignored). Off by default.

## `checkConcurrency` (type: `integer`):

Parallel link checks. Requests to the same host are still spaced by the delay below.

## `requestDelayMs` (type: `integer`):

Minimum gap between requests to the same host. A larger robots.txt Crawl-delay (up to 10 s) overrides this.

## `requestTimeoutSecs` (type: `integer`):

Time to first response headers for each request. Slow servers become 'timeout' rows, never failed runs.

## `timeBudgetSecs` (type: `integer`):

Soft limit for the whole run. When it passes, the actor stops fetching and returns what it has with status 'partial-time-budget'.

## `userAgent` (type: `string`):

Sent on every request and matched against robots.txt (the default token is SitemapLinkInventoryBot).

## `monitorMode` (type: `boolean`):

Recurring-run mode: return only sitemap URLs that are new or whose lastmod changed since the last monitor run; sites without lastmod only report new URLs. Best with Check mode off. The first run is a baseline and returns everything in scope. Unchanged records are not returned and not charged. Schedule the actor (Schedules tab) to get a recurring feed. Each row gets a changeType of new or changed.

## `reportRemoved` (type: `boolean`):

With monitor mode on, also emit one row with changeType "removed" for URLs that were in a site's sitemap in an earlier run and are gone now, only for sites whose sitemaps were read completely this run. Removed rows are free: they are not charged and never count toward your charge limit or max results. Nothing is reported as removed on the baseline run, when the fetch was incomplete or hit a cap, or when your input filters changed since the baseline (the next run re-learns the baseline instead). Default off.

## `monitorName` (type: `string`):

Name of the saved baseline. Use a different name per watch list so schedules do not overwrite each other. Default is 'default'.

## Actor input object example

```json
{
  "startUrls": [
    "https://crawlee.dev"
  ],
  "maxUrlsPerSite": 50,
  "checkMode": "off",
  "maxSitemapFiles": 3,
  "includeDuplicates": false,
  "maxLinksToCheck": 100,
  "checkIndexability": true,
  "checkCanonical": false,
  "checkConcurrency": 5,
  "requestDelayMs": 300,
  "requestTimeoutSecs": 20,
  "timeBudgetSecs": 540,
  "userAgent": "SitemapLinkInventoryBot/1.0 (+https://apify.com/transparent_meteorite)",
  "monitorMode": false,
  "reportRemoved": false,
  "monitorName": "default"
}
```

# Actor output Schema

## `overview` (type: `string`):

No description

## `summaries` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://crawlee.dev"
    ],
    "maxUrlsPerSite": 50,
    "checkMode": "off",
    "maxSitemapFiles": 3
};

// Run the Actor and wait for it to finish
const run = await client.actor("transparent_meteorite/sitemap-link-inventory").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://crawlee.dev"],
    "maxUrlsPerSite": 50,
    "checkMode": "off",
    "maxSitemapFiles": 3,
}

# Run the Actor and wait for it to finish
run = client.actor("transparent_meteorite/sitemap-link-inventory").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://crawlee.dev"
  ],
  "maxUrlsPerSite": 50,
  "checkMode": "off",
  "maxSitemapFiles": 3
}' |
apify call transparent_meteorite/sitemap-link-inventory --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,transparent_meteorite/sitemap-link-inventory"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/JZZL7210O5lLZZ8b2/builds/lBxsGOYjJhwB7D7FX/openapi.json