# Sitemap URL Extractor: All Website URLs and Status Codes (`fernandosord/sitemap-url-extractor`) Actor

Give a website or a sitemap link and get every URL in its sitemaps, found through robots.txt and the usual paths, indexes and .gz files followed. Each URL comes with lastmod, priority, image count and hreflang alternates. Filter by pattern or date, and optionally check each URL's HTTP status.

- **URL**: https://apify.com/fernandosord/sitemap-url-extractor.md
- **Developed by:** [Plainsource](https://apify.com/fernandosord) (community)
- **Categories:** SEO tools, Developer tools
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 urls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

This sitemap URL extractor finds every sitemap of a website and lists every URL in them. Give it a website (`example.com`) and it reads the `Sitemap:` lines in robots.txt, checks the usual places such as `/sitemap.xml`, `/sitemap_index.xml` and `/wp-sitemap.xml`, follows every sitemap index and unpacks `.gz` files. Give it a sitemap link and it starts there. Each URL comes back with its lastmod date, changefreq, priority, the sitemap it came from, the number of images and videos, hreflang alternates and Google News title and date. Turn on the status check to find broken and redirecting URLs.

- **Finds sitemaps for you.** No need to know where a site keeps them. robots.txt and nine common paths are checked, and nested indexes are followed up to six levels deep.
- **Handles real-world sitemaps.** Gzip files, image, video and news extensions, hreflang `xhtml:link` alternates, plain text sitemaps (one URL per line) and RSS or Atom feeds used as sitemaps.
- **Filters before you pay.** Keep only `/blog/`, drop `*?page=*`, use a regular expression, or keep only URLs modified in the last 30 days. Filtered URLs are not saved and cost nothing. With a date filter, sitemap files that are older than the cut-off are skipped without being downloaded.
- **Optional status check.** Each URL gets a HEAD request (with a GET fallback for servers that refuse HEAD) and you get its status code, where it redirects and the final status.

### What data you get

| Field | What it holds |
|---|---|
| `site` | The website the URL was found for |
| `url` | The page URL from the sitemap |
| `lastmod`, `changefreq`, `priority` | The sitemap values, when the site provides them |
| `sourceSitemap` | The sitemap file the URL was listed in |
| `imageCount`, `images` | Number of images from image sitemaps, and their URLs with **Include image URLs** |
| `videoCount` | Number of videos from video sitemaps |
| `alternates` | hreflang alternates: language code and URL |
| `newsTitle`, `newsPublishedAt` | Title and publication date from Google News sitemaps |
| `statusCode`, `finalStatusCode`, `finalUrl`, `redirected`, `statusError` | With **Check the HTTP status**: the URL's own status (301, 404...), the status after redirects, where it ends up, and the network error if it could not be reached |

The run's key-value store gets a `SUMMARY` record with, for each site: every sitemap found (type, how it was found, URL count, child sitemaps), the common paths that had no sitemap, errors such as HTTP 403 or a sitemap that returned a web page, and how many URLs were skipped as duplicates or by your filters. With the status check on, it also counts URLs by status code.

### How to use it

1. Add websites or sitemap URLs to **Websites or sitemap URLs**. Many sites in one run are fine.
2. Set **Max URLs** to cap the run. Each saved URL is one result.
3. Optional: add **Only URLs matching** or **Skip URLs matching** patterns, a date in **Only URLs modified after**, or turn on **Check the HTTP status of each URL**.
4. Run it and download the list as CSV, Excel or JSON, or call it from the API, Make, Zapier, n8n or an AI agent through the Apify MCP server.

Patterns come in four kinds:

| Pattern | Matches |
|---|---|
| `/blog/` | URLs whose path starts with /blog/ |
| `*/products/*` | A glob on the full URL; `*` matches anything |
| `/[0-9]{4}/[0-9]{2}/$/` | A regular expression between slashes (it must use a regex symbol, so `/blog/` stays a path) |
| `pricing` | URLs that contain the word |

#### Example input

```json
{
  "startUrls": [
    "https://wordpress.org/news",
    "https://www.theguardian.com/sitemaps/news.xml",
    "https://www.deepl.com/sitemap.xml"
  ],
  "maxUrlsPerSite": 100,
  "checkStatus": true,
  "includePatterns": ["/news/", "*theguardian.com/*", "/de/"]
}
```

#### Example output (from a real run, one URL per site, alternates shortened)

```json
[
  {
    "site": "https://www.theguardian.com",
    "url": "https://www.theguardian.com/media/commentisfree/2026/oct/30/liberal-senator-sarah-henderson-mistake-abc-gaza-stringer-ntwnfb",
    "lastmod": "2026-10-30T02:44:00Z",
    "changefreq": "hourly",
    "priority": null,
    "sourceSitemap": "https://www.theguardian.com/sitemaps/news.xml",
    "imageCount": 1,
    "videoCount": 0,
    "alternates": [],
    "newsTitle": "Liberal senator admits mistake after grilling ABC execs about Gaza journalist’s ‘Hamas’ ID",
    "newsPublishedAt": "2026-10-30T02:37:22Z",
    "statusCode": 200,
    "finalStatusCode": 200,
    "finalUrl": "https://www.theguardian.com/media/commentisfree/2026/oct/30/liberal-senator-sarah-henderson-mistake-abc-gaza-stringer-ntwnfb",
    "redirected": false,
    "statusError": null
  },
  {
    "site": "https://www.deepl.com",
    "url": "https://www.deepl.com/de/translator",
    "lastmod": null,
    "changefreq": null,
    "priority": null,
    "sourceSitemap": "https://www.deepl.com/p-JidqPmeasZMtTlNiGpFf0gzd.xml",
    "imageCount": 0,
    "videoCount": 0,
    "alternates": [
      { "hreflang": "x-default", "url": "https://www.deepl.com/translator" },
      { "hreflang": "pt-br", "url": "https://www.deepl.com/pt-BR/translator" }
    ],
    "newsTitle": null,
    "newsPublishedAt": null,
    "statusCode": 200,
    "finalStatusCode": 200,
    "finalUrl": "https://www.deepl.com/de/translator",
    "redirected": false,
    "statusError": null
  },
  {
    "site": "https://wordpress.org",
    "url": "https://wordpress.org/news/prototype-of-the-wayfinder-in-action/",
    "lastmod": "2023-06-27T16:50:02Z",
    "changefreq": null,
    "priority": null,
    "sourceSitemap": "https://wordpress.org/news/video-sitemap-1.xml",
    "imageCount": 0,
    "videoCount": 1,
    "alternates": [],
    "newsTitle": null,
    "newsPublishedAt": null,
    "statusCode": 301,
    "finalStatusCode": 200,
    "finalUrl": "https://wordpress.org/news/files/2023/06/Prototype-of-the-Wayfinder-in-action.mp4",
    "redirected": true,
    "statusError": null
  }
]
```

That run found 300 URLs across the three sites, checked all of them in 23 seconds, and flagged 4 that redirect.

### How much does it cost?

You pay $0.001 per URL saved, plus $0.0005 once per run. The status check is included at no extra cost. URLs removed by your filters, duplicates listed in more than one sitemap, and sites where no sitemap could be found cost nothing.

1,000 URLs cost $1.00, and 10,000 URLs cost $10.00. Set **Max URLs** to cap a run on a large site.

### Use cases

- **SEO audits:** get the full list of URLs a site submits to Google, then check which of them return 404 or redirect. Sitemaps should only list final 200 URLs.
- **Content inventories and migrations:** export every page with its last modified date before a redesign or a CMS move, and check the redirects afterwards.
- **Feed other tools:** use the URL list as input for a crawler, a screenshot tool, a page speed test or an AI extraction run.
- **Track new pages:** schedule a run with **Only URLs modified after** set to `1 day` to get the pages a competitor or news site added or changed since yesterday.
- **hreflang checks:** see which language versions each page declares.

### FAQ

**What if a site has no sitemap?**
The Actor checks robots.txt and the common paths. If none of them has a sitemap, the site is listed in `SUMMARY` with that reason and costs nothing. It does not crawl the site's links to build a URL list; for that, use a crawler.

**Does it follow sitemap indexes on other domains?**
Yes. Sitemap URLs listed in robots.txt or in an index are followed wherever they point, and page URLs on other hosts are kept, because a sitemap can list pages on a subdomain or another domain the site owns.

**What if I give a website with a path, like example.com/blog?**
robots.txt is read at the domain root, so you get every sitemap of the domain, and `/blog/sitemap.xml` and similar paths are checked too. Add `/blog/` to **Only URLs matching** to keep just that section.

**Why is lastmod empty for some URLs?**
The site did not include it. lastmod, changefreq and priority are optional in the sitemap protocol, and many sites leave them out. With a date filter, URLs without lastmod are dropped unless you turn on **Keep URLs without a lastmod date**.

**How fast is the status check, and will it get me blocked?**
It runs 10 checks in parallel by default (up to 50). One test run checked 300 URLs on three sites in 23 seconds. If a site answers 429 Too Many Requests, the Actor waits and retries, but lower **Status checks in parallel** for sites that rate-limit. A site that blocks datacenter traffic may answer 403 to every request; try Apify Proxy for those.

**What does the status check not tell me?**
It reads only the HTTP status and redirects. It does not render JavaScript, so a page that returns 200 but shows "not found" in the browser (a soft 404) is reported as 200.

**Are there size limits?**
A sitemap file can hold up to 50,000 URLs by the protocol, and the Actor reads files of that size. It stops after 10,000 sitemap files per site as a safety limit.

### Other Actors by this developer

- [SEO Site Audit](https://apify.com/fernandosord/seo-site-audit): crawls a site and lists broken links, redirect chains, missing titles and descriptions, canonical, hreflang and sitemap errors.
- [Website Change Monitor](https://apify.com/fernandosord/website-change-monitor): watch the pages from your sitemap and get the exact lines that changed, with alerts by email, Slack or webhook.
- [Lighthouse and PageSpeed Audit](https://apify.com/fernandosord/lighthouse-page-speed-monitor): Lighthouse scores and Core Web Vitals for many URLs, no PageSpeed API key.
- [Website Tech Stack Detector](https://apify.com/fernandosord/website-tech-stack-detector): the CMS, analytics, frameworks and hosting behind a list of sites.
- [Website Screenshot and PDF](https://apify.com/fernandosord/website-screenshot-pdf): full-page screenshots or PDFs of a list of URLs.

# Actor input Schema

## `startUrls` (type: `array`):

Any mix of websites (example.com, https://www.example.com/blog) and direct sitemap links (https://example.com/sitemap.xml, .xml.gz, .txt). For a website, the Actor reads the Sitemap lines in robots.txt and checks the usual paths such as /sitemap.xml, /sitemap_index.xml and /wp-sitemap.xml, then follows every sitemap index.

## `maxUrls` (type: `integer`):

The run stops after saving this many URLs in total. Each saved URL is one result, so this also caps the cost.

## `maxUrlsPerSite` (type: `integer`):

Optional. Stop reading a site after this many URLs and move on to the next one. Leave empty for no per-site limit.

## `includePatterns` (type: `array`):

Optional. Keep only URLs that match at least one pattern. Three kinds: a path start such as /blog/ (matches the URL path), a glob with \* such as */products/* or https://example.com/*/2026/*, or a regular expression between slashes such as /\[0-9]{4}/\[0-9]{2}/$/ (it must use a regex symbol such as \[ ], ( ), | or $, so a plain path like /blog/ stays a path start). A plain word such as pricing keeps URLs that contain it.

## `excludePatterns` (type: `array`):

Optional. Drop URLs that match any of these patterns. Same pattern kinds as above, for example /tag/, *?page=* or /[.](pdf|jpg)$/i.

## `modifiedAfter` (type: `string`):

Optional. Uses each URL's lastmod date from the sitemap. A date (2026-01-31) or a period (7 days, 3 months). Sitemap files whose own lastmod is older are skipped without downloading.

## `modifiedBefore` (type: `string`):

Optional. A date (2026-01-31) or a period (1 year means older than a year).

## `includeUrlsWithoutLastmod` (type: `boolean`):

Only matters with a date filter. Many sitemaps leave lastmod out; by default those URLs are dropped when you filter by date.

## `includeImageUrls` (type: `boolean`):

Adds the list of image URLs from image sitemaps to each result. The image count is always included.

## `checkStatus` (type: `boolean`):

Requests every saved URL (HEAD, with a GET fallback for servers that refuse HEAD) and adds statusCode, finalStatusCode, finalUrl and redirected. Finds broken (404) and redirecting URLs in your sitemap. Makes the run slower; the price per URL stays the same.

## `statusConcurrency` (type: `integer`):

How many URLs are checked at the same time. Lower it if the site starts answering 429 Too Many Requests.

## `requestTimeoutSecs` (type: `integer`):

How long to wait for one status check. Sitemap downloads get twice as long.

## `proxyConfiguration` (type: `object`):

Optional. Most sitemaps are public and work without a proxy. Turn on Apify Proxy if a site blocks the requests (HTTP 403 in SUMMARY).

## Actor input object example

```json
{
  "startUrls": [
    "https://crawlee.dev"
  ],
  "maxUrls": 200,
  "includeUrlsWithoutLastmod": false,
  "includeImageUrls": false,
  "checkStatus": false,
  "statusConcurrency": 10,
  "requestTimeoutSecs": 30,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `results` (type: `string`):

One item per URL found in the sitemaps.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://crawlee.dev"
    ],
    "maxUrls": 200,
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("fernandosord/sitemap-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://crawlee.dev"],
    "maxUrls": 200,
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("fernandosord/sitemap-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://crawlee.dev"
  ],
  "maxUrls": 200,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call fernandosord/sitemap-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,fernandosord/sitemap-url-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/4h4tYsmAWFcU5RQ3N/builds/jY5SLOctFTvHHOQ4r/openapi.json